@angular-devkit/architect/src/jobs/README.md

# Description

Jobs is the Angular DevKit subsystem for scheduling and running generic functions with clearly
typed inputs and outputs. A `Job` instance is a function associated with metadata. You can
schedule a job, synchronize it with other jobs, and use it to schedule other jobs.

The whole API is serializable, allowing you to use a Node Stream or message channel to
communicate between the job and the job scheduler.

Jobs are lazy, cold, and guaranteed to execute exactly once when scheduled. Subscribing to a job
returns messages from the point where the job is at.

## Argument, Input, Output and Channels

A job receives a single argument when scheduled and can also listen to an input channel. It can
emit multiple outputs, and can also provide multiple output channels that emit asynchronous JSON
messages, which can be typed.

The I/O model is like that of an executable, where the argument corresponds to arguments on the
command line, the input channel to STDIN, the output channel to STDOUT, and the channels
would be additional output streams.

## LifeCycle

A `Job` goes through multiple LifeCycle messages before its completion;

1. `JobState.Queued`. The job was queued and is waiting. This is the default state from the
   scheduler.
1. `JobState.Ready`. The job's dependencies (see
   ["Synchronizing and Dependencies"](#Dependencies)) are done running, the argument is
   validated, and the job is ready to execute.
1. `JobState.Started`. The argument has been validated, the job has been called and is running.
   This is handled by the job itself (or `createJobHandler()`).
1. `JobState.Ended`. The job has ended and is done running. This is handled by the job itself (or
   `createJobHandler()`).
1. `JobState.Errored`. A unrecoverable error happened.

Each state (except `Queued`) corresponds to a `JobOutboundMessage` on the `outboundBus` observable
that triggers the state change. The `Scheduler` emits the `Ready` and `Errored` messages; the job
implementation should not emit them, and if it does they are filtered out. You can listen for
these messages or use the corresponding state member.

The job implementation should emit the `Start` and `End` messages when it is starting the job logic
itself. Only the first `Start` and `End` messages will be forwarded. Any more will be filtered out.

The `Queued` state is set as the job is scheduled, so there is no need to listen for the message.

## `Job<OutputType>` Object

The `Job` object that is returned when you schedule a job provides access to the job's status and
utilities for tracking and modifying the job.

1. `id`. A unique symbol that can be used as a Map key.
1. `description`. The description of the job from the scheduler. See `JobDescription` object.
1. `argument`. The argument value that was used to start the job.
1. `input`. An `Observer` that can be used to send validated inputs to the job itself.
1. `output`. An `Observable<OutputType>` that filters out messages to get only the returned output
   of a job.
1. `promise`. A promise that waits for the last output of a job. Returns the last value outputted
   (or no value if there's no last value).
1. `state`. The current state of the job (see `LifeCycle`).
1. `channels`. A map of side channels the user can listen to as `Observable`.
1. `ping()`. A function that can be used to ping the job, receiving a `Promise` for when the ping
   is answered.
1. `stop()`. Sends a `stop` input to the job, which suggests to stop the job. The job itself can
   choose to ignore this message.
1. `inboundBus`. The raw input `Observer<JobInboundMessage>`. This can be used to send messages to
   the `context.inboundBus` observable in the job. These are `JobInboundMessage` messages. See
   ["Communicating With Jobs"](#Communicating).
1. `outboundBus`. The raw output `Observable<JobOutput>`. This can be used to listen to messages
   from the job. See ["Communicating With Jobs"](#Communicating).

## `JobHandlerContext<I, O>` Object

The `JobHandlerContext<>` is passed to the job handler code in addition to its argument. The
context contains the following members:

1. `description`. The description of the job. Its name and schemas.
1. `scheduler`. A `Scheduler<>` instance that can be used to create additional jobs.
1. `dependencies`. A generic list of other job instances that were run as dependencies when
   scheduling this job. Their `id` is not guaranteed to match the `id` of the `Job<>` instance
   itself (those `Job<>`s might just be proxies). The state of those `Job<>` is guaranteed to be
   `JobState.Ended`, as `JobState.Errored` would have prevented this handler from running.
1. `inboundBus`. The raw input observable, complement of the `inboundBus` observer from the `Job<>`.

# Examples

An example of a job that adds all input together and return the output value. We use a
simple synchronous job registry and a simple job scheduler.

```typescript
import { jobs } from '@angular-devkit/core';

const add = jobs.createJobHandle<number[], number>((input) =>
  input.reduce((total, curr) => total + curr, 0),
);

// Register the job in a SimpleJobRegistry. Different registries have different API.
const registry = new jobs.SimpleJobRegistry();
const scheduler = new jobs.SimpleScheduler(registry);
registry.register(add, {
  name: 'add',
  input: { type: 'array', items: { type: 'number' } },
  output: { type: 'number' },
});

scheduler
  .schedule('add', [1, 2, 3, 4])
  .promise.then((output) => console.log('1 + 2 + 3 + 4 is ' + output));
```

# Creating Jobs

A job is at its core a function with a description object attached to it. The description object
stores the JSON schemas used to validate the types of the argument passed in, the input and
output values. By default, a job accepts and can output any JSON object.

```typescript
import { Observable } from 'rxjs';
import { jobs } from '@angular-devkit/core';

const argument = {
  type: 'array',
  items: { type: 'number' },
};
const output = {
  type: 'number',
};

export function add(argument: number[]): Observable<jobs.JobOutboundMessage<number>> {
  return new Observable((o) => {
    o.next({ kind: jobs.JobOutboundMessageKind.Start });
    o.next({
      kind: jobs.JobOutboundMessageKind.Output,
      output: argument.reduce((total, curr) => total + curr, 0),
    });
    o.next({ kind: jobs.JobOutboundMessageKind.End });
    o.complete();
  });
}

// Add a property to `add` to make it officially a JobHandler. The Job system does not recognize
// any function as a JobHandler.
add.jobDescription = {
  argument: argument,
  output: output,
};

// Call the job with an array as argument, and log its output.
declare const scheduler: jobs.Scheduler;
scheduler.schedule('add', [1, 2, 3, 4]).output.subscribe((x) => console.log(x)); // Will output 10.
```

This is a lot of boilerplate, so we made some helpers to improve readability and manage argument,
input and output automatically:

```typescript
// Add is a JobHandler function, like the above.
export const add = jobs.createJobHandler<number[], number>((argument) =>
  argument.reduce((total, curr) => total + curr, 0),
);

// Schedule like above.
```

You can also return a Promise or an Observable, as jobs are asynchronous. This helper will set
start and end messages appropriately. It will also manage channels automatically (see below).

A more complex job can be declared like this:

```typescript
import { Observable } from 'rxjs';
import { jobs } from '@angular-devkit/core';

// Show progress with each count in a separate output channel. Output "more" in a channel.
export const count = jobs.createJobHandler<number, number>(
  // Receive a context that contains additional methods to create channels.
  (argument: number, { createChannel }) =>
    new Observable<number>((o) => {
      const side = createChannel('side', { type: 'string', const: 'more' });
      const progress = createChannel('progress', { type: 'number' });
      let i = 0;
      function doCount() {
        o.next(i++);
        progress.next(i / argument);
        side.next('more');

        if (i < argument) {
          setTimeout(doCount, 100);
        } else {
          o.complete();
        }
      }
      setTimeout(doCount, 100);
    }),
  {
    argument: { type: 'number' },
    output: { type: 'number' },
  },
);

// Get a hold of a scheduler that refers to the job above.
declare const scheduler: jobs.Scheduler;

const job = scheduler.schedule('count', 0);
job.getChannel('side').subscribe((x) => console.log(x));
// You can type a channel too. Messages will be filtered out.
job.getChannel<number>('progress', { type: 'number' }).subscribe((x) => console.log(x));
```

## <a name="Communicating"></a>Communicating With Jobs

Jobs can be started and updated in a separate process or thread, and as such communication with a
job should avoid using global objects (which might not be shared). The jobs API and schedulers
provide 2 communication streams (one for input and the other for output), named `inboundBus` and
`outboundBus`.

### Raw Input Stream

The `schedule()` function returns a `Job<>` interface that contains a `inboundBus` member of type
`Observer<JobInboundMessage>`. All messages sent _to_ the job goes through this stream. The `kind`
member of the `JobInboundMessage` interface dictates what kind of message it is sending:

1. `JobInboundMessageKind.Ping`. A simple message that should be answered with
   `JobOutboundMessageKind.Pong` when the job is responsive. The `id` field of the message should
   be used when returning `Pong`.
1. `JobInboundMessageKind.Stop`. The job should be stopped. This is used when
   cancelling/unsubscribing from the `output` (or by calling `stop()`). Any inputs or outputs
   after this message will be ignored.
1. `JobInboundMessageKind.Input` is used when sending inputs to a job. These correspond to the
   `next` methods of an `Observer` and are reported to the job through its `context.input`
   Observable. There is no way to communicate an error to the job.

Using the `createJobHandler()` helper, all those messages are automatically handled by the
boilerplate code. If you need direct access to raw inputs, you should subscribe to the
`context.inboundBus` Observable.

### Raw Output Stream

The `Job<>` interface also contains a `outboundBus` member (of type
`Observable<JobOutboundMessage<O>>` where `O` is the typed output of the job) which is the output
complement of `inboundBus`. All messages sent _from_ the job goes through this stream. The `kind`
member of the `JobOutboundMessage<O>` interface dictates what kind of message it is sending:

1. `JobOutboundMessageKind.Create`. The `Job<>` was created, its dependencies are done, and the
   library is validating Argument and calling the internal job code.
1. `JobOutboundMessageKind.Start`. The job code itself should send that message when started.
   `createJobHandler()` will do it automatically.
1. `JobOutboundMessageKind.End`. The job has ended. This is done by the job itself and should always
   be sent when completed. The scheduler will listen to this message to set the state and unblock
   dependent jobs. `createJobHandler()` automatically send this message.
1. `JobOutboundMessageKind.Pong`. The job should answer a `JobInboundMessageKind.Ping` message with
   this. Automatically done by `createJobHandler()`.
1. `JobOutboundMessageKind.Output`. An `Output` has been generated by the job.
1. `JobOutboundMessageKind.ChannelMessage`, `JobOutboundMessageKind.ChannelError` and
   `JobOutboundMessageKind.ChannelComplete` are used for output channels. These correspond to the
   `next`, `error` and `complete` methods of an `Observer` and are available to the callee through
   the `job.channels` map of Observable.

Those messages can be accessed directly through the `job.outboundBus` member. The job itself should
return an `Observable<JobOutboundMessage<O>>`. The `createJobHandler()` helper handles most of use
cases of this and makes it easier for jobs to handle this.

## Job Dispatchers

Dispatchers are a helper that redirect to different jobs given conditions. To create a job
dispatcher, use the `createDispatcher()` function:

```typescript
import { jobs } from '@angular-devkit/core';

// A dispatcher that installs node modules given a user's preference.
const dispatcher = jobs.createDispatcher({
  name: 'node-install',
  argument: { properties: { moduleName: { type: 'string' } } },
  output: { type: 'boolean' },
});

const npmInstall = jobs.createJobHandler(/* ... */, { name: 'npm-install' });
const yarnInstall = jobs.createJobHandler(/* ... */, { name: 'yarn-install' });
const pnpmInstall = jobs.createJobHandler(/* ... */, { name: 'pnpm-install' });

declare const registry: jobs.SimpleJobRegistry;
registry.register(dispatcher);
registry.register(npmInstall);
registry.register(yarnInstall);
registry.register(pnpmInstall);

// Default to npm.
dispatcher.setDefaultDelegate(npmInstall.name);
// If the user is asking for yarn over npm, uses it.
dispatcher.addConditionalDelegate(() => userWantsYarn, yarnInstall.name);
```

## Execution Strategy

Jobs are always run in parallel and will always start, but many helper functions are provided
when creating a job to help you control the execution strategy;

1. `serialize()`. Multiple runs of this job will be queued with each others.
1. `memoize(replayMessages = false)` will create a job, or reuse the same job when inputs are
   matching. If the inputs don't match, a new job will be started and its outputs will be stored.

These strategies can be used when creating the job:

```typescript
// Same input and output as above.

export const add = jobs.strategy.memoize()(
  jobs.createJobHandler<number[], number>((argument) =>
    argument.reduce((total, curr) => total + curr, 0),
  ),
);
```

Strategies can be reused to synchronize between jobs. For example, given jobs `jobA` and `jobB`,
you can reuse the strategy to serialize both jobs together;

```typescript
const strategy = jobs.strategy.serialize();
const jobA = strategy(jobs.createJobHandler(...));
const jobB = strategy(jobs.createJobHandler(...));
```

Even further, we can have package A and package B run in serialization, and B and C also be
serialized. Running A and C will run in parallel, while running B will wait for both A and C
to finish.

```typescript
const strategy1 = jobs.strategy.serialize();
const strategy2 = jobs.strategy.serialize();
const jobA = strategy1(jobs.createJobHandler(...));
const jobB = strategy1(strategy2(jobs.createJobHandler(...)));
const jobC = strategy2(jobs.createJobHandler(...));
```

# Scheduling Jobs

Jobs can be scheduled using a `Scheduler` interface, which contains a `schedule()` method:

```typescript
interface Scheduler {
  /**
   * Schedule a job to be run, using its name.
   * @param name The name of job to be run.
   * @param argument The argument to send to the job when starting it.
   * @param options Scheduling options.
   * @returns The Job being run.
   */
  schedule<I extends MinimumInputValueT, O extends MinimumOutputValueT>(
    name: JobName,
    argument: I,
    options?: ScheduleJobOptions,
  ): Job<JsonValue, O>;
}
```

The scheduler also has a `getDescription()` method to get a `JobDescription` object for a certain
name; that description contains schemas for the argument, input, output, and other channels:

```typescript
interface Scheduler {
  /**
   * Get a job description for a named job.
   *
   * @param name The name of the job.
   * @returns A description, or null if the job cannot be scheduled.
   */
  getDescription(name: JobName): JobDescription | null;

  /**
   * Returns true if the job name has been registered.
   * @param name The name of the job.
   * @returns True if the job exists, false otherwise.
   */
  has(name: JobName): boolean;
}
```

Finally, the scheduler interface has a `pause()` method to stop scheduling. This will queue all
jobs and wait for the unpause function to be called before unblocking all the jobs scheduled.
This does not affect already running jobs.

```typescript
interface Scheduler {
  /**
   * Pause the scheduler, temporary queueing _new_ jobs. Returns a resume function that should be
   * used to resume execution. If multiple `pause()` were called, all their resume functions must
   * be called before the Scheduler actually starts new jobs. Additional calls to the same resume
   * function will have no effect.
   *
   * Jobs already running are NOT paused. This is pausing the scheduler only.
   *
   * @returns A function that can be run to resume the scheduler. If multiple `pause()` calls
   *          were made, all their return function must be called (in any order) before the
   *          scheduler can resume.
   */
  pause(): () => void;
}
```

## <a name="Dependencies"></a>Synchronizing and Dependencies

When scheduling jobs, it is often necessary to run jobs after certain other jobs are finished.
This is done through the `dependencies` options in the `schedule()` method.

These jobs will also be passed to the job being scheduled, through its context. This can be
useful if, for example, the output of those jobs are of a known type, or have known side channels.

An example of this would be a compiler that needs to know the output directory of other compilers
before it, in a tool chain.

### Dependencies

When scheduling jobs, the user can add a `dependencies` field to the scheduling options. The
scheduler will wait for those dependencies to finish before running the job, and pass those jobs
in the context of the job.

### Accessing Dependencies

Jobs are called with a `JobHandlerContext` as a second argument, which contains a
`dependencies: Job<JsonValue>[]` member which contains all dependencies that were used when
scheduling the job. Those aren't fully typed as they are determined by the user, and not the job
itself. They also can contain jobs that are not finished, and the job should use the `state`
member of the job itself before trying to access its content.

### Scheduler Sub Jobs

The `JobHandlerContext` also contains a `scheduler` member which can be used to schedule jobs
using the same scheduler that was used for the job. This allows jobs to call other jobs
and wait for them to end.

## Available Schedulers

The Core Angular DevKit library provides 2 implementations for the `Scheduler` interface:

## SimpleJobRegistry

Available in the jobs namespace. A registry that accept job registration, and can also schedule
jobs.

```typescript
import { jobs } from '@angular-devkit/core';

const add = jobs.createJobHandler<number[], number>((argument) =>
  argument.reduce((total, curr) => total + curr, 0),
);

// Register the job in a SimpleJobRegistry. Different registries have different API.
const registry = new jobs.SimpleJobRegistry();
const scheduler = new SimpleJobScheduler(registry);

registry.register(add, {
  name: 'add',
  argument: { type: 'array', items: { type: 'number' } },
  output: { type: 'number' },
});

scheduler.schedule('add', [1, 2, 3, 4]);
```

## NodeModuleJobRegistry

Available through `@angular-devkit/core/node`.

A scheduler that loads jobs using their node package names. These jobs need to use the
`createJobHandler()` helper and report their argument/input/output schemas that way.

```typescript
declare const registry: NodeModuleJobRegistry;
const scheduler = new SimpleJobScheduler(registry);

scheduler.schedule('some-node-package#someExport', 'input');
```

# Gotchas

1. Deadlocking Dependencies  
   It is impossible to add dependencies to an already running job, but it is entirely possible to
   get locked between jobs. Be aware of your own dependencies.

1. Using `job.promise`  
   `job.promise` waits for the job to ends. Don't rely on it unless you know the job is not
   watching and running for a long time. If you aren't sure, use
   `job.output.pipe(first()).toPromise()` instead which will return the first next output,
   regardless of whether the job watches and rerun or not.

# FAQ

1. Laziness  
   A job is lazy until executed, but its messages will be replayed when resubscribed.

1. Serialize Strategy vs Dependencies  
   Strategies are functions that transform the execution of a job, and can be used when
   declaring the job, or registering it. Dependencies, on the other hand, are listed when
   scheduling a job to order jobs during scheduling.

   A job has no control over the way it's scheduled, and its dependencies. It can, however,
   declare that it shouldn't run at the same time as itself. Alternatively, a user could
   schedule a job twice and imply that the second run should wait for the first to finish. In
   practice, this would be equivalent to having the job be serialized, but the important detail
   is in _whom_ is defining the rules; using the `serialize()` strategy, the job implementation
   is, while when using dependencies, the user is.

   The user does not need to know how to job needs to synchronize with itself, and the job does
   not need to know how it synchronizes with other jobs that it doesn't know about. That's part
   of the strength of this system as every job can be developed in a vacuum, only caring about
   its contracts (argument, input and output) and its own synchronization.