kettle/docs/DataSources.md

---
title: DataSources
layout: default
category: Kettle
---

## DataSources

A DataSource is an Infusion component which meets a simple contract for read/write access to indexed data. DataSource is a simple semantic, broadly the same as that
encoded in [CRUD](https://en.wikipedia.org/wiki/Create,_read,_update_and_delete), although the current DataSource semantic does not provide explicitly for deletion.

The concrete DataSources in Kettle provide support for HTTP endpoints (with a particular variety specialised for accessing CouchDB databases with CRUDlike semantics) as well as the filesystem, with
an emphasis on JSON payloads.

The DataSource API is drawn from the following two methods – a read-only DataSource will just implement `get`, and a writeable DataSource will implement both `get` and `set`:

    /* @param directModel {Object} A JSON structure holding the "coordinates" of the state to be read -  
     * this model is morally equivalent to (the substitutable parts of) a file path or URL
     * @param options {Object} [Optional] A JSON structure holding configuration options good for just 
     * this request. These will be specially interpreted by the particular concrete grade of DataSource 
     * – there are no options valid across all implementations of this grade.
     * @return {Promise} A promise representing successful or unsuccessful resolution of the read state
     */
    dataSource.get(directModel, options);
    /* @param directModel {Object} As for get
     * @param model {Object} The state to be written to the coordinates
     * @param options {Object} [Optional] A JSON structure holding configuration options good for just 
     * this request. These will be specially interpreted by the 
     * particular concrete grade of DataSource – there are no options valid across all implementations 
     * of this grade. For example, a URL DataSource will accept an option `writeMethod` which will 
     * allow the user to determine which HTTP method (PUT or POST) will be used to implement the write
     * operation.
     * @return {Promise} A promise representing resolution of the written state,
     * which may also optionally resolve to any returned payload from the write process
     */
    dataSource.set(directModel, model, options);
    
### Simple example of using an HTTP dataSource

In this example we define and instantiate a simple HTTP-backed dataSource accepting one argument to configure a URL segment:

```javascript
var fluid = require("infusion"),
    kettle = require("../../kettle.js"),
    examples = fluid.registerNamespace("examples");


fluid.defaults("examples.httpDataSource", {
    gradeNames: "kettle.dataSource.URL",
    url: "http://jsonplaceholder.typicode.com/posts/%postId",
    termMap: {
        postId: "%directPostId"
    }
});

var myDataSource = examples.httpDataSource();
var promise = myDataSource.get({directPostId: 42});

promise.then(function (response) {
    console.log("Got dataSource response of ", response);
}, function (error) {
    console.error("Got dataSource error response of ", error);
});
```

You can run this snippet from our code samples by running `node simpleDataSource.js` from [examples/simpleDataSource](../examples/simpleDataSource) in our samples area.
This contacts the useful JSON placeholder API service at [`jsonplaceholder.typicode.com`](http://jsonplaceholder.typicode.com/) to retrieve a small JSON document holding some placeholder text. If you get
a 404 or an error, please contact us and we'll update this sample to contact a new service. 

An interesting element in this snippet is the `termMap` configured as options of our dataSource. This sets up an indirection between the `directModel` supplied as the 
argument to the `dataSource.get` call, and the URL issued in the HTTP request. The keys in the `termMap` are interpolation variables in the URL, which in the URL are
prefixed by `%`. The values in the `termMap` represent either

* Plain values to be interpolated as strings directly into the URL, or
* If the first character of the value in the `termMap` is %, the remainder of the string represents a path which will be dereferenced from the `directModel` argument to the current `set` or `get` request.  

In addition, if the term value has the prefix `noencode:`, it will be interpolated without any [URI encoding](https://developer.mozilla.org/en/docs/Web/JavaScript/Reference/Global_Objects/encodeURIComponent).

We document these configuration options in the next section:

### Configuration options accepted by `kettle.dataSource.URL`


<table>
    <thead>
        <tr>
            <th colspan="3">Supported configurable options for a <code>kettle.dataSource.URL</code></th>
        </tr>
        <tr>
            <th>Option Path</th>
            <th>Type</th>
            <th>Description</th>
        </tr>
    </thead>
    <tbody>
        <tr>
            <td><code>writable</code></td>
            <td><code>Boolean</code> (default: <code>false</code>)</td>
            <td>If this option is set to <code>true</code>, a <code>set</code> method will be fabricated for this dataSource – otherwise, it will implement only a <code>get</code> method.</td>
        </tr>
        <tr>
            <td><code>writeMethod</code></td>
            <td><code>String</code> (default: <code>PUT</code>)</td>
            <td>The HTTP method to be used when the <code>set</code> method is operated on this writable DataSource (with <code>writable: true</code>). This defaults to <code>PUT</code> but
            <code>POST</code> is another option. Note that this option can also be supplied within the <code>options</code> argument to the <code>set</code> method itself.</td>
        </tr>
        <tr>
            <td><code>url</code></td>
            <td><code>String</code></td>
            <td>A URL template, with interpolable elements expressed by terms beginning with the <code>%</code> character, for the URL which will be operated by the <code>get</code> and 
            <code>set</code> methods of this dataSource.</td>
        </tr>
        <tr>
            <td><code>termMap</code></td>
            <td><code>Object</code> (map of <code>String</code> to <code>String</code>)</td>
            <td>A map, of which the keys are some of the interpolation terms held in the <code>url</code> string, and the values will be used to perform the interpolation. If a value begins with <code>%</code>, the remainder of the string 
            represents a <a href="http://docs.fluidproject.org/infusion/development/FrameworkConcepts.html#el-paths">path</a> into the <code>directModel</code> argument
            accepted by the <code>get</code> and <code>set</code> methods of the DataSource. By default any such values looked up will be <a href="https://developer.mozilla.org/en/docs/Web/JavaScript/Reference/Global_Objects/encodeURIComponent">
            URI Encoded</a> before being interpolated into the URL – unless their value in the termMap is prefixed by the string <code>noencode:</code>.</td>
        </tr>
        <tr>
            <td><code>notFoundIsEmpty</code></td>
            <td><code>Boolean</code></a> (default: <code>false</code>)</td>
            <td>If this option is set to <code>true</code>, a fetch of a nonexistent resource (that is, a nonexistent file, or an HTTP resource giving a 404) will result in a <code>resolve</code> with an empty
            payload rather than a <code>reject</code> response.</td>
        </tr>
        <tr>
            <td><code>components.encoding.type</code></td>
            <td><code>String</code> (grade name)</td>
            <td>A <code>kettle.dataSource.URL</code> has a subcomponent named <code>encoding</code> which the user can override in order to choose the encoding used to read and write the <code>model</code>
            object to and from the textual form in persistence. This defaults to <code>kettle.dataSource.encoding.JSON</code>. Other builtin encodings are <code>kettle.dataSource.encoding.formenc</code> operating
            HTML <a href="http://www.w3.org/TR/html401/interact/forms.html#didx-applicationx-www-form-urlencoded">form encoding</code> and <code>kettle.dataSource.encoding.none</code> which applies no encoding.
            More details in <a href="#using-content-encodings-with-a-datasource">Using Content Encodings with a DataSource</a>.</td>
        </tr>
        <tr>
            <td><code>setResponseTransforms</code></td>
            <td><code>Array of String</code></a> (default: <code>["encoding"]</code>)</td>
            <td>Contains a list of the namespaces of the transform elements (see section <a href="#transforming-promise-chains">transforming promise chains</a> that are to be applied if there is a response payload
            from the <code>set</code> method, which is often the case with an HTTP backend. With a JSON encoding these encoding typically happens symmetrically - with a JSON request one will receive a JSON response - 
            however, with other encoding such as <a href="http://www.w3.org/TR/html401/interact/forms.html#didx-applicationx-www-form-urlencoded">form encoding</a> this is often not the case and one might like to
            defeat the effect of trying to decode the HTTP response as a form. In this case, for example, one can override <code>setResponseTransforms</code> with the empty array <code>[]</code>. </td>
        </tr>
        <tr>
            <td><code>charEncoding</code></td>
            <td><code>String</code> (default: <code>utf8</code>)</td>
            <td>The character encoding of the incoming HTTP stream used to convert its data to characters - this will be sent directly to the <a href="https://nodejs.org/api/stream.html#stream_readable_setencoding_encoding">setEncoding</code> method of
            the response stream</td>
        </tr>
        <tr>
            <td><code>invokers.resolveUrl</code></td>
            <td><a href="http://docs.fluidproject.org/infusion/development/Invokers.html"><code>IoC Invoker</code></a> (default: <code>kettle.dataSource.URL.resolveUrl</code>)</td>
            <td>This invoker can be overridden to customise the process of building the url for a dataSource request. The default implementation uses an invocation of 
            <a href="http://docs.fluidproject.org/infusion/development/CoreAPI.html#fluid-stringtemplate-template-terms-"><code>fluid.stringTemplate</code></a> to interpolate elements from <code>termMap</code> and the <code>directModel</code>
            argument into the template string held in <code>url</code>. By overriding this invoker, the user can implement a strategy of their choosing. The supplied arguments to the invoker consist of the values 
            <code>(url, termMap, directModel)</code> taken from these options and the dataSource request arguments, but the override can replace these with any IoC-sourced values in the invoker definition.</td>
        </tr>
    </tbody>
</table>

In addition, a `kettle.dataSource.URL` component will accept any options accepted by node's native 
[`http.request`](https://nodejs.org/api/http.html#http_http_request_options_callback) constructor – supported in addition to the above are 
`protocol`, `host`, `port`, `headers`, `hostname`, `family`, `localAddress`, `socketPath`, `auth` and `agent`. All of these options will be overriden by options of the same names supplied as the `options` object 
supplied as the last argument to the dataSource's `get` and `set` methods. This is a good way, for example, to send custom HTTP headers along with a URL dataSource request. 
Note that any of these component-level options (e.g. `port`, `protocol`, etc.) that can be derived from parsing the `url` option will override the value from the url. Compare this setup with
the very similar one operated in the testing framework for [`kettle.test.request.http`](KettleTestingFramework.md#kettle.test.request.http).

### Configuration options accepted by `kettle.dataSource.file`

An alternative dataSource implementation is `kettle.dataSource.file` - this is backed by the node filesystem API to allow files to be read and written in various encodings. The interpolation support based on `termMap`
is very similar to that for `kettle.dataSource.URL`, but with the location template option named `path` representing an absolute filesystem path rather than the `url` property of `kettle.dataSource.URL` representing
a URL. 

Exactly the same scheme based on the subcomponent named `encoding` can be used to control content encoding for a `kettle.dataSource.file` as for a `kettle.dataSource.URL`. Similarly, `kettle.dataSource.file` supports
a further option named `charEncoding` which can select between various of the character encodings supported by node.js.

<table>
    <thead>
        <tr>
            <th colspan="3">Supported configurable options for a <code>kettle.dataSource.file</code></th>
        </tr>
        <tr>
            <th>Option Path</th>
            <th>Type</th>
            <th>Description</th>
        </tr>
    </thead>
    <tbody>
        <tr>
            <td><code>writable</code></td>
            <td><code>Boolean</code> (default: <code>false</code>)</td>
            <td>If this option is set to <code>true</code>, a <code>set</code> method will be fabricated for this dataSource – otherwise, it will implement only a <code>get</code> method.</td>
        </tr>
        <tr>
            <td><code>path</code></td>
            <td><code>String</code></td>
            <td>An (absolute) file path template, with interpolable elements expressed by terms beginning with the <code>%</code> character, for the file which will be read and written the <code>get</code> and 
            <code>set</code> methods of this dataSource.</td>
        </tr>
        <tr>
            <td><code>termMap</code></td>
            <td><code>Object</code> (map of <code>String</code> to <code>String</code>)</td>
            <td>A map, of which the keys are some of the interpolation terms held in the <code>url</code> string, and the values, if prefixed by <code>%</code> are paths into the <code>directModel</code> argument
            accepted by the <code>get</code> and <code>set</code> methods of the DataSource.</td>
        </tr>
        <tr>
            <td><code>charEncoding</code></td>
            <td><code>String</code> (default: <code>utf8</code></td>
            <td>The character encoding of the file used to convert its data to characters - one of the values supported by the <a href="https://nodejs.org/api/fs.html#fs_fs_createreadstream_path_options">node filesystem API</a> - 
            values it advertises include <code>utf8</code>, <code>ascii</code> or <code>based64</code>. There is also evidence of support for <code>ucs2</code>.</td>
        </tr>
    </tbody>
</table>

A helpful mixin grade for `kettle.dataSource.file` is `kettle.dataSource.file.moduleTerms` which will allow interpolation by any module name registered with the Infusion module system 
[`fluid.module.register`](http://docs.fluidproject.org/infusion/development/NodeAPI.html#fluid-module-register-name-basedir-modulerequire-) - e.g. `%kettle/tests/data/couchDataSourceError.json`.
 

### Using content encodings with a DataSource

`kettle.dataSource.URL` has a subcomponent named `encoding` which the user can override in order to choose the content encoding used to convert the model seen at the `get/set` API to the textual (character) form in which it is
transmitted by the dataSource. The encoding subcomponent will also correctly set the [`Content-Type`](http://www.w3.org/Protocols/rfc1341/4_Content-Type.html) header of the outgoing HTTP request in the
case of a `set` request. The encoding defaults to a JSON encoding represented by a subcomponent of type `kettle.dataSource.encoding.JSON`. Here is an example of choosing a different encoding to submit 
[form encoded](http://www.w3.org/TR/html401/interact/forms.html#didx-applicationx-www-form-urlencoded) data to an HTTP endpoint:

```javascript
fluid.defaults("examples.formDataSource", {
    gradeNames: "kettle.dataSource.URL",
    url: "http://httpbin.org/post",
    writable: true,
    writeMethod: "POST",
    components: {
        encoding: {
            type: "kettle.dataSource.encoding.formenc"
        }
    },
    setResponseTransforms: [] // Do not parse the "set" response as formenc - it is in fact JSON
});

var myDataSource = examples.formDataSource();
var promise = myDataSource.set(null, {myField1: "myValue1", myField2: "myValue2"});

promise.then(function (response) {
    console.log("Got dataSource response of ", JSON.parse(response));
}, function (error) {
    console.error("Got dataSource error response of ", error);
});
```

In this example we set up a form-encoded, writable dataSource targetted at the popular HTTP testing site `httpbin.org` sending a simple payload encoding two form elements. We use Kettle's built-in form encoding
grade by configuring an `encoding` subcomponent name `kettle.dataSource.encoding.formenc`. You can try out this
sample live in its place in the [examples directory](examples/formDataSource/formDataSource.js). Note that since this particular endpoint sends a JSON response rather than a form-encoded response,
we need to defeat the dataSource's attempt to apply the inverse decoding in the response by writing `setResponseTransforms: []`.

### Built-in content encodings 

Kettle features three built-in content encoding grades which can be configured as the subcomponent of a dataSource named `encoding` in order to determine what encoding it applies to models. They are described in this table:

|Grade name| Encoding type | Content-Type header |
|----------|---------------|----------------|
|`kettle.dataSource.encoding.JSON`|[JSON](http://json.org)|`application/json`|
|`kettle.dataSource.encoding.JSON5`|[JSON5](http://json5.org)|`application/json5`|
|`kettle.dataSource.encoding.formenc`|[form encoding](http://www.w3.org/TR/html401/interact/forms.html#didx-applicationx-www-form-urlencoded)|`application/x-www-form-urlencoded`|
|`kettle.dataSource.encoding.none`|No encoding|`text/plain`|

### Elements of an encoding component

You can operate a custom encoding by implementing a grade with the following elements, and using it as the `encoding` subcomponent in place of one of the built-in implementations in the above table:

|Member name| Type | Description |
|-----------|------|-------------|
|`parse`|`Function (String) -> Any`| Parses the textual form of the data from its encoded form into the in-memory form|
|`render`|`Function (Any) -> String`| Renders the in-memory form of the data into its textual form|
|`contentType`|`String`| Holds the value that should be supplied in the [`Content-Type`](http://www.w3.org/Protocols/rfc1341/4_Content-Type.html) of an outgoing HTTP request whose body is encoded in this form|

### The `kettle.dataSource.CouchDB` mixin grade

Kettle includes a further mixin grade, `kettle.dataSource.CouchDB`, which is suitable for reading and writing to the [`doc`](http://docs.couchdb.org/en/1.6.1/api/document/common.html) URL space of a [CouchDB](http://couchdb.apache.org/) database.
This can be applied to either a `kettle.dataSource.URL` or a `kettle.dataSource.file` (the latter clearly only useful for testing purposes).
This is a basic implementation which simply adapts the base documents in this API to a simple CRUD contract, taking care of:

* Packaging and unpackaging the special `_id` and `_rev` fields which appear at top level in a CouchDB document
    * The user's document is in fact escaped in a top-level path named `value` to avoid conflicts between its keys and any of those of the CouchDB machinery. If you wish to change this behavior, you can do so by providing different [model transformation rules](http://docs.fluidproject.org/infusion/development/ModelTransformationAPI.html) in `options.rules.readPayload` and `options.rules.writePayload`.
* Applying a "read-before-write" of the `_rev` field to minimise (but not eliminate completely) the possibility for a Couch-level conflict

This grade is not properly tested and still carries some (though very small) risk of a conflict during update – it should be used with caution. Please contact the development team if
you are interested in improved Couch-specific functionality.

## Advanced implementation notes on DataSources

In this section are a few notes for advanced users of DataSources, who are interested in extending their functionality or else in issuing I/O in Kettle by other means.

### Transforming promise chains

The detailed implementation of the Kettle DataSource is structured around a particular device taken from the Infusion Promises library, the concept of a ["transforming promise chain"](http://docs.fluidproject.org/infusion/development/PromisesAPI.html#fluid-promise-firetransformevent-event-payload-options-). The core
DataSource grade implements two events, `onRead` and and `onWrite`. These events are fired during the `get` and `set` operations of the DataSource, respectively. 
These events are better described as "pseudoevents" since they are not fired in the conventional way – rather than each event 
listener receiving the same signature, each instead receives the payload returned by the previous listener – it may then transform this payload and produce its own return in the form
of a promise. Any promise rejection terminates the listener notification chain and propagates the failure to the caller. The DataSource implementation in fact fires these events by invoking
the [`fireTransformEvent`](http://docs.fluidproject.org/infusion/development/PromisesAPI.html#fluid-promise-firetransformevent-event-payload-options-) function from Infusion's Promises API. 

The virtue of this implementation strategy is that extra stages of processing
for the DataSource can be inserted and removed from any part of the processing chain by means of supplying suitable event [priorities](http://docs.fluidproject.org/infusion/development/Priorities.html) to
the event's [listeners](http://docs.fluidproject.org/infusion/development/InfusionEventSystem.html#registering-a-listener-to-an-event). Both the JSON encoding/decoding and CouchDB wrapping/unwrapping
facilities for the DataSources are implemented in terms of event listeners of this type, rather than in terms of conditional implementation code. This is a powerful and open
implementation strategy which we plan to extend in future.

### Callback wrapping in DataSources

It's important that Kettle's inbuilt DataSources are used whenever possible when performing I/O from a Kettle application, since it is crucial that any running implementation
code is always properly contextualised by its appropriate [request component](RequestHandlersAndApps.md#request-components). Kettle guarantees that the [IoC context](http://docs.fluidproject.org/infusion/development/Contexts.html) `{request}` 
will always be resolvable onto the appropriate request component from any code executing within that request. If arbitrary callbacks are supplied to node I/O APIs, the code executing in them
will not be properly contextualised. If for some reason a DataSource is not appropriate, you can manually wrap any callbacks that you use by supplying them to the API `kettle.wrapCallback`.
[Get in touch](../README.md#getting-started-and-community) with the dev team if you find yourself in this situation.