1 |
|
2 | fbp-spec [![Build Status](https://secure.travis-ci.org/flowbased/fbp-spec.png?branch=master)](http://travis-ci.org/flowbased/fbp-spec)
|
3 | [![Greenkeeper badge](https://badges.greenkeeper.io/flowbased/fbp-spec.svg)](https://greenkeeper.io/)
|
4 | =========
|
5 |
|
6 | A runtime-independent test framework for Flow Based Programming (FBP) component and graphs,
|
7 | using declarative, data-driven testing.
|
8 |
|
9 | One can use fbp-spec to do testing at multiple levels,
|
10 | each approximately corresponding to the different architectural levels of Flow Based Programming:
|
11 |
|
12 | * Unit (FBP component/subgraph)
|
13 | * Integration (FBP graph)
|
14 | * System (FBP runtime)
|
15 |
|
16 | ## Status:
|
17 |
|
18 | **0.1.x: First production version**
|
19 |
|
20 | * Used by several projects, including [Ingress Table](https://github.com/c-base/ingress-table)
|
21 | * Tested with several FBP runtimes: [NoFlo](https://noflojs.org), [MicroFlo](https://microflo.org), [Python example](https://github.com/flowbased/protocol-examples), [MsgFlo](http://msgflo.org)
|
22 | * Runners available for contious integration (CLI, Mocha) and interactively (in [Flowhub](https://flowhub.org))
|
23 |
|
24 | ## Purpose & Scope
|
25 |
|
26 | Note: `fbp-spec` is intended for use by application and component-library developers.
|
27 |
|
28 | The following is considered out-of-scope:
|
29 |
|
30 | * Testing conformance with the FBP protocol. Instead use [fbp-protocol](https://github.com/flowbased/fbp-protocol)
|
31 | * Testing an FBP runtime/engine itself. Instead use a testing framework for your particular runtime language/environment.
|
32 |
|
33 | ## License
|
34 |
|
35 | The [MIT license](./LICENSE.md)
|
36 |
|
37 | # Usage
|
38 |
|
39 | ## Installing
|
40 |
|
41 | Set up fbp-spec as an NPM dependency
|
42 |
|
43 | npm install --save-dev fbp-spec
|
44 |
|
45 | or, install it globally. Useful if you just want the commandline tool.
|
46 |
|
47 | npm install -g fbp-spec
|
48 |
|
49 | ## Writing tests
|
50 |
|
51 | Each declared test suite loads an FBP component (or graph) fixture,
|
52 | and runs a set of test cases by sending a set of input data
|
53 | to input ports and verifying the output data against the expected results.
|
54 |
|
55 | ```YAML
|
56 | name: "Simple example of passing tests"
|
57 | topic: "core/Repeat"
|
58 | fixture:
|
59 | type: 'fbp'
|
60 | data: |
|
61 | INPORT=it.IN:IN
|
62 | OUTPORT=f.OUT:OUT
|
63 | it(core/Repeat) OUT -> IN f(core/Repeat)
|
64 |
|
65 | cases:
|
66 | -
|
67 | name: 'sending a boolean'
|
68 | assertion: 'should repeat the same'
|
69 | inputs:
|
70 | in: true
|
71 | expect:
|
72 | out:
|
73 | equals: true
|
74 | -
|
75 | name: 'sending a number'
|
76 | assertion: 'should repeat the same'
|
77 | inputs:
|
78 | in: 1000
|
79 | expect:
|
80 | out:
|
81 | equals: 1000
|
82 | ```
|
83 |
|
84 | #### Multiple ports
|
85 |
|
86 | You can send data to multiple inports and check expectations on multiple ports per testcase:
|
87 |
|
88 | ```YAML
|
89 | -
|
90 | name: '1 active track toggled high'
|
91 | assertion: 'should give value1 color'
|
92 | inputs:
|
93 | tracks: 1
|
94 | animation: [
|
95 | 0, # track idx
|
96 | "0xEE00EE", # val0
|
97 | "0xAA00AA", # val1
|
98 | 200, # period
|
99 | 50, # dutycycle
|
100 | 0, # offset
|
101 | 500 ] # duration
|
102 | clock: 250
|
103 | expect:
|
104 | clock:
|
105 | equals: 250
|
106 | value:
|
107 | equals: [0, 0x00AA] # FIXME: truncated
|
108 | ```
|
109 |
|
110 | #### Sequence of packets
|
111 |
|
112 | For testing components with state, you can sending multiple input packets
|
113 | [in sequence](https://github.com/flowbased/fbp-spec/blob/master/examples/sequences.yaml).
|
114 |
|
115 | ```YAML
|
116 | -
|
117 | name: 'sequence of data using spacy notation'
|
118 | assertion: 'should pass'
|
119 | inputs:
|
120 | -
|
121 | in: true
|
122 | -
|
123 | in: false
|
124 | expect:
|
125 | -
|
126 | out:
|
127 | equals: true
|
128 | -
|
129 | out:
|
130 | equals: false
|
131 | ```
|
132 |
|
133 | #### Extract data using path
|
134 |
|
135 | With `path` you can specify a [JSONPath](http://goessner.net/articles/JsonPath/)
|
136 | to extract the piece(s) of data the assertions will be ran against:
|
137 |
|
138 | ```YAML
|
139 | -
|
140 | name: 'select single value'
|
141 | assertion: 'should pass'
|
142 | inputs:
|
143 | in: { outer: { inner: { foo: 'bar' } } }
|
144 | expect:
|
145 | out:
|
146 | path: '$.outer.inner.foo'
|
147 | equals: 'bar'
|
148 | -
|
149 | name: 'selecting many correct values'
|
150 | assertion: 'should pass'
|
151 | inputs:
|
152 | in:
|
153 | outer:
|
154 | first: { foo: 'bar' }
|
155 | second: { foo: 'bar' }
|
156 | expect:
|
157 | out:
|
158 | path: '$.outer.*.foo'
|
159 | equals: 'bar'
|
160 | ```
|
161 |
|
162 | #### Skipping tests
|
163 |
|
164 | Setting `skip` property on a testcase or suite, will cause it to not be ran.
|
165 | Should contain a message of the reason for skipping.
|
166 |
|
167 | ```YAML
|
168 | -
|
169 | name: 'a test that is skipped'
|
170 | assertion: 'will not be ran'
|
171 | inputs:
|
172 | in: 1000
|
173 | expect:
|
174 | out:
|
175 | equals: 1000
|
176 | skip: 'not implemented yet'
|
177 | ```
|
178 |
|
179 | #### Using fixtures
|
180 |
|
181 | One can use testing-specific components in the fixture, to simplify
|
182 | driving the unit under test with complex inputs and performing complex assertions.
|
183 |
|
184 | ```YAML
|
185 | fixture:
|
186 | type: 'fbp'
|
187 | data: |
|
188 | INPORT=imagename.IN:NAME
|
189 | INPORT=testee.PARAM:PARAM
|
190 | INPORT=reference.IN:REFERENCE
|
191 | OUTPORT=compare.OUT:SIMILARITY
|
192 |
|
193 | generate(test/GenerateTestImage) OUT -> IN testee(my/Component)
|
194 | testee OUT -> ACTUAL compare(test/CompareImage)
|
195 | reference(test/ReadReferenceImage) OUT -> REFERENCE compare
|
196 | cases:
|
197 | -
|
198 | name: 'testing complex data with custom components fixture'
|
199 | assertion: 'should pass'
|
200 | inputs:
|
201 | name: someimage
|
202 | param: 100
|
203 | reference: someimage-100-result
|
204 | expect:
|
205 | similarity:
|
206 | above: 0.99
|
207 | ```
|
208 |
|
209 | #### Supported assertions
|
210 |
|
211 | Instead of `equals` you can use any of the supported assertion predicates. Examples include:
|
212 |
|
213 | type
|
214 | above
|
215 | below
|
216 | contains
|
217 | haveKeys
|
218 | includeKeys
|
219 |
|
220 | For a full set of assertions, see [the schema](https://github.com/flowbased/fbp-spec/blob/master/schemata/expectation.yaml)
|
221 |
|
222 | #### More
|
223 |
|
224 | A comprehensive set of examples can be found under [./examples](./examples).
|
225 | For the detailed definition of the dataformat for tests, see [schemata/](./schemata/).
|
226 |
|
227 |
|
228 | ## Running tests with fbp-spec commandline tool
|
229 |
|
230 | The simplest and most universal way of running tests is with the `fbp-spec` commandline tool.
|
231 |
|
232 | $ fbp-spec --address ws://localhost:3333 examples/multisuite-failandpass.yaml
|
233 |
|
234 | MultiSuite, failing tests
|
235 | sending a boolean with wrong expect
|
236 | should fail: ✗ Error: expected true to deeply equal false
|
237 | sending a number with wrong expect
|
238 | should fail: ✗ Error: expected 1000 to deeply equal 1003
|
239 | MultiSuite, passing tests
|
240 | sending a boolean
|
241 | should repeat the same: ✓
|
242 | sending a number
|
243 | should repeat the same: ✓
|
244 |
|
245 | The `--command` options can be used to specify a command which will start the runtime under test:
|
246 |
|
247 | fbp-spec --command "python2 protocol-examples/python/runtime.py"
|
248 |
|
249 | It sets the exit status to non-zero on failure, so is suitable for integrating into a `Makefile` or similar.
|
250 |
|
251 | ## Running tests by integrating with Mocha
|
252 |
|
253 | [Mocha](http://mochajs.org/) iss a popular test runner framework for JavaScript/CoffeeScript on browser and node.js.
|
254 |
|
255 | Since fbp-spec communicates with your runtime over a network protocol,
|
256 | you can use this also when your project is not JavaScript-based.
|
257 | The Mocha runner is for instance [used in microflo-core](https://github.com/microflo/microflo-core/blob/master/spec/ComponentTests.coffee)
|
258 | to test C++ components for microcontrollers & embedded devices.
|
259 |
|
260 | You can have your fbp-spec tests run in Mocha by calling the `fbpspec.mocha.run()` function, in a file which is
|
261 | executed with the standard Mocha runner. Eg. `mocha --reporter spec tests/fbpspecs.js`
|
262 |
|
263 | // fbpspecs.js
|
264 | fbpspec = require('fbp-spec');
|
265 |
|
266 | rt = {
|
267 | protocol: "websocket",
|
268 | address: "ws://localhost:3569",
|
269 | secret: "py3k", // Optional. If needed to connect/authenticate to runtime
|
270 | command: 'python2 protocol-examples/python/runtime.py' // Optional. Can be used to start runtime automatically
|
271 | };
|
272 | fbpspec.mocha.run(rt, './examples/simple-passing.yaml', { starttimeout: 1000 });
|
273 |
|
274 | The tests can be specified as a list of files, or directories.
|
275 | You can use the standard `grep` option of Mocha to run only some tests.
|
276 |
|
277 | For CoffeScript example, see [./spec/mocha.coffee](./spec/mocha.coffee).
|
278 |
|
279 | ## Running tests interactively in Flowhub
|
280 |
|
281 | [Flowhub](http://app.flowhub.io) IDE (version 0.11 and later) has integrated support for fbp-spec. No installation is required.
|
282 |
|
283 | * Open existing project, or create a new one
|
284 | * Open a component, and write/copypaste in a test in the `Tests` panel
|
285 | * Ensure you have a runtime set up, and connected
|
286 |
|
287 | When you make changes to your project (components,graphs) or tests, Flowhub will now automatically (re-)run your tests.
|
288 | You can see the status in the top-right corner. Clicking on it brings up more details.
|
289 |
|
290 | ## Generating tests programatically
|
291 |
|
292 | The test-format defined by fbp-spec is fairly generic and versatile. It is intended primarily as
|
293 | a format one directly specifies tests in, but can also be generated from other sources.
|
294 |
|
295 | Sometimes data-driven testing, one does a large amount of very similar tests,
|
296 | with multiple test-cases per set of input data.
|
297 | By capturing only the unique parts of testcases in a specialied data-structure (JSON, YAML, etc),
|
298 | and then transforming this into standard `fbp-spec` files with some code, adding/removing
|
299 | cases becomes even easier.
|
300 | For instance in `imgflo-server`, [testcases](https://github.com/jonnor/imgflo-server/blob/master/spec/graphtests.yaml)
|
301 | can be defined by providing a name, an URL and a reference result (a file with naming convention based on name).
|
302 |
|
303 | Similarly, one can generate testcases using fuzzing, schema-based, model-based or similar tools.
|
304 |
|
305 | ## Integrating test runner in an application
|
306 |
|
307 | The test runner code is accessible as a JavaScript library,
|
308 | and can be integrated into other apps (like Flowhub does).
|
309 | See examples of [commandline](./src/cli.coffee) and [webappp](./ui/main.coffee) usage.
|
310 |
|
311 | ## Add supporting for a new runtime
|
312 |
|
313 | You need to implement the [FBP network protocol](https://github.com/flowbased/fbp-protocol).
|
314 | At least the `protocol:runtime`, `protocol:graph`, and `protocol:network` capabilities are required.
|
315 |
|
316 | All transports supported by [fbp-protocol-client]((https://github.com/flowbased/fbp-protocol))
|
317 | are supported by fbp-spec, including WebSocket, WebRTC, and iframe/postMessage.
|
318 |
|
319 | fbp-spec is intended to be used with flow-based and dataflow-programming,
|
320 | but might be useful also outside these programming paradigms. Try it out!
|
321 |
|
322 | ## Writing a test runner in another language
|
323 |
|
324 | As long as you stay compatible with the [fbp-spec testformat](./schemata/)
|
325 | and [FBP protocol](http://noflojs.org/documentation/protocol/),
|
326 | you can implement a compatible runner in any programming language.
|
327 |
|
328 | You can consider the fbp-spec code (in CoffeeScript) as a reference implementation.
|
329 |
|