1 | fbp-spec [![Build Status](https://secure.travis-ci.org/flowbased/fbp-spec.png?branch=master)](http://travis-ci.org/flowbased/fbp-spec)
|
2 | =========
|
3 |
|
4 | A runtime-independent test framework for Flow Based Programming (FBP) component and graphs,
|
5 | using declarative, data-driven testing.
|
6 |
|
7 | One can use fbp-spec to do testing at multiple levels,
|
8 | each approximately corresponding to the different architectural levels of Flow Based Programming:
|
9 |
|
10 | * Unit (FBP component/subgraph)
|
11 | * Integration (FBP graph)
|
12 | * System (FBP runtime)
|
13 |
|
14 | ## Status:
|
15 |
|
16 | **In production**
|
17 |
|
18 | * Used by several projects, including [Ingress Table](https://github.com/c-base/ingress-table) and [BIG IoT NoFlo bridge](https://github.com/flowhub/bigiot-bridge)
|
19 | * Tested with several FBP runtimes: [NoFlo](https://noflojs.org), [MicroFlo](https://microflo.org), [Python example](https://github.com/flowbased/protocol-examples), [MsgFlo](http://msgflo.org)
|
20 | * Runners available for contious integration (CLI, Mocha) and interactively (in [Flowhub](https://flowhub.org))
|
21 |
|
22 | ## Purpose & Scope
|
23 |
|
24 | Note: `fbp-spec` is intended for use by application and component-library developers.
|
25 |
|
26 | The following is considered out-of-scope:
|
27 |
|
28 | * Testing conformance with the FBP protocol. Instead use [fbp-protocol](https://github.com/flowbased/fbp-protocol)
|
29 | * Testing an FBP runtime/engine itself. Instead use a testing framework for your particular runtime language/environment.
|
30 |
|
31 | ## License
|
32 |
|
33 | The [MIT license](./LICENSE.md)
|
34 |
|
35 | # Usage
|
36 |
|
37 | ## Installing
|
38 |
|
39 | Set up fbp-spec as an NPM dependency
|
40 |
|
41 | npm install --save-dev fbp-spec
|
42 |
|
43 | or, install it globally. Useful if you just want the commandline tool.
|
44 |
|
45 | npm install -g fbp-spec
|
46 |
|
47 | ## Writing tests
|
48 |
|
49 | Each declared test suite loads an FBP component (or graph) fixture,
|
50 | and runs a set of test cases by sending a set of input data
|
51 | to input ports and verifying the output data against the expected results.
|
52 |
|
53 | ```YAML
|
54 | name: "Simple example of passing tests"
|
55 | topic: "core/Repeat"
|
56 | fixture:
|
57 | type: 'fbp'
|
58 | data: |
|
59 | INPORT=it.IN:IN
|
60 | OUTPORT=f.OUT:OUT
|
61 | it(core/Repeat) OUT -> IN f(core/Repeat)
|
62 |
|
63 | cases:
|
64 | -
|
65 | name: 'sending a boolean'
|
66 | assertion: 'should repeat the same'
|
67 | inputs:
|
68 | in: true
|
69 | expect:
|
70 | out:
|
71 | equals: true
|
72 | -
|
73 | name: 'sending a number'
|
74 | assertion: 'should repeat the same'
|
75 | inputs:
|
76 | in: 1000
|
77 | expect:
|
78 | out:
|
79 | equals: 1000
|
80 | ```
|
81 |
|
82 | #### Multiple ports
|
83 |
|
84 | You can send data to multiple inports and check expectations on multiple ports per testcase:
|
85 |
|
86 | ```YAML
|
87 | -
|
88 | name: '1 active track toggled high'
|
89 | assertion: 'should give value1 color'
|
90 | inputs:
|
91 | tracks: 1
|
92 | animation: [
|
93 | 0, # track idx
|
94 | "0xEE00EE", # val0
|
95 | "0xAA00AA", # val1
|
96 | 200, # period
|
97 | 50, # dutycycle
|
98 | 0, # offset
|
99 | 500 ] # duration
|
100 | clock: 250
|
101 | expect:
|
102 | clock:
|
103 | equals: 250
|
104 | value:
|
105 | equals: [0, 0x00AA] # FIXME: truncated
|
106 | ```
|
107 |
|
108 | #### Sequence of packets
|
109 |
|
110 | For testing components with state, you can sending multiple input packets
|
111 | [in sequence](https://github.com/flowbased/fbp-spec/blob/master/examples/sequences.yaml).
|
112 |
|
113 | ```YAML
|
114 | -
|
115 | name: 'sequence of data using spacy notation'
|
116 | assertion: 'should pass'
|
117 | inputs:
|
118 | -
|
119 | in: true
|
120 | -
|
121 | in: false
|
122 | expect:
|
123 | -
|
124 | out:
|
125 | equals: true
|
126 | -
|
127 | out:
|
128 | equals: false
|
129 | ```
|
130 |
|
131 | #### Extract data using path
|
132 |
|
133 | With `path` you can specify a [JSONPath](http://goessner.net/articles/JsonPath/)
|
134 | to extract the piece(s) of data the assertions will be ran against:
|
135 |
|
136 | ```YAML
|
137 | -
|
138 | name: 'select single value'
|
139 | assertion: 'should pass'
|
140 | inputs:
|
141 | in: { outer: { inner: { foo: 'bar' } } }
|
142 | expect:
|
143 | out:
|
144 | path: '$.outer.inner.foo'
|
145 | equals: 'bar'
|
146 | -
|
147 | name: 'selecting many correct values'
|
148 | assertion: 'should pass'
|
149 | inputs:
|
150 | in:
|
151 | outer:
|
152 | first: { foo: 'bar' }
|
153 | second: { foo: 'bar' }
|
154 | expect:
|
155 | out:
|
156 | path: '$.outer.*.foo'
|
157 | equals: 'bar'
|
158 | ```
|
159 |
|
160 | #### Skipping tests
|
161 |
|
162 | Setting `skip` property on a testcase or suite, will cause it to not be ran.
|
163 | Should contain a message of the reason for skipping.
|
164 |
|
165 | ```YAML
|
166 | -
|
167 | name: 'a test that is skipped'
|
168 | assertion: 'will not be ran'
|
169 | inputs:
|
170 | in: 1000
|
171 | expect:
|
172 | out:
|
173 | equals: 1000
|
174 | skip: 'not implemented yet'
|
175 | ```
|
176 |
|
177 | #### Using fixtures
|
178 |
|
179 | One can use testing-specific components in the fixture, to simplify
|
180 | driving the unit under test with complex inputs and performing complex assertions.
|
181 |
|
182 | ```YAML
|
183 | fixture:
|
184 | type: 'fbp'
|
185 | data: |
|
186 | INPORT=imagename.IN:NAME
|
187 | INPORT=testee.PARAM:PARAM
|
188 | INPORT=reference.IN:REFERENCE
|
189 | OUTPORT=compare.OUT:SIMILARITY
|
190 |
|
191 | generate(test/GenerateTestImage) OUT -> IN testee(my/Component)
|
192 | testee OUT -> ACTUAL compare(test/CompareImage)
|
193 | reference(test/ReadReferenceImage) OUT -> REFERENCE compare
|
194 | cases:
|
195 | -
|
196 | name: 'testing complex data with custom components fixture'
|
197 | assertion: 'should pass'
|
198 | inputs:
|
199 | name: someimage
|
200 | param: 100
|
201 | reference: someimage-100-result
|
202 | expect:
|
203 | similarity:
|
204 | above: 0.99
|
205 | ```
|
206 |
|
207 | #### Supported assertions
|
208 |
|
209 | Instead of `equals` you can use any of the supported assertion predicates. Examples include:
|
210 |
|
211 | type
|
212 | above
|
213 | below
|
214 | contains
|
215 | haveKeys
|
216 | includeKeys
|
217 |
|
218 | For a full set of assertions, see [the schema](https://github.com/flowbased/fbp-spec/blob/master/schemata/expectation.yaml)
|
219 |
|
220 | #### More
|
221 |
|
222 | A comprehensive set of examples can be found under [./examples](./examples).
|
223 | For the detailed definition of the dataformat for tests, see [schemata/](./schemata/).
|
224 |
|
225 |
|
226 | ## Running tests with fbp-spec commandline tool
|
227 |
|
228 | The simplest and most universal way of running tests is with the `fbp-spec` commandline tool.
|
229 |
|
230 | $ fbp-spec --address ws://localhost:3333 examples/multisuite-failandpass.yaml
|
231 |
|
232 | MultiSuite, failing tests
|
233 | sending a boolean with wrong expect
|
234 | should fail: ✗ Error: expected true to deeply equal false
|
235 | sending a number with wrong expect
|
236 | should fail: ✗ Error: expected 1000 to deeply equal 1003
|
237 | MultiSuite, passing tests
|
238 | sending a boolean
|
239 | should repeat the same: ✓
|
240 | sending a number
|
241 | should repeat the same: ✓
|
242 |
|
243 | The `--command` options can be used to specify a command which will start the runtime under test:
|
244 |
|
245 | fbp-spec --command "python2 protocol-examples/python/runtime.py"
|
246 |
|
247 | It sets the exit status to non-zero on failure, so is suitable for integrating into a `Makefile` or similar.
|
248 |
|
249 | ## Running tests by integrating with Mocha
|
250 |
|
251 | [Mocha](http://mochajs.org/) iss a popular test runner framework for JavaScript/CoffeeScript on browser and node.js.
|
252 |
|
253 | Since fbp-spec communicates with your runtime over a network protocol,
|
254 | you can use this also when your project is not JavaScript-based.
|
255 | The Mocha runner is for instance [used in microflo-core](https://github.com/microflo/microflo-core/blob/master/spec/ComponentTests.coffee)
|
256 | to test C++ components for microcontrollers & embedded devices.
|
257 |
|
258 | You can have your fbp-spec tests run in Mocha by calling the `fbpspec.mocha.run()` function, in a file which is
|
259 | executed with the standard Mocha runner. Eg. `mocha --reporter spec tests/fbpspecs.js`
|
260 |
|
261 | // fbpspecs.js
|
262 | fbpspec = require('fbp-spec');
|
263 |
|
264 | rt = {
|
265 | protocol: "websocket",
|
266 | address: "ws://localhost:3569",
|
267 | secret: "py3k", // Optional. If needed to connect/authenticate to runtime
|
268 | command: 'python2 protocol-examples/python/runtime.py' // Optional. Can be used to start runtime automatically
|
269 | };
|
270 | fbpspec.mocha.run(rt, './examples/simple-passing.yaml', { starttimeout: 1000 });
|
271 |
|
272 | The tests can be specified as a list of files, or directories.
|
273 | You can use the standard `grep` option of Mocha to run only some tests.
|
274 |
|
275 | For CoffeScript example, see [./spec/mocha.coffee](./spec/mocha.coffee).
|
276 |
|
277 | ## Running tests interactively in Flowhub
|
278 |
|
279 | [Flowhub](http://app.flowhub.io) IDE (version 0.11 and later) has integrated support for fbp-spec. No installation is required.
|
280 |
|
281 | * Open existing project, or create a new one
|
282 | * Open a component, and write/copypaste in a test in the `Tests` panel
|
283 | * Ensure you have a runtime set up, and connected
|
284 |
|
285 | When you make changes to your project (components,graphs) or tests, Flowhub will now automatically (re-)run your tests.
|
286 | You can see the status in the top-right corner. Clicking on it brings up more details.
|
287 |
|
288 | ## Generating tests programatically
|
289 |
|
290 | The test-format defined by fbp-spec is fairly generic and versatile. It is intended primarily as
|
291 | a format one directly specifies tests in, but can also be generated from other sources.
|
292 |
|
293 | Sometimes data-driven testing, one does a large amount of very similar tests,
|
294 | with multiple test-cases per set of input data.
|
295 | By capturing only the unique parts of testcases in a specialied data-structure (JSON, YAML, etc),
|
296 | and then transforming this into standard `fbp-spec` files with some code, adding/removing
|
297 | cases becomes even easier.
|
298 | For instance in `imgflo-server`, [testcases](https://github.com/jonnor/imgflo-server/blob/master/spec/graphtests.yaml)
|
299 | can be defined by providing a name, an URL and a reference result (a file with naming convention based on name).
|
300 |
|
301 | Similarly, one can generate testcases using fuzzing, schema-based, model-based or similar tools.
|
302 |
|
303 | ## Integrating test runner in an application
|
304 |
|
305 | The test runner code is accessible as a JavaScript library,
|
306 | and can be integrated into other apps (like Flowhub does).
|
307 | See examples of [commandline](./src/cli.coffee) and [webappp](./ui/main.coffee) usage.
|
308 |
|
309 | ## Add supporting for a new runtime
|
310 |
|
311 | You need to implement the [FBP network protocol](https://github.com/flowbased/fbp-protocol).
|
312 | At least the `protocol:runtime`, `protocol:graph`, and `protocol:network` capabilities are required.
|
313 |
|
314 | All transports supported by [fbp-protocol-client]((https://github.com/flowbased/fbp-protocol))
|
315 | are supported by fbp-spec, including WebSocket, WebRTC, and iframe/postMessage.
|
316 |
|
317 | fbp-spec is intended to be used with flow-based and dataflow-programming,
|
318 | but might be useful also outside these programming paradigms. Try it out!
|
319 |
|
320 | ## Writing a test runner in another language
|
321 |
|
322 | As long as you stay compatible with the [fbp-spec testformat](./schemata/)
|
323 | and [FBP protocol](http://noflojs.org/documentation/protocol/),
|
324 | you can implement a compatible runner in any programming language.
|
325 |
|
326 | You can consider the fbp-spec code (in CoffeeScript) as a reference implementation.
|
327 |
|