1 |
|
2 | <img align="right" width="20%" height="20%" src="https://raw.githubusercontent.com/sumanjs/suman-docs/master/images/suman-hex.png">
|
3 |
|
4 | [![npm version](https://badge.fury.io/js/suman.svg)](https://badge.fury.io/js/suman)
|
5 |
|
6 | [![Greenkeeper badge](https://badges.greenkeeper.io/sumanjs/suman.svg)](https://greenkeeper.io/)
|
7 |
|
8 | [![Build Status](https://travis-ci.org/sumanjs/suman.svg?branch=master)](https://travis-ci.org/sumanjs/suman)
|
9 |
|
10 | [![CII Best Practices](https://bestpractices.coreinfrastructure.org/projects/1072/badge)](https://bestpractices.coreinfrastructure.org/projects/1072)
|
11 |
|
12 | [![Coverage Status](https://coveralls.io/repos/github/sumanjs/suman/badge.svg?branch=master)](https://coveralls.io/github/sumanjs/suman?branch=master)
|
13 |
|
14 |
|
15 | # Suman: universal test runner - run tests in the language of your choice.
|
16 |
|
17 | Designed to run tests written in any language - because Suman runs tests as child processes - just write TAP to stdout
|
18 | via the runtime of choice (Golang, Java, Python, Node.js, Bash, whatever).
|
19 |
|
20 | <p>
|
21 | Suman is intended for unit testing in the browser, for backend and system testing,
|
22 | for end-to-end testing with Selenium, etc.
|
23 |
|
24 | <p>
|
25 | Originally designed for TypeScript and Babel transpilation, Suman was generified to support
|
26 | compiling from source to target for <i>any</i> language. Suman is written in TypeScript, and has strong support for TS.
|
27 | All in all, Suman is a batteries-included test runner which encourages testing best practices through its design.
|
28 | Starting with Suman is very easy, but you will find it has extremely powerful features, as your demands increase.
|
29 |
|
30 | <p>
|
31 | <b>Perhaps most importantly, Suman makes testing significantly faster for both CPU bound and I/O bound test suites.</b>
|
32 | <b>The speed benefits are generally most useful during test development</b>
|
33 |
|
34 | <p>
|
35 | As final points, Suman is designed to be universal and not just another JS framework - it's designed for the long-haul.
|
36 | <p>
|
37 | Elegance and utility of reporting output is paramount to the Suman philosophy.
|
38 | Reporters are the best place to contribute for people interested in creating beautiful test output.
|
39 | As they say (visual) design is what really separates contenders in a world of surplus.
|
40 | <p>
|
41 |
|
42 |
|
43 | # Purpose of project
|
44 | > I wrote Suman because I found test runners and test harnesses in the Node.js ecosystem to be lacking.
|
45 | > I consider Mocha to be woeful, and AVA to be slow, under-featured and too tightly coupled with Babel.
|
46 | > If your team is interested in speeding up your testing cycles, Suman is the absolute right place to look for answers.
|
47 | > Suman is designed to be 'better all-around' than AVA, TapJS and Mocha, etc. Reading the issue tracker for Mocha
|
48 | > made it very apparent that Mocha was never going to be vastly improved. If you look at the Karma codebase, you
|
49 | > also find that it's not designed with Webpack, TypeScript or Babel in mind, etc.
|
50 | >
|
51 |
|
52 | # The Suman Challenge
|
53 |
|
54 | If you can find a problem that Mocha or AVA has, which Suman does not solve, I will find some reward for you.
|
55 | <br>
|
56 | Suman was written so that it would suffer from none of the problems that existing test runners have.
|
57 |
|
58 |
|
59 | ### ► Disclaimers:
|
60 | >
|
61 | > Suman supports Node versions >= 6.0.0, since it makes heavy use of the Proxy class and Symbol primitives.
|
62 | >
|
63 | > Windows support is on the roadmap, but will not be ready anytime soon. Currently, MacOS and *nix support only.
|
64 | >
|
65 | > Browser will be supported, but not until ~Summer 2018.
|
66 | >
|
67 |
|
68 | ----------------------------------------------------------------------------------------
|
69 |
|
70 | # Expected official release date
|
71 |
|
72 | Suman is not officially released yet - expected release date ~October 2017.
|
73 | Until then, expect bugs, missing docs, etc. Proceed at your own risk :D
|
74 |
|
75 |
|
76 | # ► Documentation
|
77 |
|
78 | The Suman docs [sumanjs.org](http://sumanjs.org "Suman Docs")
|
79 |
|
80 | -----------------------------------------------------------------------------------------
|
81 |
|
82 |
|
83 | ## Suman is made up of two independent pieces:
|
84 |
|
85 | 1. A CLI that can run a test in any language, and is designed to handle transpilation/compilation for any language as well.
|
86 | 2. A super powerful test harness that can be used with JavaScript/Node.js tests. This test harness is highly recommended
|
87 | and has strong support for TypeScript.
|
88 |
|
89 | The CLI can be used to run tests in any language; on the other hand,
|
90 | the test harness, only works with Node.js and browser based JS.
|
91 |
|
92 | You do not need the suman CLI to run suman tests. <br>
|
93 | You do not need to run suman tests to use the CLI. <br>
|
94 | They are completely independent, while obviously being designed to work great together. <br>
|
95 |
|
96 | <p>
|
97 |
|
98 | --------------------------------------------------------------------------------------------
|
99 |
|
100 | ## What makes Suman better and different
|
101 |
|
102 | <div> Everything about Suman is designed for parallelization. </div>
|
103 |
|
104 | Suman is not just better than test runners in Node.js and JS land - it improves on test runners written
|
105 | for other languages as well. This is primarily because the Suman CLI is built so that you can have complete control
|
106 | over the parellization of your tests. You can select which tests should exclude each other and run in series,
|
107 | and which tests should run in parallel with each other. In the _long run_ however, all tests should run in parallel
|
108 | in their own container with their own resources. In the _long long run_, each container should run on its own hardware,
|
109 | all in the name of speed. Suman has an experimental feature called "suman groups" which will allow you to group your
|
110 | tests together and assign a group to a container. Eventually the dream is support the assignment of each container
|
111 | to its own hardware, using different cloud platforms.
|
112 |
|
113 | ___
|
114 |
|
115 | # Top 5 reasons to use Suman, instead of Mocha or AVA
|
116 |
|
117 | 1. Run tests in any language or executable, not just Node.js/JavaScript.
|
118 |
|
119 | 2. => You can run Mocha tests and AVA tests using the Suman CLI, but not the other way around!
|
120 |
|
121 | 3. Easily containerize any test process in development, using `suman --ctrz x.js`
|
122 |
|
123 | 4. Imagine you're a senior developer and a more junior developer joins the team,
|
124 | and they write a new test that brings down the whole CI/CD pipeline. With Suman, tests don't directly
|
125 | interact because they are run in separate processes; in Mocha, not so much. Avoid that fateful day.
|
126 | Using Suman, it will be clear which test process is causing the fatal problem; but with a single process
|
127 | it will not necessarily be clear that it is the junior developer's test which caused the issue.
|
128 |
|
129 | 5. Suman is much faster than AVA, because Suman does not require transpilation.
|
130 |
|
131 | <p>
|
132 |
|
133 | # Suman CLI features
|
134 |
|
135 | ```console
|
136 | $ suman>
|
137 | ```
|
138 |
|
139 | <p>
|
140 | <div> ✓ <b style="color:purple"> tests can all run in parallel, in separate processes </b> </div>
|
141 | <div> ✓ <b style="color:purple"> agnostic </b> => want to learn a new language? write a test script with language X, then run the test with Suman.</div>
|
142 | <div> ✓ <b style="color:purple"> flexible, generic, robust </b> => CLI can run JS tests directly, or in a child process </div>
|
143 | <div> ✓ <b style="color:purple"> flexible, generic, robust </b> => Composability => Suman tests can run other Suman tests (in child processes). Sumanception! </div>
|
144 | <div> ✓ <b style="color:purple"> test isolation </b> => each test can run in its own process </div>
|
145 | <div> ✓ <b style="color:purple"> test independence </b> => easily run only one test at a time (unlike other Node.js test runners...)</div>
|
146 | <div> ✓ <b style="color:purple"> "nodeable test scripts" </b> => run individual JS tests with the node.js executable</div>
|
147 | <div> ✓ <b style="color:purple"> supports unit testing in the browser </b> (tested on Chrome and Firefox)</div>
|
148 | <div> ✓ execute tests written in <b style="color:purple">any</b> language, use write TAP to stdout</div>
|
149 | <div> ✓ synchronous <b style="color:purple">*and*</b> asynchronous reporters (!) Your reporter can be async.</div>
|
150 | <div> ✓ execute tests written in <b style="color:purple">any</b> language, use write TAP to stdout</div>
|
151 | <div> ✓ Complete control => You *can* run JS unit tests all in the same process for speed, as needed. </div>
|
152 | <div> ✓ <b style="color:purple"> bash completion is available for the Suman CLI :D </b>
|
153 | </p>
|
154 |
|
155 |
|
156 | ## Suman test harness features
|
157 |
|
158 | ```javascript
|
159 | import * as suman from 'suman';
|
160 | ```
|
161 |
|
162 | These are the features for creating tests in JavaScript/TypeScript.
|
163 |
|
164 | <p>
|
165 | <div> ✓ <b style="color:purple"> fully asynchronous </b> => allows for easy, dynamic test case creation
|
166 | <div> ✓ <b style="color:purple"> agnostic </b> => works with your favorite assertion library; Suman is also bundled with Chai, if you wish to use that.
|
167 | <div> ✓ <b style="color:purple"> no globals </b> => no global variables as part of test harness - therefore JS tests can be run with Node.js executable </div>
|
168 | <div> ✓ <b style="color:purple"> flexible, generic, robust </b> => CLI can run JS tests directly, or in a child process </div>
|
169 | <div> ✓ <b style="color:purple"> test isolation </b> => each test can run in its own process </div>
|
170 | <div> ✓ <b style="color:purple"> declarative style </b> => declare (sync and async) dependencies for each test, and only load those</div>
|
171 | <div> ✓ <b style="color:purple"> supports unit testing in the browser </b> (tested on Chrome and Firefox)</div>
|
172 | <div> ✓ <b style="color:purple"> supports observables (RxJS5) </b> </div>
|
173 | <div> ✓ only <b style="color:purple">18mbs </b> on filesystem as npm install -D</div>
|
174 | <div> ✓ works with <b style="color:purple"> Selenium </b> (use selenium-webdriver, wd, or webdriver.io)</div>
|
175 | <div> ✓ Built-in watch features => Watch files, and run tests on changes </div>
|
176 | <div> ✓ Complete control => You *can* run unit tests all in the same process for speed, as needed. </div>
|
177 | </p>
|
178 |
|
179 | ---
|
180 |
|
181 | ## Suman is a singular test runner focused on Node.js and front-end JavaScript, but is both generic and robust so that it can run tests in any runtime or language
|
182 |
|
183 |
|
184 | Suman is written with Node.js, and is focused on testing Node.js code,
|
185 | but can run tests written in any language, not just JavaScript. This is
|
186 | because it can run tests in child processes and collect results using TAP (Test Anything Protocol, via stdout), IPC, or websockets.
|
187 |
|
188 | Suman can run a test in any language which exposes a script with a hashbang or a binary entrypoint file (e.g. Golang or C).
|
189 | To run Java tests, where Java does not compile to binary and where you cannot put a hashbang in a .class file,
|
190 | you would need to call Java from a shell script.
|
191 |
|
192 | It is designed for maximum test performance, via careful parallelization at
|
193 | every juncture in the testing process/pipeline.
|
194 |
|
195 | -----
|
196 |
|
197 |
|
198 | # ► Installation
|
199 |
|
200 | <i> For command line tools:</i>
|
201 | ## ```$ npm install -g suman```
|
202 |
|
203 | => **Please** do *not* use sudo to install suman globally; if you need to use sudo, then something is probably wrong
|
204 | => See: https://docs.npmjs.com/getting-started/fixing-npm-permissions
|
205 | => To avoid any problems with permissions, Suman recommends usage of NVM
|
206 |
|
207 | <i> For test suites in your project:</i>
|
208 | ## ```$ cd <project-root> && suman --init```
|
209 |
|
210 | * To avoid global NPM modules see: "Local installations only"
|
211 |
|
212 | => to convert a Mocha test or whole directory(s) of Mocha tests to Suman tests use <br>
|
213 | ### ```$ suman --convert --src=<src-file/src-dir> --dest=<dest-dir>```
|
214 |
|
215 | => to simply install Suman as dev-dependency in any project you can use ```$ npm install -D suman```, <br>
|
216 | however ```$ suman --init``` is the much preferred way to initialized suman in a given project, because it will
|
217 | add a directory to your project which you can move to a location of your choosing.
|
218 |
|
219 |
|
220 | ### ► Local installations only =>
|
221 |
|
222 | If you wish to avoid global NPM module installations, we commend you, see:
|
223 | [http://sumanjs.org/tutorial-11-advanced-installation.html/](http://sumanjs.org/tutorial-11-advanced-installation.html "Suman Docs: Advanced Installation")
|
224 |
|
225 |
|
226 | ## Example commands
|
227 |
|
228 | ```bash
|
229 |
|
230 | suman test/**/*.py # run python tests
|
231 |
|
232 | suman test/**/*.rb # run ruby tests
|
233 |
|
234 | suman test/**/*.sh test/**/*.go # run bash and golang tests
|
235 |
|
236 | suman test/src/*.spec.js --concurrency=6 # run the matching tests, no more than 6 Node.js processes at a time.
|
237 |
|
238 | suman -w project # run a set of tests when a project file changes
|
239 |
|
240 | suman -w tests # when developing a test, run it upon changes
|
241 |
|
242 |
|
243 | ```
|
244 |
|
245 | ## The Suman Story
|
246 |
|
247 | The human side of this endeavor. (A link to docs will be here soon.);
|
248 |
|
249 |
|
250 | ### Suman uses several intelligent pieces of setup:
|
251 |
|
252 | Global installations of Suman simply look for local installations to run. So if you run the suman command installed via npm install -g suman, that CLI
|
253 | will just invoke the local installation of suman in any given project - this prevents any potential
|
254 | conflicts if there is a difference between the global/local module versions. Excellent!
|
255 |
|
256 | If you use NVM and switch between Node.js versions, you can use bash functions (provided by SumanJS) which will
|
257 | denecessitate the need for any global installations of Suman at all.
|
258 |
|
259 | Suman is designed to interop well with the most common libraries in the ecosystem for handling asynchronous code.
|
260 |
|
261 |
|
262 | ## Simple example
|
263 |
|
264 | ```js
|
265 |
|
266 | import * as suman from 'suman';
|
267 | const {Test} = suman.init(module);
|
268 |
|
269 | Test.create('example', (baz, http, beforeEach, context, inject, foo, x, beforeAll) => {
|
270 |
|
271 | // Suman uses simple old-school style JavaScript DI
|
272 | // we have injected some core modules by name (http, assert, path)
|
273 | // we have also injected a module from our own project, baz
|
274 |
|
275 | inject('bar', () => {
|
276 | return baz(foo).then(v => {
|
277 | return v.filter(val => val.isGreen())
|
278 | })
|
279 | })
|
280 |
|
281 | beforeAll(h => {
|
282 | return x.anything().then(function(v){
|
283 | h.assert(typeof v === 'boolean');
|
284 | h.$inject.v = v;
|
285 | });
|
286 | });
|
287 |
|
288 | beforeEach(t => {
|
289 | t.data.v = (t.value.v * 2) + t.$inject.v;
|
290 | })
|
291 |
|
292 | context('foo', {mode: 'series'}, (bar, it) => {
|
293 |
|
294 | it('a', {value: 5}, t => {
|
295 | t.assert.equal(t.title,'a')
|
296 | t.assert.equal(t.data.v,10)
|
297 | })
|
298 |
|
299 | it('b', t => {
|
300 | t.assert.equal(t.title, 'b')
|
301 | })
|
302 |
|
303 | it('c', t => {
|
304 | t.assert.equal(t.title, 'c')
|
305 | })
|
306 |
|
307 | context('nested child', {mode: 'parallel'}, (bar, it) => {
|
308 |
|
309 | it('a', t => {
|
310 | t.assert.equal(t.title, 'a')
|
311 | })
|
312 |
|
313 | it('b', t => {
|
314 | t.assert.equal(t.title, 'b')
|
315 | })
|
316 |
|
317 | it('c', t => {
|
318 | t.assert.equal(t.title, 'c')
|
319 | })
|
320 |
|
321 | })
|
322 |
|
323 | })
|
324 |
|
325 | })
|
326 |
|
327 | ```
|
328 |
|
329 |
|
330 | # ► Purpose
|
331 |
|
332 | The purpose of the Suman library is to provide the most sophisticated test runner in the Node.js ecosystem,
|
333 | with a better developer experience, better features, higher performance, improved debuggability,
|
334 | and more expressiveness than AVA, Mocha, and Tape.
|
335 | Suman is a first-rate library and we hope you take the time to compare its capabilities with AVA, Mocha, Tape, TapJS, etc.
|
336 | One of the more exciting things about Suman is that it can run tests written in any language; all you have to do
|
337 | is output TAP (Test Anything Protocol).
|
338 |
|
339 | The primary aims are:
|
340 |
|
341 | * Developer experience and test debuggability are above ALL else
|
342 | * Provide a beautiful and intuitive API
|
343 | * Solve all major and minor problems in the Mocha API, specifically
|
344 | * Borrow some of the best features from Mocha, AVA and Tape
|
345 | * Make tests run faster by leveraging concurrency provided by async I/O *and* separate Node.js processes
|
346 | * _Isolate_ tests by running them in separate processes, so they do not share memory nor interact directly
|
347 | * Make tests _independent_, so that you can easily run one test at a time (damn you Mocha!).
|
348 | * Make debugging your test files easier; this is achieved by allowing for the running of tests with the plain-old node executable,
|
349 | this makes Suman tests "node-able"
|
350 | * Provide cleaner output, so that developer logging output is not necessarily mixed with test result =>
|
351 | achieved by using child processes and ignoring stdout.
|
352 | * Add the missing features from Mocha, Tape and AVA, while simplifying portions of the Mocha API and doing
|
353 | away with (implicit) global variables.
|
354 | * Stick closely to the popular Mocha API, so that automatic conversion is possible from Mocha to Suman,
|
355 | and that transition is as seamless as possible for the developer - you do *not* have to learn a whole new API!
|
356 | * Allow for maximum dynamicism so that Suman can match all use cases of users.
|
357 | * Allow users to create Suman-runnable tests in any languages - this possibility is created by running tests in child processes.
|
358 | Writing tests in the form of shell scripts is something that almost every project could benefit from and now you can run these
|
359 | tests through your universal test runner - Suman!
|
360 | * Composability - Suman tests should be able to run suman tests as child processes, ad inifitem, without hiccups.
|
361 |
|
362 |
|
363 | # ► Test Framework Comparison
|
364 |
|
365 | ## The Table of Goodness
|
366 |
|
367 |
|
368 | | | Node-able | Supports ES6/ES7 features | Supports test isolation using multiple Node.js processes | Concurrency within suites | Dependency Injection |
|
369 | |---------|---------------------------------------------------------------------------|---------------------------------------|-----------------------------------------------------------|---------------------------|----------------------|
|
370 | | Mocha | No | No | No | No | No |
|
371 | | Jasmine | No | No | No | No | No |
|
372 | | Tape | Yes | No | No | No | No |
|
373 | | AVA | No, requires transpilation first | Yes | Yes | Yes | No |
|
374 | | Suman | Yep, you can run any given test suite with the plain old node executable | Yep, Suman will support all features | Yep | Yep | Yep |
|
375 |
|
376 |
|
377 | ## Le Matrix of Madness
|
378 |
|
379 | | | Implicit globals | Forces you to use their assertion library madness | Confusing JS contexts madness | Developer debugging / console.log output mixed with test output madness | no concurrency madness |
|
380 | |---------|------------------|----------------------------------------------------|-----------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------|------------------------|
|
381 | | Mocha | Yes | No | Yes | Yes | Yes |
|
382 | | Jasmine | Yes | No | Yes | Yes | Yes |
|
383 | | Tape | No | Yes | No | Yes | Yes |
|
384 | | AVA | No | Yes | No | ? | No |
|
385 | | Suman | Nope | Nope, Suman is completely assertion-lib agnostic | Nope, Suman greatly simplifies the context puzzle that Mocha provided | Nope, Suman runner uses silent option with child_process so your output doesn't mix with the test results | Nope |
|
386 |
|
387 |
|
388 | ## The reasons why Mocha and its peers need a replacement are clear:
|
389 |
|
390 | * In Mocha, Tape and Jasmine test files were not run in separate processes (necessary for isolation, speed and independence of test results)
|
391 | * Using Mocha, Tape and Jasmine, not only was everything run in a single process, but all test cases and hooks were also run in series, which takes unnecessary amounts of time for tests utilizing async I/O
|
392 | * Mocha prescribed no solution to the problem of starting up the services necessary to do system/integration testing - using Mocha/Tape/AVA it is up to the developer to manually start those services,
|
393 | which makes automated testing much more difficult.
|
394 | * Single process test runners like Mocha face out-of-memory issues - https://github.com/mochajs/mocha/issues/2555, these issues are
|
395 | much much less likely to occur if tests are split into multiple processes
|
396 | * Mocha and Jasmine could not move forward with ES6/ES7 features due to certain software patterns used (globals and complex context binding)
|
397 | * a BIG ONE: clean reporting - at the command line, using Mocha and Jasmine, logging/debugging output by the developer would obfuscate the test results, nullifying any advantage of reporting tools. Suman has a simple
|
398 | trick up its sleeve to allow for 100% clean reporting for any test or group of tests.
|
399 | * Mocha and Jasmine could not have certain useful serverside features, because they were also constained by running in the browser.
|
400 | * Mocha lacked real ability to do true dynamic testing (meaning, registering a dynamic number of it() test cases) => e.g., make a network call, get X values, create a test case for each.
|
401 | * Mocha had confusing and obfuscated context values (values for 'this'), which we have greatly simplified, allowing for the usage of arrow functions, etc
|
402 | * Mocha, Jasmine and Tape lack some other nice features that are implemented by Suman. For example, Tape's ability to pre-load modules using the command line
|
403 | before running tests is nowhere near as powerful or easy to use as the dependency injection ability of this library.
|
404 | * Using Mocha, Tape, Jasmine you could not easily pass data to tests, to reuse the same test code for different scenarios; Suman allows you to pass dynamic data
|
405 | to tests using dependency injection.
|
406 | * cannot call tests programmatically without wrapping Mocha test suite in a function, Suman allows you to call tests programmiatcally, without having
|
407 | to wrap tests in a function.
|
408 | * After writing a few Mocha tests, developers will find it's very difficult to properly run only one test - the "only" feature can help,
|
409 | but there are so many bugs that can crop up because of this, especially if you have some global hooks that need to run before your "only" test needs
|
410 | to run. Suman solves this problem ridiculously well, because Suman was designed to solve this very problem from the ground up.
|
411 |
|
412 |
|
413 | ## Suman Philosophy
|
414 |
|
415 | * "Just works" - no need for addons or plugins, unless you want to write a custom reporter
|
416 | * Fail-fast
|
417 | * Stick to Node core modules
|
418 | * Unlike AVA, you don't need to transpile with Babel if you don't want to: _as ES6 generators + Promises
|
419 | can give you the same coding patterns as ES7 async/await_
|
420 | * Provide a full-featured, non-dumbed-down API that's easy to get started with, and
|
421 | both powerful and intuitive to use over the long-run.
|
422 | * Listen to what the community wants.
|
423 | * Leverage Javascript's strengths (ahem *closures*)
|
424 | * Don't be lazy.
|
425 | * As Suman is a command line application, we can utilize a more functional programming style
|
426 | * Details matter, and developer experience matters most.
|
427 |
|
428 |
|
429 | # ► Suman features in detail:
|
430 |
|
431 | * <b> basics </b>
|
432 | * => tdd/bdd interfaces
|
433 | * => easy migration from Mocha (it's automated using the --convert option)
|
434 | * => extremely powerful features, while aiming to be straightforward, clean, concise, consistent and accurate;
|
435 | * => designed with ES6 and ES7 in mind, including Promises, generators and async/await
|
436 |
|
437 |
|
438 | * <b> Improved mechanics, syntax and semantics </b>
|
439 | * singular param (t) is used for all hooks and test cases, in the style of AVA
|
440 | * Pass data from test cases directly to hooks using the t.value option of a test case
|
441 | * Pass data from hooks directly to test cases using the t.data value
|
442 | (neither are possible with Mocha, and are very much a missing feature)
|
443 | * encapsulation and immutability are utilized much more effectively than with Mocha etc
|
444 | * instead of a "grep" option like Mocha's, we have "match" because we are filtering input not output!
|
445 |
|
446 |
|
447 | * <b> Full-blown concurrency</b>
|
448 | * your tests will run much faster, especially if they are doing lots of network I/O
|
449 | * test files are run in separate Node.js processes for speed, isolation and independence
|
450 | * test cases and hooks in any given suite can be run concurrently, when using asynchronous I/O,
|
451 | using the "parallel" option in your code
|
452 | * capability to control maximum number of processes running at a time (cap it at 6 processes, or 18 processes or whatever)
|
453 | * capability to add constaints to prevent any given pair of tests from running at the same time, (if two different tests access the same
|
454 | external resource for example, and you don't want the interaction to cause false negatives).
|
455 |
|
456 |
|
457 | * <b> Improved reporting </b>
|
458 | * using the Suman test runner, you can prevent any logging output from mixing with test reports, by redirecting stdout from the child process(es).
|
459 | * Future effort: Suman includes a built-in web reporter that you can use to share test results with your team, using the Suman server
|
460 | * Future effort: Suman server provides ability to store past test results (backdata) and view test results chronologically with browser to look at trends
|
461 | * Future effort: testing backdata is stored in a local SQLite database
|
462 | which will allow you to run real queries on your test results, and share results with your team.)
|
463 |
|
464 | * <b> Automatic test execution and/or test transpilation </b>
|
465 | * Using ```suman --watch``` you can execute test files or transpile test files as you write them
|
466 | * Suman watcher processes run your tests and pipe stdout/stderr to log files which you tail with a terminal or browser window
|
467 | * Running tests on the fly is a major portion of the optimal Suman workflow, and makes it all the more fun.
|
468 |
|
469 |
|
470 | * <b> Dynamicism </b>
|
471 | * If there's anything you want to do with a test runner, you can do it with Suman
|
472 | * Test files themselves allow for all sorts of dynamic behavior, dynamic and asynchronous test case generation, etc
|
473 | * You can call tests programmatically and use them as macros
|
474 | * Suman tests can create child processes which call Suman tests, etc etc.
|
475 |
|
476 |
|
477 | * <b> Use suman.once.js to run hooks before the test runner starts </b>
|
478 | * these hooks can be used to start the services necessary for any given test file to run successfully; they only run once no matter how many tests are run, are only run
|
479 | if tests declare so. They offer a lighter weight option than containers for starting up the necessary servers in your backend system.
|
480 | * Your suman.once.js file can augment the behavior of container build scripts etc, to help start up services necessary for testing to commence
|
481 |
|
482 |
|
483 | * <b> Very simple but powerful dependency injection (DI/IoC)</b>
|
484 | * Inject dependencies sourced synchronously or asynchronously
|
485 | * Most useful for injecting values acquired _asynchronously_, such as successful network connections and database values
|
486 | * Inspired by familiar tools such as Angular and RequireJS
|
487 | * Inject any core/"built-in" Node.js module by name, with zero configuration
|
488 | * DI is used throughout the library, and relieves the burden on the developer to remember order of parameters
|
489 | * Inject network values, test dependencies and library dependencies
|
490 |
|
491 |
|
492 | * <b> Test runner tuning </b>
|
493 | * Add contraints to prevent any given pair of tests from running at the same time
|
494 | * Cap the total number of processes running at the same time
|
495 | * Suman 'once' feature gives the developer the option to run checks to see if all necessary network components are live before running any given test
|
496 |
|
497 |
|
498 | * <b> Easy migration from Mocha </b>
|
499 | * Suman includes a command line option to convert whole directories or individual Mocha tests to Suman tests
|
500 | * before/after/beforeEach/afterEach hooks behave just like in Mocha
|
501 | * solid command line tools and better grep facilities than predecessors
|
502 | * skip/only also work like Mocha
|
503 |
|
504 |
|
505 | * <b> Optional but first-rate transpilation features </b>
|
506 | * Suman support for tranpilation is first-rate
|
507 | * Suman allows you to use "babel-register" to transpile your sources on the fly or transpile a src directory to a target directory
|
508 | * If you have less than ~20 tests, the recommended way to incorporate Babel in your testing is to simply transpile your entire "test" directory to "test-target"
|
509 | * Using a "test-target" directory instead of "babel-register" allows for better debugging, performance and transparency in your testing system
|
510 | * If you have more than 20 tests, Suman does not recommend transpilation, due to the performance penalty.
|
511 | * For enterprise usage, we don't expect many teams to use transpilation features anyway; it's more likely individual developers who may
|
512 | wish to use Babel to transpile tests for a library.
|
513 |
|
514 |
|
515 | * <b> Log files help you debug </b>
|
516 |
|
517 | * Using Suman tests, you have 3 options as to how you execute a test file: ```1. node x.js, 2. suman x.js and 3. suman --runner x.js```
|
518 | * Using the Suman runner, your test file(s) all run in child processes. This can make debugging harder, but Suman sends all stderr from the child processes
|
519 | to a single log file to help you debug.
|
520 | * Normally, to debug a test, you would run the test with node or using suman without using the runner, but if you see a problem with a particular test that only
|
521 | occurs when using the runner, then the log files will help.
|
522 |
|
523 | * <b> Freedom: Suman is not highly opinionated, but gives you powerful features</b>
|
524 | * Suman prefers the standard core assert Node module (Suman has unopinionated assertions), but like Mocha you can use any assertion lib that throws errors
|
525 | * Callbacks, promises, async/await, generators and event-emitters/streams are supported in any test case or hook.
|
526 |
|
527 |
|
528 | ## Suman design
|
529 |
|
530 | * no implicit globals in test suite files, which were avoided due to the problems they caused for Jasmine and Mocha.
|
531 | * Suman uses domains to isolate errors in asynchronous tests and hooks, and currently this is the only solution to this problem at the moment.
|
532 | Lab, the test runner for Hapi servers, also uses domains for this same purpose, and using domains allows for much better coding patterns (avoiding globals in
|
533 | the suman library.) Domains are facing deprecation, and Suman will replace domains with whichever suitable replacement is chosen by the Node.js core team,
|
534 | although after talking with many people near to core, domains will be supported for quite some time. As it stands, _Suman is a perfect use case for domains,
|
535 | as untrapped errors are supposed to be thrown in test code_ (assertions, etc), and the developer experience in this library will be better than any other test library because of the use of domains, as they basically
|
536 | guarantee that we can pin an error to a particular test case or hook, no matter where the error originated from.
|
537 |
|
538 | ## Details matter
|
539 |
|
540 | * we designed Suman with details in mind
|
541 | * fewer logical branches and contingencies than Mocha WRT rules for hooks
|
542 | * much better semantics, with new standard functions alongside Mocha's 'done' callback: 'ctn', 'pass', 'fail' and 'fatal' are new functions
|
543 | each with a unique purpose and meaning, and done is still in Suman's API with the same meaning as Mocha!
|
544 | * friendly error messages, that also get sent to suman-stderr.log for reference
|
545 | * when debugging, (the debug flag is set) timeouts will automatically be set to 'infinity'
|
546 |
|
547 |
|
548 | ## We can say with confidence that Suman is the most powerful test framework for serverside JavaScript on planet Earth
|
549 | => as it gives the developer total control and access to a very large set of features,
|
550 | with the explicit goal of being bug-free first, full-featured second.
|
551 |
|
552 |
|
553 | ### More Suman Examples
|
554 |
|
555 | * See the documentation @ sumanjs.org
|
556 | * You can also read: https://medium.com/@the1mills/introducing-suman-a-node-js-testing-library-20fdae524cd
|
557 |
|
558 | ### S.L.A.
|
559 |
|
560 | The Service Level Agreement is that Suman will constantly be up-to-date with the newest features available via the node executable.
|
561 | We will focus on what's in Node and not what's available in Babel or other transpilers. That being said, we will also work to ensure Babel features are also supported,
|
562 | but we will primarily focus on making Suman completely bug-free when it comes to the latest version of Node, not the latest abilities of Babel or the like.
|
563 | By the time any ES6/ES7/ES8 feature is available in Node, it will be supported by Suman. We want to emphasize the utility of the option of running things
|
564 | with the plain old Node executable, as opposed to adding the complexity of transpilation.
|
565 |
|
566 |
|
567 | ### Execution modes for a single test file
|
568 |
|
569 | You can execute a test file with the plain ```node``` executable, with ```$ suman``` and with ```$ suman --runner```
|
570 |
|
571 | Here are the differences between the 3 options:
|
572 |
|
573 | | | $ node a.test.js | $ suman a.test.js | $ suman --runner a.test.js |
|
574 | |-------------------------|-------------------|--------------------|-----------------------------|
|
575 | | command line options | no | yes | yes |
|
576 | | runs multiple processes | no | no | yes |
|
577 | | suppress stdout/stderr | no | no | yes |
|
578 | | easy to debug? | easy | medium | hard
|
579 |
|
580 |
|
581 | In order to run multiple files, you must use ```$ suman --runner```; the above table
|
582 | only pertains to running a single test file (usually when developing a particular test.)
|
583 |
|
584 |
|
585 | ### FAQ
|
586 |
|
587 | TBD
|
588 |
|
589 |
|
590 | ## Important aside - How is Suman better than AVA?
|
591 |
|
592 | It should be abundantly clear why Suman is better than Mocha, but how is Suman better than AVA?
|
593 | (If it's not clear why Suman is better than Mocha then scroll back up).
|
594 | Suman borrows some excellent features from Mocha that AVA seems to ignore, including the ability
|
595 | to use nested describe blocks for more control and preventing the sharing of scope within tests. AVA basically
|
596 | co-opted Tape and added concurrency. Suman co-opted Mocha, added concurrency, better reporting, dependency injection and
|
597 | less confusing contexts for callbacks. Suman has more powerful facilities for asynchronous testing than AVA due to Mocha/Jasmine-style hooks
|
598 | and nested describes. Dependency injection also makes Suman extremely convenient and fun to use, compared to AVA.
|
599 | Suman is simply more powerful and richer in features than AVA.
|
600 |
|
601 | * AVA test are not "node-able" - you *cannot* run them with node directly; Suman tests are node-able, which makes
|
602 | debugging so much easier and intuitive! Note that Suman does make is easy for developers to debug child processes.,
|
603 | giving them built-in tools to do so.
|
604 |
|
605 | ![alt text](https://github.com/ORESoftware/suman-private/blob/dev/images/ava-prob.png)
|
606 |
|
607 | * AVA requires Babel transpilation, which adds unnecessary complexity for test environments, and is also much slower
|
608 | * AVA does not handle errors thrown in asynchronous code gracefully, Suman is much better in this regard.
|
609 | * AVA does not feature the nested describes of Mocha or Suman, which limits the expressiveness of the library
|
610 | tremendously!
|
611 | * AVA expects you to use its assertion library, whereas Suman will accept usage of any assertion library that
|
612 | you are already familiar with.
|
613 | * Furthermore, AVA does not prescribe solutions to common test problems -
|
614 |
|
615 | * registering dynamic test cases (given a dynamic value acquired asynchronously)
|
616 | * starting up necessary services before running tests
|
617 | * injecting different dependencies into test files
|
618 |
|
619 |
|
620 | Alternatively, with Suman:
|
621 |
|
622 | *Suman has nested describe blocks, which are imperative for non-trivial tests; a simple but common example is a before hook that you want
|
623 | to run only for a certain subset of tests in your test file. The before hook and tests would go in a nested describe block.
|
624 | *Babel transpilation is totally optional - you can achieve the async/await pattern with generators and promises alone, you don't need ES7 for this
|
625 | *Suman uses domains to correctly map runtime/assertion errors to test cases and hooks,
|
626 | which provides a much more reliable and well-designed piece of software because it can handle any error that gets thrown, not just assertion errors.
|
627 |
|
628 |
|
629 | ### Contributing - Testing and pull requests
|
630 |
|
631 | (Please see contributing.md)
|
632 |
|
633 |
|
634 |
|
635 | ![alt text](https://raw.githubusercontent.com/sumanjs/suman-docs/master/images/suman.png "Suman Primary Logo")
|
636 |
|
\ | No newline at end of file |