1 | # HappyPack (beta) [![Build Status](https://travis-ci.org/amireh/happypack.svg?branch=master)](https://travis-ci.org/amireh/happypack) [![codecov.io](https://codecov.io/github/amireh/happypack/coverage.svg?branch=master)](https://codecov.io/github/amireh/happypack?branch=master)
|
2 |
|
3 | _In a nutshell:_
|
4 |
|
5 | HappyPack makes webpack builds faster by allowing you to transform multiple
|
6 | files _in parallel_.
|
7 |
|
8 | See "How it works" below for more details.
|
9 |
|
10 | ## Motivation
|
11 |
|
12 | - webpack initial build times are horrifying in large codebases (3k+ modules)
|
13 | - something that works against both a one-time build (e.g. for a CI) and
|
14 | continuous builds (i.e. `--watch` during development)
|
15 |
|
16 | ## Usage
|
17 |
|
18 | ```
|
19 | npm install --save-dev happypack
|
20 | ```
|
21 |
|
22 | In your `webpack.config.js`, you need to use the plugin and tell it of the
|
23 | loaders it should use to transform the sources. ~~Note that you must specify
|
24 | the absolute paths for these loaders as we do not use webpack's loader resolver
|
25 | at this point.~~
|
26 |
|
27 | ```javascript
|
28 | var HappyPack = require('happypack');
|
29 |
|
30 | exports.plugins = [
|
31 | new HappyPack({
|
32 | // loaders is the only required parameter:
|
33 | loaders: [ 'babel?presets[]=es2015' ],
|
34 |
|
35 | // customize as needed, see Configuration below
|
36 | })
|
37 | ];
|
38 | ```
|
39 |
|
40 | Now you replace your current loaders with HappyPack's loader (possibly use an
|
41 | env variable to enable HappyPack):
|
42 |
|
43 | ```javascript
|
44 | exports.module = {
|
45 | loaders: {
|
46 | test: /.js$/,
|
47 | loader: 'happypack/loader',
|
48 | include: [
|
49 | // ...
|
50 | ],
|
51 | }
|
52 | };
|
53 | ```
|
54 |
|
55 | That's it. Now sources that match `.js$` will be handed off to happypack which
|
56 | will transform them in parallel using the loaders you specified.
|
57 |
|
58 | ## Configuration
|
59 |
|
60 | These are the parameters you can pass to the plugin when you instantiate it.
|
61 |
|
62 | ### `loaders: Array.<String|Object{path: String, query: String}>`
|
63 |
|
64 | Each loader entry consists of the name or path of loader that would
|
65 | transform the files and an optional query string to pass to it. This looks
|
66 | similar to what you'd pass to webpack's `loader` config.
|
67 |
|
68 | > **NOTE**
|
69 | >
|
70 | > HappyPack doesn't work with *all* webpack loaders as some loader API are not
|
71 | > supported.
|
72 | >
|
73 | > See [this wiki page](https://github.com/amireh/happypack/wiki/Webpack-Loader-API-Support)
|
74 | > for more details on current Loader API support.
|
75 |
|
76 | It is possible to omit this value and have HappyPack automatically infer the
|
77 | loaders it should use, see "Inferring loaders" below for more information
|
78 |
|
79 | Defaults to: `null`
|
80 |
|
81 | ### `id: String`
|
82 |
|
83 | A unique id for this happy plugin. This is used to generate the default cache
|
84 | name and is used by the loader to know which plugin it's supposed to talk to.
|
85 |
|
86 | Normally, you would not need to specify this unless you have more than one
|
87 | HappyPack plugin defined, in which case you'll need distinct IDs to tell them
|
88 | apart. See "Using multiple instances" below for more information on that.
|
89 |
|
90 | Defaults to: "1"
|
91 |
|
92 | ### `enabled: Boolean`
|
93 |
|
94 | Whether the plugin should be activated. This is for convenience when you want
|
95 | to conditionally disable HappyPack based on, for example, an environment
|
96 | variable.
|
97 |
|
98 | Defaults to `true`
|
99 |
|
100 | ### `tempDir: String`
|
101 |
|
102 | Path to a folder where happy's cache and junk files will be kept. It's safe to
|
103 | remove this folder after a run but not during it!
|
104 |
|
105 | Defaults to: `.happypack/`
|
106 |
|
107 | ### `cache: Boolean`
|
108 |
|
109 | Whether HappyPack should re-use the compiled version of source files if their
|
110 | contents have not changed since the last compilation time (assuming they have
|
111 | been compiled, of course.)
|
112 |
|
113 | Recommended!
|
114 |
|
115 | Defaults to: `true`
|
116 |
|
117 | ### `cachePath: String`
|
118 |
|
119 | Path to a file where the JSON cache will be saved to disk and read from on
|
120 | successive webpack runs.
|
121 |
|
122 | Defaults to `.happypack/cache--[id].json`
|
123 |
|
124 | ### `cacheContext: Object`
|
125 |
|
126 | An object that is used to invalidate the cache between runs based on whatever
|
127 | variables that might affect the transformation of your sources, like `NODE_ENV`
|
128 | for example.
|
129 |
|
130 | You should provide this if you perform different builds based on some external
|
131 | parameters. **THIS OBJECT MUST BE JSON-SERIALIZABLE**.
|
132 |
|
133 | Defaults to: `{}`
|
134 |
|
135 | ### `threads: Number`
|
136 |
|
137 | This number indicates how many Node VMs HappyPack will spawn for compiling
|
138 | the source files. After a lot of tinkering, I found 4 to yield the best
|
139 | results. There's certainly a diminishing return on this value and increasing
|
140 | beyond 8 actually slowed things down for me.
|
141 |
|
142 | Keep in mind that this is only relevant when performing **the initial build**
|
143 | as HappyPack will switch into a synchronous mode afterwards (i.e. in `watch`
|
144 | mode.) Also, if we're using the cache and the compiled versions are indeed
|
145 | cached, the threads will be idle.
|
146 |
|
147 | ### `threadPool: HappyThreadPool`
|
148 |
|
149 | A custom thread-pool to use for retrieving worker threads. Normally, this
|
150 | is managed internally by each `HappyPlugin` instance, but you may override
|
151 | this behavior for better results.
|
152 |
|
153 | See "Shared thread pools" below for more information about this.
|
154 |
|
155 | Defaults to: `null`
|
156 |
|
157 | ## How it works
|
158 |
|
159 | ![A diagram showing the flow between HappyPack's components](doc/HappyPack_Workflow.png)
|
160 |
|
161 | HappyPack sits between webpack and your primary source files (like JS sources)
|
162 | where the bulk of loader transformations happen. Every time webpack resolves
|
163 | a module, HappyPack will take it and all its dependencies, find out if they
|
164 | need to be compiled[1], and if they do, it distributes those files to multiple
|
165 | worker "threads".
|
166 |
|
167 | Those threads are actually simple node processes that invoke your transformer.
|
168 | When the compiled version is retrieved, HappyPack serves it to its loader and
|
169 | eventually your chunk.
|
170 |
|
171 | [1] When HappyPack successfully compiles a source file, it keeps track of its
|
172 | mtime so that it can re-use it on successive builds if the contents have not
|
173 | changed. This is a fast and somewhat reliable approach, and definitely much
|
174 | faster than re-applying the transformers on every build.
|
175 |
|
176 | ## Inferring loaders
|
177 |
|
178 | For configuration convenience, we can instruct a HappyPack plugin to use the
|
179 | loaders specified in webpack's original `module.loaders` config by defining a
|
180 | `happy: ...` option on that specific loader.
|
181 |
|
182 | For example, the following config will cause HappyPack to work with the `babel`
|
183 | loader against all `.js` source files:
|
184 |
|
185 | ```javascript
|
186 | // @file: webpack.config.js
|
187 | module.exports = {
|
188 | module: {
|
189 | loaders: [{
|
190 | test: /\.js$/,
|
191 | loader: 'babel',
|
192 | happy: { id: 'js' } // <-- see this
|
193 | }]
|
194 | },
|
195 |
|
196 | plugins: [
|
197 | new HappyPack({ id: 'js' }) // no need to specify loaders manually, yay!
|
198 | ]
|
199 | }
|
200 | ```
|
201 |
|
202 | > **Disclaimer**
|
203 | >
|
204 | > Using this method to configure loaders will cause HappyPack to **overwrite**
|
205 | > webpack's loader options objects to replace the source loaders with happy's
|
206 | > loader at run-time (ie, the config will be irreversibly mutated!).
|
207 | >
|
208 | > I did not find any way to work around this, so be warned!
|
209 |
|
210 | ## Using multiple instances
|
211 |
|
212 | It's possible to define multiple HappyPack plugins for different types of
|
213 | sources/transformations. Just pass in a unique id for each plugin and make
|
214 | sure you pass it their loaders. For example:
|
215 |
|
216 | ```javascript
|
217 | // @file webpack.config.js
|
218 | exports.plugins = [
|
219 | new HappyPack({
|
220 | id: 'jsx',
|
221 | threads: 4,
|
222 | loaders: [ 'babel-loader' ]
|
223 | }),
|
224 |
|
225 | new HappyPack({
|
226 | id: 'coffeescripts',
|
227 | threads: 2,
|
228 | loaders: [ 'coffee-loader' ]
|
229 | })
|
230 | ];
|
231 |
|
232 | exports.module.loaders = [
|
233 | {
|
234 | test: /\.js$/,
|
235 | loaders: 'happypack/loader?id=jsx'
|
236 | },
|
237 |
|
238 | {
|
239 | test: /\.coffee$/,
|
240 | loader: 'happypack/loader?id=coffeescripts'
|
241 | },
|
242 | ]
|
243 | ```
|
244 |
|
245 | Now `.js` files will be handled by the first Happy plugin which will use
|
246 | `babel-loader` to transform them, while `.coffee` files will be handled
|
247 | by the second one using the `coffee-loader` as a transformer.
|
248 |
|
249 | Note that each plugin will properly use different cache files as the default
|
250 | cache file names include the plugin IDs, so you don't need to override them
|
251 | manually. Yay!
|
252 |
|
253 | ## Shared thread pools
|
254 |
|
255 | Normally, each `HappyPlugin` instance you create internally manages its own
|
256 | threads which are used to compile sources. However, if you have a number of
|
257 | plugins, it can be more optimal to create a thread pool yourself and then
|
258 | configure the instances to share that pool, minimizing the idle time of
|
259 | threads within it.
|
260 |
|
261 | Here's an example of using a custom pool of 5 threads that will be shared
|
262 | between loaders for both JS and SCSS/LESS/whatever sources:
|
263 |
|
264 | ```javascript
|
265 | // @file: webpack.config.js
|
266 | var HappyPack = require('happypack');
|
267 | var happyThreadPool = HappyPack.ThreadPool({ size: 5 });
|
268 |
|
269 | module.exports = {
|
270 | // ...
|
271 | plugins: [
|
272 | new HappyPack({
|
273 | id: 'js',
|
274 | threadPool: happyThreadPool
|
275 | }),
|
276 |
|
277 | new HappyPack({
|
278 | id: 'styles',
|
279 | threadPool: happyThreadPool
|
280 | })
|
281 | ]
|
282 | };
|
283 | ```
|
284 |
|
285 | ## Benchmarks
|
286 |
|
287 | For the main repository I tested on, which had around 3067 modules, the build time went down from 39 seconds to a whopping ~10 seconds when there was yet no
|
288 | cache. Successive builds now take between 6 and 7 seconds.
|
289 |
|
290 | Here's a rundown of the various states the build was performed in:
|
291 |
|
292 | Elapsed (ms) | Happy? | Cache enabled? | Cache present? | Using DLLs? |
|
293 | ------------ | ------- | -------------- | -------------- | ----------- |
|
294 | 39851 | NO | N/A | N/A | NO |
|
295 | 37393 | NO | N/A | N/A | YES |
|
296 | 14605 | YES | NO | N/A | NO |
|
297 | 13925 | YES | YES | NO | NO |
|
298 | 11877 | YES | YES | YES | NO |
|
299 | 9228 | YES | NO | N/A | YES |
|
300 | 9597 | YES | YES | NO | YES |
|
301 | 6975 | YES | YES | YES | YES |
|
302 |
|
303 | The builds above were run on Linux over a machine with 12 cores.
|
304 |
|
305 | _TODO: test against other projects_
|
306 |
|
307 | ## Changes
|
308 |
|
309 | **2.0.5**
|
310 |
|
311 | - now using [mkdirp](https://github.com/substack/node-mkdirp) for creating the temp directory to support nested ones
|
312 |
|
313 | **2.0.4**
|
314 |
|
315 | - fix an issue with cache not being utilized on node v0.10 (`fs.statSync` doesn't exist with that name there - thanks to @XVincentX)
|
316 |
|
317 | **2.0.2**
|
318 |
|
319 | - Fixed an issue that was causing loaders running in foreground to not receive the compiler options
|
320 |
|
321 | **2.0.1**
|
322 |
|
323 | - Package in NPM is now compact
|
324 |
|
325 | **2.0.0**
|
326 |
|
327 | - Pitching loader support
|
328 | - More complete loader API support
|
329 | - More convenient configuration interface
|
330 |
|
331 | **1.1.4**
|
332 |
|
333 | - Fixed an issue where the cache was being improperly invalidated due to `cacheContext` not being stored properly (#17, thanks to @blowery)
|
334 |
|
335 | **1.1.3**
|
336 |
|
337 | - Fixed an issue where the initial cache was not being saved properly
|
338 |
|
339 | **1.1.2**
|
340 |
|
341 | - Fixed an issue on old node versions (0.10) with the EventEmitter API (#10)
|
342 | - Fixed an issue that was breaking the compiler if an invalid `threads` option
|
343 | was passed (evaluating to `NaN`)
|
344 |
|
345 | **1.1.1**
|
346 |
|
347 | - Unrecognized and invalid config parameters will cause the process to abort.
|
348 | - The active version is logged on launch.
|
349 |
|
350 | **1.1.0**
|
351 |
|
352 | - now supporting basic webpack loaders
|
353 | - dropped the `transformer` parameter as it's no longer needed
|
354 | - `cache` now defaults to `true`
|
355 | - now using a forking model utilizing node.js's `process.fork()` for cleaner
|
356 | threading code
|
357 |
|
358 | **1.0.2**
|
359 |
|
360 | - the loader will now accept IDs that aren't just numbers
|