1 | # make-fetch-happen
|
2 | [![npm version](https://img.shields.io/npm/v/make-fetch-happen.svg)](https://npm.im/make-fetch-happen) [![license](https://img.shields.io/npm/l/make-fetch-happen.svg)](https://npm.im/make-fetch-happen) [![Travis](https://img.shields.io/travis/npm/make-fetch-happen.svg)](https://travis-ci.org/npm/make-fetch-happen) [![Coverage Status](https://coveralls.io/repos/github/npm/make-fetch-happen/badge.svg?branch=latest)](https://coveralls.io/github/npm/make-fetch-happen?branch=latest)
|
3 |
|
4 | [`make-fetch-happen`](https://github.com/npm/make-fetch-happen) is a Node.js
|
5 | library that wraps [`minipass-fetch`](https://github.com/npm/minipass-fetch) with additional
|
6 | features [`minipass-fetch`](https://github.com/npm/minipass-fetch) doesn't intend to include, including HTTP Cache support, request
|
7 | pooling, proxies, retries, [and more](#features)!
|
8 |
|
9 | ## Install
|
10 |
|
11 | `$ npm install --save make-fetch-happen`
|
12 |
|
13 | ## Table of Contents
|
14 |
|
15 | * [Example](#example)
|
16 | * [Features](#features)
|
17 | * [Contributing](#contributing)
|
18 | * [API](#api)
|
19 | * [`fetch`](#fetch)
|
20 | * [`fetch.defaults`](#fetch-defaults)
|
21 | * [`minipass-fetch` options](#minipass-fetch-options)
|
22 | * [`make-fetch-happen` options](#extra-options)
|
23 | * [`opts.cachePath`](#opts-cache-path)
|
24 | * [`opts.cache`](#opts-cache)
|
25 | * [`opts.cacheAdditionalHeaders`](#opts-cache-additional-headers)
|
26 | * [`opts.proxy`](#opts-proxy)
|
27 | * [`opts.noProxy`](#opts-no-proxy)
|
28 | * [`opts.ca, opts.cert, opts.key`](#https-opts)
|
29 | * [`opts.maxSockets`](#opts-max-sockets)
|
30 | * [`opts.retry`](#opts-retry)
|
31 | * [`opts.onRetry`](#opts-onretry)
|
32 | * [`opts.integrity`](#opts-integrity)
|
33 | * [`opts.dns`](#opts-dns)
|
34 | * [Message From Our Sponsors](#wow)
|
35 |
|
36 | ### Example
|
37 |
|
38 | ```javascript
|
39 | const fetch = require('make-fetch-happen').defaults({
|
40 | cachePath: './my-cache' // path where cache will be written (and read)
|
41 | })
|
42 |
|
43 | fetch('https://registry.npmjs.org/make-fetch-happen').then(res => {
|
44 | return res.json() // download the body as JSON
|
45 | }).then(body => {
|
46 | console.log(`got ${body.name} from web`)
|
47 | return fetch('https://registry.npmjs.org/make-fetch-happen', {
|
48 | cache: 'no-cache' // forces a conditional request
|
49 | })
|
50 | }).then(res => {
|
51 | console.log(res.status) // 304! cache validated!
|
52 | return res.json().then(body => {
|
53 | console.log(`got ${body.name} from cache`)
|
54 | })
|
55 | })
|
56 | ```
|
57 |
|
58 | ### Features
|
59 |
|
60 | * Builds around [`minipass-fetch`](https://npm.im/minipass-fetch) for the core [`fetch` API](https://fetch.spec.whatwg.org) implementation
|
61 | * Request pooling out of the box
|
62 | * Quite fast, really
|
63 | * Automatic HTTP-semantics-aware request retries
|
64 | * Cache-fallback automatic "offline mode"
|
65 | * Built-in request caching following full HTTP caching rules (`Cache-Control`, `ETag`, `304`s, cache fallback on error, etc).
|
66 | * Node.js Stream support
|
67 | * Transparent gzip and deflate support
|
68 | * [Subresource Integrity](https://developer.mozilla.org/en-US/docs/Web/Security/Subresource_Integrity) support
|
69 | * Proxy support (http, https, socks, socks4, socks5. via [`@npmcli/agent`](https://npm.im/@npmcli/agent))
|
70 | * DNS cache (via ([`@npmcli/agent`](https://npm.im/@npmcli/agent))
|
71 |
|
72 | #### <a name="fetch"></a> `> fetch(uriOrRequest, [opts]) -> Promise<Response>`
|
73 |
|
74 | This function implements most of the [`fetch` API](https://developer.mozilla.org/en-US/docs/Web/API/WindowOrWorkerGlobalScope/fetch): given a `uri` string or a `Request` instance, it will fire off an http request and return a Promise containing the relevant response.
|
75 |
|
76 | If `opts` is provided, the [`minipass-fetch`-specific options](#minipass-fetch-options) will be passed to that library. There are also [additional options](#extra-options) specific to make-fetch-happen that add various features, such as HTTP caching, integrity verification, proxy support, and more.
|
77 |
|
78 | ##### Example
|
79 |
|
80 | ```javascript
|
81 | fetch('https://google.com').then(res => res.buffer())
|
82 | ```
|
83 |
|
84 | #### <a name="fetch-defaults"></a> `> fetch.defaults([defaultUrl], [defaultOpts])`
|
85 |
|
86 | Returns a new `fetch` function that will call `make-fetch-happen` using `defaultUrl` and `defaultOpts` as default values to any calls.
|
87 |
|
88 | A defaulted `fetch` will also have a `.defaults()` method, so they can be chained.
|
89 |
|
90 | ##### Example
|
91 |
|
92 | ```javascript
|
93 | const fetch = require('make-fetch-happen').defaults({
|
94 | cachePath: './my-local-cache'
|
95 | })
|
96 |
|
97 | fetch('https://registry.npmjs.org/make-fetch-happen') // will always use the cache
|
98 | ```
|
99 |
|
100 | #### <a name="minipass-fetch-options"></a> `> minipass-fetch options`
|
101 |
|
102 | The following options for `minipass-fetch` are used as-is:
|
103 |
|
104 | * method
|
105 | * body
|
106 | * redirect
|
107 | * follow
|
108 | * timeout
|
109 | * compress
|
110 | * size
|
111 |
|
112 | These other options are modified or augmented by make-fetch-happen:
|
113 |
|
114 | * headers - Default `User-Agent` set to make-fetch happen. `Connection` is set to `keep-alive` or `close` automatically depending on `opts.agent`.
|
115 |
|
116 | For more details, see [the documentation for `minipass-fetch` itself](https://github.com/npm/minipass-fetch#options).
|
117 |
|
118 | #### <a name="extra-options"></a> `> make-fetch-happen options`
|
119 |
|
120 | make-fetch-happen augments the `minipass-fetch` API with additional features available through extra options. The following extra options are available:
|
121 |
|
122 | * [`opts.cachePath`](#opts-cache-path) - Cache target to read/write
|
123 | * [`opts.cache`](#opts-cache) - `fetch` cache mode. Controls cache *behavior*.
|
124 | * [`opts.cacheAdditionalHeaders`](#opts-cache-additional-headers) - Store additional headers in the cache
|
125 | * [`opts.proxy`](#opts-proxy) - Proxy agent
|
126 | * [`opts.noProxy`](#opts-no-proxy) - Domain segments to disable proxying for.
|
127 | * [`opts.ca, opts.cert, opts.key, opts.strictSSL`](#https-opts)
|
128 | * [`opts.localAddress`](#opts-local-address)
|
129 | * [`opts.maxSockets`](#opts-max-sockets)
|
130 | * [`opts.retry`](#opts-retry) - Request retry settings
|
131 | * [`opts.onRetry`](#opts-onretry) - a function called whenever a retry is attempted
|
132 | * [`opts.integrity`](#opts-integrity) - [Subresource Integrity](https://developer.mozilla.org/en-US/docs/Web/Security/Subresource_Integrity) metadata.
|
133 | * [`opts.dns`](#opts-dns) - DNS cache options
|
134 | * [`opts.agent`](#opts-agent) - http/https/proxy/socks agent options. See [`@npmcli/agent`](https://npm.im/@npmcli/agent) for more info.
|
135 |
|
136 | #### <a name="opts-cache-path"></a> `> opts.cachePath`
|
137 |
|
138 | A string `Path` to be used as the cache root for [`cacache`](https://npm.im/cacache).
|
139 |
|
140 | **NOTE**: Requests will not be cached unless their response bodies are consumed. You will need to use one of the `res.json()`, `res.buffer()`, etc methods on the response, or drain the `res.body` stream, in order for it to be written.
|
141 |
|
142 | The default cache manager also adds the following headers to cached responses:
|
143 |
|
144 | * `X-Local-Cache`: Path to the cache the content was found in
|
145 | * `X-Local-Cache-Key`: Unique cache entry key for this response
|
146 | * `X-Local-Cache-Mode`: Always `stream` to indicate how the response was read from cacache
|
147 | * `X-Local-Cache-Hash`: Specific integrity hash for the cached entry
|
148 | * `X-Local-Cache-Status`: One of `miss`, `hit`, `stale`, `revalidated`, `updated`, or `skip` to signal how the response was created
|
149 | * `X-Local-Cache-Time`: UTCString of the cache insertion time for the entry
|
150 |
|
151 | Using [`cacache`](https://npm.im/cacache), a call like this may be used to
|
152 | manually fetch the cached entry:
|
153 |
|
154 | ```javascript
|
155 | const h = response.headers
|
156 | cacache.get(h.get('x-local-cache'), h.get('x-local-cache-key'))
|
157 |
|
158 | // grab content only, directly:
|
159 | cacache.get.byDigest(h.get('x-local-cache'), h.get('x-local-cache-hash'))
|
160 | ```
|
161 |
|
162 | ##### Example
|
163 |
|
164 | ```javascript
|
165 | fetch('https://registry.npmjs.org/make-fetch-happen', {
|
166 | cachePath: './my-local-cache'
|
167 | }) // -> 200-level response will be written to disk
|
168 | ```
|
169 |
|
170 | #### <a name="opts-cache"></a> `> opts.cache`
|
171 |
|
172 | This option follows the standard `fetch` API cache option. This option will do nothing if [`opts.cachePath`](#opts-cache-path) is null. The following values are accepted (as strings):
|
173 |
|
174 | * `default` - Fetch will inspect the HTTP cache on the way to the network. If there is a fresh response it will be used. If there is a stale response a conditional request will be created, and a normal request otherwise. It then updates the HTTP cache with the response. If the revalidation request fails (for example, on a 500 or if you're offline), the stale response will be returned.
|
175 | * `no-store` - Fetch behaves as if there is no HTTP cache at all.
|
176 | * `reload` - Fetch behaves as if there is no HTTP cache on the way to the network. Ergo, it creates a normal request and updates the HTTP cache with the response.
|
177 | * `no-cache` - Fetch creates a conditional request if there is a response in the HTTP cache and a normal request otherwise. It then updates the HTTP cache with the response.
|
178 | * `force-cache` - Fetch uses any response in the HTTP cache matching the request, not paying attention to staleness. If there was no response, it creates a normal request and updates the HTTP cache with the response.
|
179 | * `only-if-cached` - Fetch uses any response in the HTTP cache matching the request, not paying attention to staleness. If there was no response, it returns a network error. (Can only be used when request’s mode is "same-origin". Any cached redirects will be followed assuming request’s redirect mode is "follow" and the redirects do not violate request’s mode.)
|
180 |
|
181 | (Note: option descriptions are taken from https://fetch.spec.whatwg.org/#http-network-or-cache-fetch)
|
182 |
|
183 | ##### Example
|
184 |
|
185 | ```javascript
|
186 | const fetch = require('make-fetch-happen').defaults({
|
187 | cachePath: './my-cache'
|
188 | })
|
189 |
|
190 | // Will error with ENOTCACHED if we haven't already cached this url
|
191 | fetch('https://registry.npmjs.org/make-fetch-happen', {
|
192 | cache: 'only-if-cached'
|
193 | })
|
194 |
|
195 | // Will refresh any local content and cache the new response
|
196 | fetch('https://registry.npmjs.org/make-fetch-happen', {
|
197 | cache: 'reload'
|
198 | })
|
199 |
|
200 | // Will use any local data, even if stale. Otherwise, will hit network.
|
201 | fetch('https://registry.npmjs.org/make-fetch-happen', {
|
202 | cache: 'force-cache'
|
203 | })
|
204 | ```
|
205 |
|
206 | #### <a name="opts-cache-additional-headers"></a> `> opts.cacheAdditionalHeaders`
|
207 |
|
208 | The following headers are always stored in the cache when present:
|
209 |
|
210 | - `cache-control`
|
211 | - `content-encoding`
|
212 | - `content-language`
|
213 | - `content-type`
|
214 | - `date`
|
215 | - `etag`
|
216 | - `expires`
|
217 | - `last-modified`
|
218 | - `link`
|
219 | - `location`
|
220 | - `pragma`
|
221 | - `vary`
|
222 |
|
223 | This option allows a user to store additional custom headers in the cache.
|
224 |
|
225 |
|
226 | ##### Example
|
227 |
|
228 | ```javascript
|
229 | fetch('https://registry.npmjs.org/make-fetch-happen', {
|
230 | cacheAdditionalHeaders: ['my-custom-header'],
|
231 | })
|
232 | ```
|
233 |
|
234 | #### <a name="opts-proxy"></a> `> opts.proxy`
|
235 |
|
236 | A string or `new url.URL()`-d URI to proxy through. Different Proxy handlers will be
|
237 | used depending on the proxy's protocol.
|
238 |
|
239 | Additionally, `process.env.HTTP_PROXY`, `process.env.HTTPS_PROXY`, and
|
240 | `process.env.PROXY` are used if present and no `opts.proxy` value is provided.
|
241 |
|
242 | (Pending) `process.env.NO_PROXY` may also be configured to skip proxying requests for all, or specific domains.
|
243 |
|
244 | ##### Example
|
245 |
|
246 | ```javascript
|
247 | fetch('https://registry.npmjs.org/make-fetch-happen', {
|
248 | proxy: 'https://corporate.yourcompany.proxy:4445'
|
249 | })
|
250 |
|
251 | fetch('https://registry.npmjs.org/make-fetch-happen', {
|
252 | proxy: {
|
253 | protocol: 'https:',
|
254 | hostname: 'corporate.yourcompany.proxy',
|
255 | port: 4445
|
256 | }
|
257 | })
|
258 | ```
|
259 |
|
260 | #### <a name="opts-no-proxy"></a> `> opts.noProxy`
|
261 |
|
262 | If present, should be a comma-separated string or an array of domain extensions
|
263 | that a proxy should _not_ be used for.
|
264 |
|
265 | This option may also be provided through `process.env.NO_PROXY`.
|
266 |
|
267 | #### <a name="https-opts"></a> `> opts.ca, opts.cert, opts.key, opts.strictSSL`
|
268 |
|
269 | These values are passed in directly to the HTTPS agent and will be used for both
|
270 | proxied and unproxied outgoing HTTPS requests. They mostly correspond to the
|
271 | same options the `https` module accepts, which will be themselves passed to
|
272 | `tls.connect()`. `opts.strictSSL` corresponds to `rejectUnauthorized`.
|
273 |
|
274 | #### <a name="opts-local-address"></a> `> opts.localAddress`
|
275 |
|
276 | Passed directly to `http` and `https` request calls. Determines the local
|
277 | address to bind to.
|
278 |
|
279 | #### <a name="opts-max-sockets"></a> `> opts.maxSockets`
|
280 |
|
281 | Default: 15
|
282 |
|
283 | Maximum number of active concurrent sockets to use for the underlying
|
284 | Http/Https/Proxy agents. This setting applies once per spawned agent.
|
285 |
|
286 | 15 is probably a _pretty good value_ for most use-cases, and balances speed
|
287 | with, uh, not knocking out people's routers. 🤓
|
288 |
|
289 | #### <a name="opts-retry"></a> `> opts.retry`
|
290 |
|
291 | An object that can be used to tune request retry settings. Retries will only be attempted on the following conditions:
|
292 |
|
293 | * Request method is NOT `POST` AND
|
294 | * Request status is one of: `408`, `420`, `429`, or any status in the 500-range. OR
|
295 | * Request errored with `ECONNRESET`, `ECONNREFUSED`, `EADDRINUSE`, `ETIMEDOUT`, or the `fetch` error `request-timeout`.
|
296 |
|
297 | The following are worth noting as explicitly not retried:
|
298 |
|
299 | * `getaddrinfo ENOTFOUND` and will be assumed to be either an unreachable domain or the user will be assumed offline. If a response is cached, it will be returned immediately.
|
300 |
|
301 | If `opts.retry` is `false`, it is equivalent to `{retries: 0}`
|
302 |
|
303 | If `opts.retry` is a number, it is equivalent to `{retries: num}`
|
304 |
|
305 | The following retry options are available if you want more control over it:
|
306 |
|
307 | * retries
|
308 | * factor
|
309 | * minTimeout
|
310 | * maxTimeout
|
311 | * randomize
|
312 |
|
313 | For details on what each of these do, refer to the [`retry`](https://npm.im/retry) documentation.
|
314 |
|
315 | ##### Example
|
316 |
|
317 | ```javascript
|
318 | fetch('https://flaky.site.com', {
|
319 | retry: {
|
320 | retries: 10,
|
321 | randomize: true
|
322 | }
|
323 | })
|
324 |
|
325 | fetch('http://reliable.site.com', {
|
326 | retry: false
|
327 | })
|
328 |
|
329 | fetch('http://one-more.site.com', {
|
330 | retry: 3
|
331 | })
|
332 | ```
|
333 |
|
334 | #### <a name="opts-onretry"></a> `> opts.onRetry`
|
335 |
|
336 | A function called with the response or error which caused the retry whenever one is attempted.
|
337 |
|
338 | ##### Example
|
339 |
|
340 | ```javascript
|
341 | fetch('https://flaky.site.com', {
|
342 | onRetry(cause) {
|
343 | console.log('we will retry because of', cause)
|
344 | }
|
345 | })
|
346 | ```
|
347 |
|
348 | #### <a name="opts-integrity"></a> `> opts.integrity`
|
349 |
|
350 | Matches the response body against the given [Subresource Integrity](https://developer.mozilla.org/en-US/docs/Web/Security/Subresource_Integrity) metadata. If verification fails, the request will fail with an `EINTEGRITY` error.
|
351 |
|
352 | `integrity` may either be a string or an [`ssri`](https://npm.im/ssri) `Integrity`-like.
|
353 |
|
354 | ##### Example
|
355 |
|
356 | ```javascript
|
357 | fetch('https://registry.npmjs.org/make-fetch-happen/-/make-fetch-happen-1.0.0.tgz', {
|
358 | integrity: 'sha1-o47j7zAYnedYFn1dF/fR9OV3z8Q='
|
359 | }) // -> ok
|
360 |
|
361 | fetch('https://malicious-registry.org/make-fetch-happen/-/make-fetch-happen-1.0.0.tgz', {
|
362 | integrity: 'sha1-o47j7zAYnedYFn1dF/fR9OV3z8Q='
|
363 | }) // Error: EINTEGRITY
|
364 | ```
|