1 | ## Understanding the analysis
|
2 |
|
3 | JavaScript is Garbage Collected language. Rather than manually freeing objects, they
|
4 | are simply "cleaned away" at some point after all references to an object have been removed.
|
5 |
|
6 | At a basic level, the Garbage Collector traverses the JavaScript objects at various intervals to find any
|
7 | "orphaned" objects (objects which no longer have any references). If there are too many
|
8 | objects, and/or too many orphaned objects this can cause performance issues – because the Garbage
|
9 | Collector uses the same thread as the JavaScript event loop. In other words, JavaScript execution
|
10 | pauses while the Garbage Collector clears away de-referenced objects.
|
11 |
|
12 | At a more detailed level, GC collection is triggered by memory activity, rather than time and
|
13 | objects are classified by the GC into young and old. "Young" objects are
|
14 | traversed (scavenged) more frequently, while "old" objects will stay in memory for longer. So there
|
15 | are actually two GC types, a frequent scavenge of new space (short lived objects) and a less regular traversal of
|
16 | old space (objects that survived enough new space scavenges).
|
17 |
|
18 | Several heuristics may trigger detection of a GC issue, but they all center around high
|
19 | memory usage.
|
20 |
|
21 | One possible cause of a detected GC issue is a memory leak, where objects are being accidentally
|
22 | allocated. However there are other (more common) cases where the is no leak but the memory strategy
|
23 | needs to be adapted.
|
24 |
|
25 | One such common case is when large objects (such as may be generated for big JSON payloads), are
|
26 | created during periods of high activity (e.g. under request load). This can cause the objects
|
27 | to be moved into old space – if they survive two (by default) GC scavenges – where they will live
|
28 | for longer due to the less frequent scavenges. Objects can then build up in "old space" and
|
29 | cause intermittent process stalling during Garbage Collection.
|
30 |
|
31 | Depending on the use case this may be solved in different ways. For instance if the goal is to write
|
32 | out serialized objects, then the output could be written to the response as strings (or buffers) directly
|
33 | instead of creating the intermediate objects (or a combined strategy where part of the object is written out
|
34 | from available state). It may just be a case that a functional approach (which is usually recommended) is
|
35 | leading to the repeated creation of very similar objects, in which case the logical flow between functions
|
36 | in a hot path could be adapted to reuse objects instead of create new objects.
|
37 |
|
38 | Another possibility is that a very high amount of short lived objects are created, filling up the
|
39 | "young" space and triggering frequent GC sweeps – if this case *isn't* an unintended memory leak,
|
40 | then then an object pooling strategy may be necessary.
|
41 |
|
42 | To solve Garbage Collection issues we have to analyse the state of our process in order to track down the
|
43 | root cause behind the high memory consumption.
|
44 |
|
45 | ## Next Steps
|
46 |
|
47 | - If the system is already deployed, mitigate the issue immediately by implementing
|
48 | HTTP 503 Service Unavailable functionality (see *Load Shedding* in **Reference**)
|
49 | - Run <code class='snippet'>node --inspect <FILENAME></code>
|
50 | - Open Chrome and navigate to [chrome://inspect](chrome://inspect)
|
51 | - Under the **Remote Target** heading, there should be a target with the official Node.js icon
|
52 | - Click the `inspect` link for that target – this will connect Chrome Devtools to the Node processes remote debug interface
|
53 | - In Devtools, select the *Memory* tab
|
54 | - Select the *Take heap snapshot* radio box, and then click *Take snapshot*
|
55 | - Put the process under load (in the same way that the process was load tested for Clinic.js)
|
56 | - Click *Profiles* in the left panel, then click *Take snapshot* again
|
57 | - Under the *HEAP SNAPSHOTS* left panel, select the second Snapshot (it will be called *Snapshot 2*)
|
58 | - Locate the dropdown box just above the "Constructor" column (most likely the dropdown box says *Summary*)
|
59 | - Click the dropdown, and select *Comparison* – this compares the before and after snapshots of the heap
|
60 | - Click the *# Delta* and/or *Size Delta* columns to sort by the difference in object counts
|
61 | or object size, categorized by constructor type
|
62 | - Use the interactive trees in the Constructor column to drill down into the specifics
|
63 | - Use the *Retainers* panel to understand the chain of object references
|
64 | + This can lead to useful clues about the origins of an object
|
65 | + Retained size (the aggregate total space used due to references *from* an object) may be important where a reference to a large amount of objects is relevant
|
66 | + Shallow size (the actual space used by the object itself) will be pertinent when there are particularly large objects in play
|
67 |
|
68 | **Advanced**: Other Devtools memory profiling functionality, Record allocation profile and Record allocation timeline may also be very helpful
|
69 |
|
70 | **Advanced**: An alternative approach is to use a generate a core dump and use
|
71 | a core dump analysis tool to list all JS objects in a core dump file (this approach isn't viable on macOS)
|
72 |
|
73 | ## Reference
|
74 |
|
75 | - Load Shedding
|
76 | + Express, Koa, Restify, `http`: [overload-protection](https://www.npmjs.com/package/overload-protection)
|
77 | + Hapi: [Server load sampleInterval option](https://hapi.dev/api/#-serveroptionsload) & [Server connections load maxEventLoopDelay](https://hapi.dev/api/#-serveroptionsload)
|
78 | + Fastify: [under-pressure](https://www.npmjs.com/package/under-pressure)
|
79 | + General: [loopbench](https://www.npmjs.com/package/loopbench)
|
80 | - [Chrome Devtools Docs: Fix Memory Problems](https://developers.google.com/web/tools/chrome-devtools/memory-problems/)
|
81 | - [Chrome Devtools Docs: Memory Terminology](https://developers.google.com/web/tools/chrome-devtools/memory-problems/memory-101)
|
82 | - [Chrome Devtools Docs: How to record heap snapshots](https://developers.google.com/web/tools/chrome-devtools/memory-problems/heap-snapshots)
|
83 | - [Node Docs: Inspector](https://nodejs.org/en/docs/inspector/)
|
84 | - **Advanced**: [Core dump analysis tool for Linux: llnode](https://github.com/nodejs/llnode)
|
85 | - **Advanced**: [Core dump analysis tool for SmartOS: mdb_v8](https://github.com/joyent/mdb_v8)
|
86 | - **Advanced**: [Core dump analysis tool for Linux which wraps SmartOS mdb](https://www.npmjs.com/package/autopsy)
|