| 1 | ## Understanding the analysis
|
| 2 |
|
| 3 | An unknown issue occurs when Clinic.js' analysis algorithms are unable to categorize the sampling results but nevertheless an issue *of some kind* has been detected.
|
| 4 |
|
| 5 | This outcome can be attributed to one of two scenarios:
|
| 6 |
|
| 7 | 1. Ambient noise – for instance, other applications using the CPU or memory – during the sampling period has polluted the results.
|
| 8 | 2. There is a genuine performance issue but `clinic doctor` doesn't recognize it.
|
| 9 |
|
| 10 | In the case of ambient noise, there may still be a specific, categorizable performance issue.
|
| 11 |
|
| 12 | We can make eliminate the possibility of ambient noise and make it easier for Clinic.js to definitively recognize the issue by:
|
| 13 |
|
| 14 | - Closing down as many applications as possible, especially applications that are CPU- or Memory- intensive.
|
| 15 | - Using the `--on-port` flag. This can reduce the chances of unknown issues because there is no time gap nor additional system activity between the server starting and the load test beginning.
|
| 16 |
|
| 17 | By way of example, instead of running `clinic -- node app.js` in one terminal and `autocannon localhost:3000` in another, it is preferable and recommended to trigger both in one command using the following command:
|
| 18 |
|
| 19 | ```sh
|
| 20 | clinic doctor --on-port="autocannon localhost:3000" -- node app.js
|
| 21 | ```
|
| 22 |
|
| 23 | An even simpler form of this is to use the `--autocannon` flag,
|
| 24 |
|
| 25 | ```sh
|
| 26 | clinic doctor --autocannon / -- node app.js
|
| 27 | ```
|
| 28 |
|
| 29 | If after taking these steps an unknown categorization continues to occur then we can instead attempt to infer the nature of the performance issue using specialist diagnostic tooling, such
|
| 30 | as `clinic flame`, `clinic bubble` or Node Inspector.
|
| 31 |
|
| 32 | ## Next Steps
|
| 33 |
|
| 34 | - First eliminate the possibility of ambient noise
|
| 35 | - Reduce noise by closing down as many other applications running on the system as possible - especially CPU or Memory intensive applications
|
| 36 | - Ensure that the `--on-port` flag is being used to trigger load testing instead of initiating load testing independently
|
| 37 | - Use `clinic bubbleprof` to create a diagram of the application's asynchronous flow (see `clinic bubbleprof --help`)
|
| 38 | - Explore the Bubbleprof diagram. Look for long lines and large circles representing persistent delays, then drill down to reveal the lines of code responsible
|
| 39 | - A common problem is the overuse or misuse of promises. `clinic bubbleprof` will visualize promise activity, make a point of looking out for it in the diagram.
|
| 40 | - Use `clinic flame` to generate a flamegraph
|
| 41 | - Run `clinic flame --help` to get started
|
| 42 | - Look for "hot" blocks, these are functions that are observed (at a higher relative frequency) to be at the top the stack per CPU sample – in other words, such functions are blocking the event loop
|
| 43 | - For memory analysis use the [`--inspect`](https://nodejs.org/en/docs/inspector) flag with the Chrome Devtools *Memory* tab.
|
| 44 | - Run `node --inspect <FILENAME>`
|
| 45 | - Open Chrome and navigate to [chrome://inspect](chrome://inspect)
|
| 46 | - Under the **Remote Target** heading, there should be a target with the official Node.js icon
|
| 47 | - Click the `inspect` link for that target – this will connect Chrome Devtools to the Node processes remote debug interface
|
| 48 | - In Devtools, select the *Memory* tab
|
| 49 | - Select the *Take heap snapshot* radio box, and then click *Take snapshot*
|
| 50 | - Put the process under load (in the same way that the process was load tested for Clinic.js)
|
| 51 | - Click *Profiles* in the left panel, then click *Take snapshot* again
|
| 52 | - Under the *HEAP SNAPSHOTS* left panel, select the second Snapshot (it will be called *Snapshot 2*)
|
| 53 | - Locate the dropdown box just above the "Constructor" column (most likely the dropdown box says *Summary*)
|
| 54 | - Click the dropdown, and select *Comparison* – this compares the before and after snapshots of the heap
|
| 55 | - Click the *# Delta* and/or *Size Delta* columns to sort by the difference in object counts
|
| 56 | or object size, categorized by constructor type
|
| 57 | - Use the interactive trees in the Constructor column to drill down into the specifics
|
| 58 | - Use the *Retainers* panel to understand the chain of object references
|
| 59 |
|
| 60 | ## Reference
|
| 61 | - [Clinic.js Flame](https://clinicjs.org/flame)
|
| 62 | - [Overview of blocking vs non-blocking](https://nodejs.org/en/docs/guides/blocking-vs-non-blocking/)
|
| 63 | - [Concurrency model and Event Loop
|
| 64 | ](https://developer.mozilla.org/en-US/docs/Web/JavaScript/EventLoop)
|
| 65 | - [Don't Block the Event Loop (or the Worker Pool)](https://nodejs.org/en/docs/guides/dont-block-the-event-loop/)
|
| 66 | - Understanding Flamegraphs and how to use 0x: [Tuning Node.js app performance with autocannon and 0x](https://www.nearform.com/blog/tuning-node-js-app-performance-with-autocannon-and-0x/)
|
| 67 | - [Clinic.js Bubbleprof](https://clinicjs.org/bubbleprof)
|
| 68 | - [Chrome Devtools Docs: Fix Memory Problems](https://developers.google.com/web/tools/chrome-devtools/memory-problems/)
|
| 69 | - [Chrome Devtools Docs: Memory Terminology](https://developers.google.com/web/tools/chrome-devtools/memory-problems/memory-101)
|
| 70 | - [Chrome Devtools Docs: How to record heap snapshots](https://developers.google.com/web/tools/chrome-devtools/memory-problems/heap-snapshots)
|
| 71 | - [Node Docs: Inspector](https://nodejs.org/en/docs/inspector/)
|
| 72 | - **Advanced**: [Core dump analysis tool for Linux: llnode](https://github.com/nodejs/llnode)
|
| 73 | - **Advanced**: [Core dump analysis tool for SmartOS: mdb_v8](https://github.com/joyent/mdb_v8)
|
| 74 | - **Advanced**: [Core dump analysis tool for Linux which wraps SmartOS mdb](https://www.npmjs.com/package/autopsy)
|