1 | ## TracerBench: Automated Chrome Tracing For Benchmarking
|
2 |
|
3 | [![Build Status](https://travis-ci.org/TracerBench/tracerbench.svg?branch=master)](https://travis-ci.org/TracerBench/tracerbench)
|
4 | [![Version](https://img.shields.io/npm/v/tracerbench.svg)](https://npmjs.org/package/tracerbench)
|
5 | [![License](https://img.shields.io/npm/l/tracerbench.svg)](https://github.com/TracerBench/tracerbench/blob/master/package.json)
|
6 |
|
7 | # TracerBench Core
|
8 |
|
9 | https://github.com/TracerBench/tracerbench/blob/master/README.md
|
10 |
|
11 |
|
12 | * [TracerBench Core](#tracerbench-core)
|
13 | * [Usage](#usage)
|
14 | * [Optional Config](#optional-config)
|
15 | * [FAQ](#faq)
|
16 | * [Commands](#commands)
|
17 |
|
18 |
|
19 | # Usage
|
20 |
|
21 |
|
22 | ```sh-session
|
23 | $ npm install -g tracerbench
|
24 | $ tracerbench COMMAND
|
25 | running command...
|
26 | $ tracerbench (-v|--version|version)
|
27 | tracerbench/3.0.10 darwin-x64 node-v12.13.1
|
28 | $ tracerbench --help [COMMAND]
|
29 | USAGE
|
30 | $ tracerbench COMMAND
|
31 | ...
|
32 | ```
|
33 |
|
34 |
|
35 | # Optional Config
|
36 |
|
37 | The optional existance of a "tbconfig.json" file in the project root will be consumed by TracerBench and specifies default command flag options. Please note this file is optional, however is strongly recommended as this drastically speeds up running TracerBench tests succinctly [Example](https://github.com/TracerBench/tracerbench/blob/master/packages/cli/test/tbconfig_base.json):
|
38 |
|
39 | ```json-c
|
40 | {
|
41 | "$schema": "https://raw.githubusercontent.com/TracerBench/tracerbench/master/packages/cli/tb-schema.json",
|
42 | // the title of the report pdf file
|
43 | "plotTitle": "tbconfig_base file",
|
44 | "tbResultsFolder": "../tracerbench-results",
|
45 | "controlURL": "https://www.tracerbench.com/",
|
46 | "experimentURL": "https://www.tracerbench.com/",
|
47 | "url": "https://www.tracerbench.com/",
|
48 | "tracingLocationSearch": "?tracing",
|
49 | "regressionThreshold": "100",
|
50 | "appName": "tracerbench",
|
51 | "cpuThrottleRate": 1,
|
52 | "network": "fast-3g",
|
53 | "emulateDevice": "iphone-x",
|
54 | "fidelity": "high",
|
55 | "markers": [
|
56 | {
|
57 | "start": "fetchStart",
|
58 | "label": "jquery"
|
59 | },
|
60 | {
|
61 | "start": "jqueryLoaded",
|
62 | "label": "ember"
|
63 | },
|
64 | {
|
65 | "start": "emberLoaded",
|
66 | "label": "application"
|
67 | },
|
68 | {
|
69 | "start": "startRouting",
|
70 | "label": "routing"
|
71 | },
|
72 | {
|
73 | "start": "willTransition",
|
74 | "label": "transition"
|
75 | },
|
76 | {
|
77 | "start": "didTransition",
|
78 | "label": "render"
|
79 | },
|
80 | {
|
81 | "start": "renderEnd",
|
82 | "label": "afterRender"
|
83 | }
|
84 | ],
|
85 | "socksPorts": [8880, 8881],
|
86 | "browserArgs": [
|
87 | "--disable-features=site-per-process,TranslateUI,BlinkGenPropertyTrees",
|
88 | "--disable-ipc-flooding-protection",
|
89 | "--v8-cache-options=none"
|
90 | ],
|
91 | "servers": [
|
92 | {
|
93 | "name": "control-tracerbench-build_1.0.1234",
|
94 | "url": "https://www.tracerbench.com/",
|
95 | "dist": "./relative-path-to-control-dist-files/",
|
96 | "har": "./relative-path-location-to-har-file/",
|
97 | "socksPort": 8880
|
98 | },
|
99 | {
|
100 | "name": "experiment-tracerbench-build_1.0.1234_v78gh95",
|
101 | "url": "https://www.tracerbench.com/",
|
102 | "dist": "./relative-path-to-experiment-dist-files/",
|
103 | "har": "./relative-path-location-to-har-file/",
|
104 | "socksPort": 8881
|
105 | }
|
106 | ]
|
107 | }
|
108 |
|
109 | ```
|
110 |
|
111 | # FAQ
|
112 |
|
113 | _What exactly is contained within the `tracerbench compare` "tracerbench-results.json"?_
|
114 | An overview of "tracerbench-results.json" is available [here](https://github.com/TracerBench/tracerbench/blob/master/README.md#trace-results)
|
115 |
|
116 | _What exactly is a HAR file?_
|
117 | HAR (HTTP Archive) is a file format used by several HTTP session tools to export the captured data. The format is basically a JSON object with a particular field distribution. In any case, please note that not all the fields are mandatory, and many times some information won't be saved to the file ["Additional insight on Google's HAR Analyzer"](https://toolbox.googleapps.com/apps/har_analyzer/)
|
118 |
|
119 | _What exactly is contained within the output file "trace.json"?_
|
120 | The file "trace.json" is leveraged by TracerBench to capture an array of trace events. The interface of an individual trace event is essentially:
|
121 |
|
122 | ```ts
|
123 | // process id
|
124 | pid: number;
|
125 | // thread id
|
126 | tid: number;
|
127 | // timestamp in μs
|
128 | ts: number;
|
129 | // event phase
|
130 | ph: TRACE_EVENT_PHASE;
|
131 | // event categories (comma delimited)
|
132 | cat: string;
|
133 | // event name
|
134 | name: string;
|
135 | // event key/value pairs
|
136 | args: { [key: string]: any } | ARGS.STRIPPED;
|
137 | // ?timestamp in μs for trace event phase complete
|
138 | dur?: number;
|
139 | // ?thread clock timestamp in μs
|
140 | tts?: number;
|
141 | // ?thread clock duration in μs for trace event phase complete
|
142 | tdur?: number;
|
143 | // ?thread clock timestamp for related async events
|
144 | use_async_tts?: number;
|
145 | // ?scope of id
|
146 | scope?: string;
|
147 | // ?event id. optionally serialized as int64
|
148 | id?: string;
|
149 | // ?scoped event ids
|
150 | id2?: | { local: string; } | { global: string; };
|
151 | // ?async event/event associations
|
152 | bp?: string;
|
153 | // ?flow binding id optionally serialized as int64
|
154 | bind_id?: string;
|
155 | // ?incoming flow flag
|
156 | flow_in?: boolean;
|
157 | // ?outgoing flow flag
|
158 | flow_out?: boolean;
|
159 | // ?scope for TRACE_EVENT_PHASE_INSTANT events
|
160 | s?: TRACE_EVENT_SCOPE;
|
161 | ```
|
162 |
|
163 | # Commands
|
164 |
|
165 |
|
166 | * [`tracerbench compare`](#tracerbench-compare)
|
167 | * [`tracerbench compare:analyze RESULTSFILE`](#tracerbench-compareanalyze-resultsfile)
|
168 | * [`tracerbench help [COMMAND]`](#tracerbench-help-command)
|
169 | * [`tracerbench marker-timings`](#tracerbench-marker-timings)
|
170 | * [`tracerbench record-har`](#tracerbench-record-har)
|
171 | * [`tracerbench report`](#tracerbench-report)
|
172 | * [`tracerbench trace`](#tracerbench-trace)
|
173 |
|
174 | ## `tracerbench compare`
|
175 |
|
176 | Compare the performance delta between an experiment and control
|
177 |
|
178 | ```
|
179 | USAGE
|
180 | $ tracerbench compare
|
181 |
|
182 | OPTIONS
|
183 | --browserArgs=browserArgs
|
184 | (required) [default:
|
185 | --crash-dumps-dir=./tmp,--disable-background-timer-throttling,--disable-dev-shm-usage,--disable-cache,--disable-v8-i
|
186 | dle-tasks,--disable-breakpad,--disable-notifications,--disable-hang-monitor,--safebrowsing-disable-auto-update,--ign
|
187 | ore-certificate-errors,--v8-cache-options=none] (Default Recommended) Additional chrome flags for the TracerBench
|
188 | render benchmark. TracerBench includes many non-configurable defaults in this category.
|
189 |
|
190 | --config=config
|
191 | Specify an alternative directory rather than the project root for the tbconfig.json. This explicit config will
|
192 | overwrite all.
|
193 |
|
194 | --controlURL=controlURL
|
195 | Control URL to visit for compare command
|
196 |
|
197 | --cpuThrottleRate=cpuThrottleRate
|
198 | (required) [default: 2] CPU throttle multiplier
|
199 |
|
200 | --debug
|
201 | Debug flag per command. Will output noisy command
|
202 |
|
203 | --emulateDevice=iphone-4|iphone-5se|iphone-678|iphone-678-plus|iphone-x|blackberry-z30|nexus-4|nexus-5|nexus-5x|nexus-
|
204 | 6|nexus-6p|pixel-2|pixel-2-xl|lg-optimus-l70|nokia-n9|nokia-lumia-520|microsoft-lumia-550|microsoft-lumia-950|galaxy-s
|
205 | -iii|galaxy-s5|kindle-fire-hdx|ipad-mini|ipad|ipad-pro|blackberry-playbook|nexus-10|nexus-7|galaxy-note-3|galaxy-note-
|
206 | ii|laptop-with-touch|laptop-with-hidpi-screen|laptop-with-mdpi-screen
|
207 | Emulate a mobile device screen size.
|
208 |
|
209 | --emulateDeviceOrientation=horizontal|vertical
|
210 | [default: vertical] Expected to be either "vertical" or "horizontal". Dictates orientation of device screen.
|
211 |
|
212 | --experimentURL=experimentURL
|
213 | Experiment URL to visit for compare command
|
214 |
|
215 | --fidelity=fidelity
|
216 | (required) [default: low] Directly correlates to the number of samples per trace. High is the longest trace time.
|
217 |
|
218 | --headless
|
219 | Run with headless chrome flags
|
220 |
|
221 | --hideAnalysis
|
222 | Hide the the analysis output in terminal
|
223 |
|
224 | --markers=markers
|
225 | (required) [default: domComplete] User Timing Markers
|
226 |
|
227 | --network=none|offline|dialup|slow-2g|2g|slow-edge|edge|slow-3g|dsl|3g|fast-3g|4g|cable|LTE|FIOS
|
228 | (required) [default: none] Simulated network conditions.
|
229 |
|
230 | --regressionThreshold=regressionThreshold
|
231 | The upper limit the experiment can regress slower in milliseconds. eg 100
|
232 |
|
233 | --report
|
234 | Generate a PDF report directly after running the compare command.
|
235 |
|
236 | --runtimeStats
|
237 | Compare command output deep-dive stats during run.
|
238 |
|
239 | --socksPorts=socksPorts
|
240 | Specify a socks proxy port as browser option for control and experiment
|
241 |
|
242 | --tbResultsFolder=tbResultsFolder
|
243 | (required) [default: ./tracerbench-results] The output folder path for all tracerbench results
|
244 |
|
245 | --tracingLocationSearch=tracingLocationSearch
|
246 | (required) [default: ?tracing] The document location search param.
|
247 | ```
|
248 |
|
249 | _See code: [dist/src/commands/compare/index.ts](https://github.com/TracerBench/tracerbench/tree/master/packages/cli/blob/v3.0.10/dist/src/commands/compare/index.ts)_
|
250 |
|
251 | ## `tracerbench compare:analyze RESULTSFILE`
|
252 |
|
253 | Run an analysis of a benchmark run from a results json file and output to terminal
|
254 |
|
255 | ```
|
256 | USAGE
|
257 | $ tracerbench compare:analyze RESULTSFILE
|
258 |
|
259 | ARGUMENTS
|
260 | RESULTSFILE Results JSON file
|
261 |
|
262 | OPTIONS
|
263 | --fidelity=fidelity (required) [default: low] Directly correlates to the number of samples per
|
264 | trace. High is the longest trace time.
|
265 |
|
266 | --regressionThreshold=regressionThreshold The upper limit the experiment can regress slower in milliseconds. eg 100
|
267 |
|
268 | --tbResultsFolder=tbResultsFolder (required) [default: ./tracerbench-results] The output folder path for all
|
269 | tracerbench results
|
270 | ```
|
271 |
|
272 | _See code: [dist/src/commands/compare/analyze.ts](https://github.com/TracerBench/tracerbench/tree/master/packages/cli/blob/v3.0.10/dist/src/commands/compare/analyze.ts)_
|
273 |
|
274 | ## `tracerbench help [COMMAND]`
|
275 |
|
276 | display help for tracerbench
|
277 |
|
278 | ```
|
279 | USAGE
|
280 | $ tracerbench help [COMMAND]
|
281 |
|
282 | ARGUMENTS
|
283 | COMMAND command to show help for
|
284 |
|
285 | OPTIONS
|
286 | --all see all commands in CLI
|
287 | ```
|
288 |
|
289 | _See code: [@oclif/plugin-help](https://github.com/oclif/plugin-help/blob/v2.2.1/src/commands/help.ts)_
|
290 |
|
291 | ## `tracerbench marker-timings`
|
292 |
|
293 | Get list of all user-timings from trace
|
294 |
|
295 | ```
|
296 | USAGE
|
297 | $ tracerbench marker-timings
|
298 |
|
299 | OPTIONS
|
300 | --filter=filter User timing marks start with
|
301 | --traceFrame=traceFrame Specify a trace insights frame
|
302 | --tracepath=tracepath (required) The path to the generated trace.json file
|
303 | --url=url (required) URL to visit for record-har, timings & trace commands
|
304 | ```
|
305 |
|
306 | _See code: [dist/src/commands/marker-timings.ts](https://github.com/TracerBench/tracerbench/tree/master/packages/cli/blob/v3.0.10/dist/src/commands/marker-timings.ts)_
|
307 |
|
308 | ## `tracerbench record-har`
|
309 |
|
310 | Generates a HAR file from a URL.
|
311 |
|
312 | ```
|
313 | USAGE
|
314 | $ tracerbench record-har
|
315 |
|
316 | OPTIONS
|
317 | --config=config Specify an alternative directory rather than the project root for the tbconfig.json. This
|
318 | explicit config will overwrite all.
|
319 |
|
320 | --cookiespath=cookiespath (required) The path to a JSON file containing cookies to authenticate against the
|
321 | correlated URL
|
322 |
|
323 | --dest=dest (required) The destination path for the generated file
|
324 |
|
325 | --filename=filename (required) [default: tracerbench] The filename for the generated file
|
326 |
|
327 | --marker=marker (required) [default: domComplete] The last marker before ending recording
|
328 |
|
329 | --url=url (required) URL to visit for record-har, timings & trace commands
|
330 | ```
|
331 |
|
332 | _See code: [dist/src/commands/record-har.ts](https://github.com/TracerBench/tracerbench/tree/master/packages/cli/blob/v3.0.10/dist/src/commands/record-har.ts)_
|
333 |
|
334 | ## `tracerbench report`
|
335 |
|
336 | Parses the output json from tracerbench and formats it into pdf and html
|
337 |
|
338 | ```
|
339 | USAGE
|
340 | $ tracerbench report
|
341 |
|
342 | OPTIONS
|
343 | --config=config Specify an alternative directory rather than the project root for the
|
344 | tbconfig.json. This explicit config will overwrite all.
|
345 |
|
346 | --tbResultsFolder=tbResultsFolder (required) [default: ./tracerbench-results] The output folder path for all
|
347 | tracerbench results
|
348 | ```
|
349 |
|
350 | _See code: [dist/src/commands/report.ts](https://github.com/TracerBench/tracerbench/tree/master/packages/cli/blob/v3.0.10/dist/src/commands/report.ts)_
|
351 |
|
352 | ## `tracerbench trace`
|
353 |
|
354 | Parses a CPU profile and aggregates time across heuristics. Can be vertically sliced with event names.
|
355 |
|
356 | ```
|
357 | USAGE
|
358 | $ tracerbench trace
|
359 |
|
360 | OPTIONS
|
361 | --cookiespath=cookiespath
|
362 | (required) The path to a JSON file containing cookies to authenticate against the correlated URL
|
363 |
|
364 | --cpuThrottleRate=cpuThrottleRate
|
365 | (required) [default: 2] CPU throttle multiplier
|
366 |
|
367 | --harpath=harpath
|
368 | (required) The path to the HTTP Archive File (HAR)
|
369 |
|
370 | --insights
|
371 | Analyze insights from command.
|
372 |
|
373 | --iterations=iterations
|
374 | (required) [default: 1] Number of runs
|
375 |
|
376 | --locations=locations
|
377 | include locations in names
|
378 |
|
379 | --network=none|offline|dialup|slow-2g|2g|slow-edge|edge|slow-3g|dsl|3g|fast-3g|4g|cable|LTE|FIOS
|
380 | [default: none] Simulated network conditions.
|
381 |
|
382 | --tbResultsFolder=tbResultsFolder
|
383 | (required) [default: ./tracerbench-results] The output folder path for all tracerbench results
|
384 |
|
385 | --url=url
|
386 | (required) URL to visit for record-har, timings & trace commands
|
387 | ```
|
388 |
|
389 | _See code: [dist/src/commands/trace.ts](https://github.com/TracerBench/tracerbench/tree/master/packages/cli/blob/v3.0.10/dist/src/commands/trace.ts)_
|
390 |
|