1 | # quickjs-emscripten
|
2 |
|
3 | Javascript/Typescript bindings for QuickJS, a modern Javascript interpreter,
|
4 | compiled to WebAssembly.
|
5 |
|
6 | - Safely evaluate untrusted Javascript (supports [most of ES2023](https://test262.fyi/#|qjs,qjs_ng)).
|
7 | - Create and manipulate values inside the QuickJS runtime ([more][values]).
|
8 | - Expose host functions to the QuickJS runtime ([more][functions]).
|
9 | - Execute synchronous code that uses asynchronous functions, with [asyncify][asyncify].
|
10 |
|
11 | [Github] | [NPM] | [API Documentation][api] | [Variants][core] | [Examples][tests]
|
12 |
|
13 | ```typescript
|
14 | import { getQuickJS } from "quickjs-emscripten"
|
15 |
|
16 | async function main() {
|
17 | const QuickJS = await getQuickJS()
|
18 | const vm = QuickJS.newContext()
|
19 |
|
20 | const world = vm.newString("world")
|
21 | vm.setProp(vm.global, "NAME", world)
|
22 | world.dispose()
|
23 |
|
24 | const result = vm.evalCode(`"Hello " + NAME + "!"`)
|
25 | if (result.error) {
|
26 | console.log("Execution failed:", vm.dump(result.error))
|
27 | result.error.dispose()
|
28 | } else {
|
29 | console.log("Success:", vm.dump(result.value))
|
30 | result.value.dispose()
|
31 | }
|
32 |
|
33 | vm.dispose()
|
34 | }
|
35 |
|
36 | main()
|
37 | ```
|
38 |
|
39 | [github]: https://github.com/justjake/quickjs-emscripten
|
40 | [npm]: https://www.npmjs.com/package/quickjs-emscripten
|
41 | [api]: https://github.com/justjake/quickjs-emscripten/blob/main/doc/packages.md
|
42 | [tests]: https://github.com/justjake/quickjs-emscripten/blob/main/packages/quickjs-emscripten/src/quickjs.test.ts
|
43 | [values]: #interfacing-with-the-interpreter
|
44 | [asyncify]: #asyncify
|
45 | [functions]: #exposing-apis
|
46 |
|
47 | - [quickjs-emscripten](#quickjs-emscripten)
|
48 | - [Usage](#usage)
|
49 | - [Safely evaluate Javascript code](#safely-evaluate-javascript-code)
|
50 | - [Interfacing with the interpreter](#interfacing-with-the-interpreter)
|
51 | - [Runtime](#runtime)
|
52 | - [EcmaScript Module Exports](#ecmascript-module-exports)
|
53 | - [Memory Management](#memory-management)
|
54 | - [`using` statement](#using-statement)
|
55 | - [Scope](#scope)
|
56 | - [`Lifetime.consume(fn)`](#lifetimeconsumefn)
|
57 | - [Exposing APIs](#exposing-apis)
|
58 | - [Promises](#promises)
|
59 | - [context.getPromiseState(handle)](#contextgetpromisestatehandle)
|
60 | - [context.resolvePromise(handle)](#contextresolvepromisehandle)
|
61 | - [Asyncify](#asyncify)
|
62 | - [Async module loader](#async-module-loader)
|
63 | - [Async on host, sync in QuickJS](#async-on-host-sync-in-quickjs)
|
64 | - [Testing your code](#testing-your-code)
|
65 | - [Packaging](#packaging)
|
66 | - [Reducing package size](#reducing-package-size)
|
67 | - [WebAssembly loading](#webassembly-loading)
|
68 | - [quickjs-ng](#quickjs-ng)
|
69 | - [Using in the browser without a build step](#using-in-the-browser-without-a-build-step)
|
70 | - [Debugging](#debugging)
|
71 | - [Supported Platforms](#supported-platforms)
|
72 | - [More Documentation](#more-documentation)
|
73 | - [Background](#background)
|
74 | - [Status \& Roadmap](#status--roadmap)
|
75 | - [Related](#related)
|
76 | - [Developing](#developing)
|
77 | - [The C parts](#the-c-parts)
|
78 | - [The Typescript parts](#the-typescript-parts)
|
79 | - [Yarn updates](#yarn-updates)
|
80 |
|
81 | ## Usage
|
82 |
|
83 | Install from `npm`: `npm install --save quickjs-emscripten` or `yarn add quickjs-emscripten`.
|
84 |
|
85 | The root entrypoint of this library is the `getQuickJS` function, which returns
|
86 | a promise that resolves to a [QuickJSWASMModule](./doc/quickjs-emscripten/classes/QuickJSWASMModule.md) when
|
87 | the QuickJS WASM module is ready.
|
88 |
|
89 | Once `getQuickJS` has been awaited at least once, you also can use the `getQuickJSSync`
|
90 | function to directly access the singleton in your synchronous code.
|
91 |
|
92 | ### Safely evaluate Javascript code
|
93 |
|
94 | See [QuickJSWASMModule.evalCode](https://github.com/justjake/quickjs-emscripten/blob/main/doc/quickjs-emscripten/classes/QuickJSWASMModule.md#evalcode)
|
95 |
|
96 | ```typescript
|
97 | import { getQuickJS, shouldInterruptAfterDeadline } from "quickjs-emscripten"
|
98 |
|
99 | getQuickJS().then((QuickJS) => {
|
100 | const result = QuickJS.evalCode("1 + 1", {
|
101 | shouldInterrupt: shouldInterruptAfterDeadline(Date.now() + 1000),
|
102 | memoryLimitBytes: 1024 * 1024,
|
103 | })
|
104 | console.log(result)
|
105 | })
|
106 | ```
|
107 |
|
108 | ### Interfacing with the interpreter
|
109 |
|
110 | You can use [QuickJSContext](https://github.com/justjake/quickjs-emscripten/blob/main/doc/quickjs-emscripten/classes/QuickJSContext.md)
|
111 | to build a scripting environment by modifying globals and exposing functions
|
112 | into the QuickJS interpreter.
|
113 |
|
114 | Each `QuickJSContext` instance has its own environment -- globals, built-in
|
115 | classes -- and actions from one context won't leak into other contexts or
|
116 | runtimes (with one exception, see [Asyncify][asyncify]).
|
117 |
|
118 | Every context is created inside a
|
119 | [QuickJSRuntime](https://github.com/justjake/quickjs-emscripten/blob/main/doc/quickjs-emscripten/classes/QuickJSRuntime.md).
|
120 | A runtime represents a Javascript heap, and you can even share values between
|
121 | contexts in the same runtime.
|
122 |
|
123 | ```typescript
|
124 | const vm = QuickJS.newContext()
|
125 | let state = 0
|
126 |
|
127 | const fnHandle = vm.newFunction("nextId", () => {
|
128 | return vm.newNumber(++state)
|
129 | })
|
130 |
|
131 | vm.setProp(vm.global, "nextId", fnHandle)
|
132 | fnHandle.dispose()
|
133 |
|
134 | const nextId = vm.unwrapResult(vm.evalCode(`nextId(); nextId(); nextId()`))
|
135 | console.log("vm result:", vm.getNumber(nextId), "native state:", state)
|
136 |
|
137 | nextId.dispose()
|
138 | vm.dispose()
|
139 | ```
|
140 |
|
141 | When you create a context from a top-level API like in the example above,
|
142 | instead of by calling `runtime.newContext()`, a runtime is automatically created
|
143 | for the lifetime of the context, and disposed of when you dispose the context.
|
144 |
|
145 | #### Runtime
|
146 |
|
147 | The runtime has APIs for CPU and memory limits that apply to all contexts within
|
148 | the runtime in aggregate. You can also use the runtime to configure EcmaScript
|
149 | module loading.
|
150 |
|
151 | ```typescript
|
152 | const runtime = QuickJS.newRuntime()
|
153 | // "Should be enough for everyone" -- attributed to B. Gates
|
154 | runtime.setMemoryLimit(1024 * 640)
|
155 | // Limit stack size
|
156 | runtime.setMaxStackSize(1024 * 320)
|
157 | // Interrupt computation after 1024 calls to the interrupt handler
|
158 | let interruptCycles = 0
|
159 | runtime.setInterruptHandler(() => ++interruptCycles > 1024)
|
160 | // Toy module system that always returns the module name
|
161 | // as the default export
|
162 | runtime.setModuleLoader((moduleName) => `export default '${moduleName}'`)
|
163 | const context = runtime.newContext()
|
164 | const ok = context.evalCode(`
|
165 | import fooName from './foo.js'
|
166 | globalThis.result = fooName
|
167 | `)
|
168 | context.unwrapResult(ok).dispose()
|
169 | // logs "foo.js"
|
170 | console.log(context.getProp(context.global, "result").consume(context.dump))
|
171 | context.dispose()
|
172 | runtime.dispose()
|
173 | ```
|
174 |
|
175 | #### EcmaScript Module Exports
|
176 |
|
177 | When you evaluate code as an ES Module, the result will be a handle to the
|
178 | module's exports, or a handle to a promise that resolves to the module's
|
179 | exports if the module depends on a top-level await.
|
180 |
|
181 | ```typescript
|
182 | const context = QuickJS.newContext()
|
183 | const result = context.evalCode(
|
184 | `
|
185 | export const name = 'Jake'
|
186 | export const favoriteBean = 'wax bean'
|
187 | export default 'potato'
|
188 | `,
|
189 | "jake.js",
|
190 | { type: "module" },
|
191 | )
|
192 | const moduleExports = context.unwrapResult(result)
|
193 | console.log(context.dump(moduleExports))
|
194 | // -> { name: 'Jake', favoriteBean: 'wax bean', default: 'potato' }
|
195 | moduleExports.dispose()
|
196 | ```
|
197 |
|
198 | ### Memory Management
|
199 |
|
200 | Many methods in this library return handles to memory allocated inside the
|
201 | WebAssembly heap. These types cannot be garbage-collected as usual in
|
202 | Javascript. Instead, you must manually manage their memory by calling a
|
203 | `.dispose()` method to free the underlying resources. Once a handle has been
|
204 | disposed, it cannot be used anymore. Note that in the example above, we call
|
205 | `.dispose()` on each handle once it is no longer needed.
|
206 |
|
207 | Calling `QuickJSContext.dispose()` will throw a RuntimeError if you've forgotten to
|
208 | dispose any handles associated with that VM, so it's good practice to create a
|
209 | new VM instance for each of your tests, and to call `vm.dispose()` at the end
|
210 | of every test.
|
211 |
|
212 | ```typescript
|
213 | const vm = QuickJS.newContext()
|
214 | const numberHandle = vm.newNumber(42)
|
215 | // Note: numberHandle not disposed, so it leaks memory.
|
216 | vm.dispose()
|
217 | // throws RuntimeError: abort(Assertion failed: list_empty(&rt->gc_obj_list), at: quickjs/quickjs.c,1963,JS_FreeRuntime)
|
218 | ```
|
219 |
|
220 | Here are some strategies to reduce the toil of calling `.dispose()` on each
|
221 | handle you create:
|
222 |
|
223 | #### `using` statement
|
224 |
|
225 | The `using` statement is a Stage 3 (as of 2023-12-29) proposal for Javascript that declares a constant variable and automatically calls the `[Symbol.dispose]()` method of an object when it goes out of scope. Read more [in this Typescript release announcement][using]. Here's the "Interfacing with the interpreter" example re-written using `using`:
|
226 |
|
227 | ```typescript
|
228 | using vm = QuickJS.newContext()
|
229 | let state = 0
|
230 |
|
231 | // The block here isn't needed for correctness, but it shows
|
232 | // how to get a tighter bound on the lifetime of `fnHandle`.
|
233 | {
|
234 | using fnHandle = vm.newFunction("nextId", () => {
|
235 | return vm.newNumber(++state)
|
236 | })
|
237 |
|
238 | vm.setProp(vm.global, "nextId", fnHandle)
|
239 | // fnHandle.dispose() is called automatically when the block exits
|
240 | }
|
241 |
|
242 | using nextId = vm.unwrapResult(vm.evalCode(`nextId(); nextId(); nextId()`))
|
243 | console.log("vm result:", vm.getNumber(nextId), "native state:", state)
|
244 | // nextId.dispose() is called automatically when the block exits
|
245 | // vm.dispose() is called automatically when the block exits
|
246 | ```
|
247 |
|
248 | [using]: https://www.typescriptlang.org/docs/handbook/release-notes/typescript-5-2.html#using-declarations-and-explicit-resource-management
|
249 |
|
250 | #### Scope
|
251 |
|
252 | A
|
253 | [`Scope`](https://github.com/justjake/quickjs-emscripten/blob/main/doc/quickjs-emscripten/classes/Scope.md#class-scope)
|
254 | instance manages a set of disposables and calls their `.dispose()`
|
255 | method in the reverse order in which they're added to the scope. Here's the
|
256 | "Interfacing with the interpreter" example re-written using `Scope`:
|
257 |
|
258 | ```typescript
|
259 | Scope.withScope((scope) => {
|
260 | const vm = scope.manage(QuickJS.newContext())
|
261 | let state = 0
|
262 |
|
263 | const fnHandle = scope.manage(
|
264 | vm.newFunction("nextId", () => {
|
265 | return vm.newNumber(++state)
|
266 | }),
|
267 | )
|
268 |
|
269 | vm.setProp(vm.global, "nextId", fnHandle)
|
270 |
|
271 | const nextId = scope.manage(vm.unwrapResult(vm.evalCode(`nextId(); nextId(); nextId()`)))
|
272 | console.log("vm result:", vm.getNumber(nextId), "native state:", state)
|
273 |
|
274 | // When the withScope block exits, it calls scope.dispose(), which in turn calls
|
275 | // the .dispose() methods of all the disposables managed by the scope.
|
276 | })
|
277 | ```
|
278 |
|
279 | You can also create `Scope` instances with `new Scope()` if you want to manage
|
280 | calling `scope.dispose()` yourself.
|
281 |
|
282 | #### `Lifetime.consume(fn)`
|
283 |
|
284 | [`Lifetime.consume`](https://github.com/justjake/quickjs-emscripten/blob/main/doc/quickjs-emscripten/classes/lifetime.md#consume)
|
285 | is sugar for the common pattern of using a handle and then
|
286 | immediately disposing of it. `Lifetime.consume` takes a `map` function that
|
287 | produces a result of any type. The `map` fuction is called with the handle,
|
288 | then the handle is disposed, then the result is returned.
|
289 |
|
290 | Here's the "Interfacing with interpreter" example re-written using `.consume()`:
|
291 |
|
292 | ```typescript
|
293 | const vm = QuickJS.newContext()
|
294 | let state = 0
|
295 |
|
296 | vm.newFunction("nextId", () => {
|
297 | return vm.newNumber(++state)
|
298 | }).consume((fnHandle) => vm.setProp(vm.global, "nextId", fnHandle))
|
299 |
|
300 | vm.unwrapResult(vm.evalCode(`nextId(); nextId(); nextId()`)).consume((nextId) =>
|
301 | console.log("vm result:", vm.getNumber(nextId), "native state:", state),
|
302 | )
|
303 |
|
304 | vm.dispose()
|
305 | ```
|
306 |
|
307 | Generally working with `Scope` leads to more straight-forward code, but
|
308 | `Lifetime.consume` can be handy sugar as part of a method call chain.
|
309 |
|
310 | ### Exposing APIs
|
311 |
|
312 | To add APIs inside the QuickJS environment, you'll need to [create objects][newObject] to
|
313 | define the shape of your API, and [add properties][setProp] and [functions][newFunction] to those objects
|
314 | to allow code inside QuickJS to call code on the host.
|
315 | The [newFunction][] documentation covers writing functions in detail.
|
316 |
|
317 | By default, no host functionality is exposed to code running inside QuickJS.
|
318 |
|
319 | ```typescript
|
320 | const vm = QuickJS.newContext()
|
321 | // `console.log`
|
322 | const logHandle = vm.newFunction("log", (...args) => {
|
323 | const nativeArgs = args.map(vm.dump)
|
324 | console.log("QuickJS:", ...nativeArgs)
|
325 | })
|
326 | // Partially implement `console` object
|
327 | const consoleHandle = vm.newObject()
|
328 | vm.setProp(consoleHandle, "log", logHandle)
|
329 | vm.setProp(vm.global, "console", consoleHandle)
|
330 | consoleHandle.dispose()
|
331 | logHandle.dispose()
|
332 |
|
333 | vm.unwrapResult(vm.evalCode(`console.log("Hello from QuickJS!")`)).dispose()
|
334 | ```
|
335 |
|
336 | [newObject]: https://github.com/justjake/quickjs-emscripten/blob/main/doc/quickjs-emscripten/classes/QuickJSContext.md#newobject
|
337 | [newFunction]: https://github.com/justjake/quickjs-emscripten/blob/main/doc/quickjs-emscripten/classes/QuickJSContext.md#newfunction
|
338 | [setProp]: https://github.com/justjake/quickjs-emscripten/blob/main/doc/quickjs-emscripten/classes/QuickJSContext.md#setprop
|
339 |
|
340 | #### Promises
|
341 |
|
342 | To expose an asynchronous function that _returns a promise_ to callers within
|
343 | QuickJS, your function can return the handle of a `QuickJSDeferredPromise`
|
344 | created via `context.newPromise()`.
|
345 |
|
346 | When you resolve a `QuickJSDeferredPromise` -- and generally whenever async
|
347 | behavior completes for the VM -- pending listeners inside QuickJS may not
|
348 | execute immediately. Your code needs to explicitly call
|
349 | `runtime.executePendingJobs()` to resume execution inside QuickJS. This API
|
350 | gives your code maximum control to _schedule_ when QuickJS will block the host's
|
351 | event loop by resuming execution.
|
352 |
|
353 | To work with QuickJS handles that contain a promise inside the environment,
|
354 | there are two options:
|
355 |
|
356 | ##### context.getPromiseState(handle)
|
357 |
|
358 | You can synchronously peek into a QuickJS promise handle and get its state
|
359 | without introducing asynchronous host code, described by the type [JSPromiseState][]:
|
360 |
|
361 | ```typescript
|
362 | type JSPromiseState =
|
363 | | { type: "pending"; error: Error }
|
364 | | { type: "fulfilled"; value: QuickJSHandle; notAPromise?: boolean }
|
365 | | { type: "rejected"; error: QuickJSHandle }
|
366 | ```
|
367 |
|
368 | The result conforms to the `SuccessOrFail` type returned by `context.evalCode`, so you can use `context.unwrapResult(context.getPromiseState(promiseHandle))` to assert a promise is settled successfully and retrieve its value. Calling `context.unwrapResult` on a pending or rejected promise will throw an error.
|
369 |
|
370 | ```typescript
|
371 | const promiseHandle = context.evalCode(`Promise.resolve(42)`)
|
372 | const resultHandle = context.unwrapResult(context.getPromiseState(promiseHandle))
|
373 | context.getNumber(resultHandle) === 42 // true
|
374 | resultHandle.dispose()
|
375 | promiseHandle.dispose()
|
376 | ```
|
377 |
|
378 | [JSPromiseState]: https://github.com/justjake/quickjs-emscripten/blob/main/doc/quickjs-emscripten/exports.md#jspromisestate
|
379 |
|
380 | ##### context.resolvePromise(handle)
|
381 |
|
382 | You can convert the QuickJSHandle into a native promise using
|
383 | `context.resolvePromise()`. Take care with this API to avoid 'deadlocks' where
|
384 | the host awaits a guest promise, but the guest cannot make progress until the
|
385 | host calls `runtime.executePendingJobs()`. The simplest way to avoid this kind
|
386 | of deadlock is to always schedule `executePendingJobs` after any promise is
|
387 | settled.
|
388 |
|
389 | ```typescript
|
390 | const vm = QuickJS.newContext()
|
391 | const fakeFileSystem = new Map([["example.txt", "Example file content"]])
|
392 |
|
393 | // Function that simulates reading data asynchronously
|
394 | const readFileHandle = vm.newFunction("readFile", (pathHandle) => {
|
395 | const path = vm.getString(pathHandle)
|
396 | const promise = vm.newPromise()
|
397 | setTimeout(() => {
|
398 | const content = fakeFileSystem.get(path)
|
399 | promise.resolve(vm.newString(content || ""))
|
400 | }, 100)
|
401 | // IMPORTANT: Once you resolve an async action inside QuickJS,
|
402 | // call runtime.executePendingJobs() to run any code that was
|
403 | // waiting on the promise or callback.
|
404 | promise.settled.then(vm.runtime.executePendingJobs)
|
405 | return promise.handle
|
406 | })
|
407 | readFileHandle.consume((handle) => vm.setProp(vm.global, "readFile", handle))
|
408 |
|
409 | // Evaluate code that uses `readFile`, which returns a promise
|
410 | const result = vm.evalCode(`(async () => {
|
411 | const content = await readFile('example.txt')
|
412 | return content.toUpperCase()
|
413 | })()`)
|
414 | const promiseHandle = vm.unwrapResult(result)
|
415 |
|
416 | // Convert the promise handle into a native promise and await it.
|
417 | // If code like this deadlocks, make sure you are calling
|
418 | // runtime.executePendingJobs appropriately.
|
419 | const resolvedResult = await vm.resolvePromise(promiseHandle)
|
420 | promiseHandle.dispose()
|
421 | const resolvedHandle = vm.unwrapResult(resolvedResult)
|
422 | console.log("Result:", vm.getString(resolvedHandle))
|
423 | resolvedHandle.dispose()
|
424 | ```
|
425 |
|
426 | #### Asyncify
|
427 |
|
428 | Sometimes, we want to create a function that's synchronous from the perspective
|
429 | of QuickJS, but prefer to implement that function _asynchronously_ in your host
|
430 | code. The most obvious use-case is for EcmaScript module loading. The underlying
|
431 | QuickJS C library expects the module loader function to return synchronously,
|
432 | but loading data synchronously in the browser or server is somewhere between "a
|
433 | bad idea" and "impossible". QuickJS also doesn't expose an API to "pause" the
|
434 | execution of a runtime, and adding such an API is tricky due to the VM's
|
435 | implementation.
|
436 |
|
437 | As a work-around, we provide an alternate build of QuickJS processed by
|
438 | Emscripten/Binaryen's [ASYNCIFY](https://emscripten.org/docs/porting/asyncify.html)
|
439 | compiler transform. Here's how Emscripten's documentation describes Asyncify:
|
440 |
|
441 | > Asyncify lets synchronous C or C++ code interact with asynchronous \[host] JavaScript. This allows things like:
|
442 | >
|
443 | > - A synchronous call in C that yields to the event loop, which allows browser events to be handled.
|
444 | > - A synchronous call in C that waits for an asynchronous operation in \[host] JS to complete.
|
445 | >
|
446 | > Asyncify automatically transforms ... code into a form that can be paused and
|
447 | > resumed ..., so that it is asynchronous (hence the name “Asyncify”) even though
|
448 | > \[it is written] in a normal synchronous way.
|
449 |
|
450 | This means we can suspend an _entire WebAssembly module_ (which could contain
|
451 | multiple runtimes and contexts) while our host Javascript loads data
|
452 | asynchronously, and then resume execution once the data load completes. This is
|
453 | a very handy superpower, but it comes with a couple of major limitations:
|
454 |
|
455 | 1. _An asyncified WebAssembly module can only suspend to wait for a single
|
456 | asynchronous call at a time_. You may call back into a suspended WebAssembly
|
457 | module eg. to create a QuickJS value to return a result, but the system will
|
458 | crash if this call tries to suspend again. Take a look at Emscripten's documentation
|
459 | on [reentrancy](https://emscripten.org/docs/porting/asyncify.html#reentrancy).
|
460 |
|
461 | 2. _Asyncified code is bigger and runs slower_. The asyncified build of
|
462 | Quickjs-emscripten library is 1M, 2x larger than the 500K of the default
|
463 | version. There may be room for further
|
464 | [optimization](https://emscripten.org/docs/porting/asyncify.html#optimizing)
|
465 | Of our build in the future.
|
466 |
|
467 | To use asyncify features, use the following functions:
|
468 |
|
469 | - [newAsyncRuntime][]: create a runtime inside a new WebAssembly module.
|
470 | - [newAsyncContext][]: create runtime and context together inside a new
|
471 | WebAssembly module.
|
472 | - [newQuickJSAsyncWASMModule][]: create an empty WebAssembly module.
|
473 |
|
474 | [newasyncruntime]: https://github.com/justjake/quickjs-emscripten/blob/main/doc/quickjs-emscripten/exports.md#newasyncruntime
|
475 | [newasynccontext]: https://github.com/justjake/quickjs-emscripten/blob/main/doc/quickjs-emscripten/exports.md#newasynccontext
|
476 | [newquickjsasyncwasmmodule]: https://github.com/justjake/quickjs-emscripten/blob/main/doc/quickjs-emscripten/exports.md#newquickjsasyncwasmmodule
|
477 |
|
478 | These functions are asynchronous because they always create a new underlying
|
479 | WebAssembly module so that each instance can suspend and resume independently,
|
480 | and instantiating a WebAssembly module is an async operation. This also adds
|
481 | substantial overhead compared to creating a runtime or context inside an
|
482 | existing module; if you only need to wait for a single async action at a time,
|
483 | you can create a single top-level module and create runtimes or contexts inside
|
484 | of it.
|
485 |
|
486 | ##### Async module loader
|
487 |
|
488 | Here's an example of valuating a script that loads React asynchronously as an ES
|
489 | module. In our example, we're loading from the filesystem for reproducibility,
|
490 | but you can use this technique to load using `fetch`.
|
491 |
|
492 | ```typescript
|
493 | const module = await newQuickJSAsyncWASMModule()
|
494 | const runtime = module.newRuntime()
|
495 | const path = await import("path")
|
496 | const { promises: fs } = await import("fs")
|
497 |
|
498 | const importsPath = path.join(__dirname, "../examples/imports") + "/"
|
499 | // Module loaders can return promises.
|
500 | // Execution will suspend until the promise resolves.
|
501 | runtime.setModuleLoader((moduleName) => {
|
502 | const modulePath = path.join(importsPath, moduleName)
|
503 | if (!modulePath.startsWith(importsPath)) {
|
504 | throw new Error("out of bounds")
|
505 | }
|
506 | console.log("loading", moduleName, "from", modulePath)
|
507 | return fs.readFile(modulePath, "utf-8")
|
508 | })
|
509 |
|
510 | // evalCodeAsync is required when execution may suspend.
|
511 | const context = runtime.newContext()
|
512 | const result = await context.evalCodeAsync(`
|
513 | import * as React from 'esm.sh/react@17'
|
514 | import * as ReactDOMServer from 'esm.sh/react-dom@17/server'
|
515 | const e = React.createElement
|
516 | globalThis.html = ReactDOMServer.renderToStaticMarkup(
|
517 | e('div', null, e('strong', null, 'Hello world!'))
|
518 | )
|
519 | `)
|
520 | context.unwrapResult(result).dispose()
|
521 | const html = context.getProp(context.global, "html").consume(context.getString)
|
522 | console.log(html) // <div><strong>Hello world!</strong></div>
|
523 | ```
|
524 |
|
525 | ##### Async on host, sync in QuickJS
|
526 |
|
527 | Here's an example of turning an async function into a sync function inside the
|
528 | VM.
|
529 |
|
530 | ```typescript
|
531 | const context = await newAsyncContext()
|
532 | const path = await import("path")
|
533 | const { promises: fs } = await import("fs")
|
534 |
|
535 | const importsPath = path.join(__dirname, "../examples/imports") + "/"
|
536 | const readFileHandle = context.newAsyncifiedFunction("readFile", async (pathHandle) => {
|
537 | const pathString = path.join(importsPath, context.getString(pathHandle))
|
538 | if (!pathString.startsWith(importsPath)) {
|
539 | throw new Error("out of bounds")
|
540 | }
|
541 | const data = await fs.readFile(pathString, "utf-8")
|
542 | return context.newString(data)
|
543 | })
|
544 | readFileHandle.consume((fn) => context.setProp(context.global, "readFile", fn))
|
545 |
|
546 | // evalCodeAsync is required when execution may suspend.
|
547 | const result = await context.evalCodeAsync(`
|
548 | // Not a promise! Sync! vvvvvvvvvvvvvvvvvvvv
|
549 | const data = JSON.parse(readFile('data.json'))
|
550 | data.map(x => x.toUpperCase()).join(' ')
|
551 | `)
|
552 | const upperCaseData = context.unwrapResult(result).consume(context.getString)
|
553 | console.log(upperCaseData) // 'VERY USEFUL DATA'
|
554 | ```
|
555 |
|
556 | ### Testing your code
|
557 |
|
558 | This library is complicated to use, so please consider automated testing your
|
559 | implementation. We highly writing your test suite to run with both the "release"
|
560 | build variant of quickjs-emscripten, and also the [DEBUG_SYNC] build variant.
|
561 | The debug sync build variant has extra instrumentation code for detecting memory
|
562 | leaks.
|
563 |
|
564 | The class [TestQuickJSWASMModule] exposes the memory leak detection API, although
|
565 | this API is only accurate when using `DEBUG_SYNC` variant.
|
566 |
|
567 | ```typescript
|
568 | // Define your test suite in a function, so that you can test against
|
569 | // different module loaders.
|
570 | function myTests(moduleLoader: () => Promise<QuickJSWASMModule>) {
|
571 | let QuickJS: TestQuickJSWASMModule
|
572 | beforeEach(async () => {
|
573 | // Get a unique TestQuickJSWASMModule instance for each test.
|
574 | const wasmModule = await moduleLoader()
|
575 | QuickJS = new TestQuickJSWASMModule(wasmModule)
|
576 | })
|
577 | afterEach(() => {
|
578 | // Assert that the test disposed all handles. The DEBUG_SYNC build
|
579 | // variant will show detailed traces for each leak.
|
580 | QuickJS.assertNoMemoryAllocated()
|
581 | })
|
582 |
|
583 | it("works well", () => {
|
584 | // TODO: write a test using QuickJS
|
585 | const context = QuickJS.newContext()
|
586 | context.unwrapResult(context.evalCode("1 + 1")).dispose()
|
587 | context.dispose()
|
588 | })
|
589 | }
|
590 |
|
591 | // Run the test suite against a matrix of module loaders.
|
592 | describe("Check for memory leaks with QuickJS DEBUG build", () => {
|
593 | const moduleLoader = memoizePromiseFactory(() => newQuickJSWASMModule(DEBUG_SYNC))
|
594 | myTests(moduleLoader)
|
595 | })
|
596 |
|
597 | describe("Realistic test with QuickJS RELEASE build", () => {
|
598 | myTests(getQuickJS)
|
599 | })
|
600 | ```
|
601 |
|
602 | For more testing examples, please explore the typescript source of [quickjs-emscripten][ts] repository.
|
603 |
|
604 | [ts]: https://github.com/justjake/quickjs-emscripten/blob/main/packages/quickjs-emscripten/src/
|
605 | [debug_sync]: https://github.com/justjake/quickjs-emscripten/blob/main/doc/quickjs-emscripten/exports.md#debug_sync
|
606 | [testquickjswasmmodule]: https://github.com/justjake/quickjs-emscripten/blob/main/doc/quickjs-emscripten/classes/TestQuickJSWASMModule.md
|
607 |
|
608 | ### Packaging
|
609 |
|
610 | The main `quickjs-emscripten` package includes several build variants of the WebAssembly module:
|
611 |
|
612 | - `RELEASE...` build variants should be used in production. They offer better performance and smaller file size compared to `DEBUG...` build variants.
|
613 | - `RELEASE_SYNC`: This is the default variant used when you don't explicitly provide one. It offers the fastest performance and smallest file size.
|
614 | - `RELEASE_ASYNC`: The default variant if you need [asyncify][] magic, which comes at a performance cost. See the asyncify docs for details.
|
615 | - `DEBUG...` build variants can be helpful during development and testing. They include source maps and assertions for catching bugs in your code. We recommend running your tests with _both_ a debug build variant and the release build variant you'll use in production.
|
616 | - `DEBUG_SYNC`: Instrumented to detect memory leaks, in addition to assertions and source maps.
|
617 | - `DEBUG_ASYNC`: An [asyncify][] variant with source maps.
|
618 |
|
619 | To use a variant, call `newQuickJSWASMModule` or `newQuickJSAsyncWASMModule` with the variant object. These functions return a promise that resolves to a [QuickJSWASMModule](./doc/quickjs-emscripten/classes/QuickJSWASMModule.md), the same as `getQuickJS`.
|
620 |
|
621 | ```typescript
|
622 | import {
|
623 | newQuickJSWASMModule,
|
624 | newQuickJSAsyncWASMModule,
|
625 | RELEASE_SYNC,
|
626 | DEBUG_SYNC,
|
627 | RELEASE_ASYNC,
|
628 | DEBUG_ASYNC,
|
629 | } from "quickjs-emscripten"
|
630 |
|
631 | const QuickJSReleaseSync = await newQuickJSWASMModule(RELEASE_SYNC)
|
632 | const QuickJSDebugSync = await newQuickJSWASMModule(DEBUG_SYNC)
|
633 | const QuickJSReleaseAsync = await newQuickJSAsyncWASMModule(RELEASE_ASYNC)
|
634 | const QuickJSDebugAsync = await newQuickJSAsyncWASMModule(DEBUG_ASYNC)
|
635 |
|
636 | for (const quickjs of [
|
637 | QuickJSReleaseSync,
|
638 | QuickJSDebugSync,
|
639 | QuickJSReleaseAsync,
|
640 | QuickJSDebugAsync,
|
641 | ]) {
|
642 | const vm = quickjs.newContext()
|
643 | const result = vm.unwrapResult(vm.evalCode("1 + 1")).consume(vm.getNumber)
|
644 | console.log(result)
|
645 | vm.dispose()
|
646 | quickjs.dispose()
|
647 | }
|
648 | ```
|
649 |
|
650 | #### Reducing package size
|
651 |
|
652 | Including 4 different copies of the WebAssembly module in the main package gives it an install size of [about 9.04mb](https://packagephobia.com/result?p=quickjs-emscripten). If you're building a CLI package or library of your own, or otherwise don't need to include 4 different variants in your `node_modules`, you can switch to the `quickjs-emscripten-core` package, which contains only the Javascript code for this library, and install one (or more) variants a-la-carte as separate packages.
|
653 |
|
654 | The most minimal setup would be to install `quickjs-emscripten-core` and `@jitl/quickjs-wasmfile-release-sync` (1.3mb total):
|
655 |
|
656 | ```bash
|
657 | yarn add quickjs-emscripten-core @jitl/quickjs-wasmfile-release-sync
|
658 | du -h node_modules
|
659 | # 640K node_modules/@jitl/quickjs-wasmfile-release-sync
|
660 | # 80K node_modules/@jitl/quickjs-ffi-types
|
661 | # 588K node_modules/quickjs-emscripten-core
|
662 | # 1.3M node_modules
|
663 | ```
|
664 |
|
665 | Then, you can use quickjs-emscripten-core's `newQuickJSWASMModuleFromVariant` to create a QuickJS module (see [the minimal example][minimal]):
|
666 |
|
667 | ```typescript
|
668 | // src/quickjs.mjs
|
669 | import { newQuickJSWASMModuleFromVariant } from "quickjs-emscripten-core"
|
670 | import RELEASE_SYNC from "@jitl/quickjs-wasmfile-release-sync"
|
671 | export const QuickJS = await newQuickJSWASMModuleFromVariant(RELEASE_SYNC)
|
672 |
|
673 | // src/app.mjs
|
674 | import { QuickJS } from "./quickjs.mjs"
|
675 | console.log(QuickJS.evalCode("1 + 1"))
|
676 | ```
|
677 |
|
678 | See the [documentation of quickjs-emscripten-core][core] for more details and the list of variant packages.
|
679 |
|
680 | [core]: https://github.com/justjake/quickjs-emscripten/blob/main/doc/quickjs-emscripten-core/README.md
|
681 |
|
682 | #### WebAssembly loading
|
683 |
|
684 | To run QuickJS, we need to load a WebAssembly module into the host Javascript runtime's memory (usually as an ArrayBuffer or TypedArray) and [compile it](https://developer.mozilla.org/en-US/docs/WebAssembly/JavaScript_interface/instantiate_static) to a [WebAssembly.Module](https://developer.mozilla.org/en-US/docs/WebAssembly/JavaScript_interface/Module). This means we need to find the file path or URI of the WebAssembly module, and then read it using an API like `fetch` (browser) or `fs.readFile` (NodeJS). `quickjs-emscripten` tries to handle this automatically using patterns like `new URL('./local-path', import.meta.url)` that work in the browser or are handled automatically by bundlers, or `__dirname` in NodeJS, but you may need to configure this manually if these don't work in your environment, or you want more control about how the WebAssembly module is loaded.
|
685 |
|
686 | To customize the loading of an existing variant, create a new variant with your loading settings using `newVariant`, passing [CustomizeVariantOptions][newVariant]. For example, you need to customize loading in Cloudflare Workers (see [the full example][cloudflare]).
|
687 |
|
688 | ```typescript
|
689 | import { newQuickJSWASMModule, DEBUG_SYNC as baseVariant, newVariant } from "quickjs-emscripten"
|
690 | import cloudflareWasmModule from "./DEBUG_SYNC.wasm"
|
691 | import cloudflareWasmModuleSourceMap from "./DEBUG_SYNC.wasm.map.txt"
|
692 |
|
693 | /**
|
694 | * We need to make a new variant that directly passes the imported WebAssembly.Module
|
695 | * to Emscripten. Normally we'd load the wasm file as bytes from a URL, but
|
696 | * that's forbidden in Cloudflare workers.
|
697 | */
|
698 | const cloudflareVariant = newVariant(baseVariant, {
|
699 | wasmModule: cloudflareWasmModule,
|
700 | wasmSourceMapData: cloudflareWasmModuleSourceMap,
|
701 | })
|
702 | ```
|
703 |
|
704 | [newVariant]: https://github.com/justjake/quickjs-emscripten/blob/main/doc/quickjs-emscripten/interfaces/CustomizeVariantOptions.md
|
705 |
|
706 | #### quickjs-ng
|
707 |
|
708 | [quickjs-ng/quickjs](https://github.com/quickjs-ng/quickjs) (aka quickjs-ng) is a fork of the original [bellard/quickjs](https://github.com/quickjs-ng/quickjs) under active development. It implements more EcmaScript standards and removes some of quickjs's custom language features like BigFloat.
|
709 |
|
710 | There are several variants of quickjs-ng available, and quickjs-emscripten may switch to using quickjs-ng by default in the future. See [the list of variants][core].
|
711 |
|
712 | #### Using in the browser without a build step
|
713 |
|
714 | You can use quickjs-emscripten directly from an HTML file in two ways:
|
715 |
|
716 | 1. Import it in an ES Module script tag
|
717 |
|
718 | ```html
|
719 | <!doctype html>
|
720 | <!-- Import from a ES Module CDN -->
|
721 | <script type="module">
|
722 | import { getQuickJS } from "https://esm.sh/quickjs-emscripten@0.25.0"
|
723 | const QuickJS = await getQuickJS()
|
724 | console.log(QuickJS.evalCode("1+1"))
|
725 | </script>
|
726 | ```
|
727 |
|
728 | 1. In edge cases, you might want to use the IIFE build which provides QuickJS as the global `QJS`. You should probably use the ES module though, any recent browser supports it.
|
729 |
|
730 | ```html
|
731 | <!doctype html>
|
732 | <!-- Add a script tag to load the library as the QJS global -->
|
733 | <script
|
734 | src="https://cdn.jsdelivr.net/npm/quickjs-emscripten@0.25.0/dist/index.global.js"
|
735 | type="text/javascript"
|
736 | ></script>
|
737 | <!-- Then use the QJS global in a script tag -->
|
738 | <script type="text/javascript">
|
739 | QJS.getQuickJS().then((QuickJS) => {
|
740 | console.log(QuickJS.evalCode("1+1"))
|
741 | })
|
742 | </script>
|
743 | ```
|
744 |
|
745 | ### Debugging
|
746 |
|
747 | - Switch to a DEBUG build variant of the WebAssembly module to see debug log messages from the C part of this library:
|
748 |
|
749 | ```typescript
|
750 | import { newQuickJSWASMModule, DEBUG_SYNC } from "quickjs-emscripten"
|
751 |
|
752 | const QuickJS = await newQuickJSWASMModule(DEBUG_SYNC)
|
753 | ```
|
754 |
|
755 | With quickjs-emscripten-core:
|
756 |
|
757 | ```typescript
|
758 | import { newQuickJSWASMModuleFromVariant } from "quickjs-emscripten-core"
|
759 | import DEBUG_SYNC from "@jitl/quickjs-wasmfile-debug-sync"
|
760 |
|
761 | const QuickJS = await newQuickJSWASMModuleFromVariant(DEBUG_SYNC)
|
762 | ```
|
763 |
|
764 | - Enable debug log messages from the Javascript part of this library with [setDebugMode][setDebugMode]:
|
765 |
|
766 | ```typescript
|
767 | import { setDebugMode } from "quickjs-emscripten"
|
768 |
|
769 | setDebugMode(true)
|
770 | ```
|
771 |
|
772 | With quickjs-emscripten-core:
|
773 |
|
774 | ```typescript
|
775 | import { setDebugMode } from "quickjs-emscripten-core"
|
776 |
|
777 | setDebugMode(true)
|
778 | ```
|
779 |
|
780 | [setDebugMode]: doc/quickjs-emscripten/exports.md#setdebugmode
|
781 |
|
782 | ### Supported Platforms
|
783 |
|
784 | `quickjs-emscripten` and related packages should work in any environment that supports ES2020.
|
785 |
|
786 | - Browsers: we estimate support for the following browser versions. See the [global-iife][iife] and [esmodule][esm-html] HTML examples.
|
787 | - Chrome 63+
|
788 | - Edge 79+
|
789 | - Safari 11.1+
|
790 | - Firefox 58+
|
791 | - NodeJS: requires v16.0.0 or later for WebAssembly compatibility. Tested with node@18. See the [node-typescript][tsx-example] and [node-minimal][minimal] examples.
|
792 | - Typescript: tested with typescript@4.5.5 and typescript@5.3.3. See the [node-typescript example][tsx-example].
|
793 | - Vite: tested with vite@5.0.10. See the [Vite/Vue example][vite].
|
794 | - Create react app: tested with react-scripts@5.0.1. See the [create-react-app example][cra].
|
795 | - Webpack: tested with webpack@5.89.0 via create-react-app.
|
796 | - Cloudflare Workers: tested with wrangler@3.22.1. See the [Cloudflare Workers example][cloudflare].
|
797 | - Deno: tested with deno 1.39.1. See the [Deno example][deno].
|
798 |
|
799 | [iife]: https://github.com/justjake/quickjs-emscripten/blob/main/examples/global-iife.html
|
800 | [esm-html]: https://github.com/justjake/quickjs-emscripten/blob/main/examples/esmodule.html
|
801 | [deno]: https://github.com/justjake/quickjs-emscripten/blob/main/examples/deno
|
802 | [vite]: https://github.com/justjake/quickjs-emscripten/blob/main/examples/vite-vue
|
803 | [cra]: https://github.com/justjake/quickjs-emscripten/blob/main/examples/create-react-app
|
804 | [cloudflare]: https://github.com/justjake/quickjs-emscripten/blob/main/examples/cloudflare-workers
|
805 | [tsx-example]: https://github.com/justjake/quickjs-emscripten/blob/main/examples/node-typescript
|
806 | [minimal]: https://github.com/justjake/quickjs-emscripten/blob/main/examples/node-minimal
|
807 |
|
808 | ### More Documentation
|
809 |
|
810 | [Github] | [NPM] | [API Documentation][api] | [Variants][core] | [Examples][tests]
|
811 |
|
812 | ## Background
|
813 |
|
814 | This was inspired by seeing https://github.com/maple3142/duktape-eval
|
815 | [on Hacker News](https://news.ycombinator.com/item?id=21946565) and Figma's
|
816 | blogposts about using building a Javascript plugin runtime:
|
817 |
|
818 | - [How Figma built the Figma plugin system](https://www.figma.com/blog/how-we-built-the-figma-plugin-system/): Describes the LowLevelJavascriptVm interface.
|
819 | - [An update on plugin security](https://www.figma.com/blog/an-update-on-plugin-security/): Figma switches to QuickJS.
|
820 |
|
821 | ## Status & Roadmap
|
822 |
|
823 | **Stability**: Because the version number of this project is below `1.0.0`,
|
824 | \*expect occasional breaking API changes.
|
825 |
|
826 | **Security**: This project makes every effort to be secure, but has not been
|
827 | audited. Please use with care in production settings.
|
828 |
|
829 | **Roadmap**: I work on this project in my free time, for fun. Here's I'm
|
830 | thinking comes next. Last updated 2022-03-18.
|
831 |
|
832 | 1. Further work on module loading APIs:
|
833 |
|
834 | - Create modules via Javascript, instead of source text.
|
835 | - Scan source text for imports, for ahead of time or concurrent loading.
|
836 | (This is possible with third-party tools, so lower priority.)
|
837 |
|
838 | 2. Higher-level tools for reading QuickJS values:
|
839 |
|
840 | - Type guard functions: `context.isArray(handle)`, `context.isPromise(handle)`, etc.
|
841 | - Iteration utilities: `context.getIterable(handle)`, `context.iterateObjectEntries(handle)`.
|
842 | This better supports user-level code to deserialize complex handle objects.
|
843 |
|
844 | 3. Higher-level tools for creating QuickJS values:
|
845 |
|
846 | - Devise a way to avoid needing to mess around with handles when setting up
|
847 | the environment.
|
848 | - Consider integrating
|
849 | [quickjs-emscripten-sync](https://github.com/reearth/quickjs-emscripten-sync)
|
850 | for automatic translation.
|
851 | - Consider class-based or interface-type-based marshalling.
|
852 |
|
853 | 4. SQLite integration.
|
854 |
|
855 | ## Related
|
856 |
|
857 | - Duktape wrapped in Wasm: https://github.com/maple3142/duktape-eval/blob/main/src/Makefile
|
858 | - QuickJS wrapped in C++: https://github.com/ftk/quickjspp
|
859 |
|
860 | ## Developing
|
861 |
|
862 | This library is implemented in two languages: C (compiled to WASM with
|
863 | Emscripten), and Typescript.
|
864 |
|
865 | You will need `node`, `yarn`, `make`, and `emscripten` to build this project.
|
866 |
|
867 | ### The C parts
|
868 |
|
869 | The ./c directory contains C code that wraps the QuickJS C library (in ./quickjs).
|
870 | Public functions (those starting with `QTS_`) in ./c/interface.c are
|
871 | automatically exported to native code (via a generated header) and to
|
872 | Typescript (via a generated FFI class). See ./generate.ts for how this works.
|
873 |
|
874 | The C code builds with `emscripten` (using `emcc`), to produce WebAssembly.
|
875 | The version of Emscripten used by the project is defined in templates/Variant.mk.
|
876 |
|
877 | - On ARM64, you should install `emscripten` on your machine. For example on macOS, `brew install emscripten`.
|
878 | - If _the correct version of emcc_ is not in your PATH, compilation falls back to using Docker.
|
879 | On ARM64, this is 10-50x slower than native compilation, but it's just fine on x64.
|
880 |
|
881 | We produce multiple build variants of the C code compiled to WebAssembly using a
|
882 | template script the ./packages directory. Each build variant uses its own copy of a Makefile
|
883 | to build the C code. The Makefile is generated from a template in ./templates/Variant.mk.
|
884 |
|
885 | Related NPM scripts:
|
886 |
|
887 | - `yarn vendor:update` updates vendor/quickjs and vendor/quickjs-ng to the latest versions on Github.
|
888 | - `yarn build:codegen` updates the ./packages from the template script `./prepareVariants.ts` and Variant.mk.
|
889 | - `yarn build:packages` builds the variant packages in parallel.
|
890 |
|
891 | ### The Typescript parts
|
892 |
|
893 | The Javascript/Typescript code is also organized into several NPM packages in ./packages:
|
894 |
|
895 | - ./packages/quickjs-ffi-types: Low-level types that define the FFI interface to the C code.
|
896 | Each variant exposes an API conforming to these types that's consumed by the higher-level library.
|
897 | - ./packages/quickjs-emscripten-core: The higher-level Typescript that implements the user-facing abstractions of the library.
|
898 | This package doesn't link directly to the WebAssembly/C code; callers must provide a build variant.
|
899 | - ./packages/quicks-emscripten: The main entrypoint of the library, which provides the `getQuickJS` function.
|
900 | This package combines quickjs-emscripten-core with platform-appropriate WebAssembly/C code.
|
901 |
|
902 | Related NPM scripts:
|
903 |
|
904 | - `yarn check` runs all available checks (build, format, tests, etc).
|
905 | - `yarn build` builds all the packages and generates the docs.
|
906 | - `yarn test` runs the tests for all packages.
|
907 | - `yarn test:fast` runs the tests using only fast build variants.
|
908 | - `yarn doc` generates the docs into `./doc`.
|
909 | - `yarn doc:serve` previews the current `./doc` in a browser.
|
910 | - `yarn prettier` formats the repo.
|
911 |
|
912 | ### Yarn updates
|
913 |
|
914 | Just run `yarn set version from sources` to upgrade the Yarn release.
|
915 |
|
\ | No newline at end of file |