1 | # pbf
|
2 |
|
3 | [![build status](https://secure.travis-ci.org/mapbox/pbf.png)](http://travis-ci.org/mapbox/pbf) [![Coverage Status](https://coveralls.io/repos/mapbox/pbf/badge.png)](https://coveralls.io/r/mapbox/pbf)
|
4 |
|
5 | A low-level, fast, ultra-lightweight (3KB gzipped) JavaScript library for decoding and encoding [protocol buffers](https://developers.google.com/protocol-buffers) (a compact binary format for structured data serialization).
|
6 |
|
7 | Designed to be a building block for writing customized decoders and encoders.
|
8 | If you need an all-purpose protobuf JS library that does most of the work for you,
|
9 | take a look at [protocol-buffers](https://github.com/mafintosh/protocol-buffers) too.
|
10 |
|
11 | ## Examples
|
12 |
|
13 | #### Using Compiled Code
|
14 |
|
15 | Install `pbf` and compile a JavaScript module from a `.proto` file:
|
16 |
|
17 | ```bash
|
18 | $ npm install -g pbf
|
19 | $ pbf test.proto > test.js
|
20 | ```
|
21 |
|
22 | Then read and write objects using the module like this:
|
23 |
|
24 | ```js
|
25 | var obj = Test.read(new Pbf(buffer)); // read
|
26 | var buffer = Test.write(obj, new Pbf()); // write
|
27 | ```
|
28 |
|
29 | #### Custom Reading
|
30 |
|
31 | ```js
|
32 | var data = new Pbf(buffer).readFields(readData, {});
|
33 |
|
34 | function readData(tag, data, pbf) {
|
35 | if (tag === 1) data.name = pbf.readString();
|
36 | else if (tag === 2) data.version = pbf.readVarint();
|
37 | else if (tag === 3) data.layer = pbf.readMessage(readLayer, {});
|
38 | }
|
39 | function readLayer(tag, layer, pbf) {
|
40 | if (tag === 1) layer.name = pbf.readString();
|
41 | else if (tag === 3) layer.size = pbf.readVarint();
|
42 | }
|
43 | ```
|
44 |
|
45 | #### Custom Writing
|
46 |
|
47 | ```js
|
48 | var pbf = new Pbf();
|
49 | writeData(data, pbf);
|
50 | var buffer = pbf.finish();
|
51 |
|
52 | function writeData(data, pbf) {
|
53 | pbf.writeStringField(1, data.name);
|
54 | pbf.writeVarintField(2, data.version);
|
55 | pbf.writeMessage(3, writeLayer, data.layer);
|
56 | }
|
57 | function writeLayer(layer, pbf) {
|
58 | pbf.writeStringField(1, layer.name);
|
59 | pbf.writeVarintField(2, layer.size);
|
60 | }
|
61 | ```
|
62 |
|
63 |
|
64 | ## Install
|
65 |
|
66 | Node and Browserify:
|
67 |
|
68 | ```bash
|
69 | npm install pbf
|
70 | ```
|
71 |
|
72 | Making a browser build:
|
73 |
|
74 | ```bash
|
75 | npm install
|
76 | npm run build-dev # dist/pbf-dev.js (development build)
|
77 | npm run build-min # dist/pbf.js (minified production build)
|
78 | ```
|
79 |
|
80 | ## API
|
81 |
|
82 | Create a `Pbf` object, optionally given a `Buffer` or `Uint8Array` as input data:
|
83 |
|
84 | ```js
|
85 | // parse a pbf file from disk in Node
|
86 | var pbf = new Pbf(fs.readFileSync('data.pbf'));
|
87 |
|
88 | // parse a pbf file in a browser after an ajax request with responseType="arraybuffer"
|
89 | var pbf = new Pbf(new Uint8Array(xhr.response));
|
90 | ```
|
91 |
|
92 | `Pbf` object properties:
|
93 |
|
94 | ```js
|
95 | pbf.length; // length of the underlying buffer
|
96 | pbf.pos; // current offset for reading or writing
|
97 | ```
|
98 |
|
99 | #### Reading
|
100 |
|
101 | Read a sequence of fields:
|
102 |
|
103 | ```js
|
104 | pbf.readFields(function (tag) {
|
105 | if (tag === 1) pbf.readVarint();
|
106 | else if (tag === 2) pbf.readString();
|
107 | else ...
|
108 | });
|
109 | ```
|
110 |
|
111 | It optionally accepts an object that will be passed to the reading function for easier construction of decoded data,
|
112 | and also passes the `Pbf` object as a third argument:
|
113 |
|
114 | ```js
|
115 | var result = pbf.readFields(callback, {})
|
116 |
|
117 | function callback(tag, result, pbf) {
|
118 | if (tag === 1) result.id = pbf.readVarint();
|
119 | }
|
120 | ```
|
121 |
|
122 | To read an embedded message, use `pbf.readMessage(fn[, obj])` (in the same way as `read`).
|
123 |
|
124 | Read values:
|
125 |
|
126 | ```js
|
127 | var value = pbf.readVarint();
|
128 | var str = pbf.readString();
|
129 | var numbers = pbf.readPackedVarint();
|
130 | ```
|
131 |
|
132 | For lazy or partial decoding, simply save the position instead of reading a value,
|
133 | then later set it back to the saved value and read:
|
134 |
|
135 | ```js
|
136 | var fooPos = -1;
|
137 | pbf.readFields(function (tag) {
|
138 | if (tag === 1) fooPos = pbf.pos;
|
139 | });
|
140 | ...
|
141 | pbf.pos = fooPos;
|
142 | pbf.readMessage(readFoo);
|
143 | ```
|
144 |
|
145 | Scalar reading methods:
|
146 |
|
147 | * `readVarint()`
|
148 | * `readSVarint()`
|
149 | * `readFixed32()`
|
150 | * `readFixed64()`
|
151 | * `readSFixed32()`
|
152 | * `readSFixed64()`
|
153 | * `readBoolean()`
|
154 | * `readFloat()`
|
155 | * `readDouble()`
|
156 | * `readString()`
|
157 | * `readBytes()`
|
158 | * `skip(value)`
|
159 |
|
160 | Packed reading methods:
|
161 |
|
162 | * `readPackedVarint()`
|
163 | * `readPackedSVarint()`
|
164 | * `readPackedFixed32()`
|
165 | * `readPackedFixed64()`
|
166 | * `readPackedSFixed32()`
|
167 | * `readPackedSFixed64()`
|
168 | * `readPackedBoolean()`
|
169 | * `readPackedFloat()`
|
170 | * `readPackedDouble()`
|
171 |
|
172 | #### Writing
|
173 |
|
174 | Write values:
|
175 |
|
176 | ```js
|
177 | pbf.writeVarint(123);
|
178 | pbf.writeString("Hello world");
|
179 | ```
|
180 |
|
181 | Write an embedded message:
|
182 |
|
183 | ```js
|
184 | pbf.writeMessage(1, writeObj, obj);
|
185 |
|
186 | function writeObj(obj, pbf) {
|
187 | pbf.writeStringField(obj.name);
|
188 | pbf.writeVarintField(obj.version);
|
189 | }
|
190 | ```
|
191 |
|
192 | Field writing methods:
|
193 |
|
194 | * `writeVarintField(tag, val)`
|
195 | * `writeSVarintField(tag, val)`
|
196 | * `writeFixed32Field(tag, val)`
|
197 | * `writeFixed64Field(tag, val)`
|
198 | * `writeSFixed32Field(tag, val)`
|
199 | * `writeSFixed64Field(tag, val)`
|
200 | * `writeBooleanField(tag, val)`
|
201 | * `writeFloatField(tag, val)`
|
202 | * `writeDoubleField(tag, val)`
|
203 | * `writeStringField(tag, val)`
|
204 | * `writeBytesField(tag, buffer)`
|
205 | * `writePacked(tag, type, items)`
|
206 | * `writeMessage(tag, pbf)`
|
207 |
|
208 | Packed field writing methods:
|
209 |
|
210 | * `writePackedVarint(tag, val)`
|
211 | * `writePackedSVarint(tag, val)`
|
212 | * `writePackedSFixed32(tag, val)`
|
213 | * `writePackedSFixed64(tag, val)`
|
214 | * `writePackedBoolean(tag, val)`
|
215 | * `writePackedFloat(tag, val)`
|
216 | * `writePackedDouble(tag, val)`
|
217 |
|
218 | Scalar writing methods:
|
219 |
|
220 | * `writeVarint(val)`
|
221 | * `writeSVarint(val)`
|
222 | * `writeSFixed32(val)`
|
223 | * `writeSFixed64(val)`
|
224 | * `writeBoolean(val)`
|
225 | * `writeFloat(val)`
|
226 | * `writeDouble(val)`
|
227 | * `writeString(val)`
|
228 | * `writeBytes(buffer)`
|
229 |
|
230 | Misc methods:
|
231 |
|
232 | * `realloc(minBytes)` - pad the underlying buffer size to accommodate the given number of bytes;
|
233 | note that the size increases exponentially, so it won't necessarily equal the size of data written
|
234 | * `finish()` - make the current buffer ready for reading and return the data as a buffer slice
|
235 | * `destroy()` - dispose the buffer
|
236 |
|
237 | For an example of a real-world usage of the library, see [vector-tile-js](https://github.com/mapbox/vector-tile-js).
|
238 |
|
239 |
|
240 | ## Proto Schema to JavaScript
|
241 |
|
242 | If installed globally, `pbf` provides a binary that compiles `proto` files into JavaScript modules. Usage:
|
243 |
|
244 | ```bash
|
245 | $ pbf <proto_path> [--no-write] [--no-read] [--browser]
|
246 | ```
|
247 |
|
248 | The `--no-write` and `--no-read` switches remove corresponding code in the output.
|
249 | The `--browser` switch makes the module work in browsers instead of Node.
|
250 |
|
251 | The resulting module exports each message by name with the following methods:
|
252 |
|
253 | * `read(pbf)` - decodes an object from the given `Pbf` instance
|
254 | * `write(obj, pbf)` - encodes an object into the given `Pbf` instance (usually empty)
|
255 |
|
256 | The resulting code is pretty short and easy to understand, so you can customize it easily.
|
257 |
|
258 |
|
259 | ## Changelog
|
260 |
|
261 | #### 1.3.0 (Feb 5, 2015)
|
262 |
|
263 | - Added `pbf` binary that compiles `.proto` files into `Pbf`-based JavaScript modules.
|
264 |
|
265 | #### 1.2.0 (Jan 5, 2015)
|
266 |
|
267 | ##### Breaking API changes
|
268 |
|
269 | - Changed `writeMessage` signature to `(tag, fn, obj)` (see example in the docs)
|
270 | for a huge encoding performance improvement.
|
271 | - Replaced `readPacked` and `writePacked` methods that accept type as a string
|
272 | with `readPackedVarint`, etc. for each type (better performance and simpler API).
|
273 |
|
274 | ##### Improvements
|
275 |
|
276 | - 5x faster encoding in Node (vector tile benchmark).
|
277 | - 40x faster encoding and 3x faster decoding in the browser (vector tile benchmark).
|
278 |
|
279 | #### 1.1.4 (Jan 2, 2015)
|
280 |
|
281 | - Significantly improved `readPacked` and `writePacked` performance (the tile reading benchmark is now 70% faster).
|
282 |
|
283 | #### 1.1.3 (Dec 26, 2014)
|
284 |
|
285 | Brings tons of improvements and fixes over the previous version (`0.0.2`).
|
286 | Basically makes the library complete.
|
287 |
|
288 | ##### Improvements
|
289 |
|
290 | - Improved performance of both reading and writing.
|
291 | - Made the browser build 3 times smaller.
|
292 | - Added convenience `readFields` and `readMessage` methods for a much easier reading API.
|
293 | - Added reading methods: `readFloat`, `readBoolean`, `readSFixed32`, `readSFixed64`.
|
294 | - Added writing methods: `writeUInt64`, `writeSFixed32`, `writeSFixed64`.
|
295 | - Improved `readDouble` and `readString` to use native Buffer methods under Node.
|
296 | - Improved `readString` and `writeString` to use HTML5 `TextEncoder` and `TextDecoder` where available.
|
297 | - Made `Pbf` `buffer` argument optional.
|
298 | - Added extensive docs and examples in the readme.
|
299 | - Added an extensive test suite that brings test coverage up to 100%.
|
300 |
|
301 | ##### Breaking API changes
|
302 |
|
303 | - Renamed `readBuffer`/`writeBuffer` to `readBytes`/`writeBytes`.
|
304 | - Renamed `readUInt32`/`writeUInt32` to `readFixed32`/`writeFixed32`, etc.
|
305 | - Renamed `writeTaggedVarint` to `writeVarintField`, etc.
|
306 | - Changed `writePacked` signature from `(type, tag, items)` to `(tag, type, items)`.
|
307 |
|
308 | ##### Bugfixes
|
309 |
|
310 | - Fixed `readVarint` to handle varints bigger than 6 bytes.
|
311 | - Fixed `readSVarint` to handle number bigger than `2^30`.
|
312 | - Fixed `writeVarint` failing on some integers.
|
313 | - Fixed `writeVarint` not throwing an error on numbers that are too big.
|
314 | - Fixed `readUInt64` always failing.
|
315 | - Fixed writing to an empty buffer always failing.
|