qunpack
=======
[![Build Status](https://api.travis-ci.org/andrasq/node-qunpack.svg?branch=master)](https://travis-ci.org/andrasq/node-qunpack?branch=master)
[![Coverage Status](https://codecov.io/github/andrasq/node-qunpack/coverage.svg?branch=master)](https://codecov.io/github/andrasq/node-qunpack?branch=master)

`Unpack` decodes a binary string into numbers and strings according to the specified format.
This implementation is a javascript-only mostly compatible work-alike of PERL and PHP
[`unpack()`](http://php.net/manual/en/function.unpack.php).

`qunpack` favors big-endian storage; I needed both signed and unsigned big-endian support,
so I store generic "native" ("machine byte order") values in
network byte order (big-endian).  Later I added full little-endian support as well,
but the default is still network byte order.

The syntax is more like PERL than PHP; the format string is
a concatenated series of conversion specifiers without names, like PERL "SL" meaning
`[ short, long ]` and not PHP "S`sname`/L`lname`" for `{ sname: short, lname: long }`.

Full signed little-endian support is now available with eg `s<` extensions.  It is also
possible to extract nested lists and nested hashes with with `qunpack`-specific extensions.

Example:

    var bytes = new Buffer([129,2,3,4,5,6,7,8,9,10,11,12]);

    // interpret bytes as a signed short, two bytes and two unsigned shorts:
    var valuesArray = qunpack.unpack('sCCSS', bytes);
    // => [ -32510, 3, 4, 0x0506, 0x0708 ]

    // interpret as 6 bytes then a little-endian unsigned long:
    var valuesArray = qunpack.unpack('C6V', bytes);
    // => [ 129, 2, 3, 4, 5, 6, 0x0a090807 ]


Api
---

### qunpack.unpack( format, bytes, [offset] )

Unpack the binary string `bytes` in the Buffer according to the `format` string.  See below for the format
specification.  Returns an array of values.  If an `offset` is given, unpack starting
that many bytes from the start of the binary string.

### qunpack.pack( format, data )

TBD.  Maybe later.


Formats
-------

The format string is a concatenated list of conversion specifiers, like for PERL
`unpack`.  Each conversion specifier is a format letter followed by an optional
count (default `1`).

Note that the `pack/unpack` conversions are incomplete, it is not possible to specify
all four combinations of signed/unsigned + little-e/big-e: endianness can be controlled
only for unsigneds and only for integers.  I needed signed big-endian support, so
this implementation interprets "machine byte order" as being big-endian.  I later
added full little-endian support, some via the `s<`, `l<` and `q<` conversion extensions.

The conversion count is interpreted as:

    [aAHZ]<length> - a `length` byte string (default 1 byte)
    <CONV><count> - `count` instances of the type specified by CONV
    x<count> - skip ahead `count` bytes
    X<count> - back up `count` bytes
    @<offset> - seek to absolute position `offset`

    Z+<count> - `count` NUL-terminated variable-length strings

The available conversion specifiers are:

    Strings, decoded as javascript utf8:

    a - NUL-padded string, retains NUL padding
    A - SPACE-padded string, trailing whitespace stripped
    Z - NUL-padded string, trailing NULs stripped
    H - hex string, high nybble first ABC => '414243'
    h - hex string, low nybble first ABC => '142434'

    Z+ - NUL-terminated variable-length string, NUL stripped (extension)

    Portable numeric conversions:

    n - unsigned 16-bit big-e short
    N - unsigned 32-bit big-e long
    J - unsigned 64-bit big-e long long, with 53 bits of precision

    v - unsigned 16-bit little-e short
    V - unsigned 32-bit little-e long
    P - unsigned 64-bit little-e long long

    Native numbers, stored in big-endian network byte order:

    c,C - signed, unsigned 8-bit char
    s,S - signed, unsigned 16-bit "native" (big-e) short (word)
    i,I - signed, unsigned "native" int (32-bit big-e)
    l,L - signed, unsigned 32-bit "native" (big-e) long (longword)
    q,Q - signed, unsigned 64-bit "native" (big-e) long long (quadword),
          extracted into a native JavaScript number.  Note that JavaScript
          numbers are 64-bit doubles and support only 53 bits of precision.
          Larger values may lose least significant bits.

    f,d - 32-bit and 64-bit "native" big-e float and double
    G,E - 32-bit and 64-bit big-e float and double
    g,e - 32-bit and 64-bit little-e float and double

    g - 32-bit little-float
    e - 64-bit littl-e double

    s<,l<,q< - signed 16-bit, 32-bit and 64-bit little-e integers (extensions)

    Grouping extensions:

    [# ... ] - subformat inside the [ ] will be gathered into a sub-array.
               The subformat may have a non-zero repeat count, in which case
               that many sub-arrays will be extracted.  A negative count is
               ignored.  It is an error for the count to be zero.
               Eg, 'C[S2]' on [1,2,3,4,5,6] => [ 0x01, [0x0203, 0x0405] ]
               EXPERIMENTAL.

    {# ... } - extracts # objects with properties determined by the named conversion
               specifiers contained inside the `{ ... }` (default 1 object).
               A negative count is ignored.  The count must not be zero.
               Eg, '{2 a:C, x:X1, b:S}' on [1,2,3,4] => [ {a:1, b:0x0102}, {a:3, b:0304} ]
               EXPERIMENTAL.

    Position control:

    x - skip a byte (NUL-fill if packing)
    X - back up a byte
    @ - seek to absolute offset (NUL-fill if packing)

Not supported conversion specifiers (for completeness):

    * - "all remaining" repetition specifier
    ( ... ) - grouping specifier

Examples:

    "a5" - extract a 5-byte NUL-padded string, eg "ABC\0\0" -> "ABC\0\0"
    "A5" - extract a 5-byte SPACE-padded string, eg "ABC  " => "ABC"
    "S2" - extract two unsigned shorts, eg "\1\2\3\4" => [ 0x0102, 0x0304 ]
    "Z+3L" - extract three variable-length NUL-terminated strings, then a 32-bit long.
             Eg, '1\0two\0three\0\1\2\3\4' => [ '1', 'two', 'three', 0x01020304 ].

Differences
-----------

- `qunpack` does not try to emulate PHP or full PERL `unpack`.  It does borrow some of
  their syntax, but favors big-endian values and provides some non-standard extensios.

- `qunpack` supports non-standard extensions like `s<` and `{# ... }` hashes

- the `unpack` format is interpreted like PERL, where `S1L1` means one short followed by
  one long, not a short stored into property named 'L1'.  The PHP syntax would be e.g.
  `S1shortName/L1longName`, returning `{ shortName: <16 bits>, longName: <32 bits> }`.

- the `Z+#` conversion is a `qunpack` extension.  It extracts `count` (default 1)
  variable-length NUL-terminated strings from the input.

- the `[# ... ]` grouping conversion is a `qunpack` extension.  It extracts `count`
  (default 1) sub-arrays with the format contained between the brackets.
  EXPERIMENTAL.

- the `{# ... }` grouping conversion is a `qunpack` extension.  It extracts `count`
  (default 1) objects with properties according to the named formats contained between
  the braces.  EXPERIMENTAL.

- the `s<` etc little-endian conversions are a `qunpack` extension.  The two-character
  format specifiers `s<`, `l<` and `q<` read signed 16-bit, 32-bit and 64-bit integers
  from

- 64-bit integer support is limited to 53 bits, since that's the precision available
  in javascript numbers.  There is no built-in support for assembling quadword longs
  out of parts.


Change Log
----------

- 0.5.2 - make {#...} hashes more tolerant of odd punctuation and spacing,
  error out on unrecognized conversion characters
- 0.5.1 - fix node-v0.10 H,h hex conversion (use integer base,bound)
- 0.5.0 - full little-endian support, fix 'h' and 'H' string length
- 0.4.0 - initial version of `{# ... }` grouping, fix H conversion, add h conversion
- 0.3.2 - support for `[# ... ]` sub-group count
- 0.3.1 - speed up by passing a around state object, fewer arguments
- 0.3.0 - `[ ... ]` sub-group parsing, also much faster due to faster state passing
- 0.2.0 - `Z+#` countable variable-length asciiz string conversion specifier
- 0.1.1 - fix c,C count, ignore negative unpack offset
- 0.1.0 - initial implementation, with unit tests


Todo
----

- implement `pack`
- make bounds errors fatal, to not slip by undetected
- omit the wrapping array if unpacking just 1 value and no count specified
  (ie, 'L2' => [0,0], 'L1' => [0], but 'L' => 0.  However, 'A6' => 'string'
  because '6' is the size, not a count.)
- speed up `{# ... }` extraction (avoid unused array)
- fix nested hash extraction
- fix multiple-subarray extraction
- maybe an option for the default storage-endian operating mode
- maybe `s>`, `l>`, `q>` explicit-endian short, long and quad support


Related Work
------------

- [`hipack`](https://npmjs.com/package/hipack) - php compatible pack/unpack, uses C++ extension, needs node sources to install
- [`php-pack`](https://npmjs.com/package/php-pack) - fork of `hipack`
- [`qmsgpack`](https://github.com/andrasq/node-q-msgpack) - experimental nodejs `msgpack` with related works bibliography