1 | # Pickling for Graphite
|
2 |
|
3 | The graphite statsd backend can optionally be configured to use pickle
|
4 | for its over-the-wire protocol.
|
5 |
|
6 | ```javascript
|
7 | { graphiteHost: "your.graphite.host",
|
8 | graphiteProtocol: "pickle" }
|
9 | ```
|
10 |
|
11 | The default is to use the graphite text protocol, which can require
|
12 | more CPU processing by the graphite endpoint.
|
13 |
|
14 | The message format expected by the graphite pickle endpoint consists
|
15 | of a header and payload.
|
16 |
|
17 | ## The Payload
|
18 |
|
19 | The message payload is a list of tuples. Each tuple contains the measurement
|
20 | for a single metric name. The measurement is encoded as a second,
|
21 | nested tuple containing timestamp and measured value.
|
22 |
|
23 | This ends up looking like:
|
24 |
|
25 | ```python
|
26 | [ ( "path.to.metric.name", ( timestamp, "value" ) ),
|
27 | ( "path.to.another.name", ( timestamp, "value" ) ) ]
|
28 | ```
|
29 |
|
30 | The graphite receiver `carbon.protocols.MetricPickleReceiver` coerces
|
31 | both the timestamp and measured value into `float`.
|
32 |
|
33 | The timestamp must be seconds since epoch encoded as a number.
|
34 |
|
35 | The measured value is encoded as a string. This may change in the
|
36 | future.
|
37 |
|
38 | We have chosen to not implement pickle's object memoization. This
|
39 | simplifies what is sent across the wire. It is not likely any
|
40 | optimization would result within a single poll cycle.
|
41 |
|
42 | Here is some Python code showing how a given set of metrics can be
|
43 | serialized in a more simple way.
|
44 |
|
45 | ```python
|
46 | import pickle
|
47 |
|
48 | metrics = [ ( "a.b.c", ( 1234L, "5678" ) ), ( "d.e.f.g", ( 1234L, "9012" ) ) ]
|
49 | pickle.dumps(metrics)
|
50 | # "(lp0\n(S'a.b.c'\np1\n(L1234L\nS'5678'\np2\ntp3\ntp4\na(S'd.e.f.g'\np5\n(L1234L\nS'9012'\np6\ntp7\ntp8\na."
|
51 |
|
52 | payload = "(l(S'a.b.c'\n(L1234L\nS'5678'\ntta(S'd.e.f.g'\n(L1234L\nS'9012'\ntta."
|
53 | pickle.loads(payload)
|
54 | # [('a.b.c', (1234L, '5678')), ('d.e.f.g', (1234L, '9012'))]
|
55 | ```
|
56 |
|
57 | The trailing `L` for long fields is unnecessary, but we are adding the
|
58 | character to match Python pickle output. It's a side-effect of
|
59 | `repr(long(1234))`.
|
60 |
|
61 | ## The Header
|
62 |
|
63 | The message header is a 32-bit integer sent over the wire as
|
64 | four-bytes. This integer must describe the length of the pickled
|
65 | payload.
|
66 |
|
67 | Here is some sample code showing how to construct the message header
|
68 | containing the payload length.
|
69 |
|
70 | ```python
|
71 | import struct
|
72 |
|
73 | payload_length = 81
|
74 | header = struct.pack("!L", payload_length)
|
75 | # '\x00\x00\x00Q'
|
76 | ```
|
77 |
|
78 | The `Q` character is equivalent to `\x81` (ASCII encoding).
|