1 | Overview [![Build Status](https://travis-ci.org/lydell/js-tokens.png?branch=master)](https://travis-ci.org/lydell/js-tokens)
|
2 | ========
|
3 |
|
4 | A regex that tokenizes JavaScript.
|
5 |
|
6 | ```js
|
7 | var jsTokens = require("js-tokens")
|
8 |
|
9 | var jsString = "var foo=opts.foo;\n..."
|
10 |
|
11 | jsString.match(jsTokens)
|
12 | // ["var", " ", "foo", "=", "opts", ".", "foo", ";", "\n", ...]
|
13 | ```
|
14 |
|
15 |
|
16 | Installation
|
17 | ============
|
18 |
|
19 | - `npm install js-tokens`
|
20 |
|
21 | ```js
|
22 | var jsTokens = require("js-tokens")
|
23 | ```
|
24 |
|
25 |
|
26 | Usage
|
27 | =====
|
28 |
|
29 | ### `jsTokens` ###
|
30 |
|
31 | A regex with the `g` flag that matches JavaScript tokens.
|
32 |
|
33 | The regex _always_ matches, even invalid JavaScript and the empty string.
|
34 |
|
35 | The next match is always directly after the previous.
|
36 |
|
37 | ### `var token = jsTokens.matchToToken(match)` ###
|
38 |
|
39 | Takes a `match` returned by `jsTokens.exec(string)`, and returns a `{type:
|
40 | String, value: String}` object. The following types are available:
|
41 |
|
42 | - string
|
43 | - comment
|
44 | - regex
|
45 | - number
|
46 | - name
|
47 | - operator
|
48 | - punctuation
|
49 | - whitespace
|
50 | - invalid
|
51 |
|
52 | Multi-line comments and strings also have a `closed` property indicating if the
|
53 | token was closed or not (see below).
|
54 |
|
55 | Comments and strings both come in several flavors. To distinguish them, check if
|
56 | the token starts with `//`, `/*`, `'`, `"` or `` ` ``.
|
57 |
|
58 | For example usage, please see this [gist].
|
59 |
|
60 | [gist]: https://gist.github.com/lydell/be49dbf80c382c473004
|
61 |
|
62 |
|
63 | Invalid code handling
|
64 | =====================
|
65 |
|
66 | Unterminated strings are still matched as strings. JavaScript strings cannot
|
67 | contain (unescaped) newlines, so unterminated strings simply end at the end of
|
68 | the line. Unterminated template strings can contain unescaped newlines, though,
|
69 | so they go on to the end of input.
|
70 |
|
71 | Unterminated multi-line comments are also still matched as comments. They
|
72 | simply go on to the end of the input.
|
73 |
|
74 | Unterminated regex literals are likely matched as division and whatever is
|
75 | inside the regex.
|
76 |
|
77 | Invalid ASCII characters have their own capturing group.
|
78 |
|
79 | Invalid non-ASCII characters are treated as names, to simplify the matching of
|
80 | names (except unicode spaces which are treated as whitespace).
|
81 |
|
82 | Regex literals may contain invalid regex syntax. They are still matched as
|
83 | regex literals. They may also contain repeated regex flags, to keep the regex
|
84 | simple.
|
85 |
|
86 | Strings may contain invalid escape sequences.
|
87 |
|
88 |
|
89 | Limitations
|
90 | ===========
|
91 |
|
92 | Tokenizing JavaScript using regexes—in fact, _one single regex_—won’t be
|
93 | perfect. But that’s not the point either.
|
94 |
|
95 | ### Template string interpolation ###
|
96 |
|
97 | Template strings are matched as single tokens, from the starting `` ` `` to then
|
98 | ending `` ` ``, including interpolations (whose tokens are not matched
|
99 | individually).
|
100 |
|
101 | Matching template string interpolations requires recursive balancing of `{` and
|
102 | `}`—something that JavaScript regexes cannot do. Only one level of nesting is
|
103 | supported.
|
104 |
|
105 | ### Division and regex literals collision ###
|
106 |
|
107 | Consider this example:
|
108 |
|
109 | ```js
|
110 | var g = 9.82
|
111 | var number = bar / 2/g
|
112 |
|
113 | var regex = / 2/g
|
114 | ```
|
115 |
|
116 | A human can easily understand that in the `number` line we’re dealing with
|
117 | division, and in the `regex` line we’re dealing with a regex literal. How come?
|
118 | Because humans can look at the whole code to put the `/` characters in context.
|
119 | A JavaScript regex cannot. It only sees forwards.
|
120 |
|
121 | When the `jsTokens` regex scans throught the above, it will see the following
|
122 | at the end of both the `number` and `regex` rows:
|
123 |
|
124 | ```js
|
125 | / 2/g
|
126 | ```
|
127 |
|
128 | It is then impossible to know if that is a regex literal, or part of an
|
129 | expression dealing with division.
|
130 |
|
131 | Here is a similar case:
|
132 |
|
133 | ```js
|
134 | foo /= 2/g
|
135 | foo(/= 2/g)
|
136 | ```
|
137 |
|
138 | The first line divides the `foo` variable with `2/g`. The second line calls the
|
139 | `foo` function with the regex literal `/= 2/g`. Again, since `jsTokens` only
|
140 | sees forwards, it cannot tell the two cases apart.
|
141 |
|
142 | There are some cases where we _can_ tell division and regex literals apart,
|
143 | though.
|
144 |
|
145 | First off, we have the simple cases where there’s only one slash in the line:
|
146 |
|
147 | ```js
|
148 | var foo = 2/g
|
149 | foo /= 2
|
150 | ```
|
151 |
|
152 | Regex literals cannot contain newlines, so the above cases are correctly
|
153 | identified as division. Things are only problematic when there are more than
|
154 | one non-comment slash in a single line.
|
155 |
|
156 | Secondly, not every character is a valid regex flag.
|
157 |
|
158 | ```js
|
159 | var number = bar / 2/e
|
160 | ```
|
161 |
|
162 | The above example is also correctly identified as division, because `e` is not
|
163 | a valid regex flag. I initially wanted to future-proof by allowing `[a-zA-Z]*`
|
164 | (any letter) as flags, but it is not worth it since it increases the amount of
|
165 | ambigous cases. So only the standard `g`, `m` and `i` flags, as well as the `y`
|
166 | flag supported by Firefox 3.6+, are allowed. This means that the above example
|
167 | will be identified as division as long as you don’t rename the `e` variable to
|
168 | some permutation of `gmiy` 1 to 4 characters long.
|
169 |
|
170 | Lastly, we can look _forward_ for information.
|
171 |
|
172 | - If the token following what looks like a regex literal is not valid after a
|
173 | regex literal, but is valid in a division expression, then the regex literal
|
174 | is treated as division instead. For example, a flagless regex cannot be
|
175 | followed by a string, number or name, but all of those three can be the
|
176 | denominator of a division.
|
177 | - Generally, if what looks like a regex literal is followed by an operator, the
|
178 | regex literal is treated as division instead. This is because regexes are
|
179 | seldomly used with operators (such as `+`, `*`, `&&` and `==`), but division
|
180 | could likely be part of such an expression.
|
181 |
|
182 | Please consult the regex source and the test cases for precise information on
|
183 | when regex or division is matched (should you need to know). In short, you
|
184 | could sum it up as:
|
185 |
|
186 | If the end of a statement looks like a regex literal (even if it isn’t), it
|
187 | will be treated as one. Otherwise it should work as expected (if you write sane
|
188 | code).
|
189 |
|
190 |
|
191 | License
|
192 | =======
|
193 |
|
194 | [The X11 (“MIT”) License](LICENSE).
|