1 | Tokenization
|
2 | ============
|
3 |
|
4 | The Firescript parser converts the input in an array of tokens first and uses it for parsing.
|
5 | This step is called [lexical analysis](https://en.wikipedia.org/wiki/Lexical_analysi)
|
6 |
|
7 | ### Interface
|
8 |
|
9 | The Tokenizer interface is as follows:
|
10 |
|
11 | ```js
|
12 | const tokenizer = new FirescriptTokenizer(options)
|
13 | const tokenStack = tokenizer.tokenize(input)
|
14 |
|
15 | // or a shorthand
|
16 |
|
17 | const tokenStack = Firescript.tokenize(input, options)
|
18 | ```
|
19 |
|
20 | ### Tokens
|
21 |
|
22 | The Firescript Tokenizer knows these tokens
|
23 |
|
24 | Token name | Description
|
25 | -------------|------------
|
26 | keyword | Keywords like **if**, **import**, **await** etc..
|
27 | identifier | variables, function and class names
|
28 | numeric | All numeric values like: **123** , **-123** , **1.5**
|
29 | literal | Strings and other values
|
30 | punctuator | Punctator or separator chars like: **[** , **{** , **(** , **.** or **,**
|
31 | operator | Chars like: **+** , **+=** , **>>**
|
32 | indention | Indention after a linebreak
|
33 | comment | Line comments
|
34 | blockComment | Block comments
|
35 | shebang | Hasbang string like **#!/usr/bin/node**
|
36 |
|
37 | ### Example
|
38 |
|
39 | A simple hello world example demonstrates how the tokenizer works.
|
40 |
|
41 | ```fire
|
42 | const greeding = 'Hello World'
|
43 | console.log(greeding)
|
44 | ```
|
45 |
|
46 | ```js
|
47 | const fs = require('fs')
|
48 | const helloWorld = fs.readFileSync('./helloWorld.fire')
|
49 |
|
50 | Firescript.tokenize(helloWorld)
|
51 | ```
|
52 |
|
53 | The output of `tokenize()` would be
|
54 |
|
55 | ```json
|
56 | [
|
57 | { "type": "keyword", "value": "const" },
|
58 | { "type": "identifier", "value": "greeding" },
|
59 | { "type": "operator", "value": "=" },
|
60 | { "type": "literal", "value": "'Hello World'" },
|
61 | { "type": "indention", "value": 0 },
|
62 | { "type": "identifier", "value": "console" },
|
63 | { "type": "punctuator", "value": "." },
|
64 | { "type": "identifier", "value": "log" },
|
65 | { "type": "punctuator", "value": "(" },
|
66 | { "type": "identifier", "value": "greeding" },
|
67 | { "type": "punctuator", "value": ")" },
|
68 | { "type": "indention", "value": 0 }
|
69 | ]
|
70 | ```
|