kitoken
Version:
Fast and versatile tokenizer for language models, supporting BPE, Unigram and WordPiece tokenization
| Filename | Content Type | Size | |
|---|---|---|---|
| 7.31 MB | |||
| 2.66 kB | |||
| 2.57 kB |
Fast and versatile tokenizer for language models, supporting BPE, Unigram and WordPiece tokenization
| Filename | Content Type | Size | |
|---|---|---|---|
| 7.31 MB | |||
| 2.66 kB | |||
| 2.57 kB |