Skip to main content
Products
Open Source
Deno
Modern runtime for JavaScript and TypeScript
Fresh
Web framework designed for the edge
JSR
TypeScript-first ESM package registry
Cloud
Deploy
Easy serverless hosting for your JavaScript projects
Subhosting
Securely run untrusted code in a scalable sandbox
Pricing
Starts free and scales with your needs
Docs
Manual
API Reference
Deno Deploy
Deno KV
Deno by Example
Modules
Current path
deno.land/x
Standard Library
Node.js & npm
JSR
Blog
Search…
⌘K
Module
x
/
tokenizer
⚙️ A simple tokenizer for deno
eliassjogreen/deno_tokenizer
Version
Latest
0.1.0
Directory
.github
235 B
examples
5 KB
.editorconfig
148 B
.eslintignore
13 B
.eslintrc
458 B
.gitignore
240 B
.prettierrc
162 B
LICENSE
1 KB
mod.ts
114 B
package.json
527 B
pattern.ts
234 B
README.md
2 KB
rule.ts
189 B
token.ts
222 B
tokenizer_test.ts
10 KB
tokenizer.ts
5 KB
tsconfig.json
465 B