Tokenizer | /tokenizer.ts | tokenizer@0.1.0

⚙️ A simple tokenizer for deno

Latest

method Tokenizer.prototype.tokenize

import { Tokenizer } from "https://deno.land/x/tokenizer@0.1.0/tokenizer.ts";

tokenize()

Tokenizes given string (default is the lexer input) to a Token array

tokenize(source: string)

tokenize(source: string, callback: (token: Token) => void)

tokenize(callback: (token: Token) => void)

Tokenizes given string (default is the lexer input) to a Token array

Parameters

source: string

source: string

callback: (token: Token) => void

callback: (token: Token) => void