Skip to main content
Module

x/tokenizer/tokenizer.ts>Tokenizer#[Symbol.iterator]

⚙️ A simple tokenizer for deno
eliassjogreen/deno_tokenizer
Latest
method Tokenizer.prototype.[Symbol.iterator]
import { Tokenizer } from "https://deno.land/x/tokenizer@0.1.0/tokenizer.ts";

Returns

IterableIterator<Token>