Token | /token.ts | tokenizer@0.1.0

⚙️ A simple tokenizer for deno

Latest

interface Token

import { type Token } from "https://deno.land/x/tokenizer@0.1.0/token.ts";

Represents the Token that will be return on a match when scanning in the Tokenizer

Properties

type: string

match: string

groups: string[]

position: { start: number; end: number; }