Skip to main content
Module

x/tokenizer/mod.ts>Rule

⚙️ A simple tokenizer for deno
interface Rule
import { type Rule } from "https://deno.land/x/tokenizer@0.1.0/mod.ts";

Represents a Rule to be scanned for in the Tokenizer

Properties

type: string
pattern: Pattern
optional
ignore: boolean