Take a Deno.Reader and perform Lexical Analysis/Tokenization on it, returning a stream of tokens.
Repository
Current version released
4 years ago
Versions
Scanner
Take a Deno.Reader and perform Lexical Analysis/Tokenization on it, returning a stream of tokens.
This library is heavily inspired by Golang’s text/scanner
Usage
import {Scanner, Token, nextTokenIs, TokenError} from 'https://deno.land/x/scanner/mod.ts'
const file = Deno.open('./my-file.language')
const scanner = new Scanner(file)
try {
for await (const token of scanner) {
if (token === Token.int) console.log('Found integer', scanner.contents)
if (token === Token.ident) console.log('Found identifier', scanner.contents)
// ... and so on
}
} finally {
file.close()
}