Tokenizer | /tokenizers/mod.ts | netsaur@0.2.13

Deno 2 is finally here 🎉️

Powerful machine learning, accelerated by WebGPU

class Tokenizer

import { Tokenizer } from "https://deno.land/x/netsaur@0.2.13/tokenizers/mod.ts";

Tokenizer class

Constructors

new

Tokenizer(id: number)

save(): string

Save the tokenizer as json

save(pretty: boolean): string

Save the tokenizer as json

tokenize(sentence: string)

Tokenize a sentence

fromJSON(json: string)

Load a tokenizer from json data