Tokenizer | /tokenizers/mod.ts | netsaur@0.3.0

Deno 2 is finally here 🎉️

Powerful machine learning, accelerated by WebGPU

class Tokenizer

import { Tokenizer } from "https://deno.land/x/netsaur@0.3.0/tokenizers/mod.ts";

Tokenizer class

Constructors

new

Tokenizer(id: number)

decode(ids: Uint32Array, skipSpecialTokens?)

Decode a sentence

encode(sentence: string)

Encode a sentence

getVocab(withAddedTokens?)

Get the vocab

getVocabSize(withAddedTokens?)

Get the vocab size

idToToken(id: number)

Get the token from an id

save(): string

Save the tokenizer as json

save(pretty: boolean): string

Save the tokenizer as json

tokenToId(token: string)

Get the id from a token

fromJSON(json: string)

Load a tokenizer from json data