Skip to main content
Using Deno in production at your company? Earn free Deno merch.
Give us feedback
Module

x/netsaur/tokenizers/mod.ts>Tokenizer#encode

Powerful machine learning, accelerated by WebGPU
denosaurs/netsaur
Latest
method Tokenizer.prototype.encode
import { Tokenizer } from "https://deno.land/x/netsaur@0.3.2-patch/tokenizers/mod.ts";

Encode a sentence to tokens

Parameters

sentence: string

sentence to tokenize

Returns

Uint32Array