Skip to main content
Deno 2 is finally here 🎉️
Learn more
Module

x/netsaur/tokenizers/mod.ts>Tokenizer#tokenize

Powerful machine learning, accelerated by WebGPU
Go to Latest
method Tokenizer.prototype.tokenize
import { Tokenizer } from "https://deno.land/x/netsaur@0.2.12/tokenizers/mod.ts";

Tokenize a sentence

Parameters

sentence: string

sentence to tokenize