import { type Token } from "https://deno.land/x/dendron_exports@v0.1.2/deps/micromark.ts";
A token: a span of chunks.
Tokens are what the core of micromark produces: the built in HTML compiler or other tools can turn them into different things.
Tokens are essentially names attached to a slice of chunks, such as
lineEndingBlank for certain line endings, or codeFenced for a whole
fenced code.
Sometimes, more info is attached to tokens, such as _open and _close
by attention (strong, emphasis) to signal whether the sequence can open
or close an attention run.
Linked tokens are used because outer constructs are parsed first. Take for example:
> *a
b*.
- The block quote marker and the space after it is parsed first
- The rest of the line is a
chunkFlowtoken - The two spaces on the second line are a
linePrefix - The rest of the line is another
chunkFlowtoken
The two chunkFlow tokens are linked together.
The chunks they span are then passed through the flow tokenizer.
Properties
Token type.
Point where the token starts.
Point where the token ends.
The previous token in a list of linked tokens.
The next token in a list of linked tokens.
Declares a token as having content of a certain type.
Connected tokenizer.
Used when dealing with linked tokens. A child tokenizer is needed to tokenize them, which is stored on those tokens.
Field to help parse attention.
Depending on the character before sequences (**), the sequence can open,
close, both, or none.
Field to help parse attention.
Depending on the character before sequences (**), the sequence can open,
close, both, or none.
Field to help parse GFM task lists.
This boolean is used internally to figure out if a token is in the first content of a list item construct.
Field to help parse containers.
This boolean is used internally to figure out if a token is a container token.
Field to help parse lists.
This boolean is used internally to figure out if a list is loose or not.
Field to help parse links.
This boolean is used internally to figure out if a link opening can’t be used (because links in links are incorrect).