encoding
Helper module for dealing with external data structures.
Binary
Implements equivalent methods to Go’s encoding/binary
package.
Available Functions:
sizeof(dataType: DataTypes): number
getNBytes(r: Deno.Reader, n: number): Promise<Uint8Array>
varnum(b: Uint8Array, o: VarnumOptions = {}): number | null
varbig(b: Uint8Array, o: VarbigOptions = {}): bigint | null
putVarnum(b: Uint8Array, x: number, o: VarnumOptions = {}): number
putVarbig(b: Uint8Array, x: bigint, o: VarbigOptions = {}): number
readVarnum(r: Deno.Reader, o: VarnumOptions = {}): Promise<number>
readVarbig(r: Deno.Reader, o: VarbigOptions = {}): Promise<bigint>
writeVarnum(w: Deno.Writer, x: number, o: VarnumOptions = {}): Promise<number>
writeVarbig(w: Deno.Writer, x: bigint, o: VarbigOptions = {}): Promise<number>
CSV
API
readMatrix
Parse the CSV from the reader
with the options provided and return
string[][]
.
parse
Parse the CSV string/buffer with the options provided. The result of this function is as follows:
- If you don’t provide
opt.skipFirstRow
andopt.columns
, it returnsstring[][]
. - If you provide
opt.skipFirstRow
oropt.columns
it returnsRecord<string, unknown>[]
.
ParseOptions
skipFirstRow: boolean;
: If you provideskipFirstRow: true
andcolumns
, the first line will be skipped. If you provideskipFirstRow: true
but notcolumns
, the first line will be skipped and used as header definitions.columns: string[] | ColumnOptions[];
: If you providestring[]
orColumnOptions[]
, those names will be used for header definition.
ColumnOptions
name: string;
: Name of the header to be used as property.
ReadOptions
comma?: string;
: Character which separates values. Default:","
.comment?: string;
: Character to start a comment. Default:"#"
.trimLeadingSpace?: boolean;
: Flag to trim the leading space of the value. Default:false
.lazyQuotes?: boolean;
: Allow unquoted quote in a quoted field or non double quoted quotes in quoted field. Default:false
.fieldsPerRecord?
: Enabling the check of fields for each row. If == 0, first row is used as referral for the number of fields.
stringify
data
is the source data to stringify. It’s an array of items which are plain objects or arrays.DataItem: Record<string, unknown> | unknown[]
const data = [ { name: "Deno", repo: { org: "denoland", name: "deno" }, runsOn: ["Rust", "TypeScript"], }, ];
columns
is a list of instructions for how to target and transform the data for each column of output. This is also where you can provide an explicit header name for the column.Column
:The most essential aspect of a column is accessing the property holding the data for that column on each object in the data array. If that member is at the top level,
Column
can simply be a property accessor, which is either astring
(if it’s a plain object) or anumber
(if it’s an array).const columns = [ "name", ];
Each property accessor will be used as the header for the column:
name Deno If the required data is not at the top level (it’s nested in other objects/arrays), then a simple property accessor won’t work, so an array of them will be required.
const columns = [ ["repo", "name"], ["repo", "org"], ];
When using arrays of property accessors, the header names inherit the value of the last accessor in each array:
name org deno denoland If the data is not already in the required output format, or a different column header is desired, then a
ColumnDetails
object type can be used for each column:fn?: (value: any) => string | Promise<string>
is an optional function to transform the targeted data into the desired formatheader?: string
is the optional value to use for the column header nameprop: PropertyAccessor | PropertyAccessor[]
is the property accessor (string
ornumber
) or array of property accessors used to access the data on each object
const columns = [ "name", { prop: ["runsOn", 0], header: "language 1", fn: (str: string) => str.toLowerCase(), }, { prop: ["runsOn", 1], header: "language 2", fn: (str: string) => str.toLowerCase(), }, ];
name language 1 language 2 Deno rust typescript
options
are options for the delimiter-separated output.headers?: boolean
: Whether or not to include the row of headers. Default:true
separator?: string
: Delimiter used to separate values. Examples:","
comma (Default)"\t"
tab"|"
pipe- etc.
Basic Usage
import { parse } from "https://deno.land/std@0.160.0/encoding/csv.ts";
const string = "a,b,c\nd,e,f";
console.log(
await parse(string, {
skipFirstRow: false,
}),
);
// output:
// [["a", "b", "c"], ["d", "e", "f"]]
import {
Column,
stringify,
} from "https://deno.land/std@0.160.0/encoding/csv.ts";
type Character = {
age: number;
name: {
first: string;
last: string;
};
};
const data: Character[] = [
{
age: 70,
name: {
first: "Rick",
last: "Sanchez",
},
},
{
age: 14,
name: {
first: "Morty",
last: "Smith",
},
},
];
let columns: Column[] = [
["name", "first"],
"age",
];
console.log(stringify(data, { columns }));
// first,age
// Rick,70
// Morty,14
TOML
This module parse TOML files. It follows as much as possible the TOML specs. Be sure to read the supported types as not every specs is supported at the moment and the handling in TypeScript side is a bit different.
Supported types and handling
- :heavy_check_mark: Keys
- :exclamation: String
- :heavy_check_mark: Multiline String
- :heavy_check_mark: Literal String
- :exclamation: Integer
- :heavy_check_mark: Float
- :heavy_check_mark: Boolean
- :heavy_check_mark: Offset Date-time
- :heavy_check_mark: Local Date-time
- :heavy_check_mark: Local Date
- :exclamation: Local Time
- :heavy_check_mark: Table
- :heavy_check_mark: Inline Table
- :exclamation: Array of Tables
:exclamation: Supported with warnings see Warning.
⚠️ Warning
String
- Regex : Due to the spec, there is no flag to detect regex properly in a TOML declaration. So the regex is stored as string.
Integer
For Binary / Octal / Hexadecimal numbers, they are stored as string to be not interpreted as Decimal.
Local Time
Because local time does not exist in JavaScript, the local time is stored as a string.
Inline Table
Inline tables are supported. See below:
animal = { type = { name = "pug" } }
## Output { animal: { type: { name: "pug" } } }
animal = { type.name = "pug" }
## Output { animal: { type : { name : "pug" } }
animal.as.leaders = "tosin"
## Output { animal: { as: { leaders: "tosin" } } }
"tosin.abasi" = "guitarist"
## Output { tosin.abasi: "guitarist" }
Array of Tables
At the moment only simple declarations like below are supported:
[[bin]]
name = "deno"
path = "cli/main.rs"
[[bin]]
name = "deno_core"
path = "src/foo.rs"
[[nib]]
name = "node"
path = "not_found"
will output:
{
"bin": [
{ "name": "deno", "path": "cli/main.rs" },
{ "name": "deno_core", "path": "src/foo.rs" }
],
"nib": [{ "name": "node", "path": "not_found" }]
}
Basic usage
import {
parse,
stringify,
} from "https://deno.land/std@0.160.0/encoding/toml.ts";
const obj = {
bin: [
{ name: "deno", path: "cli/main.rs" },
{ name: "deno_core", path: "src/foo.rs" },
],
nib: [{ name: "node", path: "not_found" }],
};
const tomlString = stringify(obj);
console.log(tomlString);
// =>
// [[bin]]
// name = "deno"
// path = "cli/main.rs"
// [[bin]]
// name = "deno_core"
// path = "src/foo.rs"
// [[nib]]
// name = "node"
// path = "not_found"
const tomlObject = parse(tomlString);
console.log(tomlObject);
// =>
// {
// bin: [
// { name: "deno", path: "cli/main.rs" },
// { name: "deno_core", path: "src/foo.rs" }
// ],
// nib: [ { name: "node", path: "not_found" } ]
// }
YAML
YAML parser / dumper for Deno.
Heavily inspired from js-yaml
.
Basic usage
parse
parses the yaml string, and stringify
dumps the given object to YAML
string.
import {
parse,
stringify,
} from "https://deno.land/std@0.160.0/encoding/yaml.ts";
const data = parse(`
foo: bar
baz:
- qux
- quux
`);
console.log(data);
// => { foo: "bar", baz: [ "qux", "quux" ] }
const yaml = stringify({ foo: "bar", baz: ["qux", "quux"] });
console.log(yaml);
// =>
// foo: bar
// baz:
// - qux
// - quux
If your YAML contains multiple documents in it, you can use parseAll
for
handling it.
import { parseAll } from "https://deno.land/std@0.160.0/encoding/yaml.ts";
const data = parseAll(`
---
id: 1
name: Alice
---
id: 2
name: Bob
---
id: 3
name: Eve
`);
console.log(data);
// => [ { id: 1, name: "Alice" }, { id: 2, name: "Bob" }, { id: 3, name: "Eve" } ]
To handle regexp
, and undefined
types, use the EXTENDED_SCHEMA
. Note that
functions are no longer supported for security reasons
import {
EXTENDED_SCHEMA,
parse,
} from "https://deno.land/std@0.160.0/encoding/yaml.ts";
const data = parse(
`
regexp:
simple: !!js/regexp foobar
modifiers: !!js/regexp /foobar/mi
undefined: !!js/undefined ~
# Disabled, see: https://github.com/denoland/deno_std/pull/1275
# function: !!js/function >
# function foobar() {
# return 'hello world!';
# }
`,
{ schema: EXTENDED_SCHEMA },
);
You can also use custom types by extending schemas.
import {
DEFAULT_SCHEMA,
parse,
Type,
} from "https://deno.land/std@0.160.0/encoding/yaml.ts";
const yaml = "...";
const MyYamlType = new Type("!myYamlType", {
kind: "sequence",
/* other type options here*/
});
const MY_SCHEMA = DEFAULT_SCHEMA.extend({ explicit: [MyYamlType] });
parse(yaml, { schema: MY_SCHEMA });
API
parse(str: string, opts?: ParserOption): unknown
Parses the YAML string with a single document.
parseAll(str: string, iterator?: Function, opts?: ParserOption): unknown
Parses the YAML string with multiple documents. If the iterator is given, it’s applied to every document instead of returning the array of parsed objects.
stringify(obj: object, opts?: DumpOption): string
Serializes object
as a YAML document.
⚠️ Limitations
binary
type is currently not stable.
More example
See: https://github.com/nodeca/js-yaml/tree/master/examples
JSON streaming
Streams JSON concatenated with line breaks or special characters. This module supports the following formats:
- JSON lines
- NDJSON
- JSON Text Sequences
- Concatenated JSON
- JSON concatenated with any delimiter
Basic usage
If you want to parse JSON separated by a delimiter, use TextLineStream
(or
TextDelimiterStream
) and JsonParseStream
. JsonParseStream
ignores chunks
consisting of spaces, tab characters, or newline characters .
// parse JSON lines or NDJSON
import { TextLineStream } from "https://deno.land/std@0.160.0/streams/mod.ts";
import { JsonParseStream } from "https://deno.land/std@0.160.0/encoding/json/stream.ts";
const url =
"https://deno.land/std@0.160.0/encoding/testdata/json/test.jsonl";
const { body } = await fetch(url);
const readable = body!
.pipeThrough(new TextDecoderStream()) // convert Uint8Array to string
.pipeThrough(new TextLineStream()) // transform into a stream where each chunk is divided by a newline
.pipeThrough(new JsonParseStream()); // parse each chunk as JSON
for await (const data of readable) {
console.log(data);
}
// parse JSON Text Sequences
import { TextDelimiterStream } from "https://deno.land/std@0.160.0/streams/mod.ts";
import { JsonParseStream } from "https://deno.land/std@0.160.0/encoding/json/stream.ts";
const url =
"https://deno.land/std@0.160.0/encoding/testdata/json/test.json-seq";
const { body } = await fetch(url);
const delimiter = "\x1E";
const readable = body!
.pipeThrough(new TextDecoderStream())
.pipeThrough(new TextDelimiterStream(delimiter)) // transform into a stream where each chunk is divided by a delimiter
.pipeThrough(new JsonParseStream());
for await (const data of readable) {
console.log(data);
}
If you want to parse
Concatenated JSON,
use ConcatenatedJsonParseStream
.
import { ConcatenatedJsonParseStream } from "https://deno.land/std@0.160.0/encoding/json/stream.ts";
const url =
"https://deno.land/std@0.160.0/encoding/testdata/json/test.concatenated-json";
const { body } = await fetch(url);
const readable = body!
.pipeThrough(new TextDecoderStream()) // convert Uint8Array to string
.pipeThrough(new ConcatenatedJsonParseStream()); // parse Concatenated JSON
for await (const data of readable) {
console.log(data);
}
Use JsonStringifyStream
to transform streaming data to
JSON lines, NDJSON,
NDJSON or
Concatenated JSON.
By default, JsonStringifyStream
adds “\n” as a suffix after each chunk.
import { readableStreamFromIterable } from "https://deno.land/std@0.160.0/streams/mod.ts";
import { JsonStringifyStream } from "https://deno.land/std@0.160.0/encoding/json/stream.ts";
const file = await Deno.open("./tmp.jsonl", { create: true, write: true });
readableStreamFromIterable([{ foo: "bar" }, { baz: 100 }])
.pipeThrough(new JsonStringifyStream()) // convert to JSON lines (ndjson)
.pipeThrough(new TextEncoderStream()) // convert a string to a Uint8Array
.pipeTo(file.writable)
.then(() => console.log("write success"));
If you want to use an arbitrary delimiter, specify prefix and suffix as options. These are added before and after chunk after stringify. To convert to JSON Text Sequences, set the prefix to the delimiter “\x1E” as options.
import { readableStreamFromIterable } from "https://deno.land/std@0.160.0/streams/mod.ts";
import { JsonStringifyStream } from "https://deno.land/std@0.160.0/encoding/json/stream.ts";
const file = await Deno.open("./tmp.jsonl", { create: true, write: true });
readableStreamFromIterable([{ foo: "bar" }, { baz: 100 }])
.pipeThrough(new JsonStringifyStream({ prefix: "\x1E", suffix: "\n" })) // convert to JSON Text Sequences
.pipeThrough(new TextEncoderStream())
.pipeTo(file.writable)
.then(() => console.log("write success"));
If you want to stream JSON lines from the server:
import { serve } from "https://deno.land/std@0.160.0/http/server.ts";
import { JsonStringifyStream } from "https://deno.land/std@0.160.0/encoding/json/stream.ts";
// A server that streams one line of JSON every second
serve(() => {
let intervalId: number | undefined;
const readable = new ReadableStream({
start(controller) {
// enqueue data once per second
intervalId = setInterval(() => {
controller.enqueue({ now: new Date() });
}, 1000);
},
cancel() {
clearInterval(intervalId);
},
});
const body = readable
.pipeThrough(new JsonStringifyStream()) // convert data to JSON lines
.pipeThrough(new TextEncoderStream()); // convert a string to a Uint8Array
return new Response(body);
});
JSONC
JSONC (JSON with Comments) parser for Deno.
API
parse(text: string, options: { allowTrailingComma?: boolean; })
Parses the JSONC string. Setting allowTrailingComma to false rejects trailing commas in objects and arrays. If parsing fails, throw a SyntaxError.
Basic usage
import * as JSONC from "https://deno.land/std@0.160.0/encoding/jsonc.ts";
console.log(JSONC.parse('{"foo": "bar", } // comment')); //=> { foo: "bar" }
console.log(JSONC.parse('{"foo": "bar", } /* comment */')); //=> { foo: "bar" }
console.log(JSONC.parse('{"foo": "bar" } // comment', {
allowTrailingComma: false,
})); //=> { foo: "bar" }
base32
RFC4648 base32 encoder/decoder for Deno.
Basic usage
encode
encodes a Uint8Array
to RFC4648 base32 representation, and decode
decodes the given RFC4648 base32 representation to a Uint8Array
.
import {
decode,
encode,
} from "https://deno.land/std@0.160.0/encoding/base32.ts";
const b32Repr = "RC2E6GA=";
const binaryData = decode(b32Repr);
console.log(binaryData);
// => Uint8Array [ 136, 180, 79, 24 ]
console.log(encode(binaryData));
// => RC2E6GA=
base64
RFC4648 base64 encoder/decoder for Deno.
Basic usage
encode
encodes a Uint8Array
to RFC4648 base64 representation, and decode
decodes the given RFC4648 base64 representation to a Uint8Array
.
import {
decode,
encode,
} from "https://deno.land/std@0.160.0/encoding/base64.ts";
const b64Repr = "Zm9vYg==";
const binaryData = decode(b64Repr);
console.log(binaryData);
// => Uint8Array [ 102, 111, 111, 98 ]
console.log(encode(binaryData));
// => Zm9vYg==
base64url
RFC4648 base64url encoder/decoder for Deno.
Basic usage
encode
encodes a Uint8Array
to RFC4648 base64url representation, and
decode
decodes the given RFC4648 base64url representation to a Uint8Array
.
import {
decode,
encode,
} from "https://deno.land/std@0.160.0/encoding/base64url.ts";
const binary = new TextEncoder().encode("foobar");
const encoded = encode(binary);
console.log(encoded);
// => "Zm9vYmFy"
console.log(decode(encoded));
// => Uint8Array(6) [ 102, 111, 111, 98, 97, 114 ]
ascii85
Ascii85/base85 encoder and decoder with support for multiple standards.
Basic usage
encode
encodes a Uint8Array
to a ascii85 representation, and decode
decodes the given ascii85 representation to a Uint8Array
.
import {
decode,
encode,
} from "https://deno.land/std@0.160.0/encoding/ascii85.ts";
const a85Repr = "LpTqp";
const binaryData = decode(a85Repr);
console.log(binaryData);
// => Uint8Array [ 136, 180, 79, 24 ]
console.log(encode(binaryData));
// => LpTqp
Specifying a standard and delimiter
By default all functions are using the most popular Adobe version of ascii85 and
not adding any delimiter. However, there are three more standards supported -
btoa (different delimiter and additional compression of 4 bytes equal to 32),
Z85 and
RFC 1924. It’s possible to use a
different encoding by specifying it in options
object as a second parameter.
Similarly, it’s possible to make encode
add a delimiter (<~
and ~>
for
Adobe, xbtoa Begin
and xbtoa End
with newlines between the delimiters and
encoded data for btoa. Checksums for btoa are not supported. Delimiters are not
supported by other encodings.)
encoding examples:
import {
decode,
encode,
} from "https://deno.land/std@0.160.0/encoding/ascii85.ts";
const binaryData = new Uint8Array([136, 180, 79, 24]);
console.log(encode(binaryData));
// => LpTqp
console.log(encode(binaryData, { standard: "Adobe", delimiter: true }));
// => <~LpTqp~>
console.log(encode(binaryData, { standard: "btoa", delimiter: true }));
/* => xbtoa Begin
LpTqp
xbtoa End */
console.log(encode(binaryData, { standard: "RFC 1924" }));
// => h_p`_
console.log(encode(binaryData, { standard: "Z85" }));
// => H{P}{
hex
hexadecimal encoder/decoder for Deno.
Basic usage
encode
encodes a Uint8Array
to hexadecimal Uint8Array
with 2 * length, and
decode
decodes the given hexadecimal Uint8Array
to a Uint8Array
.
import {
decode,
encode,
} from "https://deno.land/std@0.160.0/encoding/hex.ts";
const binary = new TextEncoder().encode("abc");
const encoded = encode(binary);
console.log(encoded);
// => Uint8Array(6) [ 54, 49, 54, 50, 54, 51 ]
console.log(decode(encoded));
// => Uint8Array(3) [ 97, 98, 99 ]