Skip to main content

kvdex

Database wrapper for Deno’s KV store. Simple and type-safe storing/retrieving of data.

Support for indexing.

Zero third-party dependencies.

Table of Contents

Models

For collections of objects, models can be defined by extending the Model type. Optional and nullable properties are allowed. If you wish to use Zod, you can create your Zod object schema and use its inferred type as your model.

NOTE: When using interfaces instead of types, sub-interfaces must also extend the Model type.

import type { Model } from "https://deno.land/x/kvdex@v0.9.0/mod.ts"

interface User extends Model {
  username: string
  age: number
  activities: string[]
  address?: {
    country: string
    city: string
    street: string
    houseNumber: number | null
  }
}

Database

kvdex() is used for creating a new database instance. It takes a Deno KV instance and a schema definition as arguments.

import { kvdex } from "https://deno.land/x/kvdex@v0.9.0/mod.ts"

const kv = await Deno.openKv()

const db = kvdex(kv, {
  numbers: (ctx) => ctx.collection<number>().build(),
  largeStrings: (ctx) => ctx.largeCollection<string>().build(),
  users: (ctx) => ctx.indexableCollection<User>().build({
    indices: {
      username: "primary" // unique
      age: "secondary" // non-unique
    }
  }),
  // Nested collections
  nested: {
    strings: (ctx) => ctx.collection<string>().build(),
  }
})

The schema definition defines collection builder functions (or nested schema definitions) which receive a builder context. Standard collections can hold any type adhering to KvValue (string, number, array, object…), large collections can hold strings, arrays and objects, while indexable collections can only hold types adhering to Model (objects). For indexable collections, primary (unique) and secondary (non-unique) indexing is supported.

Collection Methods

find()

Retrieve a single document with the given id from the KV store. The id must adhere to the type KvId. This method takes an optional options argument that can be used to set the consistency mode.

const userDoc1 = await db.users.find(123)

const userDoc2 = await db.users.find(123n)

const userDoc3 = await db.users.find("oliver", {
  consistency: "eventual", // "strong" by default
})

findMany()

Retrieve multiple documents with the given array of ids from the KV store. The ids must adhere to the type KvId. This method takes an optional options argument that can be used to set the consistency mode.

const userDocs1 = await db.users.findMany(["abc", 123, 123n])

const userDocs2 = await db.users.findMany(["abc", 123, 123n], {
  consistency: "eventual", // "strong" by default
})

add()

Add a new document to the KV store with an auto-generated id (uuid). Upon completion, a CommitResult object will be returned with the document id, versionstamp and ok flag.

const result = await db.users.add({
  username: "oliver",
  age: 24,
  activities: ["skiing", "running"],
  address: {
    country: "Norway",
    city: "Bergen",
    street: "Sesame",
    houseNumber: null,
  },
})

console.log(result.id) // f897e3cf-bd6d-44ac-8c36-d7ab97a82d77

addMany()

Add multiple document entries to the KV store with auto-generated ids (uuid). Upon completion, a list of CommitResult objects will be returned.

// Adds 5 new document entries to the KV store.
await results = await db.numbers.addMany(1, 2, 3, 4, 5)

// Only adds the first entry, as "username" is defined as a primary index and cannot have duplicates
await results = await db.users.addMany(
  {
    username: "oli",
    age: 24
  },
  {
    username: "oli",
    age: 56
  }
)

set()

Add a new document to the KV store with a given id of type KvId. Upon completion, a CommitResult object will be returned with the document id, versionstamp and ok flag.

const result = await db.numbers.set("id_1", 2048)

console.log(result.id) // "id_1"

update()

Update the value of an exisiting document in the KV store. For primitive values, arrays and built-in objects (Date, RegExp, etc.), this method overrides the exisiting data with the new value. For custom objects (Models), this method performs a partial update, merging the new value with the existing data. Upon completion, a CommitResult object will be returned with the document id, versionstamp and ok flag.

// Updates the document with a new value
const result1 = await db.numbers.update("num1", 42)

// Partial update, only updates the age field
const result2 = await db.users.update("user1", {
  age: 67,
})

delete()

Delete one or more documents with the given ids from the KV store.

await db.users.delete("f897e3cf-bd6d-44ac-8c36-d7ab97a82d77")

await db.users.delete("user1", "user2", "user3")

deleteMany()

Delete multiple documents from the KV store without specifying ids. It takes an optional options argument that can be used for filtering of documents to be deleted, and pagination. If no options are given, “deleteMany” will delete all documents in the collection.

// Deletes all user documents
await db.users.deleteMany()

// Deletes all user documents where the user's age is above 20
await db.users.deleteMany({
  filter: (doc) => doc.value.age > 20,
})

// Deletes the first 10 user documents in the KV store
await db.users.deleteMany({
  limit: 10,
})

// Deletes the last 10 user documents in the KV store
await db.users.deleteMany({
  limit: 10,
  reverse: true,
})

getMany()

Retrieve multiple documents from the KV store. It takes an optional options argument that can be used for filtering of documents to be retrieved, and pagination. If no options are given, “getMany” will retrieve all documents in the collection.

// Retrieves all user documents
const { result } = await db.users.getMany()

// Retrieves all user documents where the user's age is above or equal to 18
const { result } = await db.users.getMany({
  filter: (doc) => doc.value.age >= 18,
})

// Retrieves the first 10 user documents in the KV store
const { result } = await db.users.getMany({
  limit: 10,
})

// Retrieves the last 10 user documents in the KV store
const { result } = await db.users.getMany({
  limit: 10,
  reverse: true,
})

forEach()

Execute a callback function for multiple documents in the KV store. It takes an optional options argument that can be used for filtering of documents and pagination. If no options are given, the callback function will be executed for all documents in the collection.

// Log the username of every user document
await db.users.forEach((doc) => console.log(doc.value.username))

// Log the username of every user that has "swimming" as an activity
await db.users.forEach((doc) => console.log(doc.value.username), {
  filter: (doc) => doc.value.activities.includes("swimming"),
})

// Log the usernames of the first 10 user documents in the KV store
await db.users.forEach((doc) => console.log(doc.value.username), {
  limit: 10,
})

// Log the usernames of the last 10 user documents in the KV store
await db.users.forEach((doc) => console.log(doc.value.username), {
  limit: 10,
  reverse: true,
})

map()

Execute a callback function for multiple documents in the KV store and retrieve the results. It takes an optional options argument that can be used for filtering of documents and pagination. If no options are given, the callback function will be executed for all documents in the collection.

// Get a list of all the ids of the user documents
const { result } = await db.users.map((doc) => doc.id)

// Get a list of all usernames of users with age > 20
const { result } = await db.users.map((doc) => doc.value.username, {
  filter: (doc) => doc.value.age > 20,
})

// Get a list of the usernames of the first 10 users in the KV store
const { result } = await db.users.forEach((doc) => doc.value.username, {
  limit: 10,
})

// Get a list of the usernames of the last 10 users in the KV store
const { result } = await db.users.forEach((doc) => doc.value.username, {
  limit: 10,
  reverse: true
})

count()

Count the number of documents in a collection. It takes an optional options argument that can be used for filtering of documents. If no options are given, it will count all documents in the collection.

// Returns the total number of user documents in the KV store
const count = await db.users.count()

// Returns the number of users with age > 20
const count = await db.users.count({
  filter: doc => doc.value.age > 20
})

enqueue()

Add data (of any type) to the collection queue to be delivered to the queue listener via db.collection.listenQueue(). The data will only be received by queue listeners on the specified collection. The method takes an optional options argument that can be used to set a delivery delay.

// Immediate delivery
await db.users.enqueue("some data")

// Delay of 2 seconds before delivery
await db.users.enqueue("some data", {
  delay: 2_000
})

listenQueue()

Listen for data from the collection queue that was enqueued with db.collection.enqueue(). Will only receive data that was enqueued to the specific collection queue. Takes a handler function as argument.

// Prints the data to console when recevied
db.users.listenQueue((data) => console.log(data))

// Sends post request when data is received
db.users.listenQueue(async (data) => {
  const dataBody = JSON.stringify(data) 

  const res = await fetch("...", {
    method: "POST",
    body: dataBody
  })

  console.log("POSTED:", dataBody, res.ok)
})

Indexable Collection Methods

Indexable collections extend the base Collection class and provide all the same methods. Note that add/set methods will always fail if an identical index entry already exists.

findByPrimaryIndex()

Find a document by a primary index.

// Finds a user document with the username = "oliver"
const userByUsername = await db.users.findByPrimaryIndex("username", "oliver")

findBySecondaryIndex()

Find documents by a secondary index. Secondary indices are not unique, and therefore the result is an array of documents. The method takes an optional options argument that can be used for filtering of documents, and pagination.

// Returns all users with age = 24
const { result } = await db.users.findBySecondaryIndex("age", 24)

// Returns all users with age = 24 AND username that starts with "o"
const { result } = await db.users.findBySecondaryIndex("age", 24, {
  filter: (doc) => doc.value.username.startsWith("o")
})

deleteByPrimaryIndex()

Delete a document by a primary index.

// Deletes user with username = "oliver"
await db.users.deleteByPrimaryIndex("username", "oliver")

deleteBySecondaryIndex()

Delete documents by a secondary index. The method takes an optional options argument that can be used for filtering of documents, and pagination.

// Deletes all users with age = 24
await db.users.deleteBySecondaryIndex("age", 24)

// Deletes all users with age = 24 AND username that starts with "o"
await db.users.deleteBySecondaryIndex("age", 24, {
  filter: (doc) => doc.value.username.startsWith("o")
})

Large Collections

Large collections are distinct from standard collections or indexable collections in that they can hold values that exceed the size limit of values in Deno KV. Value types are limited to strings, arrays of number, boolean or LargeKvValue, and objects with LargeKvValue properties. All base collection methods are available for large collections. Document values are divided accross multiple Deno KV entries, which impacts the performance of most operations. Only use this collection type if you think your document values will exceed the approximately 65KB size limit.

Database Methods

These are methods which can be found at the top level of your database object, and perform operations across multiple collections.

countAll()

Count the total number of documents across all collections. It takes an optional options argument that can be used to set the consistency mode.

// Gets the total number of documents in the KV store across all collections
const count = await db.countAll()

deleteAll()

Delete all documents across all collections. It takes an optional options argument that can be used to set the consistency mode.

// Deletes all documents in the KV store across all collections
await db.deleteAll()

enqueue()

Add data (of any type) to the database queue to be delivered to the queue listener via db.listenQueue(). The data will only be received by queue listeners on the database queue. The method takes an optional options argument that can be used to set a delivery delay.

// Immediate delivery
await db.enqueue("some data")

// Delay of 2 seconds before delivery
await db.enqueue("some data", {
  delay: 2_000
})

listenQueue()

Listen for data from the database queue that was enqueued with db.enqueue(). Will only receive data that was enqueued to the database queue. Takes a handler function as argument.

// Prints the data to console when recevied
db.listenQueue((data) => console.log(data))

// Sends post request when data is received
db.listenQueue(async (data) => {
  const dataBody = JSON.stringify(data) 

  const res = await fetch("...", {
    method: "POST",
    body: dataBody
  })

  console.log("POSTED:", dataBody, res.ok)
})

atomic()

Initiate an atomic operation. The method takes a selection function as argument for selecting the initial collection context.

db.atomic((schema) => schema.users)

Atomic Operations

Atomic operations allow for executing multiple mutations as a single atomic transaction. This means that documents can be checked for changes before committing the mutations, in which case the operation will fail. It also ensures that either all mutations succeed, or they all fail.

To initiate an atomic operation, call “atomic” on the database object. The method expects a selector for selecting the collection that the subsequent mutation actions will be performed on. Mutations can be performed on documents from multiple collections in a single atomic operation by calling “select” at any point in the building chain to switch the collection context. To execute the operation, call “commit” at the end of the chain. An atomic operation returns a Deno.KvCommitResult object if successful, and Deno.KvCommitError if not.

NOTE: For indexable collections, any operations performing deletes will not be truly atomic in the sense that it performs a single isolated operation. The reason for this being that the document data must be read before performing the initial delete operation, to then perform another delete operation for the index entries. If the initial operation fails, the index entries will not be deleted. To avoid collisions and errors related to indexing, an atomic operation will always fail if it is trying to delete and write to the same indexable collection. It will also fail if trying to set/add a document with existing index entries.

Without checking

// Deletes and adds an entry to the bigints collection
const result1 = await db
  .atomic((schema) => schema.numbers)
  .delete("id_1")
  .set("id_2", 100)
  .commit()

// Adds 2 new entries to the strings collection and 1 new entry to the users collection
const result2 = await db
  .atomic((schema) => schema.numbers)
  .add(1)
  .add(2)
  .select((schema) => schema.users)
  .set("user_1", {
    username: "oliver",
    age: 24,
    activities: ["skiing", "running"],
    address: {
      country: "Norway",
      city: "Bergen",
      street: "Sesame",
      houseNumber: 42,
    },
  })
  .commit()

// Will fail and return Deno.KvCommitError because it is trying
// to both add and delete from an indexable collection
const result3 = await db
  .atomic((schema) => schema.users)
  .delete("user_1")
  .set("user_1", {
    username: "oliver",
    age: 24,
    activities: ["skiing", "running"],
    address: {
      country: "Norway",
      city: "Bergen",
      street: "Sesame",
      houseNumber: 42,
    },
  })
  .commit()

With checking

// Only adds 10 to the value when it has not been changed after being read
let result = null
while (!result || !result.ok) {
  const { id, versionstamp, value } = await db.numbers.find("id")

  result = await db
    .atomic((schema) => schema.numbers)
    .check({
      id,
      versionstamp,
    })
    .set(id, value + 10)
    .commit()
}

Utils

Additional utility functions.

flatten()

Flatten documents with a value of type Model. Only flattens the first layer of the document, meaning the result will be an object containing: id, versionstamp and all the entries in the document value.

import { flatten } from "https://deno.land/x/kvdex@v0.9.0/mod.ts"

// We assume the document exists in the KV store
const doc = await db.users.find(123n)

const flattened = flatten(doc)

// Document:
// {
//   id,
//   versionstamp,
//   value
// }

// Flattened:
// {
//   id,
//   versionstamp,
//   ...userDocument.value
// }

Development

Any contributions are welcomed and appreciated. How to contribute:

  • Clone this repository
  • Add feature / Refactor
  • Add or refactor tests as needed
  • Run tests using deno task test
  • Prepare code (lint + format + test) using deno task prep
  • Open Pull Request

This project aims at having as high test coverage as possible to improve code quality and to avoid breaking features when refactoring. Therefore it is encouraged that any feature contributions are also accompanied by relevant unit tests to ensure those features remain stable.

The goal of kvdex is to provide a type safe, higher level API to Deno KV, while trying to retain as much of the native functionality as possible. Additionally, this module should be light-weight and should not rely on any third-party dependencies. Please kleep this in mind when making any contributions.

License

Published under MIT License