Popular
Latest
123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343344345346347348349350351352353354355356357358359360361362363364365366367368369370371372373374375376377378379380381382383384385386387388389390391392393394395396397398399400401402403404405406407408409410411412413414415416417418419420421422423424425426427428429430431432433434435436437438439440441442443444445446447448449450451452453454455456457458459460461462463464465466467468469470471472473474475476477478479480481482483484485486487488489490491492493494495496497498499500501502503504505506507508509510511512513514515516517518519520521522523524525526527528529530531532533534535536537538539540541542543544545546547548549550551552553554555556557558559560561562563564565566567568569570571572573574575576577578579580581582583584585586587588589590591592593594595596597598599600601602603604605606607608609610611612613614615616617618619620621622623624625626627628629630631632633634635636637638639640641642643644645646647648649650651652653654655656657658659660661662663664665666667668669670671672673674675676677678679680681682683684685686687688689690691692693694695696697698699700701702703704705706707708709710711712713714715716717718719720721722723724725726727728729730731732733734735736737738739740741742743744745746747748749750751752753754755756757758759760761762763764765766767768769770771772773774775776777778779780781782783784785786787788789790791792793794795796797798799800801802803804805806807808809810811812813814815816817818819820821822823824825826827828829830831832833834835836837838839840841842843844845846847848849850851852853854855856857858859860861862863864865866867868869870<!DOCTYPE HTML><html lang="en" class="light" dir="ltr"> <head> <!-- Book generated using mdBook --> <meta charset="UTF-8"> <title>proc</title> <meta name="robots" content="noindex">
<!-- Custom HTML head --> <meta name="description" content=""> <meta name="viewport" content="width=device-width, initial-scale=1"> <meta name="theme-color" content="#ffffff">
<link rel="icon" href="favicon.svg"> <link rel="shortcut icon" href="favicon.png"> <link rel="stylesheet" href="css/variables.css"> <link rel="stylesheet" href="css/general.css"> <link rel="stylesheet" href="css/chrome.css"> <link rel="stylesheet" href="css/print.css" media="print">
<!-- Fonts --> <link rel="stylesheet" href="FontAwesome/css/font-awesome.css"> <link rel="stylesheet" href="fonts/fonts.css">
<!-- Highlight.js Stylesheets --> <link rel="stylesheet" href="highlight.css"> <link rel="stylesheet" href="tomorrow-night.css"> <link rel="stylesheet" href="ayu-highlight.css">
<!-- Custom theme stylesheets -->
</head> <body class="sidebar-visible no-js"> <div id="body-container"> <!-- Provide site root to javascript --> <script> var path_to_root = ""; var default_theme = window.matchMedia("(prefers-color-scheme: dark)").matches ? "navy" : "light"; </script>
<!-- Work around some values being stored in localStorage wrapped in quotes --> <script> try { var theme = localStorage.getItem('mdbook-theme'); var sidebar = localStorage.getItem('mdbook-sidebar');
if (theme.startsWith('"') && theme.endsWith('"')) { localStorage.setItem('mdbook-theme', theme.slice(1, theme.length - 1)); }
if (sidebar.startsWith('"') && sidebar.endsWith('"')) { localStorage.setItem('mdbook-sidebar', sidebar.slice(1, sidebar.length - 1)); } } catch (e) { } </script>
<!-- Set the theme before any content is loaded, prevents flash --> <script> var theme; try { theme = localStorage.getItem('mdbook-theme'); } catch(e) { } if (theme === null || theme === undefined) { theme = default_theme; } var html = document.querySelector('html'); html.classList.remove('light') html.classList.add(theme); var body = document.querySelector('body'); body.classList.remove('no-js') body.classList.add('js'); </script>
<input type="checkbox" id="sidebar-toggle-anchor" class="hidden">
<!-- Hide / unhide sidebar before it is displayed --> <script> var body = document.querySelector('body'); var sidebar = null; var sidebar_toggle = document.getElementById("sidebar-toggle-anchor"); if (document.body.clientWidth >= 1080) { try { sidebar = localStorage.getItem('mdbook-sidebar'); } catch(e) { } sidebar = sidebar || 'visible'; } else { sidebar = 'hidden'; } sidebar_toggle.checked = sidebar === 'visible'; body.classList.remove('sidebar-visible'); body.classList.add("sidebar-" + sidebar); </script>
<nav id="sidebar" class="sidebar" aria-label="Table of contents"> <div class="sidebar-scrollbox"> <ol class="chapter"><li class="spacer"></li><li class="chapter-item expanded affix "><a href="introduction.html">Introduction</a></li><li class="chapter-item expanded affix "><li class="part-title">Processes</li><li class="chapter-item expanded "><a href="process-run.html"><strong aria-hidden="true">1.</strong> Running a Process</a></li><li class="chapter-item expanded "><a href="process-output.html"><strong aria-hidden="true">2.</strong> Output</a></li><li class="chapter-item expanded "><a href="process-input.html"><strong aria-hidden="true">3.</strong> Input</a></li><li class="chapter-item expanded "><a href="process-stderr.html"><strong aria-hidden="true">4.</strong> Stderr and Error Handling</a></li><li class="chapter-item expanded affix "><li class="part-title">Input/Output</li><li class="chapter-item expanded "><a href="io/read.html"><strong aria-hidden="true">5.</strong> Reading Stuff</a></li><li class="chapter-item expanded "><div><strong aria-hidden="true">6.</strong> Writing Stuff</div></li><li class="chapter-item expanded affix "><li class="part-title">Higher Order Functions for AsyncIterable</li><li class="chapter-item expanded "><div><strong aria-hidden="true">7.</strong> Enumeration</div></li><li class="chapter-item expanded "><div><strong aria-hidden="true">8.</strong> Compatibility with Streams</div></li><li class="chapter-item expanded "><a href="performance.html"><strong aria-hidden="true">9.</strong> Performance</a></li><li class="chapter-item expanded affix "><li class="part-title">Concurrency and Parallel Processing</li><li class="chapter-item expanded affix "><li class="part-title">Miscellaneous</li><li class="chapter-item expanded "><a href="misc/sleep.html"><strong aria-hidden="true">10.</strong> Sleep</a></li><li class="chapter-item expanded "><div><strong aria-hidden="true">11.</strong> Range</div></li><li class="chapter-item expanded "><div><strong aria-hidden="true">12.</strong> Shuffle</div></li><li class="chapter-item expanded affix "><li class="part-title">Concepts</li><li class="chapter-item expanded "><a href="text-data.html"><strong aria-hidden="true">13.</strong> Working with Text Data</a></li><li class="chapter-item expanded "><a href="transform.html"><strong aria-hidden="true">14.</strong> Transformers</a></li><li class="chapter-item expanded affix "><li class="part-title">Examples</li><li class="chapter-item expanded "><div><strong aria-hidden="true">15.</strong> Embed a Bash Script</div></li><li class="chapter-item expanded "><a href="example-counting-words.html"><strong aria-hidden="true">16.</strong> Counting Words</a></li><li class="chapter-item expanded "><a href="example-concurrent-processing.html"><strong aria-hidden="true">17.</strong> Concurrent Processing</a></li><li class="chapter-item expanded "><a href="example-io.html"><strong aria-hidden="true">18.</strong> Input and Output</a></li></ol> </div> <div id="sidebar-resize-handle" class="sidebar-resize-handle"> <div class="sidebar-resize-indicator"></div> </div> </nav>
<!-- Track and set sidebar scroll position --> <script> var sidebarScrollbox = document.querySelector('#sidebar .sidebar-scrollbox'); sidebarScrollbox.addEventListener('click', function(e) { if (e.target.tagName === 'A') { sessionStorage.setItem('sidebar-scroll', sidebarScrollbox.scrollTop); } }, { passive: true }); var sidebarScrollTop = sessionStorage.getItem('sidebar-scroll'); sessionStorage.removeItem('sidebar-scroll'); if (sidebarScrollTop) { // preserve sidebar scroll position when navigating via links within sidebar sidebarScrollbox.scrollTop = sidebarScrollTop; } else { // scroll sidebar to current active section when navigating via "next/previous chapter" buttons var activeSection = document.querySelector('#sidebar .active'); if (activeSection) { activeSection.scrollIntoView({ block: 'center' }); } } </script>
<div id="page-wrapper" class="page-wrapper">
<div class="page"> <div id="menu-bar-hover-placeholder"></div> <div id="menu-bar" class="menu-bar sticky"> <div class="left-buttons"> <label id="sidebar-toggle" class="icon-button" for="sidebar-toggle-anchor" title="Toggle Table of Contents" aria-label="Toggle Table of Contents" aria-controls="sidebar"> <i class="fa fa-bars"></i> </label> <button id="theme-toggle" class="icon-button" type="button" title="Change theme" aria-label="Change theme" aria-haspopup="true" aria-expanded="false" aria-controls="theme-list"> <i class="fa fa-paint-brush"></i> </button> <ul id="theme-list" class="theme-popup" aria-label="Themes" role="menu"> <li role="none"><button role="menuitem" class="theme" id="light">Light</button></li> <li role="none"><button role="menuitem" class="theme" id="rust">Rust</button></li> <li role="none"><button role="menuitem" class="theme" id="coal">Coal</button></li> <li role="none"><button role="menuitem" class="theme" id="navy">Navy</button></li> <li role="none"><button role="menuitem" class="theme" id="ayu">Ayu</button></li> </ul> <button id="search-toggle" class="icon-button" type="button" title="Search. (Shortkey: s)" aria-label="Toggle Searchbar" aria-expanded="false" aria-keyshortcuts="S" aria-controls="searchbar"> <i class="fa fa-search"></i> </button> </div>
<h1 class="menu-title">proc</h1>
<div class="right-buttons"> <a href="print.html" title="Print this book" aria-label="Print this book"> <i id="print-button" class="fa fa-print"></i> </a>
</div> </div>
<div id="search-wrapper" class="hidden"> <form id="searchbar-outer" class="searchbar-outer"> <input type="search" id="searchbar" name="searchbar" placeholder="Search this book ..." aria-controls="searchresults-outer" aria-describedby="searchresults-header"> </form> <div id="searchresults-outer" class="searchresults-outer hidden"> <div id="searchresults-header" class="searchresults-header"></div> <ul id="searchresults"> </ul> </div> </div>
<!-- Apply ARIA attributes after the sidebar and the sidebar toggle button are added to the DOM --> <script> document.getElementById('sidebar-toggle').setAttribute('aria-expanded', sidebar === 'visible'); document.getElementById('sidebar').setAttribute('aria-hidden', sidebar !== 'visible'); Array.from(document.querySelectorAll('#sidebar a')).forEach(function(link) { link.setAttribute('tabIndex', sidebar === 'visible' ? 0 : -1); }); </script>
<div id="content" class="content"> <main> <h1 id="proc-0218"><a class="header" href="#proc-0218"><code>proc 0.21.8</code></a></h1><p><code>proc</code> let's you use child processes with<a href="https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/AsyncIterator"><code>AsyncIterable</code></a>instead of the streams API, and it includes a library of higher-order functionsfor <code>AsyncIterator</code> via<a href="https://deno.land/x/proc@0.21.8/mod.ts?s=Enumerable"><code>Enumerable</code></a> thatroughly matches what you can do with an array (<code>map</code>, <code>filter</code>, <code>find</code>), but forasynchronous code.</p><p><code>proc</code> simplifies the process of converting a <code>bash</code> script into a Denoapplication. The intention is to make writing code that uses lots of IO andchild processes <em>almost</em> as easy as shell scripting, but you also get propererror handling, type checking, and Deno's security-by-default.</p><p><a href="https://deno.land/x/proc@0.21.8/mod.ts">Developer Documentation</a></p><h2 id="usage"><a class="header" href="#usage">Usage</a></h2><pre><code class="language-typescript">import { run } from "https://deno.land/x/proc@0.21.8/mod.ts";</code></pre><h2 id="a-simple-example"><a class="header" href="#a-simple-example">A Simple Example</a></h2><p>Run <code>ls -la</code> as a child process. Decode <code>stdout</code> as lines of text. Print toconsole.</p><pre><code class="language-typescript">await run("ls", "-la").toStdout();</code></pre><h2 id="a-better-example"><a class="header" href="#a-better-example">A Better Example</a></h2><p>Don't worry about understanding everything in this example yet. This shows alittle of what is possible using <code>proc</code>.</p><p>Given the text for <em>War and Peace</em>:</p><ul><li>Read the file into an <code>AsyncIterable</code> of <code>Uint8Array</code>.</li><li>Uncompress it (the file is GZ'd).</li><li>Convert to lowercase using JavaScript, because the JavaScript conversion ismore correct than the one in <code>tr</code>.</li><li><code>grep</code> out all the words on word boundaries.</li><li><code>tee</code> this into two streams (<code>AsyncIterable</code> of <code>Uint8Array</code>) of words.<ul><li>Count the total number of words.</li><li>Use <code>sort</code> with <code>uniq</code> to count the unique words.</li></ul></li></ul><pre><code class="language-typescript">const [words1, words2] = read( fromFileUrl(import.meta.resolve("./warandpeace.txt.gz")),) .transform(gunzip) .lines .map((line) => line.toLocaleLowerCase()) .run({ buffer: true }, "grep", "-oE", "(\\w|')+") .tee();
const [uniqueWords, totalWords] = await Promise.all([ words1.run("sort").run("uniq").lines.count(), words2.lines.count(),]);
console.log(`Total: ${totalWords.toLocaleString()}`);console.log(`Unique: ${uniqueWords.toLocaleString()}`);</code></pre><p>Up to the point where we run <code>Promise.all</code>, this is asynchronous, streaming,lazily evaluated code. It is trivially running three child processes (<code>grep</code>,<code>sort</code>, and <code>uniq</code>), a <code>DecompressionStream</code> transform, and in-process logic tonormalize to lower-case. This is all happening concurrently, mostly in parallel,one buffer, one line, or one word at a time.</p><div style="break-before: page; page-break-before: always;"></div><h1 id="running-a-process"><a class="header" href="#running-a-process">Running a Process</a></h1><p><code>proc</code> lets you run a process from Deno with as little boilerplate as possible.</p><pre><code class="language-typescript">import { run } from "https://deno.land/x/proc@0.21.8/mod.ts";</code></pre><p>To <code>ls -la</code>:</p><pre><code class="language-typescript">await run("ls", "-la").toStdout();</code></pre><p>To capture the lines as an array:</p><pre><code class="language-typescript">const lines: string[] = await run("ls", "-la").lines.collect();</code></pre><h2 id="create-a-command-programmatically"><a class="header" href="#create-a-command-programmatically">Create a Command Programmatically</a></h2><pre><code class="language-typescript">import { Cmd, run } from "https://deno.land/x/proc@0.21.8/mod.ts";</code></pre><p>A command requires that the first parameter be defined, and that it be either astring or a URL. Additional parameters are string values. This doesn't quite fitthe signature of an array. Use<a href="https://deno.land/x/proc@0.21.8/mod.ts?s=Cmd">Cmd</a> as the type of the array.This can be spread into <code>run</code>.</p><pre><code class="language-typescript">// Assume options.all is a defined boolean.
const cmd: Cmd = ["ls"];if (options.all) { ls.push("-la");}
await run(...cmd).toStdout();</code></pre><p><em>The command array is type <code>Cmd</code>, not <code>string[]</code>. You need to declare thisexplicitly.</em></p><div style="break-before: page; page-break-before: always;"></div><h1 id="output"><a class="header" href="#output">Output</a></h1><p>Process standard output, or <em>stdout</em>, is an <code>AsyncIterable<Uint8Array></code>.</p><p>This can be efficiently piped to another process with <code>run</code>:</p><pre><code class="language-typescript">// Count the words and print the result.await run("echo", "Hello, world.").run("wc", "-w").toStdout();</code></pre><p>You can't assume much about the data you are receiving from a process. It may bewritten out line by line, or it may be in large or small chunks.</p><pre><code class="language-typescript">await run("echo", "Hello, world.").forEach((it) => console.dir(it));
// Uint8Array(14) [// 72, 101, 108, 108, 111,// 44, 32, 119, 111, 114,// 108, 100, 46, 10// ]</code></pre><p>That's not very useful. Let's try again, converting to text.</p><pre><code class="language-typescript">await run("echo", "Hello,\nworld.").lines.forEach((it) => console.dir(it));
// Hello,// world.</code></pre><p>To convert the lines to an array, <code>collect</code> them.</p><pre><code class="language-typescript">const data: string[] = await run("echo", "Hello,\nworld.").lines.collect();console.dir(data);
// [ "Hello,", "world." ]</code></pre><p>If you just want to dump the output from the child process to stdout, there isan easy way to do that.</p><pre><code class="language-typescript">await run("echo", "Hello, world.").toStdout();
// Hello, world.</code></pre><div style="break-before: page; page-break-before: always;"></div><h1 id="input"><a class="header" href="#input">Input</a></h1><p><code>proc</code> supports standard input (<em>stdin</em>) of processes as<code>AsyncIterable<Uint8Array | Uint8Array[] | string | string[]></code>. This means thatyou can pass in text data or byte data.</p><p>Note that for every <code>string</code> value (including each <code>string</code> in a <code>string[]</code>),<code>proc</code> will insert a line-feed character. This is not done for byte data in<code>Uint8Array</code> form, of course. If you need to use text data without the automaticline-feed characters, you will need to convert to bytes.</p><p><code>enumerate</code> is a wrapper function that creates an <code>AsyncIterable</code> withhigher-order functions. In the example, I am using it to iterate over a few<code>Uint8Array</code> instances that, together, spell out "Hello, world." This isproviding stdin to <code>wc -w</code>.</p><pre><code class="language-typescript">// Count the words in 'Hello, world."await enumerate([ new Uint8Array([72, 101, 108, 108, 111, 44, 32]), new Uint8Array([119, 111, 114, 108, 100, 46, 10]),]).run("wc", "-w").toStdout();
// 2</code></pre><p>This also works with text data. <code>proc</code> converts strings to bytes automatically.</p><pre><code class="language-typescript">await enumerate(["Hello, world."]).run("wc", "-w").toStdout();
// 2</code></pre><div style="break-before: page; page-break-before: always;"></div><h1 id="stderr-and-error-handling"><a class="header" href="#stderr-and-error-handling">Stderr and Error Handling</a></h1><p>Standard input and standard output from a process are handled directly asiterable data. There is a third data stream, standard error, that is a bit of anoutlier. Standard error is meant to be used either purely for error text fromthe process or for some combination of logging and errors.</p><p>We are going to discuss how to handle standard error and how this relates toerror handling in the <code>proc</code> library. There are examples if you want to skipahead.</p><p>Default behavior of <code>stderr</code> and errors:</p><ul><li>all process <code>stderr</code> will be written to <code>Deno.stderr</code></li><li>any exit code other than 0 will throw an<a href="https://deno.land/x/proc@0.21.8/mod.ts?s=ExitCodeError"><code>ExitCodeError</code></a></li><li>if the process ends due to a signal, it will throw a<a href="https://deno.land/x/proc@0.21.8/mod.ts?s=SignalError"><code>SignalError</code></a></li><li>an error coming from upstream (<code>stdin</code>) will be wrapped in an<a href="https://deno.land/x/proc@0.21.8/mod.ts?s=UpstreamError"><code>UpstreamError</code></a></li></ul><p>While the default behaviors are usually adequate, these can be overridden. Thereis no standard for standard error, so it may take some effort to get the resultsyou want.</p><h2 id="taking-control-of-stderr"><a class="header" href="#taking-control-of-stderr">Taking Control of Stderr</a></h2><p>You can capture stderr by defining <code>fnStderr</code> in the process options. Thisexample adds a timestamp and colors the stderr text red.</p><pre><code class="language-typescript">const decoratedStderr: ProcessOptions<void> = { fnStderr: async (stderr) => { for await (const line of stderr.lines) { console.error(`${gray(new Date().toISOString())} ${red(line)}`); } },};
await run( { ...decoratedStderr }, "bash", "-c", ` echo "This goes to stderr." >&2 echo "This goes to stdout." `,).toStdout();</code></pre><h2 id="reinterpreting-process-errors"><a class="header" href="#reinterpreting-process-errors">Reinterpreting Process Errors</a></h2><p>Catch and reinterpret exit code error, no stderr scraping.</p><h2 id="throwing-errors-based-on-stderr"><a class="header" href="#throwing-errors-based-on-stderr">Throwing Errors based on Stderr</a></h2><p>Scrape stderr to throw an error. Simple version. Mention the "contract" withprocess that all lines of stdout should be printed, or logged, or something -where ever you put it, make sure nothing gets dropped. So error goes at the end,once all lines have been processed.</p><h2 id="throwing-errors-based-on-stderr-advanced"><a class="header" href="#throwing-errors-based-on-stderr-advanced">Throwing Errors based on Stderr (Advanced)</a></h2><p>Scrape stderr to throw an error. Full version.</p><div style="break-before: page; page-break-before: always;"></div><h1 id="reading-data"><a class="header" href="#reading-data">Reading Data</a></h1><p><a href="io/"><code>enumerate</code></a> works with any iterable, including a <code>ReadableStream</code> (which is an <code>AsyncIterable</code>).</p><h2 id="reading-from-stdin"><a class="header" href="#reading-from-stdin">Reading from <code>stdin</code></a></h2><p>Deno provides <code>Deno.stdin.readable</code> which gives you a <code>stdin</code> as a <code>ReadableStream<Uint8Array></code>. We canwrap this with <code>enumerate(...)</code> to convert to lines of text (strings).</p><p>Text of <code>example.ts</code>:</p><pre><code class="language-typescript">import { enumerate } from "https://deno.land/x/proc@0.21.8/mod.ts";
for await (const line of enumerate(Deno.stdin.readable).lines) { console.log(line);}</code></pre><p>To print War and Peace, line by line, to console:</p><pre><code class="language-shell">zcat warandpeace.txt.gz | deno run example.ts</code></pre><p>This operation will consume <code>stdin</code> and close it.</p><div style="break-before: page; page-break-before: always;"></div><h1 id="performance"><a class="header" href="#performance">Performance</a></h1><p>A few notes on performance.</p><h2 id="does-performance-matter"><a class="header" href="#does-performance-matter">Does Performance Matter?</a></h2><p>For 90% of the code you write, the bottom line is that performance does notmatter. For example, if you have some code that reads configuration on startupand dumps it into an object, that code might be complicated, but it won't matterif it runs in 10 milliseconds or 100 nanoseconds. Write clear code first andoptimize once things are working. Follow this process, and you will quicklyfigure out which things do and don't matter.</p><h2 id="the-cost-of-iteration"><a class="header" href="#the-cost-of-iteration">The Cost of Iteration</a></h2><p>We use iteration everywhere. Doing it wrong can kill your performance. Doing itright can get you close to (single threaded) C performance. This is a quicksummary of what you can expect. To keep it short, I am just going to cover thehigh points and not show my work.</p><p>The fastest code you can write in pure JavaScript looks like<a href="https://en.wikipedia.org/wiki/Asm.js">asm.js</a>. If you stick to <code>for</code> loops thatcount and index simple types or data object lookups in arrays or numbers intyped-arrays (like <code>Uint8Array</code>), you can expect that code to run at or nearsingle-threaded C speed.</p><p>Expect <code>for...of</code> with iterables and generators to be about 10x slower. Thisincludes array methods like <code>map</code>, <code>filter</code>, and <code>reduce</code>. Anything that has tocall a function in a loop is going to have extra overhead.</p><p>Promise-driven asynchronous code is another 10x slower, or 100x slower than the<code>asm.js</code>-style code. This affects code written using <code>proc</code>, particularly<code>Enumerable</code>.</p><p>So does this mean you have to always use <code>asm.js</code> syntax? Not at all. <code>for...of</code>syntax and array methods make for cleaner code, and asynchronous operations arethe whole reason we're here. Iteration performance is mostly about the innerloops. If your inner loops are tight, a little less efficiency in the outerloops won't matter much. Write clean code first. When things are working, lookfor opportunities to make it faster. Often this will mean a little profiling andrewriting a few routines in <code>asm.js</code> style. If you do it right, you should beable to get very good performance along with readable code.</p><p><a href="https://medium.com/netscape/async-iterators-these-promises-are-killing-my-performance-4767df03d85b">Async Iterators: These Promises Are Killing My Performance!</a>on Medium and supporting benchmarks in<a href="https://github.com/danvk/async-iteration">async-iteration</a> on Github.</p><p><a href="https://madelinemiller.dev/blog/javascript-promise-overhead/">The Performance Overhead of JavaScript Promises and Async Await</a>shows a couple of examples that isolate the performance difference to overheaddue to promises.</p><div style="break-before: page; page-break-before: always;"></div><h1 id="sleep"><a class="header" href="#sleep"><a href="https://deno.land/x/proc@0.21.8/mod.ts?s=sleep">sleep</a></a></h1><p><code>sleep</code> returns a <code>Promise</code> that resolves after a specified number ofmilliseconds.</p><pre><code class="language-typescript">console.log("Program starts");await sleep(2000); // Pauses the execution for 2000 millisecondsconsole.log("Program resumes after 2 seconds");</code></pre><div style="break-before: page; page-break-before: always;"></div><h1 id="working-with-text-data"><a class="header" href="#working-with-text-data">Working with Text Data</a></h1><p>Streaming data doesn't have to be line-delimited text, but it probably will bemost of the time. Many *nix tools work with this type of data or some variationof it.</p><p>Line-delimited text data is simply:</p><ul><li><code>utf-8</code> encoded bytes</li><li>logically separated into lines with <code>\n</code> or alternately <code>\r\n</code> (Windows style)characters</li></ul><p>Here is how you process text data in <code>proc</code>.</p><h2 id="utf-8-lines"><a class="header" href="#utf-8-lines"><code>UTF-8</code> Lines</a></h2><p>This is the "normal" way to work with line-delimited text. It should be a goodsolution most of the time.</p><p>The<a href="https://deno.land/x/proc@0.21.8/mod3.ts?s=ProcessEnumerable#accessor_lines">lines</a>method converts a line at a time.</p><pre><code class="language-typescript">await run("ls", "-la") .lines .forEach((it) => console.log(it));</code></pre><p>Alternately you can use<a href="https://deno.land/x/proc@0.21.8/mod3.ts?s=Enumerable#method_transform_0">transform</a>with the <a href="https://deno.land/x/proc@0.21.8/mod3.ts?s=toLines">toLines</a>transformer function.</p><pre><code class="language-typescript">await read(resolve("./warandpeace.txt.gz")) .transform(toLines) .forEach((it) => console.log(it));</code></pre><p>The<a href="https://deno.land/x/proc@0.21.8/mod3.ts?s=Enumerable#method_run_0">Enumerable.run</a>method will automatically treat <code>string</code> values as lines, adding <code>\n</code> to themand converting back into <code>utf-8</code> encoded bytes.</p><p>Note that this <em>always</em> assumes string data passed to it is line-delimited. Ifthat isn't the case (you may be working with buffered text data that is notdelimited at all, for example), you <strong>must</strong> convert text data back to<code>Uint8Array</code> yourself or <code>\n</code> characters will be added.</p><h2 id="traditional-text-and-lines"><a class="header" href="#traditional-text-and-lines">Traditional Text and Lines</a></h2><p>Deno provides a native<a href="https://deno.land/api?s=TextDecoderStream">TextDecoderStream</a> to bulk-convert<code>Uint8Array</code> data into text. The boundaries are arbitrary. The characters willalways be correct, but this can break within a word or within acontrol-character sequence. <code>TextDecoderStream</code> supports many standard characterencodings.</p><p>To parse this data into lines, Deno provides<a href="https://deno.land/std/streams/mod.ts?s=TextLineStream">TextLineStream</a>. Thissplits the data into lines on <code>\n</code> and optionally <code>\r</code>.</p><p>These are meant to be used together to convert to text then split into lines.</p><p>The traditional stream implementation is a little slower than the<code>utf-8</code>-specialized transformers, but they support different character encodingsand allow some flexibility in defining the split.</p><pre><code class="language-typescript">await read(resolve("./warandpeace.txt.gz")) .transform(gunzip) .transform(new TextDecoderStream()) .transform(new TextLineStream()) .map((line) => line.toLowerCase()) .forEach((line) => console.log(line));</code></pre><p>Note that most of the library assumes strings and arrays of strings representline data. For text that is not divided on lines, you can use<a href="https://deno.land/api?s=TextEncoderStream">TextEncoderStream</a> to convert backto <code>utf-8</code> bytes. Note that unlike <code>TextDecoderStream</code> this does not supportmultiple encodings. This is in line with the official specification.</p><h2 id="not-all-text-data-is-text-data"><a class="header" href="#not-all-text-data-is-text-data">Not All Text Data is Text Data</a></h2><p>There are many command-line utilities that use ANSI color and positionsequences, as well as raw carriage-returns (<code>\r</code>) to enhance the user experienceat the console. The codes are normally interpreted by the terminal, but if youdump them to file, you can see they make a mess. You've probably seen this inlog files before.</p><p>This type of streamed text data can't be strictly interpreted as lines. You maybe able to hack around the fluff. Use<a href="https://deno.land/std/fmt/colors.ts?doc=&s=stripColor">stripColor</a> (Deno <code>std</code>library) to remove ANSI escape codes from strings. If the utility is using raw<code>\r</code>, you may have to deal with that as well.</p><p>The best solution is to turn off color and progress for command-line utilitiesyou use for processing. This is not always possible (Debian <code>apt</code> is a famousexample of this).</p><p>Reference the <a href="https://en.wikipedia.org/wiki/ANSI_escape_code">ANSI escape code</a>wiki page.</p><p>You can <em>always</em> get around this problem by never attempting to split on lines.</p><pre><code class="language-typescript">await run("apt", "install", "build-essential") .writeTo(Deno.stdout.writable, { noclose: true });</code></pre><div style="break-before: page; page-break-before: always;"></div><h2 id="transformers"><a class="header" href="#transformers">Transformers</a></h2><p><code>proc</code> ships with some useful<a href="https://deno.land/x/proc@0.21.8/src/transformers.ts">transformers</a>.</p><p>A transformer is a <em>plain-old JavaScript function</em> with this signature:</p><pre><code class="language-typescript">type Transformer<T, U> = (it: AsyncIterable<T>) => AsyncIterable<U>;</code></pre><p>Transformers are functions (and may be defined using asynchronous generatorfunctions). You can compose them into new functions relatively easily. The<a href="https://deno.land/x/proc@0.21.8/mod3.ts?s=Enumerable#method_transform_0">transform</a>operation is like <code>pipeThrough</code> in streaming.</p><p>A transformer transforms objects from one type to another. It is like <code>map</code> butwith with complete control over the whole stream of data - including controlover error handling.</p><p>You can create a transformer using an asynchronous generator. This one willtransform strings to lower-case:</p><pre><code class="language-typescript">async function* toLower(texts: AsyncIterable<string>) { for await (const text of texts) { yield text.toLocaleLowerCase(); }}</code></pre><p>Here it is in action:</p><pre><code class="language-typescript">const lowered = await enumerable(["A", "B", "C"]) .transform(toLower) .collect();
assertEquals(lowered, ["a", "b", "c"], "Transformed to lower-case.");</code></pre><div style="break-before: page; page-break-before: always;"></div><h1 id="counting-words"><a class="header" href="#counting-words">Counting Words <!-- omit from toc --></a></h1><ul><li><a href="example-counting-words.html#direct-translation-from-bash">Direct Translation from Bash</a></li><li><a href="example-counting-words.html#embedding-a-shell-script">Embedding a Shell Script</a></li><li><a href="example-counting-words.html#doing-all-the-work-in-deno">Doing All the Work in Deno</a><ul><li><a href="example-counting-words.html#transformer-for-unique-words">Transformer for Unique Words</a></li><li><a href="example-counting-words.html#transformer-to-split-into-words">Transformer to Split into Words</a></li><li><a href="example-counting-words.html#putting-it-all-together">Putting It All Together</a></li></ul></li></ul><p>This shell script counts total and unique words:</p><pre><code class="language-shell">#!/bin/bashset -e
# total word countzcat ./warandpeace.txt.gz \ | tr '[:upper:]' '[:lower:]' \ | grep -oE "(\\w|'|’|-)+" \ | wc -l
#count unique wordszcat ./warandpeace.txt.gz \ | tr '[:upper:]' '[:lower:]' \ | grep -oE "(\\w|'|’|-)+" \ | sort \ | uniq \ | wc -l</code></pre><p>There are multiple approaches to doing the same thing in Deno using <code>proc</code>. Youcan run this in-process as a pure Typescript/JavaScript solution, run it as ashell script, or translate each command in the shell script into <code>run</code> methods.</p><blockquote><p>⚠️ The <code>tr</code> used to convert to lowercase <strong>is not</strong> fully unicode compliant.Expect counts to be a little different between this code and the code thatuses JavaScript's <code>.toLocaleLowercase()</code>, which <strong>is</strong> fully unicodecompliant.</p></blockquote><h2 id="direct-translation-from-bash"><a class="header" href="#direct-translation-from-bash">Direct Translation from Bash</a></h2><p>This is the equivalent to the shell script using <code>proc</code> methods. Thissubstitutes <code>gunzip</code> for <code>zcat</code>, translates each output to a number, and runsthe operations concurrently (and in parallel) - since that is easy to do.Otherwise it is doing exactly the same thing.</p><p>Otherwise, this is a direct translation where <code>proc</code> just controls the streamingfrom process to process. All the same child processes are being launched.</p><pre><code class="language-typescript">const [total, unique] = await Promise.all([ read(fromFileUrl(import.meta.resolve("./warandpeace.txt.gz"))) .run("gunzip") .run("tr", "[:upper:]", "[:lower:]") .run("grep", "-oE", "(\\w|'|’|-)+") .run("wc", "-l") .lines .map((n) => parseInt(n, 10)) .first,
read(fromFileUrl(import.meta.resolve("./warandpeace.txt.gz"))) .run("gunzip") .run("tr", "[:upper:]", "[:lower:]") .run("grep", "-oE", "(\\w|'|’|-)+") .run("sort") .run("uniq") .run("wc", "-l") .lines .map((n) => parseInt(n, 10)) .first,]);
console.log(total);console.log(unique);</code></pre><h2 id="embedding-a-shell-script"><a class="header" href="#embedding-a-shell-script">Embedding a Shell Script</a></h2><p>Another approach is to embed a shell script. No translation required here. Thisis a <code>bash</code> script run using <code>/bin/bash</code>. This moves the entire workload and itsmanagement into other processes. Consider this solution if your application isdoing lots of other things concurrently.</p><p>Note that you give up some control over error handling with this approach, so besure to test for the types of errors you think you may encounter. Shell scriptsare notorious for edge-case bugs - which is why we reach for a "real"programming language when things start to get complex.</p><p>This is also a simple example of a generated script. We are injecting the fullpath of our text file as determined by the Deno script.</p><p>This example shows the total count.</p><pre><code class="language-typescript">await run( "/bin/bash", "-c", ` set -e zcat "${fromFileUrl(import.meta.resolve("./warandpeace.txt.gz"))}" \ | tr '[:upper:]' '[:lower:]' \ | grep -oE "(\\w|'|’|-)+" \ | wc -l `,) .lines .forEach((line) => console.log(line));</code></pre><h2 id="doing-all-the-work-in-deno"><a class="header" href="#doing-all-the-work-in-deno">Doing All the Work in Deno</a></h2><p>This is a streaming solution staying fully in Deno, in a singleTypescript/JavaScript VM (not using child processes at all). The avoids (mostof) the memory overhead that would be needed to process the document in memory(non-streaming), and it is fast.</p><p>This demonstrates <em>transformer-composition</em> in <code>proc</code>. Because transformers arejust functions of iterable collections, you can compose them into logical unitsthe same way you would any other code.</p><h3 id="transformer-for-unique-words"><a class="header" href="#transformer-for-unique-words">Transformer for Unique Words</a></h3><p>We could shell out to <code>sort</code> and <code>uniq</code>, but this way is much faster. It onlyneeds a little extra memory. It dumps the words, one at a time, into a <code>Set</code>.Then it yields the contents of the <code>Set</code>.</p><p>The set of unique words is much smaller than the original document, so thememory required is quite small.</p><pre><code class="language-typescript">export async function* distinct(words: AsyncIterable<string>) { const uniqueWords = new Set(); for await (const word of words) { uniqueWords.add(word); } yield* uniqueWords;}</code></pre><h3 id="transformer-to-split-into-words"><a class="header" href="#transformer-to-split-into-words">Transformer to Split into Words</a></h3><p>Convert each line to lower case. Use <code>Regex</code> to split the line into words.Remove anything without a character (all symbols), anything with a number, and"CHAPTER" titles. The symbol characters in the regular expression are specificto the test document and probably won't work generally.</p><p>The document we are targeting, <code>./warandpeace.txt.gz</code>, uses extended unicodeletters and a few unicode symbols as well. We know that the Typescript solutionbelow works correctly with unicode characters (note the <code>u</code> flag on the regularexpression). Some of the *nix utilities were written a long time ago and stilldo not support unicode. In particular, <code>tr</code> does not translate case correctlyall of the time, and I am not sure what <code>grep</code> is doing - it sort of works, butthe regular expression language has subtle differences to what I am used to. Abenefit of working in a tightly spec'd language like Typescript is you know whatyour code should be doing at all times. The counts are very close, but they arenot exactly the same, so we know something is a little bit off with <code>tr</code> and/or<code>grep</code>.</p><pre><code class="language-typescript">export function split(lines: AsyncIterable<string>) { return enumerate(lines) .map((it) => it.toLocaleLowerCase()) .flatMap((it) => [...it.matchAll(/(\p{L}|\p{N}|['’-])+/gu)] .map((a) => a[0]) ) .filterNot((it) => /^['’-]+$/.test(it) || /[0-9]/.test(it) || /CHAPTER/.test(it) );}</code></pre><h3 id="putting-it-all-together"><a class="header" href="#putting-it-all-together">Putting It All Together</a></h3><p>Read the file. Uncompress it and convert to lines (<code>string</code>). Use thetransformer function we created earlier, <code>split</code>, to split into words.</p><pre><code class="language-typescript">const words = read( fromFileUrl(import.meta.resolve("./warandpeace.txt.gz")),) .transform(gunzip) .transform(toLines) .transform(split);</code></pre><p>Now we need to get (1) a count of all words and (2) a count of unique words. Wecan use <code>tee</code> to create two copies of the stream - since we have to count twice.This gets around the limitation of being able to use an iterable only once andmeans we don't have to do extra work splitting the document into words twotimes.</p><pre><code class="language-typescript">const [w1, w2] = words.tee();</code></pre><p>We can count the words in the first copy directly. For the second copy, we usethe <code>distinct</code> transformer before counting.</p><pre><code class="language-typescript">const [count, unique] = await Promise.all([ w1.count(), w2.transform(distinct).count(),]);
console.log(`Total word count: ${count.toLocaleString()}`);console.log(`Unique word count: ${unique.toLocaleString()}`);</code></pre><p>The results:</p><pre><code>Total word count: 563,977Unique word count: 18,609</code></pre><p>Clean, readable code. Understandable error handling. Fast. The only downside isthat the processing is done in-process (we only have one thread to work with inJavaScript). If you are doing other things at the same time, this will slow themdown.</p><div style="break-before: page; page-break-before: always;"></div><h1 id="concurrent-processes"><a class="header" href="#concurrent-processes">Concurrent Processes</a></h1><p><code>proc</code> supports concurrent operations with controlled (limited) concurrency.This is a way to run child processes in parallel without swamping your server.</p><p>If you have to work with S3 buckets, you know it is time consuming to determinehow much storage space you are using/paying for, and where you are using themost storage. <code>proc</code> makes it possible to run <code>ls --summarize</code> with parallelismmatching the number of CPU cores available (or whatever concurrency youspecify). The specific methods that support concurrent operations are<a href="https://deno.land/x/proc@0.21.8/mod3.ts?s=Enumerable&p=prototype.concurrentMap">.concurrentMap()</a>and<a href="https://deno.land/x/proc@0.21.8/mod3.ts?s=Enumerable&p=prototype.concurrentUnorderedMap">.concurrentUnorderedMap()</a>.</p><p>To list the <code>s3</code> buckets in your AWS account from terminal:</p><pre><code class="language-sh">aws s3 ls</code></pre><p>The result looks something like this:</p><pre><code>2013-07-11 17:08:50 mybucket2013-07-24 14:55:44 mybucket2</code></pre><p>Get all the bucket names in the account:</p><pre><code class="language-typescript">const buckets = await run("aws", "s3", "ls") .map((b) => b.split(/\s+/g, 3)) .map((b) => b[b.length - 1]) .collect();</code></pre><p>This is the shell command to get the total storage size in bytes from terminal:</p><pre><code class="language-shell">aws s3 ls s3://mybucket --recursive --summarize</code></pre><p>This will list all objects in the bucket, and we can ignore most of this. At theend of the operation, we are looking for a line that looks like this:</p><pre><code>Total Size: 2.9 MiB</code></pre><p>This is potentially a long-running operation (some buckets have a lot ofobjects), so we want to run this for many buckets at once, in parallel, andreport the results as soon as they are available.</p><pre><code class="language-typescript">enumerate(buckets).concurrentUnorderedMap( async (bucket) => { const answer: string = await run( "nice", "-19", "aws", "s3", "ls", `s3://${bucket}`, "--recursive", "--summarize") .filter(line => line.includes("Total Size:")) .map(line => line.trim()) .first;
return {bucket, answer}; }.forEach(({bucket, answer}) => console.log(`${bucket}\t${answer}`)))</code></pre><p>Use <code>nice</code> because <em>this will eat your server otherwise.</em> The method<a href="https://deno.land/x/proc@0.21.8/mod3.ts?s=Enumerable&p=prototype.concurrentUnorderedMap">.concurrentUnorderedMap()</a>will, by default, run one process for each CPU available concurrently until allwork is done.</p><p>The result will look something like this:</p><pre><code>mybucket Total Size: 2.9 MiBmybucket2 Total Size: 30.2 MiB</code></pre><div style="break-before: page; page-break-before: always;"></div><h1 id="input-and-output"><a class="header" href="#input-and-output">Input and Output</a></h1><p>Use <code>ReadableStream</code> and <code>WritableStream</code> from Deno APIs for input and output.</p><h2 id="write-to-stdout"><a class="header" href="#write-to-stdout">Write to Stdout</a></h2><p>Write to <code>stdout</code> a line at a time using <code>console.log</code>.</p><pre><code class="language-typescript">await range({ to: 3 }) .forEach((line) => console.log(line.toString()));</code></pre><p>Write to <code>stdout</code> as a <code>WritableStream</code>. In the case of <code>stdout</code>, we don't closeit. To use <code>writeTo</code>, the data has to be in <code>Uint8Array</code> form. This also addsoutput buffering to consolidate the write operations into larger chunks.</p><p><code>Deno.stdout.writable</code> is a <code>WritableStream</code>.</p><pre><code class="language-typescript">await range({ to: 10000 }) .map((n) => n.toString()) .transform(toBytes) .transform(buffer(8192)) .writeTo(Deno.stdout.writable, { noclose: true });</code></pre><p>Run a child process and stream output directly to <code>stdout</code>. This has noconversion to lines and no additional buffering, so it will also work with ANSIescape codes and positioning characters.</p><pre><code class="language-typescript">await run("ls", "-la") .writeTo(Deno.stdout.writable, { noclose: true });</code></pre><h2 id="read-from-stdin"><a class="header" href="#read-from-stdin">Read from Stdin</a></h2><p>Read <code>stdin</code>. Uncompress it and convert to lines (<code>string</code>). Remove all theblank lines. Count them. Print the count.</p><p><code>Deno.stdin.readable</code> is a <code>ReadableStream</code> which is an<code>AsyncIterable<Uint8Array></code>.</p><pre><code class="language-typescript">console.log( await enumerate(Deno.stdin.readable) .transform(gunzip) .lines .filter((line) => line.trim().length === 0) .count(),);</code></pre>
</main>
<nav class="nav-wrapper" aria-label="Page navigation"> <!-- Mobile navigation buttons -->
<div style="clear: both"></div> </nav> </div> </div>
<nav class="nav-wide-wrapper" aria-label="Page navigation">
</nav>
</div>
<script> window.playground_copyable = true; </script>
<script src="elasticlunr.min.js"></script> <script src="mark.min.js"></script> <script src="searcher.js"></script>
<script src="clipboard.min.js"></script> <script src="highlight.js"></script> <script src="book.js"></script>
<!-- Custom JS scripts -->
<script> window.addEventListener('load', function() { window.setTimeout(window.print, 100); }); </script>
</div> </body></html>