Pagecraft

Recipes

Several operations in one pass, over many files, with a manifest saying what came from what.

A Recipe is a list of operations applied in order. It is data — a name and some steps — so it can be written down, saved, shared, checked into a repository and run from a shell script, and none of that needs any code.

{
  "name": "Tidy an invoice",
  "steps": [
    { "operation": "ocr", "options": { "languages": ["eng"] }, "weight": 8 },
    { "operation": "compress", "options": { "mode": "quality", "imageQuality": 70 } },
    { "operation": "page-numbers", "options": { "template": "{n} / {total}" } }
  ]
}

pagecraft recipe tidy.json 'invoices/*.pdf' --output out/

Forty invoices go in. Each one is read, compressed and numbered, and out comes forty files and a manifest.json saying which came from what.

Why it is not just a shell loop

Three reasons, and the third is the one that matters.

One mechanism does the row and the column. A step that takes one file at a time is applied to each file in turn; a step that takes several — merge — receives what the step before it produced, all of it. So [merge, page-numbers] over forty files is one numbered bundle, and [ocr, compress] over forty files is forty tidied invoices, and neither needs a mode of its own: how the files are shared out between steps is decided by the Operations themselves (ADR-0015).

Nothing touches a disk in between. Each step’s Artifacts are opened as the next step’s Documents in memory. Running it in a browser tab is the same run as running it in a shell, which is what lets the web app offer it at all.

One failure is one file’s failure. A run over forty invoices where the seventeenth will not open hands back thirty-nine files and a manifest entry saying what happened to the seventeenth, at which step, with the remedy. The alternative — a loop that stops, or one that swallows the error and leaves you counting files — is how a batch job quietly loses a document.

From code

import { buildRecipe, runRecipe } from '@pagecraft/core';

const recipe = buildRecipe({
  name: 'Tidy an invoice',
  steps: [
    { operation: 'ocr', options: { languages: ['eng'] }, weight: 8 },
    { operation: 'compress', options: { mode: 'quality', imageQuality: 70 } },
  ],
});

const { artifacts, manifest } = await runRecipe(recipe, documents, {
  capabilities: { recognizer, rasterizer },
  onProgress: ({ value, message }) => bar.set(value, message),
});

buildRecipe is where a recipe is checked, and it is checked before anything runs: that every operation exists, that its options are valid, and that each step can actually read what the step before it produced. A recipe whose second step reads PDFs after a first step that produces images is refused at build time, naming both steps — rather than forty files in, on the file that got there first.

recipeFromJson does the same from the text of a file.

The manifest

manifest.entries;
// [{ name: 'invoice-01-numbered.pdf', kind: 'pdf', byteLength: 20481,
//    from: ['invoice-01.pdf'] }, …]

manifest.failures;
// [{ documents: ['invoice-17.pdf'], step: 1, operation: 'ocr',
//    code: 'CORRUPT_DOCUMENT', message: '…', remedy: '…' }]

entries runs in step with artifacts, so a file and its account of itself are the same index. A zip of thirty PDFs whose names were decided four steps ago is otherwise a puzzle, which is why the archive carries the manifest inside it:

import { archiveRecipeRun } from '@pagecraft/core';

const zip = archiveRecipeRun({ artifacts, manifest });

On the command line you get the same thing without asking. Given a directory with --output, a run leaves the files and a manifest.json beside them; written to standard output or to one filename, it is a zip holding both.

Weights, and why a progress bar needs them

A recipe that reads a scan and then numbers its pages spends almost all of its time in the first step. A bar divided evenly between the two sits at nothing for a minute and then jumps to the end, which is worse than no bar at all.

Nothing can work out that ratio by itself, so the recipe says it:

{ "operation": "ocr", "weight": 8 }

Weights are relative and default to 1. They affect nothing but the bar.

Saved recipes on the site

The recipe surface on the web app builds the same JSON, from the same registry, with the same check that each step can follow the last. A recipe saved there is a list of steps rather than a file — the documents are yours and stay yours — and it can be exported, checked into a repository, and run by the CLI without change.

What a recipe cannot do

  • Branch. There is no condition, no loop and no variable. A recipe is a list. Anything that needs a decision in the middle is a script that calls the engine twice, and that is the honest shape for it.
  • Reach outside itself. A step cannot name a file that was not given to the run, or write one anywhere but into the run’s own results.
  • Undo a step. Each step reads what the last one produced. If step two rasterizes, step three is working on pictures, and the text is gone — buildRecipe will tell you so, but it will not stop you meaning it.

Where to go next