ADR-0014: Handwriting is read by a multimodal model the workspace chooses, behind one interface

Status: accepted · 2026-09-20

Context

The scan plan (2026-09-20) asked for a photographed handwritten page to become a Markdown page, keeping the formatting a person wrote on paper — # headings, - lists, [ ] tasks, **bold**, tables ruled by hand — with the photo straightened and shrunk on the device before upload.

Three questions had cheap answers and right ones.

What reads the page. OCR engines do not read handwriting: Tesseract is printed text only; the classical pipeline (TrOCR handwritten behind a line segmenter) returns lines with no layout and no markers. What reads a notebook page and returns structure — a heading is a heading because it is bigger and alone on its line — is a multimodal model. In 2026 that is Gemini 2.5 Flash, or an open model such as Qwen2.5-VL behind vLLM, at a fifth of a cent to a cent a page.

Whose model. Bladbase could hold one key and pay for every page. That makes inference a cost of the product, a secret to rotate, and a single provider every customer is bound to — and a consumer Gemini subscription, which many people already pay for, exposes no API at all.

Where the photo is corrected. OpenCV.js is the obvious document scanner in a browser. Its embind layer calls new Function at load, which the app’s Content-Security-Policy forbids (ADR on the CSP hardening: script-src carries no 'unsafe-eval', and will not for a 12 MB dependency).

Decision

One Reader interface (packages/ai), three implementations, chosen by the workspace rather than by the install — a Gemini API key the workspace or the person brought, Gemini on Vertex AI in a self-hosted install’s own project, or any OpenAI-compatible endpoint — with one transcription prompt and one parser that are Bladbase’s. Bladbase never holds the key.

• Settings → AI has three states: configured (the workspace’s own provider), members may use their own key (the default), and off, which beats every personal key. A personal key is used only for jobs the person starts, never for another member’s. Resolution — workspace key → the plan’s included allowance → the starter’s own key → none — is one pure function, shown on the Scan screen before a photo is taken and recorded on the page (page.scan.keySource).
• Keys are validated on save by one small call and stored encrypted under the same key as the Drive tokens; the settings page shows the last four characters and never the key.
• The prompt keeps, never writes: markers as written, order as written, no correction, [?] for an unreadable word. A second prompt (clean up) resolves [?] against the image and normalises markers as a revision.
• The device corrects the photo in plain canvas JS: the sheet as the largest bright region, its corners as that region’s extreme points, a homography to flatten it, grey with the contrast stretched, 2 200 px on the long edge, JPEG q0.85. No OpenCV, no WebAssembly, no CSP change.
• Usage is counted per month per key (aiUsage/{workspaceId}/{YYYY-MM}), in one unit, AI pages; the hosted service’s included allowance is a plan limit (workspace.aiPagesPerMonth) reserved up front and settled after, so two jobs cannot both squeeze under a cap.

Assumptions

• A workspace that wants this has, or can get, a Gemini API key — free tier included — or runs its own project. “Bring your own key” is acceptable to the teams Bladbase is for.
• Gemini 2.5 Flash’s handwriting reading stays good enough that the open route is a choice, not a necessity.
• The plain-JS sheet detection handles the common case — a lighter sheet on a darker surface — and the draggable corners handle the rest.

Invalidation triggers

• Customers refusing to bring a key in numbers, and the included allowance becoming the only path used — then Bladbase is the provider after all and the pricing model has to say so.
• A provider’s terms changing so that photos sent under a customer’s own key are not handled as the settings page says.
• Sheet detection failing on most real photos (the spike on real notebook pages is the check).

Blast radius

• packages/ai (the interface, the prompt), packages/domain/src/ai.ts (settings, resolution, the unit), apps/worker/src/scan.ts, the Scan screen and Settings → AI, the plan limit and the operator console.

Alternatives rejected

• A Bladbase key for everyone. A cost, a secret and a lock-in, in that order.
• OAuth sign-in to the Gemini API. Needs a GCP project with quota; a developer flow, not “log in with my account”.
• OCR engines (Tesseract, TrOCR + Kraken). No handwriting, or no structure.
• OpenCV.js in the browser. Refused by the CSP; not worth 'unsafe-eval'.
• Reading on the device. No on-device model reads handwriting well enough yet; the plan’s later native-app section keeps the option.

Escape plan

Two-way door. A fourth reader is a class; a Bladbase-paid default is the included provider turned on by a plan; the prompt is one file.

Consequences

• The hosted service pays for nothing unless the operator sets an included key, and then only what the plans allow.
• Every scanned page says which model and whose key read it.
• The scan and metered plans’ proof against real notebook pages depends on a key in Settings → AI on the Bladbase workspace, which Dustin holds.

What happened

• 2026-09-21 — first real page. A phone photo of handwritten Stake Conference notes, on the workspace’s own Gemini key (gemini-3.8-flash): read in 27 seconds, heading and speaker names kept, every bullet and a nested one in order, no [?], nothing invented. The reading-quality assumption held on the first try. The sheet was found without dragging corners.
• The same page then lost its text — not to the reader. The collab service had not been deployed with the new details node, so the editor’s room could not load; the editor opened empty and, on leaving, saved that emptiness over the page. The revisions kept the transcription. Two fixes: collab deployed, and the editor now stays read-only until the room’s document arrives and never saves from a room it did not sync with (77d8c57). A lesson for the blast radius: the editor schema is shared with collab — a new node ships to all four services or to none.