ERP & Automation
From Inbox to ERP: Anatomy of a Document Pipeline
This article is for the people who have to build it or own it: IT leadership, ERP managers, developers. It describes the architecture of a document pipeline — PDF document in, validated ERP record out — as concretely as possible without a codebase. For the management perspective, see the guide to document automation; this is about the how.
The central design statement up front: the language model is the smallest component of the system. Anyone who understands “AI extraction” as an API call with a PDF in the prompt is building a demo. What makes the system production-grade is everything around the model — deterministic routing, provenance, validation, escalation, and feedback.
Three design principles
1. Minimally predetermined. The only things defined up front are the target schema (the fields of the ERP document model) and universal invariants (arithmetic, required fields). Everything layout- and sender-specific is recognized at runtime — no templates, no per-sender rules. Every sender-specific special case you avoid is maintenance saved over the system’s entire lifetime.
2. The model transcribes, it doesn’t calculate. The LLM transfers only values that appear verbatim in the document. Derived quantities — net prices, line totals, checksums — are computed by deterministic code. Language models are strong readers and unreliable calculators; this division of labor exploits both properties.
3. The quality target is “zero undetected errors,” not “100% accuracy.” Raw accuracy of 100% is unattainable with free-form layouts (here’s why). The architecture is therefore designed to detect every doubtful value and route it out — not to claim infallibility.
Stage 0 — Identity and routing
Before any model runs, every incoming file gets an identity: its SHA-256 hash. Checked against a unique column in the target database, this settles the duplicate question structurally — the same purchase order, emailed twice or forwarded again, is flagged as a duplicate before any compute is spent and before a double import can happen.
Then a file router branches by input type: already structured inputs — CSV attachments, genuine e-invoices (XRechnung, ZUGFeRD XML) — are parsed deterministically and mapped straight onto the target schema. No LLM, no uncertainty. The AI path is reserved exclusively for what actually needs it: unstructured PDFs. This switch sounds trivial and is one of the most effective decisions in the architecture — it keeps the probabilistic share of the system as small as possible.
Stage 1 — Parsing: layout to Markdown, with provenance
A layout parser (proven choice: Docling, open source) converts the PDF into structured Markdown — tables as tables, reading order resolved, on CPU in seconds. For born-digital PDFs, the normal case in B2B traffic, OCR is skipped entirely: the existing text layer is read directly, which is faster and less error-prone than any recognition step.
The Markdown is stored in full — as the provenance anchor. Every value extracted later must be verifiable against this raw text; in addition, the original text line is carried along for each line item. Without this anchoring, grounding (Stage 3) is impossible and every debugging session becomes archaeology.
What awaits the parser in real-world documents is easy to underestimate. Four recurring traps: item descriptions spanning three lines (item line, size line, EAN line — one position, three table rows); page breaks in the middle of the line-item table, complete with a carry-over row that looks like a line item but isn’t; two “Total” rows — with and without VAT — of which exactly one belongs in the target field; and Swiss number formats with apostrophes as thousands separators (1’234.56). A parser that preserves table structure defuses half of these cases; the mapping has to catch the rest with precise rules.
Stage 2 — Mapping: two schemas, constrained decoding, rules with reasons
The core of this stage is a deliberate separation into two schemas with different roles:
- The extraction schema is what the LLM sees. It contains only fields that appear verbatim in the document — header data, line items, and the source line for each item. Every field is nullable: what isn’t in the text is null, not guessed.
- The validation and database model is never seen by the LLM. This is where Decimal amounts, computed fields, confidence score, and status live — assigned by the pipeline, never by the model.
The output is enforced via constrained decoding: a grammar generated from the schema restricts token selection, so the model cannot produce invalid JSON. Two practical subtleties: first, the grammar guarantees syntax only, never values — it replaces not a single check. Second, only the final output should be constrained; forcing a model to also think in grammar degrades results (known in the literature as the alignment tax).
For the system prompt, one simple discipline applies: every rule exists because of a concrete, observed error. Typical examples: “Total is the final amount excluding VAT — if ‘Total excl. VAT’ appears in the text, take exactly that value” (because the model otherwise grabs the gross total); “Numbers as plain JSON numbers, strip apostrophe thousands separators”; “Take dates verbatim, don’t reformat” (normalization is done by deterministic code, which can throw errors instead of papering over them). Prompts built this way stay short, justified, and testable — prompt folklore has no place in a pipeline.
Stage 3 — Validation: the part that creates the trust
This is where “extracted” becomes “verified.” Three families of checks, all deterministic, all in ordinary, unit-testable code:
- Arithmetic with Decimal and explicit tolerances. Quantity × unit price, discounts included, must yield the line total; the line totals must yield the document total. Calculations use Decimal (never float), with defined rounding and two tolerances: one rounding unit per line, slightly more cumulatively at header level — real documents round per line. These invariants make the most common error class — digit errors in numeric fields — mathematically visible.
- Grounding against the source. Every value must be locatable in the stored raw text; the source line carried along per item is re-anchored against the original (fuzzy match on the raw text, not trust in the model’s hypothesis). What cannot be verified counts as not extracted. This catches hallucinations even where arithmetic has no grip.
- Master data and consistency checks. Item numbers against the item master (note: your own numbering scheme — the sender’s customer-facing numbers vary and are carried along, but not enforced against the master), partners against master data, currency, date logic, completeness of the line-item count.
From the results, the pipeline assigns a score and a status — for instance: auto-processed, review required, failed, duplicate. Crucially: self-healing only with evidence. If the system corrects a value on its own (say, via the arithmetic), the correction is accepted only if the corrected value is verifiable in the source text — otherwise the case goes to a human. A system that “repairs” its own errors with plausible guesses is more dangerous than one that reports them.
Stage 4 — Escalation: vision and reasoning as the exception, not the default
When checks fail, the pipeline escalates in stages — and only now do the heavier tools come in: a vision model reads documents with a broken or missing text layer (scans, photos, exotic PDF generators) directly from the image. A reasoning pass resolves ambiguous cases — think freely first, then map the result into the schema as a separate step.
Why not do it this way from the start? Because both do more harm than good on the standard path: the mapping task is transcription, not inference — reasoning on numeric fields increases the tendency to hallucinate, costs a multiple in latency, and sits poorly with constrained decoding. Vision is needlessly expensive for born-digital PDFs with a clean text layer and harder to audit than text parsing. Escalation is the place for heavy artillery; the standard path stays lean, fast, and traceable.
Stage 5 — A human in the loop who learns
The remaining cases land in a review interface: document on the left, extracted and validated data on the right, deviations highlighted, correction in seconds. Two properties determine the value of this stage: first, it must be pre-filled — review means checking, not data entry. Second, every correction flows back into the pipeline as an example (few-shot examples in the prompt, additions to the test set) — the review rate is not a constant but a falling curve.
Evaluation and operations: without a gold set, everything is anecdote
Measurement happens against a gold set: real documents with manually defined expected output, built in two waves (≈10 documents for the pipeline logic, ≈100 for threshold calibration) — necessarily including held-out senders never seen during tuning, because the real question is generalization to the unknown. Useful metrics: field accuracy, row-level line-item accuracy (the weak spot of every extraction), document exact match, and the straight-through rate. And one test that must never be missing: a deliberately injected error has to be caught by the checks — a validation layer that never fires is decoration.
In operation, quality is monitored per sender, but never routed per sender: routing decisions are made on signals alone (confidence, validation and grounding results), while sender labels make visible in monitoring whether someone is systematically performing worse. That keeps the system template-free without being blind.
On the deployment side, the whole thing is unspectacular — and that’s the compliment: an inference server (vLLM or Ollama) with the mapping model, workers for parsing/pipeline/validation, a queue, a review UI; all as Docker Compose, GPU passthrough via the NVIDIA Container Toolkit. On hardware of the DGX Spark class (128 GB unified memory), the mapping and vision models stay resident simultaneously — escalations don’t wait for model swaps. The hardware and sovereignty questions in detail are beyond the scope of this article.
Conclusion
A production-grade document pipeline is 20 percent model and 80 percent system architecture: deterministic routing, provenance from the first byte, a tight extraction schema, validation with mathematics and source grounding, escalation driven by signals, and a review path that learns. None of it is magic — it’s solid software engineering around a probabilistic component. And that’s exactly why it works: the model is allowed to be wrong, and the system notices.
kitun builds pipelines like this as custom ERP modules — on-premise, with open-weights models, gold-set evaluation, and no per-document pricing. Architecture questions about your own document intake: happy to discuss in a 20-minute call.