An invoice extraction can return perfectly valid JSON and still be unusable. The supplier may be wrong, a page may be missing, or the line items may not explain the total. Before syncing AI-extracted invoices to an ERP or accounting system, validate the document, the critical fields, and the resulting business record separately.
OCR, or optical character recognition, turns visible text into machine-readable text. An extraction model goes further by assigning meaning to fields such as invoice number, due date, and total. Neither step should silently decide that a document is approved for payment.
A useful workflow lets software propose the data, runs explicit checks, and gives reviewers the evidence needed to resolve exceptions quickly.
The real question: why is the JSON plausible but wrong?
In an r/ollama discussion about local invoice extraction, u/burzaiscoming described a system handling varied PDFs, scans, multipage invoices, and separate files that could belong to one invoice. The reported errors included misread product names, missing tax identifiers, inconsistent totals, and ambiguous extra charges.
One reply from u/HotEstablishment7184 recommended preserving page-level evidence and treating historical supplier data as a reasonableness check rather than a substitute for the document. A second commenter, who disclosed an evaluation-product affiliation, recommended field-level testing against labelled examples.
This is a small discussion, not a measured accuracy study. It nevertheless identifies a concrete design problem: clean output formatting can hide unreliable values. The post's local-processing preference also matters; a proposed architecture should respect where the business permits its documents to be processed.
Define what “correct” means before choosing a model
List the fields your downstream process actually needs, how they should be represented, and what happens when they are missing. Different systems may require different date formats, supplier identifiers, or line-item structures.
Separate raw evidence from normalized values. Preserve the text as printed while storing a parsed date or decimal amount alongside it. That makes it possible to investigate whether a mistake came from reading the document or transforming the extracted value.
Do not let “required” mean “the model must fill it somehow.” A required field that cannot be established should stop the affected operation or route it to review. An invented invoice number is worse than a visible missing-value exception.
Have the person responsible for the business process approve these rules. The developer can implement validation, but should not guess which discrepancies the business is willing to accept.
Check the document and its pages first
Establish whether the input is an invoice, receipt, statement, purchase order, credit note, or another document. Similar layouts can contain very different business meanings. A statement listing several invoices should not automatically become one new invoice.
Track the original file, page count, and any page grouping. If three attachments might belong to one invoice, require evidence for the grouping rather than joining files because they arrived in the same email. Keep the original ordering and identifiers available for review.
Inspect image quality before adding model complexity. Amazon Textract's guidance recommends high-quality images, ideally at least 150 DPI, and warns against unnecessary conversion or downsampling of already supported documents. That is product guidance, not a guarantee of accuracy at a particular resolution. Amazon Textract best practices
A missing second page cannot be repaired by a better prompt. The appropriate result may be to request the complete document.
Keep field evidence beside the proposed value
For critical fields, retain the source page and the relevant text or image region where the extraction system makes that available. A reviewer should be able to inspect the invoice total without scrolling through every attachment.
Microsoft's Document Intelligence invoice model, for example, extracts key fields and line items from supported invoice documents into structured JSON. Its documentation describes inputs including scanned documents, phone images, and digital PDFs. These are product capabilities, not independent proof of your own extraction accuracy. Microsoft invoice-model documentation
The review interface should distinguish missing, uncertain, and conflicting values. A blank field because it is not printed is different from a field that could not be read. A model suggestion that conflicts with a supplier record needs a different explanation again.
If an extraction route cannot provide a precise source region, make that limitation visible. Do not manufacture a page reference merely to make the result appear traceable.
Match the supplier without guessing
Resolve the document's supplier to the correct internal supplier record using approved identifiers and matching rules. A similar trading name is only one signal. Two suppliers can have similar names, and one supplier may use different layouts or addresses.
Keep the proposed match separate from the extracted document fields. If the match is uncertain, reviewers should see the candidates and the reason for uncertainty before a record is created.
Treat new bank details or other consequential changes as a separate verification workflow. A document extraction should not silently overwrite an established supplier record simply because the latest attachment contains a different value. Preserve the existing approval process for such changes.
Historical data can flag an unusual value, but it should not rewrite the source to make it look familiar. If the document and history disagree, retain both and investigate.
Reconcile the arithmetic the invoice actually shows
Check whether line amounts, discounts, shipping, tax, and other printed adjustments explain the displayed total. Use the document's structure rather than assuming every supplier uses the same formula.
Here is a hypothetical example. Two line amounts total 240.00. The document shows a 20.00 discount, a 10.00 delivery charge, and 29.90 tax, producing 259.90. If the extraction reports a total of 289.90, the discrepancy should be visible.
Those numbers illustrate reconciliation only; the tax amount is stipulated for the example, not advice about a jurisdiction's tax treatment. Your accounting rules and rounding tolerances should come from the responsible business owner.
A matching total does not prove every field is correct. Two offsetting extraction errors can still sum to the expected number, and the supplier or invoice identifier can remain wrong. Use arithmetic as one check among several.
When values disagree, do not ask the model to invent the missing amount that makes the equation balance. Show the discrepancy and its source evidence to the reviewer.
Make confidence scores earn their role
A confidence score is useful only if you understand what it refers to and how it behaves on your documents. A page-level score, a word-level score, and a model's self-reported certainty are not interchangeable.
Amazon's guidance says thresholds should reflect the application and sensitivity to errors. It does not establish one universal cutoff that makes every invoice safe. Test candidate thresholds against a reviewed sample and inspect the errors that still pass.
For example, a very clear but wrong supplier match may receive high extraction confidence because the printed name was read correctly. The failure is in entity matching, not OCR. Likewise, a confidently read total does not show that every page arrived.
Route on explicit reasons as well as scores: missing page, unresolved supplier, mismatched total, duplicate candidate, or unsupported currency. That gives reviewers a useful explanation and helps developers identify which part of the system needs improvement.
Bring representative invoices and the steps your team performs after reading them. We can map extraction, validation, review, and integration around the actual work.
Understand what an invoice benchmark does and does not prove
The 2026 Invoice Haystack preprint introduces 1,500 anonymized invoice images and 200 question-answer pairs to study retrieval among visually similar documents. Its abstract reports 60.0% Recall@1 for the proposed method on its Invoice Haystack-500 evaluation. Invoice Haystack paper
That is a document-retrieval result, not an invoice-field accuracy rate or a payment-approval success rate. It is useful here because it highlights a separate challenge: finding the right invoice among similar-looking records before answering a question about it.
Do not apply that percentage to your extraction pipeline. If your workflow processes one uploaded invoice at a time, its task differs from searching a large collection. If it retrieves supporting invoices from an archive, evaluate that retrieval step separately.
A benchmark is informative when its task and denominator are clear. A vendor's single “accuracy” number should prompt the same questions.
Build a pilot that includes the inconvenient documents
Use a permitted sample from the real workflow. Include common suppliers, new suppliers, clean PDFs, poor scans, multipage documents, revisions, credit notes, and inputs with missing information. Keep document handling within the business's access and processing requirements.
Have reviewers establish reference values before using the sample to compare systems. Resolve disagreements in the reference set; otherwise, the evaluation may score the same correct extraction differently depending on who reviews it.
Keep some documents out of prompt tuning and rule development. The final evaluation should include examples the implementation was not adjusted to memorize.
Measure critical fields individually and record whether the whole document can proceed. A system can read hundreds of line-item descriptions correctly while getting the one total or supplier identifier that controls the transaction wrong.
Use a scorecard with explicit denominators
Suppose a hypothetical pilot has 100 invoices. Each has five critical fields, so there are 500 critical-field checks. If 490 are correct, field accuracy is 98%. If the ten errors occur on ten different invoices, only ninety invoices have all five critical fields correct.
Neither result tells you whether the exception rules caught those ten invoices. That is a third question. A useful scorecard separates:
| Metric | What it tells you |
|---|---|
| Correct critical fields / checked critical fields | Field-level extraction quality |
| Fully correct documents / reviewed documents | Document-level completeness |
| Erroneous documents flagged / erroneous documents | How often review rules catch errors |
| Correct documents unnecessarily flagged / correct documents | Avoidable review burden |
| Review minutes / processed documents | Human effort introduced by the workflow |
| Incorrect records reaching the destination | Failure of the end-to-end controls |
The numbers above are illustrative, not claimed production results. Repeat the scorecard by document type so strong performance on clean PDFs does not conceal weak performance on scans.
Prevent a second import of the same invoice
An exact file fingerprint can identify a repeated attachment, but it will not necessarily identify a rescan or reformatted copy of the same invoice. Combine file identity with an approved business-level match, such as supplier plus invoice identifier.
Define how revisions, credit notes, and reused numbering patterns should be handled. An invoice number alone may not be unique across suppliers. A filename is even weaker because email systems and users can rename attachments.
Also test retries after an uncertain write. If the ERP created the draft but the integration lost the response, repeating creation can produce another draft. Store the destination record ID and reconcile the outcome before retrying.
The same integration principle appears in preventing recurring CRM duplicates. Repeated delivery should produce one intended business result, with uncertainty visible rather than silently duplicated.
Design the review queue as part of the product
Show reviewers the reason an invoice stopped, the original evidence, the proposed correction, and any related record. Let them correct a field without retyping the entire invoice, while preserving who changed it and why.
Give unresolved cases an owner. A queue that grows invisibly can delay work even when extraction is fast. Track its age and distinguish missing-document requests from cases awaiting an internal decision.
Feed reviewed errors back into evaluation. If a supplier changes its layout, add representative examples and verify that the fix does not harm other suppliers. Do not reduce review thresholds solely to make the queue look smaller.
Calculate net effort using the same document mix before and after automation. Include reviewing, correcting, chasing missing pages, resolving duplicates, and maintaining the integration. Faster extraction is valuable only if the complete workflow improves.
Connect the destination after the checks work
Begin with reviewed draft records and verify the field mapping, attachments, supplier relationship, and destination status. Preserve the established approval process. Extraction permission should not automatically confer authority to approve or pay.
The existing guide to reconciling CRM invoices with QuickBooks covers a related downstream problem. This validation checklist addresses the earlier point where document evidence becomes structured data.
The right first milestone is a small, representative batch that reaches the destination correctly, with exceptions explained and review effort measured. Once that works, expand deliberately. Reliable invoice automation is a complete path from source document to verified record, not a JSON response that merely looks finished.