Building AI systems for business documents and knowledge

Key takeaways

Task scope: Extracting specified fields, answering questions and completing a packet review require different evidence and different success criteria.
Evidence coverage: A relevant passage can support an answer. Establishing that no amendment or exception was missed requires a defined source collection and a coverage procedure.
Document meaning: Preserve document versions, page locations, hierarchy, table headers, units and effective dates throughout processing. A valid output schema does not establish correct extraction.
Evaluation and economics: Measure complete task outcomes, unresolved cases and reviewer correction effort. Parser scores and the cost of a model call describe only parts of the system.

Organizations use documents to record commitments, define procedures and preserve decisions. Turning that material into a useful AI system requires deciding what an output must establish. A searchable passage, an extracted amount and a recommendation to an approver carry different evidential obligations.

Consider a supplier review. An assistant might reproduce a maintenance fee exactly and cite its location, yet use an agreement that a later amendment changed. It might apply the amendment before its effective date, overlook a required attachment or disclose an internal policy to someone without access. Each failure can survive a demonstration in which the answer looks plausible and the citation opens correctly.

The implementation has to preserve meaning from the original document through extraction, evidence selection and the final review. This paper explains how to choose among ordinary search, structured extraction, full-document context and retrieval-assisted generation, and how to evaluate the result at each stage.

What must the output establish?

Specified-field extraction asks for an identified value: a policy number, invoice date or recurring fee. The field definition must resolve questions such as whether “amount” means a monthly charge, a subtotal or a tax-inclusive total. Extracting a policy number says little about interpreting the policy’s coverage or exceptions.

Question answering asks the system to identify and combine information relevant to a question. The answer may depend on several documents, but the user may only need a particular procedure or explanation. Complete packet review adds another obligation: determine whether the required material is present and examine all applicable requirements and exceptions within the agreed scope.

Task	Evidence obligation	Evaluation that matches the task
Specified-field extraction	Recover the requested value with its unit, period and source location.	Required-field accuracy and recall; wrong units, dates and missing outputs.
Collection question answering	Assemble sufficient authorized evidence and produce a supported answer.	Evidence retrieval, answer correctness, citation support and appropriate abstention.
Complete packet review	Account for expected documents, governing versions and every defined review requirement.	Missing-document detection, required-evidence coverage, conflict handling and correct disposition or escalation.

Table 1. Proposed evaluation targets for three document tasks. These are design distinctions, not reported benchmark results.

Finding no matching passage is an observation about a search. A conclusion that no qualifying exception exists needs a stronger basis: which documents belong to the case, which versions apply, what was successfully processed and what review requirements were examined. Even a full-packet review remains bounded by the packet definition. It cannot certify the absence of an undisclosed agreement outside the accessible record.

Running example, entirely fictional: Alder Facilities reviews a supplier’s maintenance agreement, a later amendment and an internal approval policy. The task is to identify the recurring fee applicable on a specified date and determine the required internal review route. The example’s dates, amounts, document references and policy rules are invented. Its scope is administrative routing, not a legal opinion or a reconstruction of client work.

Establish that task with the people who prepare and use the review. Inspect a previously accepted output and a case returned for missing evidence. Ask which decision the recipient must make, which omissions would change it and who can resolve a missing or conflicting source. Those answers define the review questions and completion criteria before a parser or retrieval method is selected.

The packet inventory comes from Alder’s document custodian, independently of search results. It identifies the executed agreement, applicable amendments and approval policy expected for the review. This inventory establishes the expected document set; the review requirements establish what must be checked within it. Both can themselves be incomplete, so the resulting coverage statement must identify their scope and date.

When should the system search, extract, read the full packet or use tools?

Ordinary search and structured extraction are useful starting points when users need original documents or consistently defined values. A searchable repository may satisfy “find the maintenance agreement.” Validated field records and a database query may satisfy recurring reporting. When an authoritative source already exposes the required structured value, parsing its PDF rendering adds another opportunity for error. Generated prose earns its place when interpretation or synthesis is part of the task.

Bounded full-document context is a candidate for a small, known packet. Supplying all of Alder’s relevant documents avoids selecting only a few passages before interpretation and keeps nearby definitions and exceptions available. The input budget must accommodate instructions, documents, images where needed and the output. If material is truncated or replaced by summaries, the system is no longer examining the complete representation it claims to examine.

Fitting the material into context does not establish that the model used it successfully. The 2024 study “Lost in the Middle” found that the tested models’ performance varied with the position of relevant information in long inputs. That historical finding motivates testing evidence position and packet length with the models actually selected for the application.

Retrieval-augmented generation, or RAG, selects source material before generating an answer. The original RAG paper by Lewis and colleagues combines a retriever with a generator for knowledge-intensive NLP tasks. For business systems, retrieval makes it possible to select evidence from a changing collection without sending the whole collection into each request. It also creates a selection failure: material omitted before generation cannot support the answer through that input.

Multi-step tool use is justified when the information need depends on an earlier result. The system might identify an agreement, request its registered amendments, then open the policy applicable to the review date. Known dependencies can use a prescribed sequence. Less predictable questions may warrant model-selected searches, with limits and an explicit unresolved outcome. Additional steps add latency and more opportunities to search the wrong scope; they do not establish completeness by themselves.

These alternatives can coexist. Alder might retrieve the correct supplier packet from a large repository, then read that bounded packet in full and calculate annualized fees in code. Selection depends on collection size and change rate, context requirements, permissions, response time and the evidence a reviewer must inspect. An interactive procedure lookup and an overnight packet review need not use the same architecture.

How can processing preserve the meaning of a document?

Ingestion should record both document identity and processing status. Keep the source-system identifier, immutable version identifier, content fingerprint, acquisition time and declared document role. Preserve the original file or an authorized reference to it. Distinguish the time a file entered the system from its signature date, effective date and period of application. A content hash can identify identical bytes; it cannot establish that a document is authoritative.

Profile the input before interpreting it: native text, scanned pages, mixed content, tables, languages, attachments and unreadable or protected sections. A failed attachment or an unprocessed page belongs in the case’s status. Silently continuing with the remaining text changes the evidence available to every downstream stage.

Recovering text may involve native PDF extraction, OCR, specialized layout models or a vision-language model reading page images. The December 2024 Docling Technical Report, describing Docling v1, illustrates a linear pipeline containing learned layout and table models, optional OCR and typed document assembly. A fixed execution sequence can therefore contain uncertain predictions. Model-free extraction can also fail through encoding, ordering or structural errors.

Five properties need separate attention. Page layout locates regions. Reading order determines their sequence. Logical hierarchy determines which heading or list governs a passage. Table topology identifies rows, columns and spanning cells. Field correctness asks whether the requested business value was recovered with the right meaning. Correctness at one level does not establish correctness at the next.

In Alder’s hypothetical amendment, a table cell contains 2.8 under a heading stating “Monthly fee, USD thousands.” The normalized value is USD 2,800 per month. Retaining the number while losing the heading produces a structurally plausible but commercially wrong result. A neighboring annual-total column, a footnote limiting the period or a header continued from a previous page can change the interpretation just as materially.

Store the raw value, normalized value, unit, period and source reference together. Keep the transformation inspectable: 2.8 x 1,000 = 2,800. If the unit or period is unclear, record that ambiguity rather than silently selecting a convention. Arithmetic can be executed reliably once its operands and meaning have been established; correct multiplication cannot repair a wrongly selected fee.

Segmentation should preserve those relationships. A passage may need its section heading, a definition elsewhere, or a table’s row and column headers. Small chunks can lose governing context; large chunks can add irrelevant material and consume the input budget. A practical option is to index small units with links to their parent section and expand them when assembling evidence. Explicit agreement-to-amendment links are useful when the task follows known document relationships. A model-extracted relationship remains a candidate until checked against its source.

Every transformation should retain a route back to the original page and region, with printed page labels distinguished from file page numbers. A summary should point to the passages it summarizes. Several summaries derived from one source are still one evidential origin.

flowchart TD
    S["Versioned source documents"] --> P["Text, layout and tables"]
    P --> F["Full packet context"]
    P --> I["Scoped retrieval"]
    F --> E["Evidence assembly"]
    I --> E
    Q["Inventory, identity and review date"] --> I
    Q --> F
    Q --> E
    E --> O["Facts, conflicts and gaps"]
    O --> R["Reviewer disposition"]
    S -. "Original page and region links" .-> O

Figure 1. Proposed processing and evidence flow for the fictional review. Full-packet context and retrieval are alternative, combinable routes. The original source remains available for checking the resulting claims; each processing stage can introduce errors.

Control flow, reproducibility, schema validity, empirical accuracy and authorization answer separate questions. A schema can require a date without proving that it is the applicable date. Repeating a run can reproduce an error. Accurate extraction does not authorize the recipient to see the result.

How should retrieval find evidence without defining its own completeness?

Lexical retrieval such as BM25 is useful for distinctive identifiers and phrases. Dense retrieval represents queries and passages as vectors, enabling matches across different wording. The Dense Passage Retrieval study documents this distinction in its open-domain QA experiments. Its combined lexical-and-dense results also vary across datasets, supporting hybrid retrieval as an alternative to test rather than a guaranteed improvement.

Hybrid search combines candidate sets. Reciprocal rank fusion, for example, combines ranking positions without assuming that lexical and vector scores use comparable scales. A subsequent reranker assesses candidates more closely. In “Passage Re-ranking with BERT”, the model processes the query and candidate passage together. This spends more computation on a smaller set, but cannot recover an amendment absent from that set.

Query transformation can help when a user’s language differs from the documents. It can also change the question. Preserve the original request, identifiers, dates and constraints while examining whether expansion improves required-evidence coverage. A transformation of “current recurring charge” must not discard “current” or replace the named supplier with a broader topic.

Retrieval for a complete review needs both relevance and enumeration. Fetch registered attachments and amendments by their relationships, then examine each review requirement across that set. Record which documents and requirements were covered, which failed and which remain unresolved. Increasing the number of retrieved passages alone does not provide this account.

The same distinction applies to organizational knowledge. At fictional Alder, “How do I request a procurement exception?” can be answered from the applicable procedure. “Which suppliers currently require an exception?” requires an appropriately scoped supplier inventory and application of the rule across it. A handful of relevant passages is insufficient evidence for a complete list.

The search infrastructure should support these requirements: metadata filtering, document relationships, update handling and the required latency. The number of stored vectors alone does not determine whether a database is suitable.

Which sources and permissions govern the answer?

Authority depends on the proposition being assessed. An executed supplier amendment can establish a changed fee. Alder’s internal policy establishes its approval route. A supplier’s promise cannot grant an employee internal approval authority, and an internal policy does not by itself change the supplier’s contractual terms.

Represent source status, effective intervals and supersession relationships explicitly where the records permit it. File modification time is inadequate as a precedence rule. If two apparently authoritative versions disagree and the available record does not resolve their relationship, preserve the conflict for the document owner or reviewer.

The evidence set must also respect the requesting user’s current entitlements. Apply authorization before source content enters a reranker, model context or user-visible result. Carry access metadata onto chunks and other derived records, and authorize subsequent source opening and cached-answer reuse. A full-document input needs the same controls as retrieval.

Permission freshness requires separate attention. Microsoft’s Azure AI Search access-control documentation, inspected October 19, 2026, describes preview token-based document controls that enforce access against permission metadata stored in the index. Keeping those permissions current depends on the documented source-specific synchronization mechanism. This implementation illustrates why authenticating a user on every request does not alone establish that the indexed access rules are current.

A system requiring rapid access revocation must define how it detects a permission change, prevents stale disclosure and verifies completion. Where synchronized permissions can lag beyond the accepted limit, use a current authorization check or withhold the affected material. Such controls establish application behavior, not legal compliance.

Deletion has a similar propagation problem. Microsoft’s Azure Storage indexer documentation distinguishes automatic change detection from explicitly configured deletion handling. More generally, account for searchable chunks, embeddings, cached responses and other retained derivatives. Historical evidence retained under a separate policy needs its own access and retention controls; ordinary retrieval should not silently resurrect withdrawn content.

Operational traces can become another copy of the documents. The OpenTelemetry project’s May 2026 observability walkthrough explains that captured prompts, outputs and tool content can contain sensitive data. Retain the evidence needed for diagnosis and review deliberately, with content minimization, access controls and a defined retention period.

Finally, document content is evidence, not authority to rewrite application instructions. An uploaded page telling an AI reader to ignore policy or export the packet must not acquire those powers. AgentDojo’s 2024 experiments study instructions injected into untrusted tool-returned data. The design implication is to separate source content from authorized instructions and restrict tools to the information task. Delimiters or an injection detector can assist, but do not establish a guarantee.

How does the review handle success, temporal applicability and missing sources?

For the ordinary fictional review, Alder’s custodian identifies three applicable sources:

Agreement A1, Schedule 1, page 8: a recurring monthly fee of USD 2,400, effective January 1, 2026.
Executed amendment M1, section 2, page 2: signed June 20, effective July 1, 2026; replaces that fee with USD 2,800 per month.
Internal policy P3, section 4.2, page 3: effective January 1, 2026; annualized recurring fees of at least USD 30,000 require procurement review. For this example, annualization means the applicable monthly fee multiplied by 12.

Successful review as of September 1. The extracted facts include both fees, their dates and the policy threshold. The supported interpretation is that M1’s fee applies: 2,800 x 12 = 33,600, triggering procurement review. No source gap remains within the stipulated inventory and narrow task. The authorized reviewer verifies the evidence and records “Route to procurement.” This disposition neither approves the supplier nor settles unrelated agreement terms.

Temporal applicability as of June 25. M1 is signed and available, but its fee is not yet effective. For a question about the charge applicable on June 25, A1 supplies USD 2,400 and an annualized amount of USD 28,800. The reviewer records that the threshold is not met on that date and identifies the July 1 change. If the question instead concerns a commitment beginning in July, the later fee governs the example’s calculation. The requested date and decision horizon determine which interpretation is responsive.

Unavailable source in a separate September 1 variant. The custodian’s inventory also lists an executed amendment M2, effective August 15, whose content is unavailable to the system. The accessible documents still support what A1 and M1 said. They no longer establish the current fee. The output preserves those historical facts, marks the current-fee interpretation unresolved and identifies the missing evidence at a disclosure level the user is permitted to see. The reviewer holds the routing determination pending authorized access or custodian clarification.

If the source is restricted, even its title or existence may be sensitive. A permitted message might simply state that the accessible material is insufficient. If conflicting executed copies are available, both should be surfaced with the unresolved authority question. Neither the latest upload nor the most confidently extracted value settles the conflict.

What does a citation establish for a reviewer?

A useful review artifact separates extracted facts, supported interpretations, unresolved questions and the reviewer’s disposition. Each consequential statement should identify the exact source version and passage, including any table heading or footnote needed to understand it. Calculated values should show their operands and calculation. Interpretations requiring several sources should identify all of them.

The ALCE study, “Enabling Large Language Models to Generate Text with Citations”, evaluates answer correctness separately from citation quality. Its citation-recall measure asks whether generated statements are supported by their cited passages. That measures support for what the answer says; it does not measure whether the answer omitted a governing amendment or exception.

Review therefore has several distinct checks: correct transcription, sufficient evidence coverage, support for each statement, resolution of conflicts and the appropriate business disposition. A citation makes a claim inspectable. Its usefulness depends on whether it opens the right version, locates the relevant material and exposes the context needed to assess it.

An answer can be correct without establishing how the model internally computed it. A generated rationale, a retrieved passage and an execution trace offer different forms of evidence about the system. For this task, the practical requirement is an externally checkable result and its source dependencies.

Abstention should specify the missing condition when disclosure permits: an unreadable fee schedule, an unresolved effective date or an unavailable amendment. It should also preserve facts that remain supported. A blanket refusal can discard useful work; a confident completion can conceal the gap that the reviewer needs to resolve.

How should evaluation distinguish component quality from complete review?

Build evaluation cases around the document families and tasks the system will encounter. Include native and scanned files, mixed layouts, continued tables, attachments, revisions and poor images in proportions reflecting expected work. Add targeted cases for material exceptions, missing pages, contradictory versions, permission changes and source withdrawal. Report these deliberate stress cases separately from the representative sample.

Reference answers should be constructed from the original sources independently of the system’s output. Reviewers identify required fields, evidence, governing versions, legitimate unresolved states and the expected disposition. Adjudicate disagreements and preserve genuine ambiguity. DocVQA provides a concrete precedent: a second annotator answered document-image questions without seeing the first answers, with mismatches subsequently reviewed.

For synthetic cases, define the facts, relationships, effective dates and expected outcomes before rendering documents. Then vary layout, scan quality and wording against that reference. This makes a lost header or missed amendment diagnosable. It does not establish how frequently those failures occur in incoming business material. Keep related templates, organizations and document versions together when separating development from held-out evaluation, so near-duplicates do not make the test artificially easy.

Different scores answer different questions. DocVQA’s answer-string similarity metric tolerates minor transcription differences. The PubTables-1M paper distinguishes table topology, content and location through its GriTS metrics. FinQA supplies financial questions with supporting facts and numerical programs, assessing results separately from program equivalence. These are useful component or task references; none evaluates Alder’s complete supplier review.

A one-character difference in an identifier, date or fee can matter greatly despite a high text-similarity score. A correct table grid can contain the wrong period’s value. Conversely, a calculation can produce the expected number from inappropriate operands. Business-field evaluation should check normalized values, units, periods and sources, using tolerances justified by the task rather than a general similarity threshold.

Compare search with manual review, field extraction with explicit calculations, bounded full-packet context and retrieval-assisted interpretation on the same assigned cases. Require each approach to meet the same task obligation and accepted final outcome. Include the human interpretation, coverage checks and corrections each approach still requires. Processing the same documents is insufficient if one approach returns extracted values and another completes the review. Compare component outputs separately when their intended purposes differ.

Add reranking or multi-step retrieval when an observed failure makes its contribution testable. Keep model, parser, prompt, chunking and index configurations identifiable. Diagnostic runs using reference transcription or reference evidence can help distinguish parsing, retrieval and interpretation failures.

The denominator must begin with all assigned cases. Include ingestion failures, timeouts, malformed outputs, omissions and unnecessary abstentions. Separately count correct abstention on genuinely unresolved cases. Build role-specific references so withholding restricted information is scored correctly and unauthorized disclosure is a failure. For evidence retrieval, measure recovery of all required evidence in the reference, not merely whether one answer-bearing passage appeared. For packet review, report wrong dispositions and missed critical exceptions alongside field-level results. A weighted aggregate can hide a failure that should block use.

Report counts and uncertainty by important document family and failure type. Repeated runs on selected cases can reveal instability. Independently assess the completed human-plus-system review as well as the machine output, including residual mistakes after review. Model-based graders may help triage, but require validation against domain reviewers for the judgments entrusted to them.

When do review effort, latency and change alter the design?

The relevant operating cost includes ingestion, processing, indexing, retrieval, generation, retries, maintenance and human review. At a stated cutoff, divide total cost across all assigned cases by the number correctly completed to obtain cost per correct completion. Use that same total divided by all assigned cases for cost per assigned case. Report both measures with completion and unresolved counts at the same cutoff, including costs incurred on failed, abandoned and still-open cases. If no case is correctly completed, report that count and the incurred cost without a finite cost-per-completion figure.

Completion depends on the defined task. Holding Alder’s routing determination when M2 is unavailable is correct handling of insufficient evidence. It leaves the fee determination and underlying business review unfinished. Count that hold as correct handling and an unresolved business case; a handoff to another reviewer does not complete the end-to-end review. Record both time to the initial result and time to an accepted final outcome, including waits for missing sources and escalation.

Verification can require nearly the same work as producing an answer. If a reviewer must reopen every document to check for omitted exceptions, fluent prose with citations may save little. A compact record of values, competing passages, calculations and coverage gaps may reduce search effort, but that is a hypothesis to measure through review time, substantive corrections and final correctness. The people reviewing outputs must have the expertise and access needed to resolve the remaining questions.

Those measurements can change architecture. Repeated extraction from stable documents may justify processing once and querying versioned records. Frequently changing sources can make index synchronization and invalidation a substantial operating cost. A larger context or additional retrieval step is worthwhile when its reduction in correction burden or missed evidence justifies its added latency and computation for the task.

Change testing should follow information dependencies. A parser update can alter reading order and table cells; a chunking or embedding change can alter retrieval; a model change can alter interpretation or abstention. Preserve the source snapshot and configuration needed to investigate a changed result. Re-run affected cases and the critical failure cases before adopting the change, then monitor newly arriving document families and reviewer corrections.

Acceptance follows the task. Extraction requires the specified values with their units, periods and source context. Collection question answering requires a correct answer supported by sufficient authorized evidence, with abstention where that evidence is insufficient. Complete review also requires accounting for expected documents, applicable versions and every defined review requirement before reaching the final disposition. For Alder, that means using the applicable fee and policy to record the correct route, or keeping the determination open when necessary evidence is unresolved. Evaluate the chosen design against its obligation, including the human work required to finish it.