How to develop and assess data extraction in AI-enabled work

Define the record before asking AI to fill it

The US Government Accountability Office's AI Accountability Framework says that data quality, reliability, representativeness, sources, origins, and processing all matter to responsible use. NIST's AI Risk Management Framework similarly calls for intended context, data selection, limitations, testing, and measurement results to be documented.

Those principles apply directly to extraction work. A worker who starts with “extract the key data” has not yet defined a verifiable task. “Key” may mean every contract obligation, only decision-relevant incidents, one row per invoice, or one row per line item. A model can create a polished table while silently mixing those units.

Before extraction begins, specify:

This turns extraction from a copying exercise into a controlled evidence transformation.

• the unit of record, such as one supplier, event, invoice, or requirement;
• the fields and their operational definitions;
• the permitted source set and authoritative version;
• how missing, conflicting, approximate, and not-applicable values are represented;
• any transformations, such as date normalization or currency conversion;
• the evidence locator required for each extracted value;
• the checks that determine whether the record is complete enough for its intended use.

Build a FIELD ledger

Use a FIELD ledger for every material extracted value:

| Element | Question | Retained evidence | | --- | --- | --- | | Field | What does this column mean, and what is explicitly excluded? | Field definition and allowed values | | Instance | Which source object or event does this row represent? | Stable row key and unit-of-record note | | Evidence | Where exactly does the value come from? | File, version, page, cell, or message locator | | Logic | Was the value copied, normalized, calculated, or inferred? | Transformation rule and intermediate values | | Disposition | Is the value verified, missing, conflicting, approximate, or not applicable? | Status, reviewer note, and next action |

The ledger should distinguish source values from derived values. If a model converts “within ten business days” into a calendar date, retain the original phrase, the calendar used, the triggering date, and the calculation. If a PDF table has a blank cell, record missing rather than assuming zero. If two sources provide different values, keep both until an authority or reconciliation rule resolves them.

For repeated extraction, create a short data dictionary. Include one positive example, one exclusion, and one edge case for each important field. Give the model that dictionary, ask it to return evidence locators with every value, and reject rows whose locators cannot be reproduced.

Worked example: a synthetic grant-reporting register

Imagine a synthetic programme analyst who must build a register from twelve grant agreements, amendment letters, and quarterly emails. The output will support an internal review of reporting obligations. This example is invented and does not describe a Two Prune customer.

The first AI-generated spreadsheet contains columns for recipient, award value, reporting date, status, and owner. It looks complete, but three problems appear. One amendment changes a deadline, two agreements express dates relative to an event, and the model fills blank owner fields with names found elsewhere in the pack.

The analyst creates a FIELD ledger and defines one row as one reporting obligation, not one agreement. “Reporting date” becomes three separate fields: source wording, normalized date, and date basis. “Owner” may be extracted only when the source assigns responsibility directly. The analyst also adds amendment status and evidence locator fields.

For a clause stating “submit the final report within 30 days of project completion,” the ledger records:

For another agreement, an amendment moves a fixed date from 30 June to 31 July. The original date remains traceable, but the current register uses the approved amendment and labels it superseding evidence. An email that says “Mina can help” is not treated as an owner assignment.

The analyst asks AI to produce candidate rows, then samples all derived dates, all amended records, and every missing or conflicting field. A pivot comparing source obligations with extracted row keys reveals that one agreement contains two separate reports but had produced only one row. The final register includes 17 obligations, not 12 agreements, and an exceptions sheet names the unresolved completion-date dependency.

The useful artifact is not simply a populated spreadsheet. It is a register whose granularity, transformations, and gaps can be traced back to the source pack.

• Field: final-report due date;
• Instance: agreement G-104, final report obligation;
• Evidence: agreement G-104, section 8.2, page 14;
• Logic: derived from the project completion date, which is not present in the approved pack;
• Disposition: missing input, so no normalized date is produced.

Practice with changing schemas

Give a learner a mixed pack containing a PDF agreement, spreadsheet, email thread, and scanned amendment. Ask for a ten-row register using a supplied data dictionary. Include one repeated obligation, one superseded value, one blank that could be mistaken for zero, one relative date, and one conflict.

After the first pass, change one field definition. For example, redefine “amount” from total award value to current-quarter eligible spend. The learner should identify which rows require re-extraction, which source fields remain reusable, and which calculations must be invalidated. They should not merely rename the column.

Useful practice behaviors include:

• defining the unit of record and field meanings before extraction;
• preserving source wording alongside normalized values;
• retaining precise evidence locators;
• separating copied, transformed, calculated, and inferred values;
• representing missing and conflicting values explicitly;
• checking coverage against the source objects, not just the completed rows;
• using AI for candidate extraction while independently testing high-risk fields.

Assess observable extraction behavior

In an assessment, provide a realistic evidence pack and a structured output that will be used for a named decision. Observe whether the participant:

Do not score extraction quality by row count alone. A smaller register with explicit gaps may be more faithful than a full table containing invented values. Fast extraction is not evidence of reliable extraction unless the record survives source-level checks.

• identifies the correct unit of record;
• defines ambiguous fields before populating them;
• retrieves the authoritative source and current version;
• records evidence locators that another reviewer can reproduce;
• preserves source values before normalization or calculation;
• distinguishes missing, zero, not applicable, conflicting, and inferred data;
• detects tables or clauses that produce multiple records;
• tests AI-generated rows rather than accepting visual completeness;
• adjusts the extraction when the schema or source authority changes;
• communicates exceptions that limit downstream use.

Verify with three reconciliations

First, perform a source-to-record reconciliation. Select source objects and confirm that every in-scope item produced the expected number of rows. This catches omissions and incorrect granularity.

Second, perform a record-to-source reconciliation. Sample rows, especially transformed and high-impact fields, and reproduce every value from the cited locator and stated logic. This catches fabricated citations and unsupported normalization.

Third, perform a distribution and exception review. Inspect duplicate keys, impossible dates, unexpected blanks, extreme values, category totals, and status counts. An anomaly is a prompt to revisit evidence, not proof that the value is wrong.

Data extraction skill is task-bounded. It does not establish that the resulting dataset is generally representative or suitable for another purpose. In a Two Prune assessment, the data dictionary, FIELD ledger, source navigation, AI interactions, reconciliation checks, revisions, and final register can contribute bounded evidence about Evidence Navigation, AI Orchestration, Verification & Risk Control, and Communication & Delivery. They do not make an employment decision.

Sources

US GAO Artificial Intelligence Accountability Framework

• NIST AI Risk Management Framework Core

Sources: [1] [2]