Measuring AI productivity in knowledge work

Choose the unit and outcome

Define whether the unit is a task, case, document, worker, team, or business process. Record the start and end of the workflow and the outcome that matters. For customer support it may be resolved cases with quality controls. For analysis it may be decision-ready outputs after review. For proposals it may be compliant sections released, not words drafted.

Separate leading activity from outcomes. Licences activated, weekly users, prompts, and generated tokens describe access or activity. They do not by themselves show that useful work increased, quality held, or costs declined. Time saved is also incomplete unless the same task boundary, quality standard, and review effort are measured.

The NBER study Generative AI at Work examined a staggered introduction of an assistant among customer-support agents and defined productivity using issues resolved per hour. Its reported effects belong to that tool, setting, population, and outcome definition. It is primary empirical evidence, not a universal productivity coefficient.

Sources: [1]

Build a productivity evidence ledger

Record the workflow version, eligible cases, baseline period, treatment or comparison, worker and task mix, AI access, actual use, output count, error or defect measure, review minutes, rework, escalations, and downstream outcome. Add costs such as licences, integration, training, governance, and additional supervision where the decision requires them.

Predefine exclusions and missing-data rules. If complex cases are routed away from the AI-enabled team, a higher throughput rate may reflect easier work. If reviewers spend unrecorded time correcting drafts, apparent creation speed may overstate the net result. If output volume rises while customer or decision quality is unknown, label the evidence incomplete.

The NIST AI RMF Measure guidance recommends selecting methods and metrics for mapped risks and documenting characteristics that cannot be measured. Although it is a risk framework rather than a productivity standard, the principle supports pairing outcome measures with uncertainty, testing, and monitoring.

Sources: [2]

Worked artifact: a synthetic policy-summary workflow

Imagine a synthetic operations team testing AI assistance for weekly policy summaries. The baseline covers eight weeks and records completed summaries, editor minutes, material corrections, late deliveries, and reader clarification requests. The trial covers the same policy categories and requires editors to classify every correction.

Drafting time falls during the trial, but the first two weeks show more source-attribution corrections. The ledger records both changes instead of announcing a time-saving percentage. After the team adds a source-link requirement and a conflict check, attribution corrections decline, while editor minutes remain slightly above the original target because reviewers inspect more links.

The final decision is to continue a bounded trial for routine updates but keep complex cross-jurisdiction summaries outside scope. The artifact contains task-mix checks, raw counts, correction categories, review time, and the continuation rule. The figures are synthetic and are not Two Prune results or a customer benchmark.

Design a credible comparison

Use random assignment when feasible, or document why another design is appropriate. If comparing before and after, account for seasonality, learning, staffing, task mix, and concurrent process changes. Report the denominator and distribution, not only the average. A small group may benefit while another experiences more rework or lower quality.

The 2025 NBER working paper Shifting Work Patterns with Generative AI reports a field experiment across firms and knowledge workers, including changes in email time and limited detected shifts in task composition. Its design illustrates why time-use outcomes and organizational outcomes should be kept distinct and why null findings matter.

Sources: [3]

Verify the result and bound the claim

Reperform calculations, inspect missing cases, compare task mix, and sample outputs against the quality rubric. Trace review and rework across the full workflow. State what was not observed and what evidence would change the decision. Repeat measurement after material changes to the model, workflow, user population, or quality standard.

This method supports a bounded productivity decision. It does not establish causal impact without an appropriate design, predict results in another workflow, or create a mature benchmark. Two Prune can provide assessment evidence about performance in a controlled simulation; it does not measure an organization's production return or automate employment decisions.