Practical guide

AI capability measurement: a decision agenda for the chief information officer

A chief information officer does not need another abstract conversation about whether employees are “good at AI.” The useful agenda connects a specific work decision to architecture, access, evidence quality, operational risk, and a controlled next step. The CIO may own the decision, influence it, review it, or collaborate on it depending on the organization. Do not assume the title owns the.

By Two Prune

A chief information officer does not need another abstract conversation about whether employees are “good at AI.” The useful agenda connects a specific work decision to architecture, access, evidence quality, operational risk, and a controlled next step. The CIO may own the decision, influence it, review it, or collaborate on it depending on the organization. Do not assume the title owns the budget, the assessment programme, workforce decisions, or every AI control. Confirm the mandate before assigning actions.

The CIO decision in one sentence

Decide whether the organization can run a defined AI-enabled work assessment with sufficient technical access, evidence integrity, data handling, and operating ownership to support the stated capability decision.

NIST’s AI Risk Management Framework organizes work through Govern, Map, Measure, and Manage and treats measurement as quantitative, qualitative, or mixed, with uncertainty and limitations documented. The US Government Accountability Office’s AI Accountability Framework groups accountability around governance, data, performance, and monitoring. These are useful lenses for a CIO agenda; neither framework supplies a universal employee score or proves that an assessment changes business outcomes. NIST AI RMF Core GAO AI Accountability Framework

Sources: [1] [2]

Use the STACK agenda

STACK turns the meeting into five decisions rather than a product presentation.

### S — Scope the workflow and decision

Ask:

Required artifact: a one-page decision charter naming the workflow, population, decision, owners, exclusions, time window, and claims boundary.

Stop if the group is trying to measure “AI readiness” without a defined task or decision. A broad label will produce a broad score with no defensible operating use.

### T — Trace architecture, data, and tool boundaries

Ask:

Required artifact: a system-and-data map with owners, data classes, transfers, retention rules, failure paths, and approved exceptions.

An access failure is an infrastructure signal, not a low-capability result. The agenda must keep those categories separate.

### A — Assess the evidence layers

Review three layers independently:

Required artifact: an evidence map showing the source, definition, owner, refresh date, missingness rule, and decision use for each measure.

Do not collapse the layers into a funnel that implies causation. Completion does not demonstrate capability; an assessment score does not establish workplace impact; an operational change does not by itself show the assessment caused it.

### C — Calibrate risk, uncertainty, and alternatives

Ask:

Compare at least these options:

| Option | What it can answer | Main limitation | | --- | --- | --- | | Self-report survey | confidence, exposure, and perceived barriers | does not directly observe work performance | | Usage analytics | tool access and defined activity | does not show judgment, verification, or output quality | | Manager observation | contextual behaviour in ordinary work | inconsistent opportunity and observer effects | | Authentic work assessment | performance on a controlled realistic task | requires clear task validity, access, scoring, and interpretation boundaries | | Operational outcome study | change in a defined business measure | needs a separate design and cannot be inferred from a single score |

Required artifact: a risk-and-alternatives register with probability or evidence strength, consequence, owner, response, and remaining uncertainty.

### K — Keep the next gate owned

End with one of four gates:

Required artifact: a decision record naming the gate, rationale, dissent, approvals, conditions, owner, due date, and next review.

  • Which authentic work task is in scope, and why does it matter now?
  • Is the decision about baseline measurement, programme design, access readiness, a learning priority, or a later re-measurement?
  • Who is included, excluded, deferred, or offered an alternative route?
  • Which claims must the evidence support, and which claims are out of bounds?
  • Who owns the final decision, who advises, and who can stop the work?
  • Which identity, device, browser, network, model, and workspace services are required?
  • What participant data, prompts, documents, event records, submissions, and scores are created?
  • Where does each data class travel, who can access it, and how long is it retained?
  • Which customer systems are integrated, and which can remain isolated for the pilot?
  • How will access failures, outages, accessibility needs, and unsupported devices be recorded?
  • Access and participation evidence: invitation, eligibility, launch, completion, withdrawal, and technical failure.
  • Assessment evidence: work process, sources reviewed, AI interactions where permitted, verification behaviour, final deliverable, rubric application, score confidence, and calibration notes.
  • Operational evidence: later workflow quality, time, risk, adoption, or other business measures collected under their own design.
  • What error, missingness, or bias would reverse the decision?
  • Which benchmark is relevant, how large is it, and is it mature enough for the proposed interpretation?
  • What evidence would trigger a manual review, re-run, exclusion, or pause?
  • Can a smaller isolated pilot answer the architecture question before an integration is built?
  • What alternatives could answer the decision with less data or operational burden?
  • Proceed: named conditions are met for the bounded pilot.
  • Revise: a repairable evidence, design, or access gap has an owner and due date.
  • Hold: a critical boundary, data, authority, or participant-protection issue is unresolved.
  • Stop: the proposed approach cannot answer the decision or creates disproportionate risk.

Worked pilot: AI-assisted IT incident summaries

Consider a synthetic assessment in which service-desk analysts review an incident packet, use approved AI support, verify the generated summary, and submit an escalation note. The organization wants a baseline for targeted learning, not an employment recommendation.

The pilot record shows 60 eligible analysts, 52 invited, 47 completed, four could not access the assessment environment, and one withdrew. Of the 47 completions, 42 have complete scoring evidence and five are held for manual review because the required source trace is missing.

The STACK agenda prevents three common errors. First, it reports the four access failures as infrastructure defects rather than participant results. Second, it treats the five held records as unresolved assessment evidence rather than assigning provisional scores. Third, it does not compare the assessment result with later incident-resolution time until the operational metric has a stable definition, comparable case mix, and separate analysis plan.

The architecture review recommends an isolated browser-based pilot with synthetic incident material, federated identity, a documented retention window, and no production ticket integration. The evidence review requires a frozen scenario version, scoring rubric, reviewer record, score-confidence field, and reason codes for exclusions and holds.

The CIO is classified as decision owner for technical access and data architecture, collaborator on assessment design, and reviewer of the evidence controls. The learning leader owns the development response; the assessment programme owner owns the measurement decision. The gate is revise until accessibility testing and the access-failure escalation path are complete. Nothing in the record claims that the assessment improved incident operations.

Copyable CIO decision record

| Decision | Required evidence | Source and owner | Failure condition | CIO role | Other accountable owner | Gate / due date | | --- | --- | --- | --- | --- | --- | --- | | Can the bounded pilot proceed? | scope, data map, access test, assessment design, exception path | stable record plus named owner | missing authority, untested access, undefined retention, or unsupported claim | owner / influencer / reviewer / collaborator | one named person | proceed / revise / hold / stop |

Objections worth resolving in the room

“We already have usage data” is answered by separating activity from observed work quality. “Managers can judge this” is answered by comparing observation consistency and evidence coverage. “The platform score is enough” is rejected because confidence, missingness, rubric fit, and benchmark maturity affect interpretation. “We need every integration before a pilot” is tested against the smallest architecture that can answer the decision. “This should predict productivity” becomes a separate operational study, not a hidden promise in the capability measurement.

Two Prune helps customers run realistic AI-enabled work simulations and produce evidence-backed assessment scores and reports. It does not make automated employment decisions, replace accountable human judgment, or establish that a score causes productivity, safety, compliance, or business performance.

Sources

NIST: AI Risk Management Framework Core

  • US Government Accountability Office: AI Accountability Framework

Sources: [1] [2]

Sources

  1. 1.NIST AI RMF Core · National Institute of Standards and Technology
  2. 2.GAO AI Accountability Framework · U.S. Government Accountability Office