Practical guide

AI capability measurement agenda for the Head of AI

A Head of AI needs a decision agenda that connects the organization's AI portfolio to observable work, accountable ownership, operating risk, and the next investment choice. Adoption counts alone cannot answer whether people can use AI well in consequential tasks, while a single assessment cannot establish organization-wide value. Use BOARD: Boundary, Outcomes, Ownership, Assurance, and.

By Two Prune

A Head of AI needs a decision agenda that connects the organization's AI portfolio to observable work, accountable ownership, operating risk, and the next investment choice. Adoption counts alone cannot answer whether people can use AI well in consequential tasks, while a single assessment cannot establish organization-wide value. Use BOARD: Boundary, Outcomes, Ownership, Assurance, and Decisions. The Head of AI is often an owner or influential collaborator for the AI portfolio, but budget, workforce, risk, learning, and functional decisions may sit elsewhere. Confirm authority rather than inferring it from the title.

The five decisions for the agenda

This agenda is for a 90-minute working session with the Head of AI, functional owner, AI enablement or learning lead, risk or governance reviewer, and an evidence-method owner. Add data, security, legal, worker representation, or procurement participants where the use requires them.

  • Which AI-enabled work contexts matter now?
  • What evidence is needed beyond access and activity?
  • Who owns performance, risk, learning, and interpretation?
  • Where would a controlled assessment add useful evidence?
  • What result would change a real decision?

B: Set the portfolio boundary

Start with three to five AI-enabled workflows, not every tool or employee. For each workflow, record:

| Workflow | Intended user | Decision or deliverable | AI role | Human control | Material failure | Current evidence | | --- | --- | --- | --- | --- | --- | --- | | proposal compliance review | proposal professional | exception and correction memo | search, compare, draft | source verification and sign-off | missed mandatory clause | usage, final review notes | | service-policy response | customer adviser | customer answer | retrieve and summarize | exception check and escalation | incorrect commitment | case audit, activity data | | variance explanation | finance analyst | management commentary | calculate, classify, draft | source reconciliation | unsupported explanation | workbook and manager review |

NIST describes governance and oversight as involving actors with management, fiduciary, or legal authority, while deployment actors make contextual decisions about production use and organizational change. Use those actor distinctions to assign work, not to assume the Head of AI personally owns every risk. NIST AI RMF: AI actor tasks

Sources: [1]

O: Define outcomes and evidence layers

For each workflow, separate five layers:

Do not collapse the layers into one “AI maturity” score. A capability assessment can contribute evidence at the third and fourth layers within a bounded task. It does not by itself show a causal operational outcome.

The OECD's 2026 employer and worker surveys use complementary perspectives on AI implementation and experience. They support asking for evidence from both organizational and participant viewpoints, while remaining self-reported survey evidence from defined countries and sectors. OECD: AI surveys of employers and workers

  • Access: approved tools, permissions, sources, and support are available.
  • Activity: people use features or workflows under defined conditions.
  • Capability: people demonstrate observable behaviors on a comparable task.
  • Work quality: outputs meet context-specific acceptance rules.
  • Operational outcome: cost, speed, risk, service, or other result changes.

Sources: [2]

O: Assign ownership and decision rights

Complete this map before discussing a pilot:

| Decision | Accountable owner | Contributors | Evidence required | Escalation | | --- | --- | --- | --- | --- | | workflow priority | functional executive | Head of AI, operations | volume, consequence, readiness | portfolio forum | | approved AI use | governance owner | legal, security, data, product | use case and control evidence | risk committee | | capability intervention | learning or functional owner | Head of AI, managers | task-bounded gap evidence | programme owner | | assessment interpretation | evidence-method owner | trained reviewers, domain expert | rubric, artifacts, missingness | independent review | | scale or stop | pilot decision owner | all relevant owners | pre-specified gate results | executive sponsor |

NIST's AI RMF Core calls for clear roles, leadership responsibility, ongoing review, documented practitioner proficiency, and defined human oversight. It is voluntary guidance and should be tailored to the actual organization. NIST AI RMF Core

Sources: [3]

A: Review assurance gaps and alternatives

Ask what can already answer the decision:

For each gap, record the least burdensome source that could resolve it. Do not commission an assessment when source repair, workflow telemetry, a manager calibration session, or a small work-sample audit would answer the question.

### Common objections

“We already have adoption analytics.” Keep them. They answer access and activity questions. Add a controlled task only when the decision requires comparable evidence about framing, source use, verification, judgment, or delivery.

“A simulation is not real work.” Correct. Report it as task-bounded evidence and test whether its findings are useful alongside real-work evidence.

“We need one enterprise score.” A single score may hide task, role, evidence, and risk differences. Start with a decision-specific evidence map and add aggregation only when constructs and samples support it.

“We cannot use production data.” Use synthetic or explicitly approved non-sensitive material and set security, retention, and access gates before the pilot.

  • platform analytics may show adoption, features, and activity;
  • work-quality review may show output defects and reviewer burden;
  • incident and support data may reveal failure patterns;
  • surveys may show confidence, access, experience, and perceived constraints;
  • manager observation may add context but vary in opportunity and standard;
  • a controlled assessment may create a comparable opportunity to observe behavior;
  • a field experiment may be required for causal operational questions.

R: Choose a bounded pilot

Select one workflow with an accountable owner, clear failure mode, available domain reviewers, and a decision scheduled within 60 days.

| Pilot field | Required answer | | --- | --- | | decision | what will change based on the result? | | participant group | who is represented and who is not? | | task | what realistic work and ambiguity will be included? | | evidence | deliverable, source trail, workflow evidence, reviewer rationale | | controls | data, access, notice, accommodations, retention, challenge route | | missingness | what becomes insufficient evidence? | | gates | continue, repair, hold, stop | | owner | who decides and who advises? | | report boundary | what may and may not be inferred? |

D: End with explicit decisions

The meeting output should contain no more than five decisions:

  • prioritized workflow and reason;
  • unanswered evidence question;
  • chosen evidence method and rejected alternatives;
  • pilot owner, controls, and gates;
  • reporting audience, date, and next decision.

Worked agenda: proposal compliance red-team pilot

A Head of AI convenes the proposal director, AI enablement lead, security reviewer, and research lead. Usage data shows broad adoption of an approved assistant, while proposal reviews still find unsupported compliance statements. The meeting does not assume AI caused the defects.

Boundary. The team selects synthetic civilian-government RFP compliance review, excluding live bids, classified material, pricing, and external communication.

Outcomes. The unresolved question is whether a consistent task reveals how proposal professionals locate mandatory clauses, use AI, verify citations, distinguish an exception, and communicate corrections. Win rate and productivity are outside the pilot claim.

Ownership. The proposal director owns the workflow and scale decision. The Head of AI is an influential collaborator and funds the bounded pilot. Security owns data gates. The research lead owns protocol and interpretation. Trained proposal reviewers apply frozen anchors.

Assurance. The team compares a controlled simulation with a retrospective defect audit and manager interviews. It chooses the simulation because the upcoming cohort creates a fair, known opportunity to observe the target behaviors. The other sources remain part of interpretation.

Decision. Run 24 voluntary sessions with two alternate task forms. Repair if critical evidence is missing in more than 10% of sessions, hold if reviewer anchors cannot be applied consistently, and continue only if the evidence changes a scheduled development or workflow decision. No result enters an employment decision.

Reporting boundaries

Report observed behavior, evidence strength, missingness, reviewer disagreement, task conditions, participant coverage, operational burden, and the decision reached. Do not report task results as enterprise readiness, training impact, future work performance, or proof that incidents will be prevented.

Two Prune can provide the controlled simulation and task-bounded evidence packet. It does not replace portfolio governance, establish operational outcomes, provide mature enterprise norms, or make employment decisions.

Sources

NIST AI RMF: Descriptions of AI actor tasks

  • NIST AI Risk Management Framework Core
  • OECD: AI surveys of employers and workers

Sources: [1] [3] [2]

Sources

  1. 1.NIST AI RMF: AI actor tasks · National Institute of Standards and Technology
  2. 2.OECD: AI surveys of employers and workers · www.oecd.org
  3. 3.NIST AI RMF Core · National Institute of Standards and Technology