Practical guide

What AI tools and usage data can tell you

A practical guide to interpreting AI tool features and telemetry without confusing activity with capability or work quality.

By Two Prune Research

AI tool telemetry can show access, activity, feature use and changes in workflow volume. It cannot by itself establish that a task was correctly framed, relevant evidence was selected, an output was verified or the final decision was sound. Interpretation improves when usage data is joined to bounded task and outcome evidence.

Read telemetry as evidence of activity and reach

Logins, active users, prompts, accepted suggestions and feature events can describe adoption patterns. They help identify who has access, where use concentrates and whether an intervention changes observable activity over time.

Those measures are valuable when the decision is about rollout, support or product utilization. They become misleading when relabeled as quality or capability without additional evidence. A high count can include trivial work, repeated corrections or activity that never reaches a usable deliverable.

Define event meaning, coverage and missingness before comparing teams. Product changes, logging configuration and differences in task mix can move a metric even when underlying capability is unchanged.

Sources: [1] [2]

Add evidence from the work itself

When the question concerns quality or capability, examine a bounded task, its source material, the submitted artifact and relevant workflow evidence. This makes it possible to ask whether the person selected appropriate evidence, challenged an output and delivered for the intended audience.

A final artifact alone can hide how an error was caught or how a claim was sourced. A process log alone can show that an action occurred without revealing the worker's reasoning. Combining them supports a richer explanation while each signal retains its limitations.

NIST emphasizes context, documentation and appropriate measurement. Apply that discipline by stating which task and conditions the evidence covers and by avoiding claims about every future workflow or permanent personal traits.

Sources: [1] [3]

Use proportionate data and accountable interpretation

Collect only the telemetry needed for a defined purpose, document access and retention, and explain how the signal will be interpreted. More detailed monitoring is not automatically more valid, and sensitive content creates additional privacy, security and trust considerations.

Before adding a data source, ask what decision it changes and whether a less intrusive signal would suffice. Separate operational troubleshooting from individual development evidence, and involve relevant privacy, security, legal or worker representatives where appropriate.

Report activity, capability and outcome measures separately before relating them. OECD research treats training, skills, working conditions and responsible adoption as connected concerns; a responsible dashboard should preserve those distinctions rather than compressing them into one score.

  • Define the question before selecting an event or metric.
  • Document coverage, missingness and product-version effects.
  • Combine telemetry with task evidence for quality claims.
  • Limit collection, access and retention to the stated purpose.

Sources: [2] [1]

Explore this topic