Prompt analytics platform guide for AI-enabled knowledge work

Define the decision before collecting prompt data

OpenTelemetry defines semantic conventions so telemetry attributes have consistent meaning across traces, metrics, logs, and events. Its generative-AI attribute registry also warns that tool arguments, tool results, and other captured content may contain sensitive information. NIST’s AI Risk Management Framework calls for measurement tied to deployment context, documented limitations, appropriate metrics, production monitoring, and feedback from relevant users and experts.

These principles point to a minimum rule: collect a signal only when it supports a named operational decision and has an approved handling path. “Capture everything in case it is useful” creates privacy, security, access, cost, and interpretation risks without guaranteeing better evidence.

Build a SIGNAL measurement contract

Use one SIGNAL contract for each monitoring question:

| Field | Question | Required artifact | | --- | --- | --- | | Scope | Which workflow, users, environment, and time window are included? | Bounded population and exclusions | | Indicator | Which event, metric, span, label, or reviewed sample represents the question? | Precise field definition and unit | | Governance | Who may collect, access, combine, export, and delete it? | Data classification, permissions, retention, and approval | | Analysis | How will raw events become a finding, and what denominator is required? | Query, aggregation, sampling, and uncertainty note | | Limits | What cannot be inferred from the signal? | Explicit non-claims and blind spots | | Next action | What decision, investigation, or review can the result trigger? | Owner, threshold, and escalation route |

Version the contract alongside instrumentation and dashboard changes. A dashboard tile without a measurement contract is only a display. It does not reveal whether the underlying event means the same thing across products, releases, teams, or time.

Worked artifact: a synthetic citation-check investigation

Imagine a synthetic research team using an AI assistant to draft short evidence summaries. A programme owner asks whether people “verify AI answers.” This example is invented and does not represent a Two Prune customer.

The analytics system records session identifiers, request duration, model identifier, token usage, tool calls, document identifiers, and whether the user opened a cited source. It does not store full prompt or response text. The first dashboard reports that 64 percent of sessions contain a source-open event.

That number does not answer the original question. A source-open event could occur before the answer, concern an unrelated document, or be followed by no comparison with the generated claim. Sessions without the event may involve knowledge checked outside the instrumented application. The platform also cannot see whether the user noticed a mismatch and corrected the final deliverable.

The team rewrites the question: “In the synthetic citation-check task, how often does the recorded workflow contain the required source-open and claim-revision sequence, and what do reviewed artifacts show about the remaining cases?” The SIGNAL contract defines the task cohort, event sequence, denominator, missing-event cases, access controls, and a sample of final artifacts for human review.

The resulting evidence has three layers:

The programme owner can use the result to investigate workflow friction or improve the exercise. The result is not a general verification score and does not prove that an employee is careful in every setting.

• telemetry identifies sessions with the expected event sequence;
• artifact review checks whether the cited claim was actually supported and whether corrections appear in the deliverable;
• participant notes and observation identify checks performed outside the captured interface.

Choose a signal model that preserves meaning

Prompt analytics often combines several data types:

Use consistent names, units, versions, and identifiers. Record which application and instrumentation produced each signal. Avoid high-cardinality or content-rich attributes unless the decision value and handling controls justify them. If a provider changes an event definition or a product moves from experimental to stable conventions, preserve the change date so trend breaks are visible.

• traces connect operations across one request or workflow;
• spans describe timed operations such as a model call or retrieval step;
• events record point-in-time occurrences such as an exception or policy action;
• metrics aggregate counts, durations, errors, or usage values;
• logs preserve structured operational records for investigation;
• session links group related activity where the application can identify a bounded session.

Apply a minimum-necessary capture review

For every proposed field, ask:

Do not assume that a vendor’s ability to capture content means the organization should enable it. Conversely, do not claim a platform lacks prompt or response visibility without checking the current product, plan, permissions, and configuration.

• Which SIGNAL decision requires it?
• Could a less sensitive event or derived value answer the same question?
• Could the field contain personal, confidential, privileged, regulated, or security-relevant information?
• Which users and services can access the raw value and each derived report?
• How long is the value needed, and how is deletion verified?
• What happens when consent, notice, contract, policy, or jurisdiction changes?
• Can a user or reviewer understand and challenge a conclusion drawn from it?

Test the analytics before trusting the dashboard

Create a small validation pack with known sessions. Include a successful call, retry, timeout, tool error, content-redaction case, missing field, multi-step workflow, and user who checks evidence outside the application. Predict the expected traces, events, and dashboard values before running the pack.

Then reconcile raw records to aggregates. Check duplicate events, clock boundaries, sampling, dropped telemetry, identity joins, bot or test traffic, model-version changes, and excluded environments. Verify access with both authorized and unauthorized test accounts. Record what the platform could not observe.

This validation is different from assessing the quality of the AI-assisted work. Operational telemetry can show what happened inside the instrumented path. A task assessment can examine the submitted artifact and available workflow evidence against a known brief. Neither source should be stretched beyond its boundary.

Interpret common metrics carefully

Prompt counts, active users, latency, error rate, token use, tool calls, and source-open events can support operating decisions. They do not independently establish judgment, verification, factual accuracy, productivity, learning transfer, or business value. Comparisons also require stable definitions and a relevant denominator. Ten errors among one hundred attempted calls means something different from ten errors among ten thousand.

Two Prune is an assessment product that can provide bounded evidence from realistic work simulations. It does not function as a general employee-monitoring platform, certify prompt analytics, or make automated employment decisions.

Sources

OpenTelemetry semantic conventions

• OpenTelemetry generative-AI attribute registry
• NIST AI Risk Management Framework Core

Sources: [1] [2] [3]