A trace is not evidence if you don't know what it missed

A trace can look complete even when it missed the event that matters.

It may show the prompt, tool calls, file changes, errors, retries, and tests. It may put them in a clean timeline. It may also use hashes to show that nobody edited the record later.

But the provider may not expose IDE context. A hook may start after the session begins. A child process may run outside the adapter. A network request may bypass the proxy.

The timeline has a start and an end, so it feels complete. That does not mean it is complete.

I realized this while talking to people using AgentTrace, the open-source tool I built for recording and replaying coding-agent sessions. The same issue came up while working on Verio, an early product idea for connecting agent sessions to engineering outcomes.

Collecting events is the easy part. The harder part is explaining what those events can prove.

What AgentTrace records

AgentTrace applies the basic idea behind strace to AI agents.

It records more than model calls. It can also record:

• user prompts;
• tool calls and results;
• commands;
• file reads and writes;
• errors and retries; and
• final responses.

The core stores these records as a flat event stream in NDJSON files. The local path has no database. The Python package has no runtime dependencies. Each session is a directory with a metadata file and an event file.

This makes the data easy to inspect and move. It does not lock the record inside a hosted dashboard.

AgentTrace supports several capture methods, including hooks, an MCP proxy, and Python instrumentation. Each method sees a different part of the session.

A hook sees the events that its host sends. An MCP proxy sees the requests that pass through it. A Python decorator sees the function it wraps.

None of them can record an event they never receive.

An intact record can still be incomplete

AgentTrace can add a SHA-256 hash chain to captured events. The audit --verify-chain command can detect changes to that chain.

This can answer:

Did someone change a captured event?

It cannot answer:

Did the adapter capture every important event?

A valid hash chain shows that the stored record is intact. It does not show that the provider sent every tool call. It cannot detect an action that happened outside the observed path.

A useful report should say both things:

Captured events passed integrity checks.
Provider coverage is partial.
IDE context was not available.
Network effects outside the MCP proxy were not observed.

This is more useful than a single "verified" badge because it shows where the record ends.

Estimates need clear labels

AgentTrace uses simple rules to remove common secrets from traces. It checks sensitive field names and patterns for API keys, GitHub tokens, JWTs, connection strings, and long hexadecimal values.

The redaction design record also explains the limits. Pattern matching can remove safe text by mistake. It can miss a secret with an unknown format. It cannot clean data written before redaction was enabled.

Cost estimates have the same problem. If a provider does not return token usage, AgentTrace estimates it from text length. The cost design record states that the estimate can be wrong by 30 to 50 percent and should not be used for billing.

These features are still useful when their limits are clear.

An estimate can help debug an expensive session. It should not become an invoice. Redaction can reduce risk. It cannot promise that a shared trace has no secrets.

Recording, control, and proof are different

AgentTrace has grown beyond session replay.

It can check recorded actions against a local policy. It can flag access to sensitive files. It can scan MCP tool descriptions for known prompt-injection patterns. Its watchdog can pause, resume, or stop the process it started. It can then write a report that shows what happened before the process stopped.

These controls do not make AgentTrace a complete security system. The repository says:

This project is experimental. Do not treat it as a production security boundary without your own threat model, controls, and testing.

A scanner is not a sandbox. A local allow list is not a central authorization service. A process monitor cannot stop an action outside the process it owns. Local identity signing can show that a record changed, but it cannot prevent someone from changing the file.

I made a similar point in The review queue is the bottleneck. A trace of activity is not proof that an action was allowed or that a result should be accepted.

For each session, we need to ask:

1. What did the tool record?
2. What did the control allow or stop?
3. What does the evidence support?

One green badge cannot answer all three.

Show the level of confidence

The AgentTrace walkthrough uses four labels:

• Capture: Verified
• Explain: Factual
• Correlate: Partial
• Improve: Human-owned

Each label makes a different claim.

Capture can be checked against the stored record. An explanation can stay close to recorded events. Correlation is less certain because pull request, deployment, or incident data may be missing. A person still decides what the team should change.

The page also says when data is missing:

IDE context was not exposed by the provider.

It does not claim access to private model reasoning:

The record does not claim hidden reasoning.

This is how evidence tools should work. Missing data should appear next to the events, not in a footnote.

Where Verio may fit

AgentTrace is the open-source recording and local analysis tool.

Verio is not a live hosted product. It is an early idea that I am testing. The question is whether teams need a way to connect agent sessions with pull requests, deployments, rollbacks, incidents, and review results.

The proposed flow is:

1. Capture the agent's work.
2. Explain the errors, retries, changes, and checks.
3. Link the session to an engineering outcome.
4. Use repeated evidence to improve the workflow.

The third step needs care.

Suppose an agent session links to a pull request that merged faster than usual. That does not prove the agent caused the faster merge. The pull request may have been smaller. The reviewer may have known that code well. The task may have been easy.

The link helps the team investigate. It does not prove cause.

Verio should not claim that it makes an agent secure, compliant, or productive. It should not become an identity system or an authorization gateway. The goal is narrower: help teams inspect agent work, judge the quality of the evidence, and study the outcomes.

That idea still needs real customer testing.

The missing feature is evidence health

AgentTrace does not yet have one clear measure of session evidence quality.

The open evidence-health issue proposes four session states: healthy, partial, unknown, and invalid.

I would add a label to each important fact:

• Observed: The adapter received the event.
• Inferred: The system worked it out from other events.
• Missing: An expected event was absent.
• Truncated: Only part of the data was kept.
• Redacted: The data was removed before storage or sharing.
• Reconstructed: The system rebuilt a likely sequence later.
• Unknown: The available data cannot answer the question.

Consider this line:

Test suite passed.

Did the adapter record the command and exit code? Did it parse an agent summary? Did it infer success because the session continued? Was the output cut short?

Those are different types of evidence. The interface should show the difference.

What needs to happen next

The open issues show the current gaps. AgentTrace still needs:

• a stable version-one review workflow;
• verified setup checks and provider capture guides;
• the evidence-health model;
• a better review interface;
• a safe evidence fixture; and
• safer defaults for sharing traces.

I would start with one test session that includes:

• a failed command;
• a retry;
• a file change;
• a test result;
• a blocked action;
• a redacted field; and
• a human correction.

The setup check should report what the adapter can capture. Replay should label each fact. The review page should show gaps beside the timeline. Sharing should remove sensitive data by default.

Only then should the product link sessions to pull requests or deployments. Outcome data is not useful if the session record cannot explain its own limits.

The useful answer may be "I don't know"

Many tools can show model calls, tool use, token counts, cost, latency, and errors. OpenTelemetry also makes this data easier to move.

The harder questions come after collection.

Which facts did the adapter record? Which did the system infer? What did the provider hide? Is the record intact? Is it complete enough for the decision at hand? Can someone share it without exposing a secret?

A good trace does not hide those questions behind a clean chart.

Sometimes the correct answer is: I don't know.