What an AI audit log has to capture

Most products in this category will tell you they have an AI audit log. Almost all of them do. The useful question is not whether one exists, it is whether somebody can answer a real question with it at nine in the morning after something went wrong the night before.

That question is usually a version of: an agent changed something it should not have. What exactly did it touch, whose access was it using, and was it even a person asking?

A log that cannot answer that is a list of events. Here is the record shape we arrived at, with the reasoning, including the parts we got wrong first.

An AI audit log needs one shape for two kinds of record

The first decision was to use a single record shape for audit events and application logs both.

The obvious design is two systems. Audit is a compliance artifact with a stable contract, application logs are operational noise, and they have different readers. Keeping them apart feels tidy.

It is tidy right up to the first incident. The moment somebody asks what happened, the answer lives half in one system and half in the other, and the person answering is correlating two timelines by eye and by timestamp. Timestamps are the worst possible join key because everything in the incident happened inside the same few seconds.

One shape, two streams. The same fields, the same field names, one query language. The streams are labeled, so you can read only the audit stream when that is what you want, and you can read both when you are trying to understand a failure.

Name the account that was actually reached

This is the property we would test first in any product in this category.

A company connects two Notion workspaces, one for engineering and one for HR. An agent updates a page. The log says: Notion, update page, succeeded.

That entry is worse than useless, because it reads like an answer. The person holding it now has to go and look in both workspaces.

So the record carries the connected account, and it takes it from the execution rather than from the request. Those two differ more often than you would expect. A tool that reaches several accounts of the same app takes an account argument, the model may omit it, and resolution happens inside the call. What the caller asked for and what the call reached are two different facts, and the one worth keeping is the second.

The same reasoning applies one level down, to arguments. Argument names and counts are recorded. Argument values never are. The names tell you the shape of what was attempted, which is what an investigation needs. The values are customer data, and an audit trail is the wrong place to accumulate a second copy of it.

Whether an AI was asking is a field, not an inference

The temptation is to work this out at read time. Requests from the assistant come through a particular path, or carry a particular user agent, or the actor id belongs to a service account you recognize, so you can tell.

You can tell until you cannot. Paths get refactored, a second surface appears that reuses the first one's plumbing, and the inference quietly starts returning the wrong answer. It returns the wrong answer silently, and it does so specifically in the situation where somebody is asking a question that matters.

So the actor kind is written at the moment of the action, from the code that knows. It is one of a closed set: a human user, the system itself, platform staff, a provisioning client, an API token, or an AI assistant. Six values, decided at the write.

There is a related case worth mentioning because it is the one that embarrassed us. When platform staff act while impersonating a customer's user, the action used to appear in that customer's own audit log as their own administrator doing it, with nothing anywhere to say otherwise. That is not a logging bug in a narrow sense. It is a log that tells the customer something false about their own organization. Impersonation is now stamped once, centrally, at the point where the request context is built, rather than at each of the hundred or so places that write an entry. Stamping it at the call sites would have meant getting it right a hundred times and then getting it right again for every new one.

Failures are the interesting half

A trail that records only what succeeded cannot answer the most common security question, which is what somebody tried to do and was prevented from doing.

So every attempt produces an entry, and the entry carries an outcome. Both the succeeded and the failed entry name the account.

Within failures, one distinction is kept sharp: a call refused by a restriction is a different category from a call that reached the third party and failed there. Only the first one is our answer. It is tempting to collapse them, since the user experience is similar, but the two lead to completely different next steps. One means governance did its job. The other means somebody's API is down or a token expired.

Classifying by matching on refusal text was the version we did not ship. It works, and it breaks the day somebody rewords a message. Refusals carry a code.

The error-text firewall

This is the part of the design that is least obvious and that we would most want another vendor to have thought about.

When a tool call fails at the third party, there is error text. It is useful text. It says the record was not found, or the field was invalid, or the token lacked a scope. The caller should see it, because it is their own data coming back to them and it is how they fix the problem.

That same text must not be written to the audit trail.

The reason is what the audit trail is connected to. It is org-visible. It is readable by the in-product assistant. And it is fanned out to whatever SIEM the customer has configured. So a remote error body reaching it is third-party payload leaving the system through the log pipe, into up to three places, none of which were part of the decision to make that API call.

Every failed call therefore carries two error strings. One is derived from the third party's response and goes to the caller only. One is written for the trail and is never derived from the request or the response. They are separate fields with separate rules, because the alternative is one field and a convention, and conventions do not survive.

Bound the field namespace

Structured logging invites a flatten. Take the metadata object, promote every key to a top-level field, and now everything is queryable without a substring scan.

The problem is that metadata keys are not all ours. Connector slugs, tool names and provisioning payloads all contribute keys, and all of them are influenced by somebody outside the system. Unbounded flattening means one organization's traffic grows the field namespace for the whole tenant, and the index that makes queries fast is the thing that pays for it.

So promotion works from a fixed allowlist, derived by reading every place that writes an entry and asking what a person would filter on. Roughly two dozen keys: the tool, the connector, the connection, the outcome, the duration, the classification, whether it was approved, the error code, and the join keys that let an AI action be traced back to the conversation and the specific tool call that produced it. Those last few have to be real fields rather than sitting inside a JSON blob, because inside the blob they are only reachable by substring scan, which is not a query.

Everything else stays in the JSON. Nothing is lost; some things are just slower to reach, which is the correct trade for a key nobody filters on.

Tenancy as a type, not a WHERE clause

The log store we use has no per-tenant authorization of its own. What separates one customer's records from another's is the tenant header the service sends. That is a thin thing to rest on, so it is enforced in the type system: the query client accepts only a scope value that can be constructed in three named ways, and there is deliberately no code path that takes a bare tenant number.

The same reasoning separates customer-visible audit records from internal application logs. That could have been a visibility field and a WHERE clause. It is a tenant boundary instead, because a dropped WHERE clause leaks and a wrong tenant returns nothing. Returning nothing is a bug you find. Leaking is a bug you find later, from someone else.

It matters more than usual here because the in-product assistant can read the audit log on a user's behalf. A scope bug would not just expose internal records, it would feed them to a model that is talking to a customer.

Two smaller things that cost us

The write must never fail the action. Every entry is written after its action has already committed, so throwing rolls nothing back; it only turns a completed action into a server error. The case that makes this concrete: creating an API token returns the raw secret exactly once. A failure at the logging step there leaves a live credential whose value nobody ever saw. So the write cannot throw. When it does fail, it fails loudly into the application stream with enough identity to reconstruct the entry, minus the metadata, which carries request payloads and does not belong in a second place. It is not retried either, because there is no idempotency key on it, and a double-written trail is no better than a gapped one.

Timestamps come from the identifier. Entry ids are time-ordered, and the record's time is recovered from the id rather than taken at ingest, so time order and id order agree. That agreement is what the read cursor's tiebreak depends on. The scar: the function that extracts it once parsed the id as hex, which was correct for the old id format and silently wrong for the new one. Every record got stamped with the beginning of the epoch. Because reads are bounded to a recent time window, that presented as an empty audit log rather than as an error. An audit log that is wrong in a way that looks like an audit log that is empty is the worst failure mode in this whole design, and it is the reason we would tell anyone evaluating a product in this category to load real data and look at it rather than reading a feature list.

Reading it, which is a separate design problem

Everything above is about the write. A trail nobody can query is a trail nobody uses, and the read path has its own decisions.

Filters have to be indexed fields, and the derivation happens at write time. A reviewer arrives with a request shaped like "all administrative actions by this person in March". That means filtering on category, actor and time. Categories and action kinds are therefore derived when the entry is written and stored on it, rather than computed during the query. Doing it at read time would mean a second implementation of the same classification, and a second implementation of anything is a second source of truth that has already drifted once.

The same values are returned on each row, so the interface does not re-derive them either. Three copies of one rule is how a filter and a label end up disagreeing on the same screen.

Actor names are resolved server-side, per page. The store holds identifiers, not names, and nobody wants to read a page of identifiers. Resolving them in batches as each page is fetched keeps that cheap. The detail that matters: a person who has left still resolves, rendered as a former member rather than dropped. An entry that disappears because its actor did is a gap in the trail at exactly the moment somebody is investigating a departure.

Searching by actor cannot be paged naively. The trail's cursor orders by the log store's own identifiers, which the member directory knows nothing about. So a search by person's name resolves the whole matching set of identifiers up front, bounded, before paging the trail. Filtering page by page would silently drop matches whose entries happen to fall on a later page.

Reads are bounded by time. Retention is a storage setting; the bound on a query is separate and exists so that an unfiltered read does not scan everything an organization has ever done. Worth knowing when a request covers an unusually long window, because the answer is to widen the bound deliberately rather than to assume the trail is short.

What to ask a vendor for

Not a screenshot. Ask for an export with real activity in it, and check five things.

1. Does an entry name the specific connected account the call reached, or just the app?
2. Is there a field that says an AI was the actor, or would you be inferring it?
3. Are failed attempts present, and can you tell a policy refusal from a third-party error?
4. Does any entry contain text that came back from a third party, or argument values?
5. Can you get it into your own SIEM, and what is actually implemented today rather than configurable?

If a product answers those five, the rest of its audit story is probably fine. If it cannot answer the first one, the log will not help you on the morning you need it.

The governance rules that decide which calls are allowed in the first place are a separate subject, written up in the rules that decide what an agent may call. What this looks like when the person asking is a compliance reviewer is in producing audit evidence for agent access. The connector catalog shows what each connection exposes as tools, which is the vocabulary these entries are written in.