Product / Evaluations
Decisions are deterministic, so agent behavior is testable. Assert the answers agents should get against published policy, review every question they asked, and gate SQL changes with query validation.
Red-team cases state the decision an agent should get — allow, deny, or conditional — and run against the live MCP tools. When policy changes, the same checks re-run and fail loudly if behavior shifts.
Evaluations
Write cases that state what an agent should be told — this use allowed, that export denied, this one conditional. Run them against the live MCP tools and assert the typed answer state, deterministically, in CI.
The per-token request log shows every question agents asked and the typed answer state each got — 90 days, allow-listed fields, query text never recorded.
Run SQL and data changes through validate_query_context before merge. The check is fail-closed: if a changed query touches a restricted table for its intent, the gate flags it before release.
Not yet shipped
Today, decisions are re-derived from current serving state and validation answers are not persisted. The next evaluation layers build on a durable decision ledger:
Decision-ledger evaluations
Score recorded decisions against policy after the fact.
Drift trends
See decision patterns shift across runs before they compound.
Policy replay
Re-run past decisions against a new policy version to see what would change.