BLACKBOX: The Flight Recorder for AI Agents

Modern aircraft carry flight data recorders because investigators need to know exactly what happened in the seconds before an incident — not a reconstructed approximation, not a best guess from logs, but a tamper-evident record of every input and output in the order it occurred.

Enterprise AI agents have no equivalent.

When an agent makes a decision that affects a customer, a credit limit, a regulatory filing, or a clinical recommendation, the enterprise typically has: a log file that may have been rotated, a database record that can be updated, and the agent's final output. The chain of reasoning — the intermediate steps, the tool calls, the evidence the agent considered, the exact prompt it saw — is gone.

The three questions enterprises cannot answer:

1. What exactly happened, in what order?
2. Was the record altered after the fact?
3. Which specific event caused the outcome?

BLACKBOX answers all three. Not by adding a better log — by changing the data structure. An append-only, SHA-256 hash-chained event store where the chain itself is the proof of integrity.

The entire platform rests on a single cryptographic guarantee:

hash = sha256(prev_hash + canonical_json(event_content))

Genesis uses prev_hash = "GENESIS". Every subsequent event commits to the previous hash. The chain is a linked list where each node contains the fingerprint of everything before it.

Replay verification:

Replay re-hashes from scratch using stored event content. If any event was altered, reordered, or removed:

• chain_valid is false
• first_broken_seq identifies the exact break point

There is no "mostly valid" chain. Either every event is intact and in order, or the chain is broken. This binary guarantee is what makes the record provable to an auditor: a valid chain means the record has not been touched since the run was sealed.

The tamper test is the definition of done. Alter a stored event. The chain must break. This test must always pass — if it fails, the product has no value.

Canonical JSON ensures consistent serialization: sorted keys, no whitespace variance, deterministic encoding. Hand-rolling serialization would introduce edge cases that silently corrupt the chain. The canonical encoder in chain.py is never bypassed.

BLACKBOX was designed on a single constraint: recording cannot require agent architecture changes. If adoption requires a sprint of refactoring, it will not happen. Three paths ensure every stack can record in minutes.

Path 1 — Python SDK (context manager)

import blackbox_sdk as bb

bb.configure(api_url="https://blackbox-production-ad23.up.railway.app")

with bb.run("run_001", model="claude-sonnet-4-6", system_prompt="...") as run:
    run.reasoning("Analysing the request…")
    run.tool_call("web_search", {"query": "Q3 earnings"}, result={"hits": []})
    run.output("Q3 earnings were $4.2M", tokens_in=520, tokens_out=180, cost_usd=0.0042)
# Run sealed automatically. Chain is provable.

pip install blackbox-sdk — the context manager handles genesis, event emission, and sealing. Explicit control for teams that want to record exactly what they choose.

Path 2 — Auto-instrumentation (zero agent changes)

import blackbox_sdk as bb

bb.configure(api_url="https://blackbox-production-ad23.up.railway.app")
bb.instrument()  # patches openai + anthropic clients globally

bb.instrument() monkey-patches the OpenAI and Anthropic Python clients at import time. Every LLM call is recorded automatically — no other changes needed. For teams that want observability without touching agent code.

Path 3 — OpenTelemetry (OTLP/HTTP JSON)

export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://blackbox-production-ad23.up.railway.app/v1/otel/traces
export OTEL_SERVICE_NAME=my-agent

BLACKBOX accepts OTLP/HTTP JSON. Works with opentelemetry-instrumentation-anthropic, opentelemetry-instrumentation-openai, LangChain, LlamaIndex, and any OTLP-compatible library. For teams already invested in OpenTelemetry infrastructure.

TypeScript / Node

import { BlackboxClient } from "@blackbox-ai/sdk";

const bb = new BlackboxClient({ apiUrl: process.env.BLACKBOX_API_URL });
const run = bb.run("my-run-001");
await run.genesis({ model_snapshot: "gpt-4o", system_prompt: "You are an analyst." });
await run.reasoning("Reviewing the data…");
await run.seal();

npm install @blackbox-ai/sdk — same event model, TypeScript types, full chain guarantee.

The CISO, CCO, and Head of Model Risk are the buyers — not the engineering team. They need to answer one question: "Can you prove the system behaved as designed on the date of the incident?"

Without BLACKBOX, the answer is always qualified: "We believe so, based on our logs." With BLACKBOX, the answer is: "Yes. Here is the sealed run. The chain is valid. Every event is intact."

The regulated-industry imperative

In finance, healthcare, and legal contexts, AI agents are crossing the line from recommendation tools to decision tools. A credit decision influenced by an agent. A clinical triage prioritization. A regulatory filing generated by an agent. In each case, the enterprise is accountable for the output — and regulators will ask how the output was produced.

The OCC, FRB, and equivalent bodies do not yet have agent-specific rules. They will. When they do, the question will be exactly the one BLACKBOX answers: can you demonstrate that the system operated as documented, that the record was not altered, and that you can reproduce the exact sequence of events that led to the outcome?

Faithful playback vs. approximate re-execution

BLACKBOX makes an explicit distinction the market has conflated:

• Faithful playback: verbatim replay of stored I/O — exact, provable, the same every time
• Approximate re-execution: feeding the same prompt to the same model and getting a similar output — not guaranteed bit-identical; providers do not guarantee deterministic output

BLACKBOX delivers faithful playback. It never claims deterministic re-execution. This distinction is the honesty constraint that makes the platform credible to a CISO.

Phase 0 — Recorder MVP ✓ Complete

The wedge. Insurance budget. Ten-second pitch: "replay the exact moment your agent went wrong."

Append-only event store, SHA-256 hash chain, faithful playback with chain_valid, tamper test. Docker Compose locally, backend on Railway, frontend dashboard on Vercel (blackbox-gold.vercel.app). PyPI package blackbox-sdk, npm package @blackbox-ai/sdk.

Phase 1 — Honest Replay + Drift Awareness ✓ Complete

Model snapshot ID stored per event — flag when model changes between runs. Daily Merkle root: anchor the chain head so a whole day is provable in one hash. PII redaction at capture: configurable masking patterns before storage.

Phase 2 — Fault Detection on the Record ✓ Complete

Citation verifier: claims in an output checked against the retrieval set, flagged inline. Policy engine: fault classes (fabricated source, hallucinated citation) trigger recorded responses. Quarantine + regenerate flow, all sealed as events. The recorder does not just store faults — it catches and contains them on-record.

Phase 3 — Provider Neutrality + Scale (Planned)

OpenTelemetry GenAI ingestion so any provider (Anthropic, OpenAI, Gemini, internal) records through one schema. ClickHouse for analytics reads at volume. Go/Bun ingestion gateway if Python overhead matters in the inference path.

Phase 4 — The Performance Layer (Planned)

Only possible once months of recorded traffic exist — the data is the moat. Leaderboard: score prompts on production traffic. Mining: cluster what users actually type, surface the best phrasing. Champion/challenger in shadow mode. Certified prompt registry → Workspace buttons + MCP into IDEs.

Phase 5 — The Upgrade Test (Planned)

Re-run verbatim recorded history against an incoming model snapshot. Classify each run: unchanged, benign regression, breaking regression. HOLD-or-PROCEED verdict. Sealed report. This converts the platform from insurance into a monthly-use product — the retention engine.