Part A · How DMA AI works

A.6The AI gateway and observability

Built in the POC

No.: A.6
Relates to DMA's book: Decision Book p.8, p.46
Part: A · the foundation

What this part does · Decision Book p.8, p.46

DMA's book asks for AI cost, traceability and decision observability: every AI decision traceable and costed, so the system can be debugged and audited, including the after-the-fact sampling of deals the AI closed on its own.

Why DMA needs it: Autonomy without an audit trail is autonomy nobody checks. DMA should be able to see what the AI did, why, and what it cost, for any single reply.

How we do it

Every AI model call, from every worker, goes through one gateway. It records the model, the tokens in and out, the cost, the time taken, the worker and version that asked, and a trace id that ties it to the conversation. The prompt and the answer are stored encrypted.
The gateway enforces the money limits: an overall AI budget for the prototype ($50), daily caps per caller, and kill switches that stop all AI calls at once. When a limit is reached, calls stop; they don't quietly carry on.
Every consequential decision (a reply, a tool call, a rule result) is also written to the decision log with its reasons, and linked to the model call that produced it, so spend is counted once.
DMA's Observe page shows it live: each chat reply as a trace of steps (input check, tools, reply, checks) with the model, tokens and cost of each; the Log page totals the spend.

The flow

It plays on its own while it's on screen; hover or use the controls to pause or step through.

Step 1 of 5

A worker needs the model

The website salesperson is about to reply to a visitor.

AI
Rule in code
Routing
Stored data
Screen

Where the data goes

The same six stages on every page. Nothing reaches Nutshell except through the write gate.

1Any worker
Source
Every model call, from any of the ten workers.
2Sync
Copies Nutshell changes into the bridge database
Not involved.
3Bridge database
A copy of the CRM data, plus what the AI works out
The call log, the decision log and the kill switches.
4AI
Claude models, only through DMA's own gateway
Claude Sonnet 5 and Claude Haiku 4.5, only through the gateway.
5Write gate
The only way back into Nutshell: checked, approved when needed, sent once
Not involved in model calls.
6Nutshell
The system of record
Not involved.

The tables behind it

Drawn from the POC's database catalogue: structure only, no data.

PK primary keyFK reference the database enforces (solid line)ID reference kept by id (dashed line)

Table	What it holds	Columns
ops.llm_calls	The gateway's record of every model call: who asked, the model, tokens, cost, time; prompt and answer encrypted.	37
ctrl.decision_log	Every consequential decision, linked to the model call and the trace.	28
ctrl.kill_switches	The switches that stop AI calls, email, CRM writes or everything at once.	9

How the tables connect (7)

Column	Points to	Kept by
ops.llm_calls.agent_version_id	agent.agent_versions	The application (by id)
ops.llm_calls.run_id	agent.runs	The application (by id)
ops.llm_calls.conversation_id	conv.conversations	The application (by id)
ops.llm_calls.decision_id	ctrl.decision_log	The application (by id)
ctrl.decision_log.agent_version_id	agent.agent_versions	The application (by id)
ctrl.decision_log.run_id	agent.runs	The application (by id)
ctrl.decision_log.conversation_id	conv.conversations	The application (by id)

Worked example

Synthetic demo data: every name, business and number is made up.

One recorded chat, as the gateway saw it

1 / 5

What you see

1.Every call in the gateway's log

The calls list shows each model call with who asked, the model, tokens, cost and time.

localhost:13301/observe

Observe → Calls: every model call with its model, tokens and cost.

What happens in the system

LLM gatewaychat
Claude Sonnet 5: 2 tokens in, 45 out · $0.0082 · 2.2 s.
LLM gatewaychat
Claude Sonnet 5: 2 tokens in, 149 out · $0.0040 · 2.3 s.
LLM gatewaychat
Claude Sonnet 5: 2 tokens in, 119 out · $0.0059 · 2.9 s.
LLM gatewayinput-classifier
Claude Haiku 4.5: 576 tokens in, 47 out · $0.0008 · 1.3 s.
Bridge databaseops.llm_calls
6 calls for this short chat, $0.022 in all.

Right-hand values come from a real run of the POC on September 25, 2026 (trace tr_muh5tbo52jgpqd0smgrf8, tr_muh5tbqv18g3rgfbfdhda…). Rows marked “sample” are illustrative.

Typical cost per call in the prototype, from the gateway's own log (averages over two days of use).

What each kind of call costs

Call	Model	Average cost	Average time
Website chat reply	Claude Sonnet 5	$0.0068	2.0 s
Input check on a visitor message	Claude Haiku 4.5	$0.0008	1.2 s
Chat note-taker	Claude Haiku 4.5	$0.0020	1.9 s
Review of one Nutshell record	Claude Haiku 4.5	$0.0027	4.1 s
Reactivation email draft	Claude Sonnet 5	$0.0110	6.9 s
Escalation brief	Claude Haiku 4.5	$0.0009	1.8 s

In the running POC

localhost:13301/observe

1 / 3Observe → Calls: every model call with its model, tokens and cost.

1The model and tokens of one call.
2Its cost.

Dashboard · Observe → Model calls (LLM gateway). Captured from the running POC on September 25, 2026; synthetic data.

See it live

Opens the exact screen in the running POC, in a new tab.

Open	Where	What to try
Observe	DMA AI dashboard	Live activity; open any reply as a trace.
Model calls	DMA AI dashboard	Every call with its tokens and cost.
Log	DMA AI dashboard	The spend total and every decision.

Status

Built in the POC

Built in the POC. This capability runs in the POC today, on synthetic data, end to end.