Section 18 · Testing, Reporting & Learning

18.6Human-reviewed model changes

Built in the POC

What DMA asked for · Decision Book p.33

The system never quietly rewrites its own sales strategy without review.

Why it matters to DMA: It is the safety rail on the learning loop; DMA's book calls it non-negotiable governance from day one.

In our words, from DMA's Product Decision Book v3, page 33.

Everything that shapes the AI's behaviour is versioned: each agent's instructions, its model and limits, the scoring configuration and every rule. A change is a new version, never an edit in place.
A new version is tested first: the AI Lab runs it against saved cases and compares it with the current one, and the website chat's safety suite runs against the draft itself.
It only goes live when a person approves it in Approvals, and not the person who wrote it (four-eyes). In the live configuration, an unapproved version simply doesn't run: the chat falls back to lead capture rather than use it.
The AI cannot edit its own configuration: no tool it can call touches prompts, rules or scoring.

It plays on its own while it's on screen; hover or use the controls to pause or step through.

Step 1 of 5

A draft

Someone writes a new version of the chat's instructions.

The same six stages on every page. Nothing reaches Nutshell except through the write gate.

1DMA's team
Source
A new version of an instruction, rule or score.
2Sync
Copies Nutshell changes into the bridge database
Not involved.
3Bridge database
A copy of the CRM data, plus what the AI works out
Every version, its tests, its approval and who gave it.
4AI
Claude models, only through DMA's own gateway
Runs only the approved version.
5Write gate
The only way back into Nutshell: checked, approved when needed, sent once
Not involved: configuration never goes to Nutshell.
6Nutshell
The system of record
Unchanged.

Drawn from the POC's database catalogue: structure only, no data.

PK primary keyFK reference the database enforces (solid line)ID reference kept by id (dashed line)

Table	What it holds	Columns
agent.prompts	Every version of every instruction, with status and approver.	14
agent.agent_versions	A released configuration: model, limits and the instructions it uses.	19
agent.eval_runs	Test runs of a version against saved cases.	26
ctrl.approvals	The approval queue, including configuration changes.	22

How the tables connect (2)

Column	Points to	Kept by
agent.agent_versions.agent_id	agent.agents	The database (foreign key)
agent.eval_runs.previous_run_id	agent.eval_runs	The database (foreign key)

Synthetic demo data: every name, business and number is made up.

1 / 3

What you see

1.The AI workers

Each agent with its model and its approved version.

localhost:13301/agents

What happens in the system

Bridge databaseagent.agent_versions · agent.prompts
One active, approved version per agent; every other version is a draft, in testing or rejected.

The website chat's instruction versions in the POC today.

Version	Status	What it means
1	Active (starting version)	The starting version, approved at setup
2	In testing	The full instructions: identity, rules, method and style; waiting for DMA's approval
3	Draft	A proposed change, not yet tested

localhost:13301/agents

1 / 3Build → Agents: each AI worker, its model and its approved version.

Dashboard · Build → Agents. Captured from the running POC on September 25, 2026; synthetic data.

Opens the exact screen in the running POC, in a new tab.

Open	Where	What to try
Agents	DMA AI dashboard	Each agent's versions and who approved them.
AI Lab	DMA AI dashboard	Test a version against saved cases.

Built in the POC

Built in the POC. This capability runs in the POC today, on synthetic data, end to end.