Section 18 · Testing, Reporting & Learning
18.5Won/lost feedback loop
Designed for Phase 2- No.
- 18.5
- DMA's reference
- Decision Book p.33
- Phase
- Phase 2
- Autonomy
- L1 · Recommend
- DMA's decision
- Yes
◆ Amended in v3 of the book.
What DMA asked for · Decision Book p.33
Every outcome improves future scoring and recommendations.
Why it matters to DMA: It is what makes the system smarter over time instead of staying static; the amendment confirms it is part of the learning loop.
In our words, from DMA's Product Decision Book v3, page 33.
How we do it
- Scheduled for Phase 2, after the deal autopsy and revenue measurement.
- Outcomes are already linked back to the decisions before them, and the backtest already replays the scoring against past outcomes (synthetic in the POC) and reports how well the score ranked the accounts that revived and the deals that were won.
- The design turns that into a loop: a new scoring configuration is proposed from the outcomes, backtested, and only goes live after a person approves it (18.6). The system never retunes itself silently.
Already in place
- Outcomes linked to decisions; the backtest with its accuracy figures on the Measurement screen.
- Versioned scoring configurations that only change through approval.
What the later phase adds
- A proposed new scoring configuration from recent outcomes.
- Its backtest shown next to the current one, for approval.
The flow
It plays on its own while it's on screen; hover or use the controls to pause or step through.
Outcomes
A quarter of won and lost deals.
- Nutshell
- AI
- Rule in code
- Person
Where the data goes
The same six stages on every page. Nothing reaches Nutshell except through the write gate.
- 1Nutshell
Source
Won and lost outcomes.
- 2Sync
Copies Nutshell changes into the bridge database
Mirrors them.
- 3Bridge database
A copy of the CRM data, plus what the AI works out
Outcomes, scoring versions and backtests.
- 4AI
Claude models, only through DMA's own gateway
Proposes; never activates.
- 5Write gate
The only way back into Nutshell: checked, approved when needed, sent once
Not involved.
- 6Nutshell
The system of record
Better tiers and next steps, after approval.
The tables behind it
Drawn from the POC's database catalogue: structure only, no data.
PK primary keyFK reference the database enforces (solid line)ID reference kept by id (dashed line)
| Table | What it holds | Columns |
|---|---|---|
| ai.backtest_runs | Each backtest: the scoring version, how well it predicted wins, and the lift of tier A over C. | 18 |
| ai.scoring_configs | Versioned scoring configurations. | 15 |
How the tables connect (1)
| Column | Points to | Kept by |
|---|---|---|
| ai.scoring_configs.approval_id | ctrl.approvals | The application (by id) |
Worked example
Synthetic demo data: every name, business and number is made up.
The backtests in the POC today (on synthetic history).
Backtest
- Runs so far
- 2
- Best ranking quality (AUC)
- 0.588 on the synthetic history (50% is chance)A real figure needs DMA's real outcomes.
In the running POC
There is no screen for this one yet: it is designed for Phase 2, as DMA's book decides. The parts listed under “Already in place” run in the POC today.
See it live
Opens the exact screen in the running POC, in a new tab.
| Open | Where | What to try |
|---|---|---|
| Backtest | DMA AI dashboard | The scoring replayed against past outcomes. |
Status
Designed for Phase 2. DMA's book schedules this for Phase 2. The design is ready and builds on parts that already run in the POC.