Section 2 · Qualification & Opportunity Scoring
2.2Learn scoring from DMA's real history
Designed for Phase 2- No.
- 2.2
- DMA's reference
- Decision Book p.11
- Phase
- Phase 2
- Autonomy
- L1 · Recommend
- DMA's decision
- Yes
What DMA asked for · Decision Book p.11
Use DMA's real won and lost deals to learn what actually predicts revenue, instead of guessing the weights.
Why it matters to DMA: It turns the score from an opinion into something backed by DMA's own outcomes.
In our words, from DMA's Product Decision Book v3, page 11.
How we do it
- DMA's book schedules this for Phase 2, after the review of the 30,000 records and clean lost-reason data. Today's scores use reasoned starting weights, as the book allows.
- The design: the backtest already built in the prototype re-scores past accounts and deals as they stood at the time, and measures how well the score ranked the ones that were won or revived. Learning adds a proposed set of weights that ranks DMA's real history better.
- A learned version is never switched on automatically (L1: it recommends). It becomes a new scoring version with its backtest result next to it, and a person at DMA approves it, exactly like any other rule change.
Already in place
- Measurement → Backtest: scores past reactivation waves and closed deals as they stood at the time, using only what the CRM held then, with deciles, top-10 precision and the winners that ranked low, and why.
- Won and lost outcomes are already linked back to the accounts and waves that produced them.
- Scoring weights are a versioned setting that only changes with a person's approval.
What the later phase adds
- Fitting the weights to DMA's real won and lost history once the record review is done.
- A proposed scoring version with its backtest comparison, for approval.
- A periodic re-check as new outcomes arrive.
The flow
It plays on its own while it's on screen; hover or use the controls to pause or step through.
DMA's history
Every closed deal and every past outreach with its outcome.
- Nutshell
- Rule in code
- AI
- Stored data
- Person
Where the data goes
The same six stages on every page. Nothing reaches Nutshell except through the write gate.
- 1Past deals
Source
Won and lost deals, and past outreach outcomes.
- 2Sync
Copies Nutshell changes into the bridge database
Mirrors closed deals from Nutshell.
- 3Bridge database
A copy of the CRM data, plus what the AI works out
Backtest runs and proposed scoring versions.
- 4AI
Claude models, only through DMA's own gateway
No model call: the backtest re-uses the stored AI reading of the notes.
- 5Write gate
The only way back into Nutshell: checked, approved when needed, sent once
Not involved.
- 6Nutshell
The system of record
Unchanged until a new version is approved.
The tables behind it
Drawn from the POC's database catalogue: structure only, no data.
PK primary keyFK reference the database enforces (solid line)ID reference kept by id (dashed line)
| Table | What it holds | Columns |
|---|---|---|
| ai.backtest_runs | Saved backtest runs with their key numbers and full report. | 18 |
| ai.scoring_configs | Scoring versions; a learned one would be a new version waiting for approval. | 15 |
How the tables connect (1)
| Column | Points to | Kept by |
|---|---|---|
| ai.scoring_configs.approval_id | ctrl.approvals | The application (by id) |
Worked example
Synthetic demo data: every name, business and number is made up.
The prototype's backtest on its synthetic history, which the learning step would build on.
Backtest today (synthetic data)
- Reactivation
- 24 of 176 accounts revived; the 10 highest-scored revived at 20%, against 14% overall
- Closed deals
- Not what this score is for: no better than chance on synthetic dealsReal outcomes are needed to judge or learn the weights.
- Learning
- Phase 2, on DMA's real history
In the running POC
There is no screen for this one yet: it is designed for Phase 2, as DMA's book decides. The parts listed under “Already in place” run in the POC today.
See it live
Opens the exact screen in the running POC, in a new tab.
| Open | Where | What to try |
|---|---|---|
| Measurement → Backtest | DMA AI dashboard | How the current score ranked past accounts and deals. |
Status
Designed for Phase 2. DMA's book schedules this for Phase 2. The design is ready and builds on parts that already run in the POC.