Section 18 · Testing, Reporting & Learning
18.2Measure actual revenue
Built — needs DMA's data- No.
- 18.2
- DMA's reference
- Decision Book p.33
- Phase
- Phase 1
- Autonomy
- L1 · Recommend
- DMA's decision
- Yes
★ Marked "non-negotiable, build first" in the book.
What DMA asked for · Decision Book p.33
Track the whole path: visitor → conversation → qualified → meeting → proposal → client → monthly revenue.
Why it matters to DMA: Without it nobody can tell whether the system works. DMA's book marks it non-negotiable, build first.
In our words, from DMA's Product Decision Book v3, page 33.
How we do it
- Every step leaves a record in the bridge database: the chat and its stage (opened, engaged, reviewed, booked), the booked meeting, the proposal, the signed contract and payment, and the won or lost outcome mirrored back from Nutshell.
- Every AI decision is logged, and outcomes are linked back to the decision that led to them, under a versioned, approved set of attribution rules.
- Reactivation is measured against a control group: each wave keeps a random, stratified 20% back (never emailed, never shown to reps, never written to Nutshell), so the lift is money that wouldn't have come anyway, with a 95% range and an honest "not meaningful yet".
- The Measurement screen shows each wave's sends, replies, meetings, pipeline dollars, won dollars and lift. In the POC two waves are real test waves and two are clearly labelled simulated history; real revenue needs DMA's real outcomes.
The flow
It plays on its own while it's on screen; hover or use the controls to pause or step through.
A first touch
A win-back email goes to a former client; 20% of the wave is held back and gets nothing.
- Person
- Stored data
- Nutshell
- Rule in code
- Screen
Where the data goes
The same six stages on every page. Nothing reaches Nutshell except through the write gate.
- 1Chat, outreach, Nutshell
Source
Every step of the path, and the final outcome.
- 2Sync
Copies Nutshell changes into the bridge database
Brings won and lost outcomes back from Nutshell.
- 3Bridge database
A copy of the CRM data, plus what the AI works out
Decisions, outcomes, experiments and the held-back control group.
- 4AI
Claude models, only through DMA's own gateway
Not needed: the numbers are computed in SQL.
- 5Write gate
The only way back into Nutshell: checked, approved when needed, sent once
Held-back accounts are never written to Nutshell.
- 6Nutshell
The system of record
Stays the source of the won and lost truth.
The tables behind it
Drawn from the POC's database catalogue: structure only, no data.
PK primary keyFK reference the database enforces (solid line)ID reference kept by id (dashed line)
| Table | What it holds | Columns |
|---|---|---|
| ctrl.experiments | Each measured wave or test: its control-group share and dates. | 17 |
| ctrl.holdout_assignments | Which accounts were treated and which were held back. | 10 |
| ctrl.decision_outcomes | Outcomes linked back to the AI decision before them. | 13 |
| ctrl.attribution_rules | The versioned, approved rules for what counts as caused by the AI. | 10 |
How the tables connect (4)
| Column | Points to | Kept by |
|---|---|---|
| ctrl.holdout_assignments.experiment_id | ctrl.experiments | The database (foreign key) |
| ctrl.holdout_assignments.decision_id | ctrl.decision_log | The application (by id) |
| ctrl.decision_outcomes.decision_id | ctrl.decision_log | The application (by id) |
| ctrl.decision_outcomes.experiment_id | ctrl.experiments | The database (foreign key) |
Worked example
Synthetic demo data: every name, business and number is made up.
Did it bring in money that wouldn't have come anyway?
What you see
1.The honest label
Simulated history is marked as such, on the screen and in the data.
What happens in the system
- Bridge database
is_simulatedEvery simulated experiment, assignment and outcome row carries a flag, so it can never be mistaken for a real result.
The waves on the Measurement screen today (two real test waves, two labelled simulated).
Reactivation waves with a control group
| Wave | Sent | Meeting rate vs held out | Pipeline | Won | Lift (95% range) |
|---|---|---|---|---|---|
| Win-back wave 1 · June 2026 (simulated) | 88 | 4.5% vs 0.0% | $66,500 | $48,500 | +5.7 pts (−9.5 to +12.6) |
| Win-back wave 2 · July 2026 (simulated) | 88 | 5.7% vs 4.5% | $9,500 | $0 | −2.3 pts (−21.4 to +7.5) |
In the running POC
Measurement: each wave against its held-back control group.
- 1Simulated history is labelled as simulated.
- 2One wave: sends, replies, meetings, pipeline, won and the lift with a 95% range.
Dashboard · Measurement. Captured from the running POC on September 25, 2026; synthetic data.
See it live
Opens the exact screen in the running POC, in a new tab.
| Open | Where | What to try |
|---|---|---|
| Measurement | DMA AI dashboard | Every wave, its control group and the lift. |
Status
Built — needs DMA's data. This capability runs in the POC today. It needs DMA's own content or data to go live; no code change is needed for that.