Section 18 of 21 · Decision Book p.32–33
18. Testing, Reporting & Learning
DMA's book asks the system to measure real revenue end to end, learn from every outcome, and never change its own strategy without review. Revenue measurement with control groups and human-reviewed changes run in the POC; the finer learning pieces are scheduled for Phase 2 and 3.
What we built for this section
- Revenue measurement with a random 20% control group per wave, lift and a 95% range, simulated history clearly labelled.
- Outcomes linked back to the AI decisions before them, under approved attribution rules.
- Versioned instructions, rules and scoring that go live only after a second person approves them.
The 6 capabilities, in the book's order
| No. | Capability | Book page | Phase | Autonomy | Status |
|---|---|---|---|---|---|
| 18.1 | A/B test individual conversation piecesTest different openers, questions and objection answers against each other, not just AI versus no AI. | p.32 | Phase 3 | L2 · Prepare, then wait for approval | Designed for Phase 3 |
| 18.2 | Measure actual revenueTrack the whole path: visitor → conversation → qualified → meeting → proposal → client → monthly revenue. | p.33 | Phase 1 | L1 · Recommend | Built — needs DMA's data |
| 18.3 | Industry cohort analysisLearn which industries and services convert best. | p.33 | Phase 2 | L1 · Recommend | Designed for Phase 2 |
| 18.4 | Drop-off analysisFind the exact point where good prospects abandon the conversation. | p.33 | Phase 2 | L1 · Recommend | Designed for Phase 2 |
| 18.5 | Won/lost feedback loopEvery outcome improves future scoring and recommendations. | p.33 | Phase 2 | L1 · Recommend | Designed for Phase 2 |
| 18.6 | Human-reviewed model changesThe system never quietly rewrites its own sales strategy without review. | p.33 | Phase 1 | L1 · Recommend | Built in the POC |