Section 18 · Testing, Reporting & Learning
18.1A/B test individual conversation pieces
Designed for Phase 3- No.
- 18.1
- DMA's reference
- Decision Book p.32
- Phase
- Phase 3
- Autonomy
- L2 · Prepare, then wait for approval
- DMA's decision
- Later
◆ Amended in v3 of the book.
What DMA asked for · Decision Book p.32
Test different openers, questions and objection answers against each other, not just AI versus no AI.
Why it matters to DMA: It is the only way the chat's conversations get better over time; the book keeps it for later, after the core revenue loop is proven.
In our words, from DMA's Product Decision Book v3, page 32.
How we do it
- DMA decided LATER (Phase 3).
- Each conversation piece already has an id (for example the SEO opener is version 1 of its template), and chats already have a place for an experiment arm.
- The design assigns each new chat to an arm at random, records which pieces it saw, and compares booked-call rates per arm with the same statistics as the reactivation waves.
- Only approved versions can be in a test; a winning version still goes through approval before it becomes the default.
Already in place
- Randomised control groups already run for every reactivation wave (18.2), with lift and a 95% range on the Measurement screen.
- Openers and answers are versioned pieces with ids, and each chat records the ones it used.
- Conversations already have fields for a variant and an experiment arm.
What the later phase adds
- Random assignment of new chats to arms.
- Per-arm booked-call rates on the Measurement screen.
The flow
It plays on its own while it's on screen; hover or use the controls to pause or step through.
Assigned
Each new chat gets opener A or B at random.
- Person
- Stored data
- Rule in code
Where the data goes
The same six stages on every page. Nothing reaches Nutshell except through the write gate.
- 1Website chat
Source
Chats in each arm.
- 2Sync
Copies Nutshell changes into the bridge database
Brings back outcomes.
- 3Bridge database
A copy of the CRM data, plus what the AI works out
The arm, the pieces used and the result.
- 4AI
Claude models, only through DMA's own gateway
Uses only approved pieces.
- 5Write gate
The only way back into Nutshell: checked, approved when needed, sent once
Not involved.
- 6Nutshell
The system of record
Unchanged.
The tables behind it
Drawn from the POC's database catalogue: structure only, no data.
PK primary keyFK reference the database enforces (solid line)ID reference kept by id (dashed line)
| Table | What it holds | Columns |
|---|---|---|
| conv.conversations | Chats, with room for the variant and experiment arm. | 22 |
| ctrl.experiments | Experiments and their allocation. | 17 |
How the tables connect (1)
| Column | Points to | Kept by |
|---|---|---|
| conv.conversations.agent_version_id | agent.agent_versions | The application (by id) |
Worked example
Synthetic demo data: every name, business and number is made up.
A designed test (synthetic example).
Opener test
| Arm | Chats | Booked-call rate |
|---|---|---|
| A: "take a quick look at your SEO" | 412 | 6.1% |
| B: "what would you like more of?" | 405 | 4.7% |
In the running POC
There is no screen for this one yet: it is designed for Phase 3, as DMA's book decides. The parts listed under “Already in place” run in the POC today.
Status
Designed for Phase 3. DMA's book schedules this for Phase 3 (decision: later). The design is ready and builds on parts that already run in the POC.