How a Selection Share number gets earned.
Every visibility tool shows a score. The question that matters is whether the score can survive being checked. This is the exact method behind a Grounds holdout read — published so a skeptical buyer, or a competitor, can find the hole in it if there is one.
Buyer prompts for a category are divided into a treated arm and a holdout arm before anything ships. The holdout is never touched by the intervention being tested — it's the counterfactual.
Once a post-intervention read exists for a plan, its treated/holdout split can't change. Re-approving with a different split after that point is refused, not silently applied — the design a result is measured against has to stay what it was measured against.
24h, 72h, 7d, 14d — treated and holdout are sampled on the same windows, across every configured engine, so drift in the models themselves doesn't get mistaken for the intervention working.
A two-proportion z-test on the treated-vs-holdout selection rate, plus a diff-in-differences against the pre-intervention baseline. The confidence label is derived from the resulting p-value — not picked from a sample-size table.
The statistics, plainly
A two-proportion z-test compares the treated arm’s selection rate against the holdout arm’s, within the same measurement window. The p-value uses the pooled standard error (the standard choice under the null hypothesis that both arms behave the same); the 95% confidence interval uses the unpooled, Wald standard error — the standard choice once the interval itself is meant to describe how different the arms might really be. The diff-in-differences carries its own interval, built from the sum of all four arms’ proportion variances (current treated, current holdout, baseline treated, baseline holdout), since they’re independent query clusters, not paired observations.
| Read | Treated | Holdout | Lift | Result |
|---|---|---|---|---|
| 14d | 45% (n=100) | 30% (n=100) | +15 pts | p=0.028, 95% CI [1.7, 28.3] |
Illustrative numbers, not a live read — but the exact formula: pooled SE gives z≈2.19, two-tailed p≈0.028; Wald SE gives the interval. Run those two proportions through any standard statistics package and you get the same answer.
- Lift, before the intervention is confirmed live — not drafted, published and verified.
- A confidence label without the p-value it came from, next to it.
- A diff-in-differences without saying so when the baseline arm is missing or small.
- Any read under n=20 without saying the sample is thin, in the same sentence as the number.
Every snapshot, scored observation, and experiment result is written to an append-only table — enforced by a Postgres trigger that rejects UPDATE and DELETE at the database level, not by application discipline. A number on a report traces back to specific, immutable rows, not a screenshot someone could have edited.
Selection Share, defined
The share of selection-bearing buyer prompts — the ones where an engine names a single winner, not just a list of options — where your brand is that pick. Cited is not selected: appearing as a source an engine drew on is a different, weaker signal than being the name it actually recommends. Selection Share is the number that moves revenue; it’s also the harder one to move, which is why it’s worth a real test rather than a monitoring dashboard’s best guess.