Interactive measurement dashboard · PAWLY · fictional pet-food D2C brand · 40 Korean regions · 104 weeks of synthetic data
PAWLY Incrementality Lab
Five questions a small advertiser asks about its ads, each answered with the study that fits it:
a geo experiment designed for power, a causal-impact readout of a pause, a check of what AI auto-bidding really added,
a media mix model calibrated with the experiments, and a cross-platform check of platform and GA4 credit. Every number is graded against the hidden truth of the simulated world.
PAWLY was chosen as a typical fast-growing small D2C advertiser whose profit depends on repeat purchases; the methods apply to any e-commerce advertiser.
All data synthetic · no real brand or customer data
Three things the ad platforms' reports got wrong
Scorecard: the study's answer, the naive answer, and the truth
"Naive" is what the advertiser would conclude without the study: the platform's reported ROAS, a before/after
comparison, or reported revenue. Intervals are 95% (chapters 1-3) and 90% (chapter 4). Each readout says whether the uplift is statistically significant (the interval excludes zero) and the smallest effect the study could detect.
One channel, six reads: what to tell the advertiser about short video
Chapter 0 · Pick the study from the question
A decision table an account team can use before promising a study. Eligibility numbers come from PAWLY's own history.
Before any study: is the data good enough to measure?
Most measurement problems at small advertisers start with data, not models. Five checks run before a study is promised.
How to read this tab
Design (left): before launching, past weeks are replayed as fake tests. Each line shows how often a test of that length catches a true revenue drop of the size on the x-axis. Where a line crosses 80% is its minimum detectable effect.
Eligibility table: turns that effect into money. "Min. detectable iROAS" is the smallest return per won the test can tell apart from zero at the planned budget.
Readout (below): each dot is a region pair; the shaded band is normal pair-to-pair noise. A dot outside it is checked with the advertiser before anything is dropped.
Power by test length
Share of replayed past windows where the 95% interval sits below zero
6 weeks (chosen)2, 4, 8 weeks
Data table
Eligibility: what each length can detect
Design options the advertiser may ask for
Same history, same A/A calibration. "Min. detectable return" = the smallest return per won each design can tell apart from zero with 80% power.
Pair diagnostic
Change in go-dark / control revenue ratio, test vs pre-period, in robust z-units
Data table
Readout
Is dropping the pair the right call? 60 re-runs on fresh noise
How to read this tab
Black line: actual daily revenue, all regions. Blue line and band: what revenue would have been without the pause (median and 95% range), learned from the weeks before.
The model leans on pet-food category searches, which follow the season but not PAWLY's ads. That is how it tells the summer dip apart from the pause.
Grey weeks are masked: during the geo test half the regions had no video, so the model must not learn from them.
Daily revenue vs counterfactual
KRW million per day, all regions
ActualCounterfactual (median)95% range
Data table (pause weeks)
Three answers to the same question
Placebo checks: quiet past windows
A model that finds effects where nothing happened cannot be trusted where something did. Both ranges contain zero, and neither probability comes near the real pause's.
How to read this tab
Both lines are indexed to 100 over the four weeks before the switch, so they share one axis.
Orange: search revenue as the ad platform reports it. Black: total revenue from PAWLY's own order system.
Histogram: the same jump measurement at 456 past dates when nothing changed. If the measured jump sits in the middle, it is ordinary noise.
Reported search revenue vs total revenue
Index, 4 weeks before the switch = 100 (7-day average)
Reported search revenueTotal revenue
Is the jump unusual?
Jumps in total revenue at 456 quiet past dates, %
Where the reported increase came from (4 weeks)
The budget increase that followed
How to read this tab
ROI chart: each channel's average return per won. Grey: the model built before any experiment. Blue: the same model calibrated with the two experiments. Black tick: the truth. Orange ring: what the ad platforms report.
Response curve: extra revenue next year if one channel's budget is scaled (x = 1 is today's budget), others unchanged. Blue band: calibrated model, 90%. Black: the truth.
Plans: each model proposes a reallocation (same total). Every plan is then run through the true world.
Cross-platform view: what each platform claims vs what it adds
Weeks 1-78. Claimed = the platform's own reported revenue. Added = incremental revenue from the calibrated model (90%).
The same question with the advertiser's GA4 data
GA4 data-driven attribution on the own site, weeks 1-78. GA4 counts each website sale once, so the question becomes each channel's share of the paid credit against its share of what ads added.
Channel ROI against the truth
Weeks 1-78, 90% intervals
Before testsAfter tests (calibrated)TruthPlatform-reported
Data table
Response curve
Extra revenue next year, KRW million
Calibrated model (median)90% rangeTruth
Marginal ROI at today's budget: where the next won earns most
Extra revenue per extra won if one channel's budget moves ±10% next year. Above 1 = room to grow; below 1 = trim first.
Budget plans, run through the true world
Model check: can the model be trusted beyond the truth we know?
Checks that also work on a real advertiser, where no truth is available: convergence, fit, an out-of-sample forecast, and agreement with the experiments.
Does a refit fix it? The same 4 weeks (101-104), three models
Chart: weekly revenue vs the out-of-sample forecast
ActualModel fitted on weeks 1-9290% predictive range
Checks that every number passed
Known limits
One simulated world. The geo test was re-run on 60 noise draws; the other chapters were checked with placebo windows, not repeated worlds.
The control series is cleaner than real ones. The category-search index that drives the causal-impact model follows true demand with 5% noise. Real covariates (search trends, competitor sales) are noisier and can be touched by the campaign, which widens intervals and can bias them.
TBR with A/A calibration instead of Trimmed Match. TBR reads the go-dark test from group totals, as in GeoX; its interval was fixed with historical A/A tests. Trimmed Match, the pair-level estimator that trims outlier pairs automatically, would be the more robust choice when pairs differ a lot; here the pair diagnostic and an advertiser check do that trimming by hand.
GA4 is simulated at channel level, from the platforms' own claims. The GA4 export applies stated rules (site share of sales, consent loss, a multiplier on each platform's claim) rather than user paths, so the direction of its mix partly follows the platform reports by construction. It shows how to read GA4 data, not an independent second proof; real GA4 paths would spread credit differently.
Korea has few usable geo units. 40 regions of comparable size is generous: matched pairs in Korea would be built from city and county clusters (si-gun-gu), with Seoul and Gyeonggi hard to split, and sales by region need shipping addresses. Where pairs cannot be matched, the playbook's fallback is a user-level conversion lift study.
The world is mine. The methods were tested against problems I chose; the log below records where they failed a check, including results that came out differently from the plan.
Problems the checks caught, and the fixes
Each entry is a check that failed, why, and what changed. Full log: docs/work_log.md.