Situation. A consumer brand selling through two channels — roughly $12M direct-to-consumer and around $10M on Amazon — spending heavily on paid social and Amazon's demand-side platform. Both platforms reported strong returns. Both sets of numbers were, in the ordinary sense, true: the platforms could show orders following ad exposure.
The number nobody had computed. Whether any of those orders were incremental. Platform attribution credits a channel for sales that would have happened regardless, and the size of that overlap is unknowable from inside the platform's own reporting. The only way to find out is to change the spend and watch what happens to total sales in the real world.
The test and how we made it readable. Geography, not attribution. We picked five metros, increased Facebook and Instagram spend fivefold there — from about 7% of total budget to 35% — and measured orders by shipping ZIP code across both Shopify and Amazon, against the rest of the country as control. Shipping address is the honest instrument: it doesn't care what the platform claims.
Before starting, we drew the line the results should follow if the reported return were real. A 500% spend increase should produce a large, visible step up in the test geos.
We ran the same design on Amazon DSP, increasing spend more than tenfold in five metros and reading Amazon shipping ZIPs.
Result. Paid social: the test regions' share of sales moved from roughly 10.9% to 12.2%. A 10% lift against a 500% spend increase — effectively nothing.
Amazon DSP: zero lift. Sales in the test geos actually drifted slightly down, within normal variation.
We then ran the inverse in a separate market, cutting spend rather than raising it. The return on the avoided spend came in at about 1.14 — meaning the marginal dollar was barely breaking even, at an incremental cost per customer of roughly $114.
What transfers. Two things, and the second is the more useful.
First: platform-reported return is not incrementality, and the gap between them is usually large. If a channel matters to your budget, the only trustworthy read is a geographic holdout measured on your own order data.
Second — and this is what the Amazon result explains — advertising can only create incremental sales where the platform is a discovery surface for your product. Buyers were arriving at Amazon already searching for the brand by name. Ads placed in front of people who have already decided don't create demand; they harvest demand that was coming anyway and then take credit for it. Before testing any channel, ask whether buyers there are discovering products or fulfilling intentions they arrived with. The answer determines whether spend can do anything at all.
