Turning Ads Off in Four Regions

I’ve spent eleven years watching agencies try to sell “statistical significance” and “multi-variate regional modeling” to business owners who just want to know if their ad spend is actually working. Most of these consultants will tell you that learning how to run a geo holdout test requires a data science degree and a six-figure software budget, which is complete nonsense. They love to bury the truth under layers of jargon because if you realized how simple the math actually is, they’d lose their grip on your monthly retainer.

I’m not here to give you a lecture or a slide deck full of useless diagrams. My promise is simple: I am going to show you how to run a geo holdout test using nothing more than a basic spreadsheet and a bit of common sense. We are going to strip away the enterprise fluff and focus on a method you can set up in a single afternoon. By the end of this, you’ll know exactly which cities to turn off and how to read the results without needing a PhD.

The Afternoon Blueprint for Marketing Incrementality Measurement

Forget the academic papers on causal inference in advertising. You don’t need a PhD to figure out if your local radio spot or Facebook campaign is actually driving sales or just riding the wave of existing demand. To do this in an afternoon, you need a simple geographic split testing methodology: pick two markets that look almost identical on paper. If you sell landscaping services in North London and South London, and both have similar average household incomes and weather patterns, you have your pair.

One city is your “test” (where you keep the ads running) and the other is your “holdout” (where you turn everything off). This is your control group selection for geo tests. Don’t overthink the math; just ensure they aren’t experiencing wildly different local events—like one city hosting a massive festival that the other isn’t.

Once the ads are toggled, stop looking at the dashboard every ten minutes. Set a date two weeks out, check your total sales in both regions, and compare the gap. If the “test” city outperformed the “holdout” city by a margin that feels larger than your usual weekly fluctuation, you’ve just achieved measuring media lift via holdouts without spending a cent on a data scientist.

Why Geographic Split Testing Methodology Beats Your Gut Feeling

Most small business owners I work with are running on pure instinct. They see a spike in sales during a campaign and assume the ads did the heavy lifting. But instinct is a liar. You might just be seeing a seasonal trend or a random surge in local interest that would have happened even if you’d spent the month sitting on your hands. Relying on “gut feeling” is how you end up pouring money into channels that aren’t actually driving growth.

If you want the truth, you need marketing incrementality measurement. This isn’t some academic exercise in causal inference in advertising; it’s simply the process of proving that your spend actually caused the sale, rather than just riding the wave of existing demand. By using a geographic split testing methodology, you stop guessing. You compare a region where you’re pushing hard against a similar region where you’re doing nothing. The difference between those two numbers is your real profit. Anything else is just expensive guesswork that leaves you wondering why your margins are shrinking despite “better” results.

Five Ways to Stop Wasting Money on Ads That Don't Work

• Pick your cities based on size, not sentiment. Don’t pick your favorite town just because you grew up there; pick two cities with similar populations and similar sales volumes so the math actually makes sense.
• Ignore the “vanity” metrics. When you’re running the test, stop looking at click-through rates or impressions. They are noise. Only look at the raw sales numbers in your holdout city versus your active city.
• Keep your variables locked. If you’re running a sale in your active zone, don’t run a separate promotion in the holdout zone. If you change too many things at once, you won’t know if it was the ad or the discount that moved the needle.
• Don’t overthink the duration. You don’t need a six-month longitudinal study. Run the test for two weeks, check the delta in sales, and move on with your life.
• Prepare for the “ugly” truth. Most small businesses find out their ads are doing about 20% less work than they thought. Don’t panic when the numbers drop; that’s just the baseline you needed to stop overspending.

The Bottom Line

Stop guessing if your ads work; pick two similar towns, kill the spend in one, and let the sales data tell you the truth.

If a testing method requires a data scientist and a month of prep, it’s a waste of your time—keep it simple enough to run yourself.

Use the results to cut the dead weight; if turning off ads in City B doesn’t tank your revenue, stop paying for those clicks immediately.

Stop Guessing and Start Testing

Look, we’ve stripped away the enterprise fluff. You don’t need a data science degree or a six-figure software budget to figure out if your ads are actually driving sales or just riding the wave of existing demand. All you need is a clear map, two comparable territories, and the discipline to stop looking at vanity metrics that make you feel good but don’t pay the rent. By picking a holdout city and comparing it against your active zones, you get a real, numeric answer to the only question that matters: is this spend actually working? If you can’t measure it with a simple spreadsheet by tea time, you’re just throwing money into a black hole.

My advice is simple: stop waiting for the “perfect” data set to arrive. It isn’t coming, and while you’re waiting, you’re likely wasting a significant chunk of your monthly budget on channels that aren’t pulling their weight. Use this afternoon to run your first test. It might feel uncomfortably basic compared to the complex models the big agencies peddle, but results beat complexity every single time. Get your numbers, make your decision, and get back to running your business. You have enough on your plate without playing mathematician; just find out what works.

Frequently Asked Questions

What if my two cities aren't actually "similar" enough to compare?

If they aren’t similar, don’t force it. You’ll end up chasing ghosts in your data. If City A is a tech hub and City B is a retirement village, the comparison is useless. Instead, look for “twin” metrics rather than identical demographics. Do they have similar baseline sales volumes? Similar seasonal swings? If the numbers move in the same rhythm normally, you can use them. If not, pick different cities. Better a messy test than a wrong one.

How long do I actually have to leave the ads turned off before the data is worth anything?

Stop looking for a magic number; it depends on your sales cycle. If you sell coffee beans, three days might suffice. If you sell enterprise software, you’ll be waiting months. My rule of thumb: wait until you have enough natural sales variance to actually see a dip. If your revenue fluctuates wildly day-to-day anyway, you’ll need a longer window to drown out the noise. Don’t turn them back on just because you’re getting impatient.

If I see a massive drop in sales, do I turn the ads back on immediately or wait it out?

Don’t panic and hit the “on” switch immediately. If you react instantly, you’re just chasing noise. A single bad afternoon isn’t a trend; it’s a statistical hiccup. Wait at least three to five days to see if the numbers stabilize. If the slump persists through the week, then you have a real problem. Data needs time to breathe. If you flip the switch every time a line dips, you’ll never actually learn anything.

Can I do this with just one specific product line, or do I have to pull the plug on everything in that area?

Stick to one product line. If you pull the plug on everything in a region, you’re not testing your marketing; you’re testing your entire business model, which is a much bigger, much more expensive mess to clean up. Pick your most consistent seller, pause the ads for that specific line in your test city, and see if the revenue dips. It’s cleaner, faster, and won’t leave you staring at a total sales crater.