The Fake Test That Wasted Your Last Three Months

I was on a group coaching call recently when a guy pulled up his campaign stats and walked me through three email variants he'd been running for months.

Different wording on the CTAs. A slightly tweaked opener on one of them. Some variation in how he phrased the subject line. He was proud of this. He'd been iterating.

I looked at the stats. Same target audience across all three. Same case study. Same offer. And the structure was identical across all three. And then I said it, and I watched him go quiet: You haven't tested anything.

He'd spent three months on this, sent hundreds of emails, and kept careful documentation of open rates and reply rates - without changing a single variable that could have explained a different outcome.

Campaigns break this way - eyes on the wrong variables, while the thing causing the problem goes untouched.

What a Test Is

A test is only a test if the variable you're changing is capable of explaining the outcome.

If you swap a variable that couldn't have caused the result to change, you're performing the idea of experimentation while your problem sits there untouched, getting worse.

Changing "Mind if I send over a few times for a quick call?" to "Would you be against me sending some consulting clients your way?" is not a test of anything meaningful. Both of those CTAs are going to the same people, pitching the same case study, making the same implicit promise. If one gets a 0.5% higher reply rate, you've learned nothing you can act on. The sample size will swamp any signal that difference could produce.

But swap your entire target audience - same email, different niche, different company size - and now you're running a test. Because the variable you changed is capable of explaining a dramatically different result.

Changing superficial things feels like work. It takes time. It requires decisions. It produces data. All of the external markers of productive experimentation are present. The only thing missing is the possibility of learning anything.

The Order You Should Be Testing In

Campaigns stall when people optimize the wrong variable.

Subject line, CTA, case study - when those are broken, they're not the starting point for your fixes.

Test these things, in this sequence, before you touch anything else:

1. Lead quality and targeting. This comes first because if you get this wrong, every other optimization you make downstream is wasted. I don't care how well-written your email is - if you're sending it to people who can't afford you, or who have no reason to care about your offer, you've burned the send. Get this right first.

2. Case study and benefit framing. Once you know your list is solid, your case study does the selling. When a campaign is broken, the case studies are soft. "'We helped our client develop their software solution in a cost-efficient way' is a vague claim that can't be evaluated. Put a number in it. We saved them 30% compared to hiring US developers. We doubled their leads from 45 per month to 178. We added $30,000 in new contracts in a single week. Get on the phone with your client and ask them what they'd say to their boss about the results. That's your case study.

3. Call to action. Only after targeting and case study are working do you start testing CTAs. And when you test them, test meaningfully different versions - not synonym swaps. Test asking for a meeting directly vs. asking to send a video vs. asking to send notes. Those are structurally different asks that will produce structurally different responses.

4. First lines. Personalization. This is last because it has the least pull of the four. A killer case study going to a bad list will fail. A mediocre first line going to a great list with a great case study will still book meetings.

The guy I was coaching had skipped straight to testing variations of #3 without ever properly addressing #1 or #2. His case study was vague. He wasn't filtering for companies that could afford to hire him - no minimum revenue threshold on the targeting at all. And then he was carefully A/B testing whether "Mind if I send some times?" outperformed "Are you free for a quick call?"

Adjusting the wing mirrors on a car with no engine.

The Case Study Problem Is Almost Always a Data Problem

In broken campaigns, the case study problem tends to look the same: the result is described without numbers, or the result described doesn't match the audience being targeted.

A guy on the same call was pitching a chatbot service. His case study mentioned a real estate investment company he'd worked with - he'd doubled their leads, from 45 per month to 178. Good numbers. But then he was sending those emails to CMOs of SaaS companies.

He already knew this was wrong. He said it out loud: "I think I messed up because I'm using a case study for a real estate company and targeting SaaS CMOs."

Right. The prospect needs to see themselves in your result. If you're going after real estate investment firms, your case study should be about real estate investment firms. "We doubled leads for a company like yours" works. "We doubled leads for a completely different type of company" - the prospect mentally checks out.

Case studies break in a second way too: wrong language. For a real estate investment company, the KPI isn't "leads." It might be deal flow. It might be qualified inquiries. It might be applications. Use the word they use for the thing they're measuring - say "leads" when they say "deal flow," and you've shown you don't understand their business.

Go ask your client: what did your boss say when you showed them these results? What metric did they care about? Use that language. Word for word.

And if you don't have a number - if you just have "we helped them" without a before and after - pick up the phone and get one before you send another email. One conversation with your best client, asking "what would you say the results were in concrete terms," gets you more than three months of subject line testing.

The Targeting Problem Compounds Everything

Targeting errors are expensive because they contaminate every test you run after them.

Send to companies that can't afford you and your reply rate drops. You interpret that as an email problem. You start testing subject lines and CTAs to fix a reply rate that was never broken in the way you thought. The email was fine all along - you were just reaching the wrong people.

One of the callers I worked with was running a PR placement service. Good offer - guaranteed placements in places like Fast Company and Inc., performance-based pricing. He was getting replies, but a lot of those replies were people asking if it was free.

His instinct was to add pricing information to the cold email itself. I talked him out of it. Putting price in a cold email hands people something to reject before they've understood what they're buying. And frankly, companies that can't tell whether a Forbes placement costs money probably can't afford it anyway.

Targeting was the fix. He was already reaching out to companies doing $1M to $50M a year in revenue - the right bracket. For those companies, $10K for a guaranteed media placement isn't a confusing proposition. They understand services cost money. The confusion was happening on calls, not in emails - the fix was a pre-call qualifier, a short line in the meeting confirmation email that sets the expectation before anyone shows up thinking they're getting something free.

The targeting was solid. The funnel was leaking in the middle, and he was about to blow up a working cold email campaign by stuffing pricing into the subject line to solve it.

How Many Tests Does It Take?

Someone asked me on this call: if a student does everything right and iterates consistently, how long before they see results?

Tests are what move you forward, not volume.

If you send 100 emails and change something meaningful, test again with 100 more, change something meaningful again - you can get results within the first week. If the offer is solid and it's going to the right people, sometimes the first hundred emails produce meetings.

But send 6,000 emails without changing anything meaningful - same list, same case study, slightly tweaked CTAs - and you can go six months and learn nothing. Sending volume without changing meaningful variables is just burning time.

My rough benchmark: if you're sending in batches of 100 and testing the right variables in order, you should see signal within 20 tests. That's 2,000 emails. Some people hit it in 5. Some take 30. But 20 is a reasonable expectation if you're disciplined about what you're changing each time.

The trap is sending 2,000 emails that are all the same test, disguised as different tests by surface-level variation.

What This Looks Like When You Get It Right

A recruiter was running cold email campaigns for blue-collar skilled labor - electrical, mechanical, technical positions. He'd done the targeting work. He'd narrowed to a niche that performed. Now he wanted to optimize.

I told him: keep the niche, keep the targeting, change the case study framing. Test a version that name-drops clients: "We recently helped [Company X] hire a qualified mechanic in under two months." Then test a version that leads with a stat: "Our average time-to-hire is 80% faster than the industry average." Then test a version that leads with impact downstream: "The mechanic we placed saved a major project for that client - the team estimates it protected over $200,000 in contract value."

Same email structure. Same target. Three different benefit statements. Each one makes a different kind of promise and will attract a different kind of person.

That's the testing matrix working correctly. You're not changing words. You're changing the claim you're making and seeing which claim the market responds to.

For lead sourcing, you can pull prospect lists from ScraperCity's B2B database, Apollo, or job boards if you're targeting based on hiring signals - the recruiter on this call was doing exactly that, pulling contacts directly from job postings to reach hiring managers at the moment they had an active need. The signal is built into the list itself - that's why it works. You can also use tools like the ScraperCity Apollo scraper to pull enriched contact data at scale without hitting manual export limits.

On the sending side, the callers on this call were split between Instantly and Omni. For high-volume campaigns where you need to rotate across multiple domains, Instantly is solid. For smaller, more managed sends where you want more campaign-level control, either works. What doesn't work is running everything through a CRM cold email tool that can't handle multi-inbox rotation - I had to tell someone on this call directly: don't use your CRM for cold outreach. It's not built for it.

The No-Brainer Offer Is Not an Email Thing

Someone asked: where does the no-brainer offer fit in? Should it go in the cold email itself?

The email has one job: get a reply or get a meeting. The no-brainer offer - the thing that makes it a stupid decision to say no - comes out on the call, once you're talking to a qualified person who's shown interest. That's what pushes them over the line - not what gets them interested in the first place.

Putting it in the email is anxiety talking. They want to pre-empt the objection. They think if they make the offer irresistible enough in writing, they won't have to sell. If your offer is irresistible, it'll close on the call. If it won't close on a call with a warm prospect, putting it in a cold email won't change that.

The email needs one compliment, one case study, one call to action. Get the meeting, then sell on the call.

If you want templates for this, I've got the top 5 cold email scripts available as a free download - they're built around this exact structure. And if you want the full system for moving someone from cold email to booked meeting to closed deal, go grab the discovery call framework.

The Harder Question

Why do people run fake tests?

The guy I was coaching at the start of this was clearly putting in work. He had spreadsheets. He had stats. And he clearly cared about the outcome.

Changing a CTA and watching it fail teaches you almost nothing - but it risks almost nothing either. Your targeting might still be right. Your case study might still be right. The thing you tested just didn't pan out.

But test a completely different target audience and watch it fail, and you're confronting the possibility that your offer doesn't work for anyone. Test a totally different case study and watch it fail, and you're confronting the possibility that your results aren't compelling enough to sell with. Those are uncomfortable things to find out.

Staying busy with surface-level tests means you never have to confront either of those possibilities. You can defer the hard discovery indefinitely.

The market will tell you the truth eventually. Ask it directly.

Get your targeting tight - right niche, right company size, right person. Build a case study with a number in it. Pick one meaningful variable and test it against a list of at least 100 contacts per variant. Then do it again.

Do it honestly.

For a deeper breakdown of the full outbound framework - targeting, case study, sequencing, and everything downstream - check out the Cold Email Manifesto. Fifteen years of cold email, compressed into one place.

Now go run a test.