Pulsar · Field guide · No. 21 · Why most tests prove nothing, how to isolate a variable, and how to kill losers on evidence instead of vibes.
Find the Ad That Actually Works
Most “testing” is guessing with a budget. Real creative testing changes one thing, funds it enough to speak, and kills on evidence — not vibes.
Why most “testing” proves nothing
Ten ads, a few pounds each, killed on day two because one had a bad afternoon. That isn’t a test — it’s noise wearing a lab coat. Four things ruin most creative tests: too many variants, too little budget each, no clean variable, and a trigger finger.
If a test can’t tell you why one ad won, it didn’t teach you anything you can repeat.
Change exactly one thing
A test is only readable if a single variable moves. Change the hook, or the angle, or the format, or the offer — never several at once — so a winner names the reason it won.
| Isolate this… | …and you learn |
|---|---|
| Hook (first 3s) | which opener stops the scroll |
| Angle (the pitch) | which promise the market wants |
| Format (video/static/UGC) | which container performs |
| Offer | what actually moves the buy decision |
Fund it enough to speak
A test needs enough impressions (and ideally conversions) to climb out of noise. Underfund it and you’re reading coin flips — the same trap as an ad set that never exits the learning phase.
Test the hook first
The first three seconds decide most of an ad’s fate. Before you polish everything, test hooks — it’s the highest-leverage variable there is (its own lecture: No. 22).
Keep the winning body, swap five openers, and let the hook rate tell you which one earns the rest of the ad a chance.
Set kill criteria before you launch
Decide the metric, the threshold and the window up front — in writing. Otherwise you’ll rationalize a loser (“give it one more day”) until it drains the budget.
Write the rule before the emotion
“If hook rate < X after N impressions, cut it.” “If CPA > target after M conversions, cut it.” A rule set in advance survives the moment you’re attached to your own idea.
Read the metric the test is actually about
Match the metric to the variable. A hook test lives on 3-second plays / thumbstop / CTR. An offer test lives on CVR and CPA. Judging a hook test by purchases (which need volume + time) just adds noise.
The testing cadence
And once you have a winner — stop editing it. Every change resets learning (No. 12). Test around the winner; don’t iterate it into the ground.