Let me make a confession.
Over two days of Sandra backtest work, I built 8 variants of her strategy. v2.0 through v2.8.
If I were a YouTube trader, I'd have told you about one of them. The best one. +510 percent. Moon.
I'd have quietly deleted the others. v2.0 lost 330 percentage points to v2.1 (+180% vs +510%). v2.2 made only +22 percent. v2.6 made +11 percent. v2.7 killed two winning positions at the bottom.
You'd never have known.
This is confirmation bias. Your own worst enemy.
What It Is
Confirmation bias: the tendency to remember your wins and forget your losses, to save your winning backtests and delete your losing ones, to publish what confirms your thesis and bury what doesn't.
Everyone does it. Myself included, unless I force myself not to.
The 8 Variants I Tested
For the Sandra strategy, here's the full list, ranked by return:
| Variant | What it did | Return | Published? |
|---|---|---|---|
| v2.1 Pure Hold | Core + dip + max 5 + HOLD | +510% | ✓ Winner |
| v2.8 Clever Dead-Horse | + weekly stop + 3/3 sunset rule | +57% | ✓ Runner-up |
| v2.3 Stop + Trend | Trailing stop with SMA20/50 filter | +44% | ✗ Hidden |
| v2.7 Smart (2/3) | Dead-horse at 2/3 criteria | +43% | ✗ Hidden |
| v2.2 Daily Stop | Raw trailing stop | +22% | ✗ Hidden |
| v2.6 Complete | Full exit on death cross | +11% | ✗ Hidden |
| v2.0 Trim Ladder | Take profits at +100/200/400% | +180% | ✗ Hidden |
| v1 CYCLE | Sell at each new high | +7% | ✗ Hidden |
If I only published v2.1, you'd think I was a genius. +510 percent strategy. Nailed it.
If I only published v2.6, you'd think I was incompetent. +11 percent over 11 years is miserable.
Neither story is the truth. The truth is: I tested 8 variants and 6 of them failed or were mediocre. I learned from all 8. That's the real story.
8 variants tested. You saw 2. Here are all of them.
Each bar is a different execution rule set over the same 11 years. Cherry-pick the green ones and I look like a genius.
If I'd only published v2.1, you'd call me a genius. If I'd only published v2.6, you'd call me an idiot. Neither is the truth — the truth is all 8 bars. Most YouTube traders show you one.
Why It Matters for You
When you see a YouTuber's "backtest," you're seeing the one that survived their confirmation filter.
The other 20 they tested and dropped? Didn't make the video.
This is why so many strategies look brilliant and perform terribly. You're seeing curated wins. Not random samples.
The Mental Trick That Fights It
Every experiment is a data point. Wins and losses are equally informative.
In fact, losses are often MORE informative. A failed variant teaches you what doesn't work. That narrows the real winning space faster than another vague win.
When I ran v2.6 (the one that exit-sold NFLX and SPOT at the 2022 bottom, losing 22k and 35k on those positions), I learned something specific: death cross signals are too noisy for growth stocks. They fire on normal corrections.
That insight informed v2.7 and v2.8. Without the v2.6 failure, I wouldn't have known to soften the rule.
Had I deleted v2.6 out of embarrassment, I'd have lost the learning.
The Rule I Now Follow
Publish every experiment.
On BearBullRadar's BotLab page, all 23 bots are live. Including the ones that lost money on 6 years of data.
The Volume Spike Bot — my embarrassing "gold nugget" that collapsed from +45 percent to -27 percent? Still there. Clearly labeled. "Overfitting example."
The VWAP Bot that made +15 percent on short data and lost -12 percent on long data? Still there.
If you delete the losers, you break the teaching. And worse, you lie to yourself about your success rate.
How to Test for Your Own Confirmation Bias
1. Count your folders. How many strategy files do you have? How many are visible on your site or in your portfolio? If the ratio is 1:10 or worse, you're filtering heavily.
2. Check your deletion pattern. When you delete a backtest result, ask why. "It was broken code" — valid. "It looked bad" — confirmation bias.
3. Review your forecasts. Did you predict the winning strategy beforehand? Or did you back-fit a story after seeing the result? If the latter, your belief in that strategy is weaker than it feels.
4. Ask a friend. Show them all your experiments. Which do they find most convincing? Compare to your pick.
The Silicon Valley Twist
Tech startups operate under extreme confirmation bias. Every founder tells a clean story of "we knew it would work." Usually they pivoted 5 times, had 3 near-deaths, and lost 10 co-founders along the way.
The story survives. The reality is forgotten. Both by founders and by listeners.
Trading is the same. The "systems that work" exist inside a graveyard of systems that didn't. If you don't remember the graveyard, you can't evaluate the survivor honestly.
My Practical System
For every trading/backtest project I do:
- Archive directory: all variants stay. Named and dated.
- Results JSON: every run writes results, even failures.
- Published results.md: full table of every variant with honest metrics.
- Blog article: summarizes winners AND losers. Explains why each failed or succeeded.
Takes twice as long. Feels naked. Helps me actually improve.
What This Means for Sandra
I could have told Sandra "your strategy makes +510 percent — congrats."
Instead I told her "your strategy makes +510 percent on the cherry-picked sample, +28 percent on an honest sample, and I tested 8 execution variants and most of them were worse than v2.1."
Her first reaction: slight disappointment. Her second reaction, 10 minutes later: relief.
Because now she knows. She trusts the picture. She can decide what to do with real money based on realistic numbers.
That is the only gift honest confirmation-bias-resistance gives: better decisions.
-> Previous: Cherry Picking -> Next: P-Hacking -> Back to pillar
Sources
- Nickerson 1998 — Confirmation Bias: A Ubiquitous Phenomenon — foundational psychology paper
- Taleb — Fooled by Randomness — book-length treatment for finance
Your Dominic, who publishes the losers because the losers teach more.
Disclaimer: Not financial advice. Past performance does not guarantee future results.
