"I tested this strategy on the 2022 bear market. It made 40 percent. Here's the proof."
Nice chart. Nice conviction. Nice trap.
The strategy probably doesn't work. The guru just chose the one period where it did, and showed you that one.
Welcome to cherry picking. The most common dishonesty in trading content.
What Cherry Picking Is
You test a strategy across many time periods. You pick the best one. You publish the best one. You don't mention the others.
Reader sees: +40 percent in 2022. Reality: +40 percent in 2022, -15 percent in 2021, +8 percent in 2020, -22 percent in 2019. Average: 2.75 percent per year. Below the market.
Cherry picked, the strategy looks brilliant. Honestly, it's mediocre.
Why It's Everywhere
Because it works. Cherry-picked backtests go viral.
"Look at this strategy during the COVID crash! +67 percent!" generates 10x more engagement than "average strategy, average returns, probably not worth your time."
The incentives are backwards. Honest content loses. Cherry-picked content wins. So the feed fills up with cherry picks.
How to Spot It
Red flag 1: a single time period. If the testimonial is "during X event" — 2020 crash, 2022 bear, 2023 rally — without the full time series, you're seeing a cherry pick.
Red flag 2: no drawdown numbers. Honest backtests show max drawdown. Cherry picks hide it.
Red flag 3: no comparison to buy-and-hold. If the strategy made 40 percent in 2022 but BTC made 60 percent... well, 40 is worse than 60. Hence no comparison shown.
Red flag 4: round numbers that are too clean. Real backtests return numbers like +43.7 percent. Cherry picks often come with ~40, ~50, ~100 — rounded for emphasis, hiding precision.
Red flag 5: no code, no data, no replication. If you can't reproduce it, it doesn't exist.
The 2022 Trap in Crypto
2022 was a bear market. Bitcoin from $69,000 to $15,500. Everything crashed.
In that environment, any strategy that held cash most of the time and shorted occasionally looked heroic. "My bot returned +30 percent while BTC lost 77 percent!"
Run the same bot on 2020-2021 (the bull run). It loses badly. Why? Because shorting in a bull kills you. But the 2022-only pitch doesn't mention 2020-2021.
The honest test of a "bear market strategy" is: does it outperform buy-and-hold over a FULL CYCLE? Usually no.
The 2020 Trap in Stocks
March 2020 COVID crash, then recovery. Anything that bought the dip looked genius. "My strategy caught the COVID bottom!"
Great. How did it do in 2018? Or 2022? Or during 2015-2019 sideways action?
Cherry pickers don't answer. Because the answer is usually "not well."
My Own Close Call
In my Sandra backtest work, I almost fell into cherry picking unintentionally.
Version 2.1 of her strategy (Core + Dip + Hold) made +510 percent over 2015-2026. Excellent.
My instinct: publish the win. Done.
My discipline: run it on a different period. Use rolling selection. See what happens.
Result: +28.5 percent. Not +510 percent. The difference isn't about the strategy. It's about which period I tested.
Had I published the +510 percent number without the rolling test, that would have been cherry picking. Not intentional, but functionally identical to what the YouTube gurus do.
The only reason I caught it: habit of running multiple time windows and universe refreshes.
The Honesty Test
When someone shows you a backtest, ask three questions:
1. "What was the performance in [different period]?" If they only tested one period, you've found a cherry pick.
2. "What was the max drawdown?" If they don't know, they didn't look — or they looked and don't want to say.
3. "How does this compare to buy-and-hold over the same time?" If the answer is "my strategy is better" without a number, they're dodging.
The Positive Test
A non-cherry-picked backtest looks boring. It goes something like:
"The strategy was tested over 2015-2026 on 500 US stocks. Annual returns ranged from -18% (2018) to +62% (2021). Average annual return 14%, vs SPY 13% — so slight alpha. Maximum drawdown -32% (Feb 2020-March 2020). Sharpe ratio 0.9. Works in bull markets, underperforms in sideways years, survives bears."
No magic. No miracles. But you can trust it.
What This Means for You
Every time you see a trading strategy pitch, mentally do this:
- Count the time periods mentioned. One? Cherry pick.
- Look for a drawdown number. Missing? Cherry pick.
- Look for the baseline comparison. Missing? Cherry pick.
If all three are missing, close the tab.
If one is missing, be suspicious. Ask in the comments.
If all three are present, it might be real. Still probably overfit (see article 4), but at least not cherry picked.
What We Do on BearBullRadar
Every backtest I publish has:
- The full time window (usually 6 years minimum, often 10+)
- Max drawdown explicit
- Benchmark comparison (SPY for stocks, HODL for crypto)
- The code is available
- When the number looks suspicious, I explicitly explain why
Example: my momentum strategy claim of +742% over 7 years. Next to it: -53% max drawdown. Benchmark: SPY +212%. Out-of-sample with monthly universe refresh. Trading costs modeled.
Not because I'm virtuous. Because the opposite is lying.
And honesty is the only moat I have.
-> Previous: Overfitting -> Next: Confirmation Bias -> Back to pillar
Sources
- Harvey, Liu — Backtesting — academic paper on cherry picking in factor research
- AQR Capital — Pitfalls of Backtesting — practitioner literature
Your Dominic, who tested every time window so you don't have to trust anyone who didn't.
Disclaimer: Not financial advice. Past performance does not guarantee future results.
