data + crypto = magic internet money
Joined February 2018
- Tweets170
- Following53
- Followers55
- Likes142
It looks much worse on hyperliquid haha. I've recently too had a drawdown on my xs neutral portfolio but not as bad as this one...
Most of you split your data the same way. First two thirds in-sample (IS), last third out-of-sample (OOS).
Then you check the IC, fwd return deciles, the Sharpe...
If the OOS numbers hold up, the strategy is "validated."
Here's how that goes wrong.
Hardly anyone checks what the market was actually doing in each part.
Say the first two thirds holds the crash, the chop and the slow bleed. The last third is one clean bull run.
You optimize on the hard part. Every parameter tweak that dodges a bad stretch pushes the IS Sharpe up. By the end, the model is great at surviving that one crash and that one chop.
Then it reaches the OOS. A bull run, where almost anything that's long makes money. It passes.
But the test never asked the real question: does this work in a hard market it wasn't tuned on? Every hard market went into the tuning. Only the easy one was left for the grade.
Before you look at a single metric, look at the chart:
- What regimes are in your IS?
- What regimes are in your OOS?
If your OOS is one easy market, the test was decided before you ran it.
Put it over a stretch that actually hurts, and let the IS take both ends.
An out-of-sample test is only as hard as the market you put in it.
Just watched one of your videos: youtu.be/KCWPOkQGLa0?t=566
You clearly overfit that strategy there aswell. You can't just grid search for the best parameters and pick the ones that work the best.
I made a whole post about how to avoid doing exactly that xitter.cf/hyperfitted/status/210…
How I use heatmaps to avoid overfitting:
Split your data in two. One part you optimize on (in-sample). The other you lock away and don't touch until the end (out-of-sample).
Run your parameter grid on IS. Then, instead of picking the single best cell, average each cell with its neighbors and pick the best average.
Why? One great cell surrounded by bad ones is usually luck. A cell whose whole neighborhood performs well is much more likely to hold up.
Then run the same grid on OOS and check whether that area still looks good.
If it does, you might have a real edge.
If it falls apart, you were fitting noise, and it's better to learn that now than with real money.
This is what a robust strategy's heatmap should look like:
The direction signal, by contrast, flipped.
Long falling OI, short building OI, over 3 days:
2022: +35 bps
2023: −17 bps
2024: −12 bps
2025: −128 bps
2026 H1: −151 bps
That's what you'd expect if the short leg keeps getting caught in squeezes.
Not one squeeze either. Among the most volatile coins, building OI had more 30%+ rallies than the middle group in 5 of 5 years, and more 30%+ drops in 5 of 5.
174 rallies of 50%+ after building OI, spread over 76 different coins.
Isn't that just volatile coins being volatile? I compared coins with similar past volatility.
Falling OI: volatility calmed down.
Building OI: it stayed high. In all 3 groups.
Most volatile third: 1.6% of building-OI coins rallied 50%+ in 3 days, vs 0.4% for the middle.
Back to the first chart. OI isn't useless. It answers a different question:
Which way vs How far
Which way: building-OI coins beat the median over the next 3 days 48% of the time. Falling OI: 48%.
How far: building OI saw 30%+ moves 2.4x as often as the middle group.
But that −45 bps came from 1% of coin-days.
Worst 1%: −136 bps
Middle 98%: −5 bps
Best 1%: +95 bps
100% of the 25 worst were building-OI coins that then exploded. MYX alone: −19 bps. It did +1,103% in 3 days.
Test 2: the contrarian version. If building OI is just crowding, fading it should pay. Each day: long the Top 80 coins where OI fell most, short where it grew most.
Result: −45 bps over 3 days [−83, −9].
Building OI did better, just as the rule says.
Test 1: the rule itself. Did rallies with rising OI beat rallies with falling OI?
On average, yes: +70 bps more over the next 3 days.
Remove the biggest 1% of moves and the gap is −0.3 bps. The typical rally faded either way.
