No. 01
Kahlus
Machine learning · EEG forecasting benchmark · Python, PyTorch
A leaked scoring rule was inflating the benchmark 126×. I found it in my own evaluation, proved the bound, and made the leak impossible to reconstruct.
Read the detail
A leakage-controlled benchmark for forecasting brain state from EEG. The finding that matters came from auditing my own evaluation: the input and target windows overlapped by 126 of 127 samples, so the headline score was measuring a model copying its input one step shifted, not forecasting anything. Rebuilt overlap-free, the identical ridge baseline moved from 0.0018 to 0.227 MSE — a 126× swing produced by the scoring rule alone, on one model and one dataset. I then derived an analytic bound on how much overlap can inflate a score, checked it across 286 configurations with zero violations, and added a guard that makes the leaking configuration impossible to construct.
| Scoring rule | Ridge MSE |
|---|---|
| Original, overlapping window | 0.0018 |
| Isolated strictly-future sample | 0.227 |
| What it measures | Forecasting skill under strict holdout |
|---|---|
| Honest result | The model loses under isolated evaluation it does not beat ridge or persistence |
| Bound | Validated across 286 configurations, 0 violations |
| Repository | 316 commits, 291 mine |








