X
@pgasawa thread (15): @pgasawa: Today, we’re releasing Continual Learning Bench…
@pgasawathreadx
TL;DR. Continual Learning Bench 1.0 tests whether AI systems actually get better with experience on ordered, stateful task sequences — and most frontier models barely beat their stateless selves.
Takeaways
- Introduces a "gain" metric: performance vs. the same system run stateless, so a strong-but-static model scores ~0 while a weaker learner scores positive.
- Tasks (e.g., Codebase Adaptation, SWE-Bench-style sequences) are expert-validated to have latent structure that can't be solved by offline training alone.
- Surprisingly strong baseline: just appending history (naive in-context learning) beats fancier approaches — but there's still major headroom on real learning.