@GeneralityLabs
Joined September 2026
  • Tweets13
  • Following0
  • Followers18
8Media
LABBench2 has been used by several labs and other orgs to measure the bio-capabilities of frontier models. We audited it and found some issues: many task sets are saturated, two are flawed, and the rest are at least moderately concerning. More below🧵
1
2
3
51
more replies
Recommendations: • Report LABBench2 per task set, not as one aggregate • Drop the saturated sets (TrialQA, PatentQA, SourceQuality, TableQA2, maybe LitQA3) unless checking for regression • Fix harness for model comps, especially for DbQA2 and SuppQA2
1
6
More in the full post, including gated transcript examples: generality.org/blog/posts/la… And the SeqQA2 post: generality.org/blog/posts/la…

LABBench2 Audit - Generality Labs

Most of LABBench2’s task sets now offer little signal, due to saturation, questionable construct validity or sensitivity to the judge model. The rest remain useful, but their results must be interp...

generality.org
6
Is GLM-5.3-Flash Mythos-level at cyber? In our latest post, we find that GLM-5.3-Flash can outperform Mythos Preview on ExploitBench at ~6% of the price. We do this by scaling up inference tokens to 1B per vulnerability.
1
2
7
222
see the post for more: generality.org/blog/posts/gl…

Is GLM-5.3-Flash Mythos-level at Cyber? - Generality Labs

We ran GLM-5.3-Flash on ExploitBench with a budget of 1 billion tokens per vulnerability. It matched Claude Mythos Preview’s performance at about 6% of the cost.

generality.org
2
122
Load more