🚀What Benchmark Design Tells Us About the Result of Step 3.5 Flash? Here's a detailed breakdown from model infra engineer & Zhihu contributor P2oileen, who worked directly on the benchmarking infrastructure. 💬"If high scores can't be reproduced, a tech report is just paper." Since June last year, he's been working on model evaluation at @StepFun_ai, focusing on one core goal: making training metrics reliable, reproducible, and trustworthy. And Step 3.5 Flash(196B MoE, A11B) did deliver with good results on different benchmarks. 🧱 Benchmarking at Scale: Lessons from the Infrastructure Our evaluation platform now integrates 300+ internal and external benchmarks. Making these scores "real" required fixing a lot of invisible problems: 1️⃣ Failure-aware evaluation API timeouts, service instability, sandbox crashes—these happen constantly at scale. Instead of silently assigning low scores, we now: • Capture failures across model calls, judgers, tokenizers, and sandboxes • Explicitly report failure rates and causes in the UI 2️⃣ Train–Inference consistency We moved from "convert weights → deploy → test" to in-training evaluation with guarded inference services, eliminating misalignment between training and evaluation. 3️⃣ Managing 300+ benchmarks without chaos Early on, one maintainer + manual reviews = daily firefighting. Now we have: • Standardized onboarding & release processes • Clear ownership for every benchmark • AI-powered reviewers • Experiment with Agents auto-integrating benchmarks 🔍 Making Scores Credible To ensure metrics actually mean something: • Reproduce competitor results internally to verify alignment • Check for data contamination by deduplicating evaluation sets • Reduce false negatives via: - Prompt engineering (e.g. \boxed{} answer formats) - Upgraded LLM-based judgers - Fixing brittle answer extractors • Statistical testing for small datasets (mean + variance) • Pretrain-specific strategies: output truncation, stop tokens, few-shot pattern stabilization • Prompt alignment is critical. They open-source all System + Question Prompts in the appendix — reproducibility matters more than prompt tricks. 🧠 Reasoning-Heavy & Long-Context Tasks Step 3.5 Flash produces long, reasoning-dense outputs. We added: • Token-level reasoning length monitoring • Streamed inference optimizations (socket buffers, keep-alive strategies) These infra-level details are unglamorous—but they're what make high scores on AIME, IMO, FRAMES actually possible. 📌 Final Takeaway After 8 months of large-model evaluation, one lesson stands out: ✨Evaluation shouldn't just "follow" training—it should slightly lead it. For Step 3.5 Flash, strong results came from carefully chosen benchmarks and well-designed evaluation protocols. Our next goal is to formalize this intuition into scalable quality checks—so evaluation certainty can offset training uncertainty. We hope this experience is useful to the community—and that it helps power Step 4 and beyond 🚀 Feel free to try Step 3.5 Flash, and happy to discuss benchmarking with fellow practitioners. 📖 Read more: zhihu.com/question/200191378… 🔗 Full evaluation protocols: github.com/stepfun-ai/Step-3… #Step35Flash #OpenSource #AI #LLM #MoE #Reasoning #Agent #AIInfra