Itay Levy (@itayoush) profile picture

Deep Learning Researcher @NVIDIA

Israel
Joined September 2020
Photo by Itay Levy (@itayoush)Photo by Itay Levy (@itayoush)Photo by Itay Levy (@itayoush)Photo by Itay Levy (@itayoush)Photo by Itay Levy (@itayoush)Photo by Itay Levy (@itayoush)Photo by Itay Levy (@itayoush)Photo by Itay Levy (@itayoush)Photo by Itay Levy (@itayoush)Photo by Itay Levy (@itayoush)
🧵 1/3 Our new open-source model, Nemotron-Labs-3-Puzzle-75B-A9B, is out 🎉 We compressed Nemotron-3-Super-120B-A12B into a smaller, faster derivative optimized for interactive deployment.
5
9
2
55
15,221
2/3 The result: 2x higher throughput under the same interactivity constraints, while maintaining strong accuracy across benchmarks. At the core is Iterative Puzzle: a compression recipe that alternates heterogeneous pruning with short recovery phases.
1
11
914
3/3 This lets the architecture search adapt as the model changes, instead of compressing everything in a single step. Check out our tech report to learn more about heterogeneous pruning, knowledge distillation, RL, quantization, and MTP training. huggingface.co/nvidia/NVIDIA…
1
2
19
882
We’re releasing Nemotron-Labs-Diffusion - the first Tri-mode LM family (3B/8B/14B) that switches between 1⃣Autoregressive, 2⃣Diffusion, and 3⃣Self-Speculation decoding by simply changing the attention pattern/mask. One model Three decoding modes. No extra draft models. No architecture changes. Just significantly better efficiency across different concurrency levels. Up to 4× higher real throughput for a single user. 🤗 HF Collection: huggingface.co/collections/n…, open license 🛜 Project page: research.nvidia.com/publicat… 📰 Tech report: bit.ly/Nemotron-Labs-Diffusi… Details below 👇
15
88
20
582
52,230
Itay Levy retweeted
Nemotron 3 Super is live! So far the most intelligent agentic reasoning model in the Nemotron family, with world leading efficiency and openness. Super particularly marks our first infra & research milestone in agentic reinforcement learning scaling up. Stay tuned for more infra, data and agentic generalization research we will open to the ecosystem. 🤗 Huggingface: lnkd.in/gWfamwwX 📜 Tech Report: lnkd.in/gRFFJxKm 🤸‍♂️NeMo-Gym (RL env data and orchestration): github.com/NVIDIA-NeMo/Gym 🤸NeMo-RL (RL training): github.com/NVIDIA-NeMo/RL
6
9
39
1,524
Nemotron 3 Super is here! 🚀 Big capability jump, especially on agentic benchmarks, while staying built for efficient inference. Released the @nvidia way: weights + training recipes + code + datasets. HF: huggingface.co/nvidia/NVIDIA… Tech report: research.nvidia.com/labs/nem…
1
5
37
1,514
Itay Levy retweeted
My last open-source project before joining xAI is just out today. Megatron Core MoE is probably the best open framework out there to seriously train mixture of experts at scale. It achieves 1233 TFLOPS/GPU for DeepSeek-V3-685B. arxiv.org/abs/2603.07685
39
101
12
970
83,005
🧵 New paper: We compressed OpenAI’s gpt-oss-120B into a smaller, faster derivative (gpt-oss-puzzle-88B) with no accuracy loss: ⚡ Up to 1.63× higher token throughput on 8×H100 ⚡ Up to 2.82× on a single H100
1
4
22
1,026
For reasoning models, tok/s isn’t sufficient because trace length can change, so we also measure request-level efficiency. On the accuracy–speed frontier, 88B outperforms 120B across all efforts, with up to 1.29× higher request rates
1
1
156
Try Llama Nemotron Ultra 253B, the smartest open reasoning model available today! 🏆 Tops scientific reasoning, complex math and coding benchmarks ⚡️ 4x higher inference throughput over DeepSeek R1. Optimized with neural architecture search and FFN fusion
1
8
384
It has been an honor for us to collaborate on the Llama-Nemotron family, creating open world-class frontier models! build.nvidia.com/nvidia/llam…
1
119
Puzzle: Distillation-Based NAS for Inference-Optimized LLMs Author's Explanation: xitter.cf/itayoush/status/186358… Overview: Puzzle introduces a distillation-based neural architecture search framework that significantly optimizes LLM inference on specific hardware, achieving a 2.17x speedup with a 98.4% retention of the original model's capabilities via blockwise local knowledge distillation and mixed-integer programming. The framework's application to Nemotron-51B enables a single NVIDIA H100 GPU to handle large batch sizes, highlighting the efficiency of the approach with only 45B training tokens needed compared to the 15T used for the original model, thus prioritizing inference performance over parameter count for model selection. Paper: arxiv.org/abs/2411.19146
Introducing Puzzle: Distillation-Based NAS for Inference-Optimized LLMs 🔗 arxiv.org/abs/2411.19146 🧵
1
2
3
559
Introducing Puzzle: Distillation-Based NAS for Inference-Optimized LLMs 🔗 arxiv.org/abs/2411.19146 🧵
1
1
1
8
789
Puzzle accelerates LLM inference on specific hardware while preserving capabilities. Using decomposed NAS and knowledge distillation, we optimize LLMs under hardware constraints, while requiring only a fraction of the original training compute
1
70
As a highlight, we present Nemotron-51B, derived from Llama-3.1-70B-Instruct, achieving 2.17× inference throughput speedup on a single NVIDIA H100 GPU while preserving 98.4% of the original model's capabilities. Dive into the details of how it all works in our new research paper!
88