Research scientist in NVIDIA. Working in creating a world you cannot distinguish from reality. DLSS5 is my first step. Opinions are my own.
Joined July 2015
- Tweets179
- Following1.8K
- Followers601
- Likes857
Pinned Tweet
Have to correct your misunderstanding of my sentence.
My original sentence: PT *faithfully* simulate the scene. The bottleneck is that the assets are a real-world approximation. This approximation leads to fake feeling.
There does exist many games with great assets, PT makes them look incredible.
Yichen Sheng retweeted
Today, Dream Labs is officially out of stealth! 🚀
What an incredible journey these past six months have been.
I’ve always believed in two things: scaling and generalization. That belief led us to introduce World Action Models (WAMs), which learn from videos of the world and use that understanding to act. With enough scale, the right architecture, and the right data, I believe we can build a general-purpose robot brain that works across different robots, tasks, and environments.
Yet despite incredible progress in robotics, most breakthroughs have remained in research labs, out of reach for the broader public.
Six months ago, @jang_yoel , @scott_e_reed , @youliangtan , and I co-founded Dream Labs with a simple dream: What if anyone could give their robot a general-purpose brain, just through an API call?
Today, we’re taking that first step. Dream-1 is now live for everyone
Dream-1 controls bimanual robots with parallel grippers it has never seen before, zero-shot, on unseen tasks in unseen environments. But what excites me most isn’t just what Dream-1 can do today. It’s what we might discover together.
With our public robot brain API, we want to put general-purpose robot intelligence into everyone’s hands. Try it on new robots, tasks, and environments we’ve never imagined. We want to empower everyone to discover their own “wow” moments with robots and explore what’s truly possible.
ChatGPT wasn’t perfect when it first launched, but it opened up AI to the world. I hope today marks a similar beginning for robotics.
Dream-1 is still a research beta. It will make mistakes, and there’s a long way to go. But someone has to take the first step.
None of this would have been possible without our amazing team. I’m incredibly proud of what we’ve accomplished in just six months and grateful to be building alongside such talented people. ❤️
And we’re just getting started. If you share our dream, come build the future of robotics with us! 🚀
Everyday capabilities for everyday robots @dreamlabs_ai
Today, Dream Labs is coming out of stealth with Dream-1, a robot foundation model that controls bimanual robots it has never seen, on tasks it was never shown, in places it has never been, from $3/hour.
Dream API is live for everyone, today. 🧵
Cheaper experiments → more hypotheses tested. That's the part that actually compounds.
The AI industry has a massive compute efficiency problem. At Physion Labs, we think it's time to challenge it.
Today, we're sharing the technology behind NexTron, our world-intelligence training engine.
Our benchmarks on matched hardware:
⚡ Up to 20.9× SFT throughput vs. DeepSpeed (H100)
⚡ Up to 6.6× SFT throughput vs. Megatron (H200)
⚡ Up to 7.2× SFT throughput vs. Megatron (B200)
⚡ Up to 3.1× end-to-end RL throughput vs. Miles–Megatron
🔥 180B model fine-tuning on just 8 H100 GPUs.
Same hardware. Dramatically higher throughput. Comparable model quality.
We believe the next frontier isn't just building bigger GPU clusters.
It's unlocking more intelligence from every GPU, every watt, and every dollar spent on training.
#PhysionLabs #NexTron #WorldIntelligence #AIInfrastructure
Yichen Sheng retweeted
Please help spread the word!
We’re hiring Research Scientists and Research Interns at NVIDIA Research to work on some of the most exciting problems in generative AI:
🌎 World Models & Video Diffusion
🧠 Diffusion Language Models
🧬 Biomolecule Design
If your research aligns with any of these areas, please apply directly through the links below.
Open positions + application links:
research.nvidia.com/labs/gen…
Yichen Sheng retweeted
“5 years of graphics progress in one toggle”
Now that people have had a chance to play with DLSS5, we can see that neural rendering has redefined real-time graphics.
Ten years of dedicated research and development paying off.
youtu.be/WY5u1-4h48Q
Yichen Sheng retweeted
Yichen Sheng retweeted
Can't wait to see some of you in Berlin!
Open models are how developers build AI they can own.
Not just weights. Model families, post-training recipes, evaluation tools, and supporting data give teams the ability to inspect, adapt, and deploy AI for real applications.
At GTC Berlin, Bryan Catanzaro, VP of Applied Deep Learning Research at NVIDIA, will show developers what that looks like in practice.
🎙️ Building Open Models That Developers Can Inspect, Adapt, and Deploy: nvda.ws/4ivv0qH
📍 #NVIDIAGTC Berlin | October 20–22, 2026
🔗 Register with code GTC-NVP8A3H2 for 25% off: nvda.ws/4j297iN
Yichen Sheng retweeted
Impressed by GPT-Astra’s CoT ! This Kung Fu demo is built from scrach in Blender and takes the reference video + simple prompt as input. The CoT reasoning really show a grasp of kinematics, joint coordination, and body motion !
Reference video comes from PhysAvater @qingqing_zhao_ @yang_zheng18
Yichen Sheng retweeted
GPT-6 Astra feels deeply personal. In summer 2024, during my internship with #NVIDIACosmos team, we started Scenethesis(research.nvidia.com/labs/cos…) with GPT-4o: an agent that could imagine a 3D scene, ground it through promt/vision, align it with physics, judge the result before replaning. Those were good old days.
Two years later, the dream has grown far beyond a single scene. Astra brings much stronger multimodal reasoning and long-horizon tool use; Blender and Unreal are becoming canvases for intelligence. Watching the base model advance this quickly is astonishing—and incredibly exciting.
The mission ahead excites me most: agents that create, simulate, and explore worlds; accumulate knowledge from experience; and keep evolving—for games, robotics, and embodied AI. Huge congratulations to the brilliant @OpenAI team. What a beautiful moment for the agentic world!
GPT-6 Astra is the most intelligent and aligned model in the world, and sets a new state of the art for computer use, browsing, software engineering, cybersecurity, science, and professional work.
openai.com/index/gpt-6-astra…
Yichen Sheng retweeted
🚀🚀 We officially release DLSS 5, a real-time, 3D-guided neural rendering stage that transforms rendered frames into vivid, realistic scenes while preserving developer-authored content and design intent. To achieve both enhanced realism and real-time rendering at up to 4K resolution, we develop a one-step pixel-space diffusion model.
Compared with offline foundation models such as GPT Image 2 and Gemini 3 Pro, DLSS 5 delivers strong realism while much more faithfully preserving the input content, identity, and scene structure.
Looking ahead, we’re continuing to push on realism, controllability, temporal stability, and efficiency. Stay tuned!
Check out our technical report for more on how we think about generative modeling for gaming:
research.nvidia.com/labs/adl…
I feel excited to finally see DLSS 5 released! As a researcher working on visual generative modeling, it’s especially meaningful to see our fundamental research make its way into a real product. This has actually been a very special journey for me—from exploring research ideas, to building our first working prototype, and then working closely with the team to turn the research, step by step, into DLSS 5. Proud of what our team has accomplished!
We’d love to hear your thoughts on how we can continue improving the technology and where generative rendering can take the future of gaming. Discussions are very welcome!
We’re publishing a deeper technical report on why generation is needed for the next leap toward photorealism.
VRAM and compute limit both the scene abstraction and the number of rays we can afford. Previous DLSS reconstructs from that representation—so even perfect reconstruction is still bounded by what the scene contains.
DLSS 5 is the first DLSS to generate beyond that abstraction and break through that ceiling, while staying grounded in the game and controlled by the artist.
We also compare it with offline generative models and renderers at similar scaling cost.
research.nvidia.com/labs/adl…
Yichen Sheng retweeted
There we go 🚀 youtu.be/5khwlu2qD9U
I feel incredibly lucky that my first project after finishing my PhD became #DLSS5.
In my first 1:1 with @edliu1105, I created this slide to show what I believed next-generation rendering could become. For me, this was the beginning of the journey.
My North Star for rendering has always been a world indistinguishable from footage captured by a real camera. The realism upper bound for existing #DLSS is path tracing (PT). PT can faithfully simulate the scene we describe—but that scene remains an approximation of the real world. As a result, it still looks fake, plastic, far from visually realistic in most of the current gaming.
The rise of generative models opened a complementary path. Early results gave us enough signal to pursue a belief: learned real-world appearance priors could work with physically based rendering—not replace it—to move beyond this ceiling.
DLSS 5 is our first step: a generative video model that operates in real time at up to 4K while remaining grounded in the authored scene.
Getting here took years. Skepticism pushed us to clarify the idea, strengthen the system, and make the evidence more rigorous. I’m deeply grateful for the leadership of @edliu1105, @drewtao and @ctnzr, and for our team’s extraordinary execution.
I’m proud that we have demonstrated a new research and design space in graphics rendering—one that deserves much more exploration. If you work in graphics, vision, or generative models, I hope you’ll read our technical report and join us in exploring it.
We’re publishing a deeper technical report on why generation is needed for the next leap toward photorealism.
VRAM and compute limit both the scene abstraction and the number of rays we can afford. Previous DLSS reconstructs from that representation—so even perfect reconstruction is still bounded by what the scene contains.
DLSS 5 is the first DLSS to generate beyond that abstraction and break through that ceiling, while staying grounded in the game and controlled by the artist.
We also compare it with offline generative models and renderers at similar scaling cost.
research.nvidia.com/labs/adl…
People are worrying about AI takes over human's jobs.
After vibe coding for months, I confirmed(a lot of times) I am the bottleneck.
So no need to scale agents, we need to scale human labors to maximize productivity actually 😅😅😅
A super cool world model that you can control the action as well.
We made an Elden Ring boss fight playable inside a real-time video world model.
Text is not a scene prompt. It is the controller.
Play Incantation:
reactor.inc/incantation
Yichen Sheng retweeted
Look forward to it!
The future of computer graphics is here, and it's built by AI. ✨
Don't miss the NVIDIA Research Keynote at SIGGRAPH 2026 for live demos and research examples showcasing breakthrough neural rendering, world models, and simulation methods across creative tools, industrial design, robotics, and autonomous systems.
Yichen Sheng retweeted
A new era of computer graphics is emerging through AI, with new speaker Edward Liu joining NVIDIA Research and Engineering leaders Jan Kautz and Ming-Yu Liu.
Join the 'SIGGRAPH 2026 Sponsored Keynote, Next Era of Graphics — Neural Rendering, World Models, and Simulation'
This session will explore the latest advances in neural rendering, world foundation models, and AI-driven simulation through breakthrough research and live demonstrations. s2026.siggraph.org/program/k…
Yichen Sheng retweeted
Glad to share our PixelDiT work to the community! Many thanks to all coauthors and contributors. This is the first research project I initiate and lead after joining NVIDIA. Thanks for the support from our ADLR team. We will continue pushing the research on pixel diffusion. Check our project page for paper, code, and weights: pixeldit.github.io
Selected as a best paper finalist at #CVPR2026: PixelDiT from NVIDIA Research
In most image generation models, a pretrained autoencoder compresses the image before any diffusion happens, causing quality loss that accumulates across the entire pipeline.
PixelDiT, or Pixel Diffusion Transformers, removes this step entirely. It's a single-stage model that learns the diffusion process directly in pixel space, end-to-end.
Yichen Sheng retweeted
Heading to #CVPR2026! I’m excited to present our work:
I-Scene: 3D Instance Models are Implicit Generalizable Spatial Learners
📍 Denver
🗓 June 6, 4:45–6:45 PM MDT
For years, learning-based interactive 3D scene generation has been shaped and constrained by 3D-FRONT. Methods learn layout distributions from this dataset, making their spatial priors tightly coupled to its limited diversity, and spatial relation statistics.
In I-Scene, we answer a key question:
❓ Where do spatial priors for interactive 3D scene generation come from when data is limited?
Our finding is surprisingly simple: strong 3D instance models already encode rich spatial priors. By reprogramming an instance-level model with scene-context attention and view-centric space, I-Scene can generalize to interactive 3D scenes without relying on heavily annotated scene datasets.
🌟 Key takeaways:
1️⃣ View-centric space matters for spatial relationships
2️⃣ Randomly composed objects can provide surprisingly strong supervision
3️⃣ Strong instance models can serve as a foundation for real-to-sim 3D scene generation
Come by and chat if you’re interested in #GenAI, #SpatialIntelligence, #RealToSim, or #EmbodiedAI!
