Research scientist in NVIDIA. Working in creating a world you cannot distinguish from reality. DLSS5 is my first step. Opinions are my own.
Joined July 2015
- Tweets179
- Following1.8K
- Followers601
- Likes857
Yichen Sheng retweeted
Today, Dream Labs is officially out of stealth! 🚀
What an incredible journey these past six months have been.
I’ve always believed in two things: scaling and generalization. That belief led us to introduce World Action Models (WAMs), which learn from videos of the world and use that understanding to act. With enough scale, the right architecture, and the right data, I believe we can build a general-purpose robot brain that works across different robots, tasks, and environments.
Yet despite incredible progress in robotics, most breakthroughs have remained in research labs, out of reach for the broader public.
Six months ago, @jang_yoel , @scott_e_reed , @youliangtan , and I co-founded Dream Labs with a simple dream: What if anyone could give their robot a general-purpose brain, just through an API call?
Today, we’re taking that first step. Dream-1 is now live for everyone
Dream-1 controls bimanual robots with parallel grippers it has never seen before, zero-shot, on unseen tasks in unseen environments. But what excites me most isn’t just what Dream-1 can do today. It’s what we might discover together.
With our public robot brain API, we want to put general-purpose robot intelligence into everyone’s hands. Try it on new robots, tasks, and environments we’ve never imagined. We want to empower everyone to discover their own “wow” moments with robots and explore what’s truly possible.
ChatGPT wasn’t perfect when it first launched, but it opened up AI to the world. I hope today marks a similar beginning for robotics.
Dream-1 is still a research beta. It will make mistakes, and there’s a long way to go. But someone has to take the first step.
None of this would have been possible without our amazing team. I’m incredibly proud of what we’ve accomplished in just six months and grateful to be building alongside such talented people. ❤️
And we’re just getting started. If you share our dream, come build the future of robotics with us! 🚀
Everyday capabilities for everyday robots @dreamlabs_ai
Today, Dream Labs is coming out of stealth with Dream-1, a robot foundation model that controls bimanual robots it has never seen, on tasks it was never shown, in places it has never been, from $3/hour.
Dream API is live for everyone, today. 🧵
Yichen Sheng retweeted
Please help spread the word!
We’re hiring Research Scientists and Research Interns at NVIDIA Research to work on some of the most exciting problems in generative AI:
🌎 World Models & Video Diffusion
🧠 Diffusion Language Models
🧬 Biomolecule Design
If your research aligns with any of these areas, please apply directly through the links below.
Open positions + application links:
research.nvidia.com/labs/gen…
Yichen Sheng retweeted
“5 years of graphics progress in one toggle”
Now that people have had a chance to play with DLSS5, we can see that neural rendering has redefined real-time graphics.
Ten years of dedicated research and development paying off.
youtu.be/WY5u1-4h48Q
Yichen Sheng retweeted
Yichen Sheng retweeted
Can't wait to see some of you in Berlin!
Open models are how developers build AI they can own.
Not just weights. Model families, post-training recipes, evaluation tools, and supporting data give teams the ability to inspect, adapt, and deploy AI for real applications.
At GTC Berlin, Bryan Catanzaro, VP of Applied Deep Learning Research at NVIDIA, will show developers what that looks like in practice.
🎙️ Building Open Models That Developers Can Inspect, Adapt, and Deploy: nvda.ws/4ivv0qH
📍 #NVIDIAGTC Berlin | October 20–22, 2026
🔗 Register with code GTC-NVP8A3H2 for 25% off: nvda.ws/4j297iN
Yichen Sheng retweeted
Impressed by GPT-Astra’s CoT ! This Kung Fu demo is built from scrach in Blender and takes the reference video + simple prompt as input. The CoT reasoning really show a grasp of kinematics, joint coordination, and body motion !
Reference video comes from PhysAvater @qingqing_zhao_ @yang_zheng18
Yichen Sheng retweeted
GPT-6 Astra feels deeply personal. In summer 2024, during my internship with #NVIDIACosmos team, we started Scenethesis(research.nvidia.com/labs/cos…) with GPT-4o: an agent that could imagine a 3D scene, ground it through promt/vision, align it with physics, judge the result before replaning. Those were good old days.
Two years later, the dream has grown far beyond a single scene. Astra brings much stronger multimodal reasoning and long-horizon tool use; Blender and Unreal are becoming canvases for intelligence. Watching the base model advance this quickly is astonishing—and incredibly exciting.
The mission ahead excites me most: agents that create, simulate, and explore worlds; accumulate knowledge from experience; and keep evolving—for games, robotics, and embodied AI. Huge congratulations to the brilliant @OpenAI team. What a beautiful moment for the agentic world!
GPT-6 Astra is the most intelligent and aligned model in the world, and sets a new state of the art for computer use, browsing, software engineering, cybersecurity, science, and professional work.
openai.com/index/gpt-6-astra…
Yichen Sheng retweeted
🚀🚀 We officially release DLSS 5, a real-time, 3D-guided neural rendering stage that transforms rendered frames into vivid, realistic scenes while preserving developer-authored content and design intent. To achieve both enhanced realism and real-time rendering at up to 4K resolution, we develop a one-step pixel-space diffusion model.
Compared with offline foundation models such as GPT Image 2 and Gemini 3 Pro, DLSS 5 delivers strong realism while much more faithfully preserving the input content, identity, and scene structure.
Looking ahead, we’re continuing to push on realism, controllability, temporal stability, and efficiency. Stay tuned!
Check out our technical report for more on how we think about generative modeling for gaming:
research.nvidia.com/labs/adl…
I feel excited to finally see DLSS 5 released! As a researcher working on visual generative modeling, it’s especially meaningful to see our fundamental research make its way into a real product. This has actually been a very special journey for me—from exploring research ideas, to building our first working prototype, and then working closely with the team to turn the research, step by step, into DLSS 5. Proud of what our team has accomplished!
We’d love to hear your thoughts on how we can continue improving the technology and where generative rendering can take the future of gaming. Discussions are very welcome!
We’re publishing a deeper technical report on why generation is needed for the next leap toward photorealism.
VRAM and compute limit both the scene abstraction and the number of rays we can afford. Previous DLSS reconstructs from that representation—so even perfect reconstruction is still bounded by what the scene contains.
DLSS 5 is the first DLSS to generate beyond that abstraction and break through that ceiling, while staying grounded in the game and controlled by the artist.
We also compare it with offline generative models and renderers at similar scaling cost.
research.nvidia.com/labs/adl…
Yichen Sheng retweeted
There we go 🚀 youtu.be/5khwlu2qD9U
Yichen Sheng retweeted
Look forward to it!
The future of computer graphics is here, and it's built by AI. ✨
Don't miss the NVIDIA Research Keynote at SIGGRAPH 2026 for live demos and research examples showcasing breakthrough neural rendering, world models, and simulation methods across creative tools, industrial design, robotics, and autonomous systems.
Yichen Sheng retweeted
A new era of computer graphics is emerging through AI, with new speaker Edward Liu joining NVIDIA Research and Engineering leaders Jan Kautz and Ming-Yu Liu.
Join the 'SIGGRAPH 2026 Sponsored Keynote, Next Era of Graphics — Neural Rendering, World Models, and Simulation'
This session will explore the latest advances in neural rendering, world foundation models, and AI-driven simulation through breakthrough research and live demonstrations. s2026.siggraph.org/program/k…
Yichen Sheng retweeted
Glad to share our PixelDiT work to the community! Many thanks to all coauthors and contributors. This is the first research project I initiate and lead after joining NVIDIA. Thanks for the support from our ADLR team. We will continue pushing the research on pixel diffusion. Check our project page for paper, code, and weights: pixeldit.github.io
Selected as a best paper finalist at #CVPR2026: PixelDiT from NVIDIA Research
In most image generation models, a pretrained autoencoder compresses the image before any diffusion happens, causing quality loss that accumulates across the entire pipeline.
PixelDiT, or Pixel Diffusion Transformers, removes this step entirely. It's a single-stage model that learns the diffusion process directly in pixel space, end-to-end.
Yichen Sheng retweeted
Heading to #CVPR2026! I’m excited to present our work:
I-Scene: 3D Instance Models are Implicit Generalizable Spatial Learners
📍 Denver
🗓 June 6, 4:45–6:45 PM MDT
For years, learning-based interactive 3D scene generation has been shaped and constrained by 3D-FRONT. Methods learn layout distributions from this dataset, making their spatial priors tightly coupled to its limited diversity, and spatial relation statistics.
In I-Scene, we answer a key question:
❓ Where do spatial priors for interactive 3D scene generation come from when data is limited?
Our finding is surprisingly simple: strong 3D instance models already encode rich spatial priors. By reprogramming an instance-level model with scene-context attention and view-centric space, I-Scene can generalize to interactive 3D scenes without relying on heavily annotated scene datasets.
🌟 Key takeaways:
1️⃣ View-centric space matters for spatial relationships
2️⃣ Randomly composed objects can provide surprisingly strong supervision
3️⃣ Strong instance models can serve as a foundation for real-to-sim 3D scene generation
Come by and chat if you’re interested in #GenAI, #SpatialIntelligence, #RealToSim, or #EmbodiedAI!
Yichen Sheng retweeted
Very impressive! We observed a similar finding in I-Scene — directly mapping 2D observations into canonical 3D space makes pixel-to-3D correspondence hard to learn. Coordinate reparameterization really matters! And that is even more important for scene generation.
luling06.github.io/I-Scene-w…
🚀🚀 Introducing Pixal3D (SIGGRAPH’26) — a new pixel-aligned image-to-3D generation paradigm for high-fidelity 3D asset creation.
Today’s Image-to-3D has become pretty good at producing plausible 3D assets. But plausibility is not enough. Fidelity is a hidden bottleneck.
❓A generated model may look “about right,” yet still fail to truly align with the input pixels. Can we make 3D generation as faithful as reconstruction, while still allowing it to complete the unseen?
Pixal3D is our answer.
💡We believe the core bottleneck behind fidelity is 2D–3D correspondence. Most 3D-native generators synthesize shapes in canonical space and inject image cues through cross-attention, forcing the model to implicitly search for which pixels correspond to which 3D regions.
🍀Pixal3D takes a different route. Instead of generating in canonical space, Pixal3D generates directly in pixel-aligned camera space — what you see is what you get. The generated 3D asset is aligned with the input view from the start.
☕️Meanwhile, Pixal3D introduces back-projection-based image condition scheme - explicitly back-projects multi-scale pixel features into 3D voxels, thus resolving the 2D-3D association problem. The input image is no longer just a prompt - it becomes a geometric anchor.
🚩Pixal3D shows that pixel-aligned 3D generation is not only feasible and scalable, but also significantly improves fidelity, pushing 3D-native generation closer to reconstruction-level faithfulness. It also naturally extends to multi-view and scene-level 3D generation.
✅Faithful to the input view. ✅Generative for the unseen.
Closer to reconstruction-level fidelity, with the creativity of 3D generation. Pixal3D also represents an effort towards the unification of 3D generation and reconstruction.
📢Paper, code, and demo are fully released — try it out and let us know your feedback!
🌐Project page: ldyang694.github.io/projects…
🤗Huggingface Demo:
huggingface.co/spaces/Tencen…
💻Code:
github.com/TencentARC/Pixal3…
📄Paper:
arxiv.org/abs/2605.10922
Yichen Sheng retweeted
Two months ago, I vaguely posted a number: 0.9 FID, one-step, pixel space.
Now it is 0.75, and can be even lower.
Many wonder how.
I thought it might end as a small FID prank: simple and deliberate.
It started with one question: can FID be optimized directly, and what does it reveal?
Introducing FD-loss.
Yichen Sheng retweeted
📒 Phota 101: Profile Setup covers best practices to help you get the most out of Phota Studio.
Profiles are at the center of Phota.
Built from your personal album, your identity models learn the details of your appearance so edits and generations across different contexts preserve your identity.
Your photos and models are owned by you and are not used for any other model training.