NVIDIA Research Scientist - Generative Graphics

Zurich
Joined November 2021
Congrats to @LukasHollein for his successful PhD defense! Lukasโ€™s thesis is on ๐†๐ž๐ง๐ž๐ซ๐š๐ญ๐ข๐ง๐  ๐Ÿ‘๐ƒ ๐–๐จ๐ซ๐ฅ๐๐ฌ ๐Ÿ๐ซ๐จ๐ฆ ๐Ÿ๐ƒ ๐†๐ž๐ง๐ž๐ซ๐š๐ญ๐ข๐ฏ๐ž ๐Œ๐จ๐๐ž๐ฅ๐ฌ. His most impactful work so far, Text2Room (ICCVโ€™23 Oral), was the first method to generate complete room-scale 3D scenes with textured geometry from text alone. The key idea is to lift pre-trained 2D text-to-image models into 3D: iteratively generate new views, estimate depth, and fuse everything into a seamless textured mesh, with a carefully designed viewpoint selection to complete the scene. This inspired many others on text-to-3D scene generation and led to a long line of follow-up works, including Ctrl-Room, LucidDreamer, WonderJourney, WorldGen, and many more. Today, the Text2Room paradigm of leveraging 2D priors has become the primary approach for both image and video diffusion models to generate 3D scenes. However, Lukas has done much more cutting-edge research. ViewDiff turns text-to-image models into 3D-consistent multi-view generators, WorldExplorer generates fully navigable 3D scenes from video diffusion models, and 3DGS-LM speeds up Gaussian Splatting optimization with Levenberg-Marquardt, and most recently World Reconstruction From Inconsistent Views. In summary, his PhD has helped define how we generate 3D worlds with generative AI. Lukas also keeps pushing the boundaries of 3D world generation. Heโ€™s now continuing his research at Apple and I'm sure we can expect more great research very soon! Check his great research: lukashoel.github.io Congrats Dr. Hรถllein, you will be deeply missed in the lab!
6
10
3
148
7,993
Happy to share that Iโ€™ve successfully defended my PhD! ๐ŸŽ“ Excited to continue this research at NVIDIA and explore what comes next in generative graphics!
Congrats to @Peter4AI for his successful PhD defense! Peterโ€™s thesis is on ๐†๐ž๐ง๐ž๐ซ๐š๐ญ๐ข๐ฏ๐ž ๐๐ซ๐ข๐จ๐ซ๐ฌ ๐Ÿ๐จ๐ซ ๐Œ๐š๐ญ๐ž๐ซ๐ข๐š๐ฅ ๐„๐ฌ๐ญ๐ข๐ฆ๐š๐ญ๐ข๐จ๐ง ๐š๐ง๐ ๐‹๐ข๐ ๐ก๐ญ๐ข๐ง๐  ๐‚๐จ๐ง๐ญ๐ซ๐จ๐ฅ. His arguably most impactful work, Intrinsic Image Diffusion, was among the first to bring diffusion models to material estimation. The key idea is that intrinsic decomposition is inherently ambiguous, so rather than regressing a single answer, it is formulated probabilistically and samples plausible decompositions, leveraging the strong image priors of diffusion models. This opened up a new line of diffusion-based inverse rendering work, including IntrinsicAnything, Marigold-IID, DiffusionRenderer, and many more. Today, generative priors are the standard approach for material estimation and relightable reconstruction. However, Peter has done much more cutting-edge research: LightIt brought explicit lighting control to diffusion models, IntrinsiX enabled direct PBR generation from text, and Intrinsic Image Fusion lifts these priors to high-quality multi-view 3D material reconstruction. In summary, his PhD has shaped how we think about materials and lighting in the age of generative AI. Iโ€™m also particularly proud that Peter is continuing this line of research as a Research Scientist at NVIDIA, e.g., his new work on Texture Space Material Diffusion and NeuMatEx. Also thanks to Ruediger Westermann and George Drettakis for being on the committee. Check out Peter's fantastic work: peter-kocsis.github.io Congrats Dr. Kocsis, it was an honor to have you in the lab!
4
1
31
877
๐Ÿ“ข Texture Space Material Diffusion We generate relightable materials from photos or text using video diffusion priors directly in texture space, scaling to 8K textures. Great project with Jacob Munkberg (@yoslber) and Jon Hasselgren! nvlabs.github.io/texdiffusioโ€ฆ
3
11
2
99
6,298
๐Ÿ“ขNeuMatEx - Material Reconstruction beyond PBR We extract neural materials from multi-view images using a Large Material Reconstruction Model combined with differentiable rendering, Great project from @kim_youwang! nvlabs.github.io/neumatex/
1
7
45
2,965
Excited to share that Iโ€™ve joined NVIDIA as a Research Scientist in Generative Graphics! ๐Ÿš€ Looking forward to working on generative rendering and the next steps in graphics with an incredible team. Grateful to everyone who helped along the way! #NVIDIAlife
24
1,449
We present Intrinsic Image Fusion in Poster Session 4 (106) on Saturday at #CVPR ๐Ÿฅณ Come by to learn more about room-scale material estimation! PS: Code is released ๐Ÿš€
๐Ÿ“ข Intrinsic Image Fusion for Multi-View 3D Material Reconstruction ๐Ÿ“ข We combine generative material priors with inverse path tracing: 1) define a parametric texture space 2) fuse monocular predictions across views into consistent textures 3) optimize low-dimensional parameters for physically-grounded reconstructions. The results are relightable PBR textures for 3D scenes: check out the result on a real-world 3D scan from the ScanNet++ dataset! ๐ŸŒpeter-kocsis.github.io/Intriโ€ฆ ๐ŸŽฅyoutu.be/-Vs3tR1Xl7k Great work by @Peter4AI @LukasHollein!
6
40
7,112
๐Ÿ“ข 3D world models from video diffusion suffer from inconsistent frames -> blurry output. Our fix: instead of naรฏve 3D reconstruction, we non-rigidly align each frame into a globally-consistent 3DGS representation. ->sharp visuals on top of any VDM! lukashoel.github.io/video_toโ€ฆ
6
81
3
519
53,514
๐Ÿ“ข๐Ÿ“ข๐Ÿ“ขData release: high-res, multi-view, OLAT face recordings ๐Ÿ“ข๐Ÿ“ข๐Ÿ“ข We captured individuals in our custom light stage with 16 high-end, global shutter cameras (72 fps) and 40 LED modules, totaling 2.8M precisely calibrated frames. We us the data for BecomingLit (#NeurIPS2025): intrinsically decomposed Gaussian avatars, enabling photorealistic and real-time relighting via hybrid neural shading. Code & Data: jonathsch.github.io/becomingโ€ฆ Great work by @jnthnschmdt, @SGiebenhain
5
40
1
236
19,550
๐Ÿ“ขPix2NPHM: Learning to Regress NPHM Reconstructions From a Single Image๐Ÿ“ข We directly regress neural parametric head models (NPHMs) from a single image โ€” fast, stable, and significantly more expressive than classical 3DMMs such as FLAME. Face tracking & 3D reconstruction are often limited by the representational capacity of PCA-based face models. By lifting NPHMs to a first-class reconstruction primitive, we enable more accurate geometry, richer expressions, and finer animation control. Pix2NPHM obtains fast and reliable NPHM reconstructions on real-world data. Inference-time optimization against surface normals and canonical point maps can further increase fidelity. Key to successful and generalized training of our ViT-based network are: (1) large-scale registration of existing 3D head datasets, and (2) self-supervised training on vast in-the-wild 2D video datasets using pseudo ground-truth surface normals. Finally, we show that geometry-aware pretraining on pixel-aligned reconstruction tasks significantly outperforms generic visual pretraining (e.g., DINO-style features) in terms of generalization. ๐ŸŒsimongiebenhain.github.io/Piโ€ฆ ๐ŸŽฅyoutu.be/MgpEJC5p1Ts Great work by @SGiebenhain, @TobiasKirschst1, @liamschoneveld, Davide Davoli, Zhe Chen
15
79
6
536
38,030
๐Ÿ“ข๐Ÿ“ข๐Ÿ“ข๐Œ๐ž๐ฌ๐ก๐‘๐ข๐ฉ๐ฉ๐ฅ๐ž: Structured Autoregressive Generation of Artist-Meshes High-fidelity, topologically complete 3D assets that expand naturally like a ripple on a surface! ๐ŸŒŠ Existing AR models often rely on sliding-window inference over truncated segments. However, this limitation breaks long-range geometric dependencies, causing holes and fragmentation. Instead, MeshRipple uses frontier-aware BFS and sparse-attention global memory to ensure coherent growth with an unbounded receptive field. -> Highly detailed-mesh generations -> Artist-like meshing quality -> Works on room-scale environments ๐ŸŒmaymhappy.github.io/MeshRippโ€ฆ ๐ŸŽฅyoutu.be/AHvaLslzXQU Great work by Junkai Lin, Hang Long, Huipeng Guo, Jielei Zhang, Jiayi Yang, Tianle Guo, Yang Yang, Jianwen Li, Wenxiao Zhang, Wei Yang
5
100
1
649
34,172
๐Ÿ“ข Intrinsic Image Fusion for Multi-View 3D Material Reconstruction We reconstruct clean and sharp relightable textures using inverse path tracing and monocular priors. Check out our project page for more results: peter-kocsis.github.io/Intriโ€ฆ
๐Ÿ“ข Intrinsic Image Fusion for Multi-View 3D Material Reconstruction ๐Ÿ“ข We combine generative material priors with inverse path tracing: 1) define a parametric texture space 2) fuse monocular predictions across views into consistent textures 3) optimize low-dimensional parameters for physically-grounded reconstructions. The results are relightable PBR textures for 3D scenes: check out the result on a real-world 3D scan from the ScanNet++ dataset! ๐ŸŒpeter-kocsis.github.io/Intriโ€ฆ ๐ŸŽฅyoutu.be/-Vs3tR1Xl7k Great work by @Peter4AI @LukasHollein!
11
713
Releasing Echo today is incredibly exciting for me โ€” because it is a critical step for generative AI, enabling the creation of virtual worlds. Echo is our first world model at SpAItial AI. It turns text or images into explorable 3D environments โ€” spaces you can move through, inspect, and build on. Seeing this work in real time still feels a bit surreal. My fascination with this goes back a long way: video games, virtual environments, and the idea of capturing the real world in 3D. As a researcher, I spent years working on 3D reconstruction, neural rendering, and scene understanding โ€” all driven by the same question: how do we teach machines to understand the world? One thing became clear over time: the biggest bottleneck isnโ€™t compute or rendering โ€” itโ€™s 3D worlds themselves. High-quality, consistent environments are expensive to create by hand and donโ€™t scale to the experiences we want to build. In particular, I believe that the ability to generate virtual worlds is ultimately key towards understanding the real world. Thatโ€™s why we founded SpAItial AI. Weโ€™re building spatial world models that combine geometric understanding with creative generation โ€” models that can generate, edit, and eventually reason about 3D environments. Echo is just the beginning. For me, this feels like the moment when decades of research finally meet the imagination that got many of us into graphics, games, 3D understanding in the first place.๐ŸŒ spaitial.ai/
๐Ÿš€ Announcing Echo โ€” our new frontier model for 3D world generation. Echo turns a simple text prompt or image into a fully explorable, 3D-consistent world. Instead of disconnected views, the result is a single, coherent spatial representation you can move through freely. This is part of a bigger shift in AI: from generating pixels and tokens to generating spaces. Echo predicts a geometry-grounded 3D scene at metric scale, meaning every novel view, depth map, and interaction comes from the same underlying world โ€” not independent hallucinations. Once generated, the world is interactive in real time. You control the camera, explore from any angle, and render instantly โ€” even on low-end hardware, directly in the browser. High-quality 3D world exploration is no longer gated by expensive equipment. Under the hood, Echo infers a physically grounded 3D representation and converts it into a renderable format. For our web demo, we use 3D Gaussian Splatting (3DGS) for fast, GPU-friendly rendering โ€” but the representation itself is flexible and can be easily adapted. Why this matters: consistent 3D worlds unlock real workflows โ€” digital twins, 3D design, game environments, robotics simulation, and more. From a single photo or a line of text, Echo builds worlds that are reliable, editable, and spatially faithful. Echo also enables scene editing and restyling. Change materials, remove or add objects, explore design variations โ€” all while preserving global 3D consistency. Editing no longer breaks the world. This is only the beginning. Echo is the foundation for future world models with dynamics, physical reasoning, and richer interaction โ€” environments that donโ€™t just look right, but behave right. Explore the generated worlds on our website and sign up for the closed beta. The era of spatial intelligence starts here. ๐ŸŒ #Echo #WorldModels #SpatialAI #3DFoundationModels Check it out: spaitial.ai/
24
100
5
626
71,358
Peter Kocsis retweeted
Radiance Meshes for Volumetric Reconstruction ๐ŸŽ‰ We made a thing! A Radiance Field that is composited out of a triangle meshes that renders accurately the underlying volumetric field, with no popping AND faster than approximate methods like Gaussian Splatting.
5
37
2
306
26,522
๐Ÿ“ข ๐™„๐™ฃ๐™ฉ๐™ง๐™ž๐™ฃ๐™จ๐™ž๐™“ @NeurIPSConf If you are excited about PBR materials, drop by at Wednesday from 4:30pm to 7:30pm (poster 4308), or feel free to dm me. PS: We have just released the training code, so you can also train your own models now! peter-kocsis.github.io/Intriโ€ฆ
๐Ÿ“ข IntrinsiX: High-Quality PBR Generation using Image Priors ๐Ÿ“ข From text input, we generate renderable PBR maps! Next to editable image generation, our predictions can be distilled into room-scale scenes using SDS for large-scale PBR texture generation. We first train separate LoRA modules for the intrinsic properties of albedo, rough/metal, normal. Then, we introduce cross-intrinsic attention using a rerendering loss with importance-weighted light sampling to enable coherent PBR generation. Our method outperforms text -> image -> PBR methods both in generalization and quality, since directly generating PBR maps does not suffer from the inherent ambiguity of intrinsic image decomposition. In addition, our design choice facilitates SDS-based PBR texture distillation. ๐ŸŒ peter-kocsis.github.io/Intriโ€ฆ ๐ŸŽฅ youtu.be/b0wVA44R93Y Great work by @Peter4AI, @LukasHollein
2
9
1,262
FLUX.2 is here - our most capable image generation & editing model to date. Multi-reference. 4MP. Production-ready. Open weights. Into the new.
161
458
245
2,925
485,239
Congrats @yawarnihal! Amazing work, really well-deserved!
Congrats to @yawarnihal for winning the @MdsiTum best paper award for his amazing ๐Œ๐ž๐ฌ๐ก๐†๐๐“ work๐ŸŽ‰ MeshGPT autoregressively generates compact, artist-style triangle meshes by tokenizing faces into a learned discrete vocabulary (VQ-style codebook) and training a decoder-only transformer to predict those face tokens โ€” because discrete tokenization + attention lets GPT-style models learn long-range geometric & topological patterns and produce coherent, high-fidelity 3D assets. MeshGPT's use cases go far beyond traditional content creation applications in computer graphics. For instance, the method was developed in collaboration with @Audi to help rapid prototyping of car designs, where explicit and precise mesh design are essential. In the research community, there have already been many follow ups such as MeshAnything, MeshXL, Meshtron, and many more - finally, we can use AI to generate high-fidelity 3D content :) Project: nihalsid.github.io/mesh-gpt/ Video: youtu.be/UV90O1_69_o
1
3
702
Peter Kocsis retweeted
๐Ÿ“ขProcGen3D: Learning Neural Procedural Graphs for Image-to-3D Reconstruction @xinyi092298 learns neural procedural graphs to generate high-fidelity 3D - MCTS-guided sampling maintains consistency with the input image, even from real images! Check it out: xzhang-t.github.io/project/Pโ€ฆ
8
52
1
237
14,715
๐Ÿ“ข๐Ÿ“ข ๐๐ž๐ซ๐œ๐‡๐ž๐š๐: ๐๐ž๐ซ๐œ๐ž๐ฉ๐ญ๐ฎ๐š๐ฅ ๐‡๐ž๐š๐ ๐Œ๐จ๐๐ž๐ฅ ๐Ÿ๐จ๐ซ ๐’๐ข๐ง๐ ๐ฅ๐ž-๐ˆ๐ฆ๐š๐ ๐ž ๐Ÿ‘๐ƒ ๐‡๐ž๐š๐ ๐‘๐ž๐œ๐จ๐ง๐ฌ๐ญ๐ซ๐ฎ๐œ๐ญ๐ข๐จ๐ง & ๐„๐๐ข๐ญ๐ข๐ง๐ ๐Ÿ“ข๐Ÿ“ข PercHead reconstructs realistic 3D heads from a single image and enables disentangled 3D editing via geometric controls and style inputs from images or text. At its core is a generalized 3D head decoder trained with perceptual supervision from DINOv2 and SAM 2.1. We find that our new perceptual loss formulation improves reconstruction fidelity compared to commonly-used methods such as LPIPS. Our trained reconstruction model is able to generate 3D-consistent heads from a single input image. Even with challenging side-view inputs, the model robustly infers missing regions for a coherent, high-fidelity output. In addition, our architecture seamlessly adapts to downstream tasks: by swapping the encoder, we can transform the model into a disentangled 3D editing pipeline. In this scenario, we can control geometry through - potentially hand-drawn - segmentation maps, and condition style via image or text prompt. We also provide an interactive GUI to enable the exploration of our editing pipeline. ๐ŸŒ antoniooroz.github.io/PercHeโ€ฆ ๐Ÿ“ฝ๏ธ youtu.be/4hFybgTk4kE Great work by @antonio_oroz and @TobiasKirschst1
5
37
5
258
18,936
Peter Kocsis retweeted
๐Ÿ“ขNew in ScanNet++: High-Res 360ยฐ Panos! @chandan__yes & @liuyuehcheng have added pano captures for 956 ScanNet++ scenes, fully aligned with the 3D meshes, DSLR, and iPhone data - multiple panos per scene Check it out: Docs kaldir.vc.in.tum.de/scannetpโ€ฆ Code github.com/scannetpp/scannetโ€ฆ
44
221
12,937