Ph.D. Student @Oxford_VGG and @amazon Frontier AI & Robotics. Building scalable sim and representation for physical AI.
Joined February 2023
- Tweets110
- Following991
- Followers885
- Likes493
Agentic real2sim2real is going to unlock many long-horizon capabilities for robotics. In this work led by Xudong, we aim to make agent-generated assets more "physically correct", with less self-penetration and better performance in downstream robotic tasks!
A couple of questions at the top of my mind:
1. Two obvious limitations of current agentic real2sim are unrealistic materials and lack of metric-accurate object scale. How much do these actually matter for downstream robotic evaluation and post-training?
2. If the goal is to improve a policy in a specific target environment, e.g., through post-training in an agent-crafted simulation, what's the right balance between diversity (generating many scene variations) and faithfulness (closely matching the target environment)?
3. Physical properties are hard to get exactly right with vision alone. What is the right interface to enable models to absorb interaction data (think end effector trajectory, force, contact, etc.)? Maybe coding (writing code to digest the raw data) would already be enough?
Looking forward to seeing more work on this front!
How can we generate and evaluate physically grounded articulated 3D assets?
🚀 Introducing ARTISAN: a multi-agent generation pipeline and a comprehensive benchmark
🔗 Project: physicalvision.github.io/art…
💻 Code and datasets: github.com/physicalvision/Ar…
RuiningLi retweeted
🚀 PhysAI is coming to ECCV 2026!
Excited to announce the 1st Workshop on Physical AI: Understanding and Building the Physical World (PhysAI) at #ECCV2026 in Malmö, Sweden!
📅 Sept. 9, 2026
🌐 Website: phys-ai-workshop.github.io/e…
RuiningLi retweeted
A bit of a backstory, Instruct-Particulate is the project that we made Articraft for.
It was very difficult to source clean 3D assets with kinematic data, so we thought about how to generate it synthetically.
🚀 Introducing Instruct-Particulate, our new model for inferring articulated structures from static 3D meshes, with significantly improved generalization to novel object categories and support for kinematic prompting.
To achieve this, we scaled our training data 40× and redesigned the model to follow kinematic prompts.
The result: diverse, realistic, simulator-compatible articulated 3D assets can now be generated directly from real-world images!
🔗 Project page: instruct-particulate.github.…
🤗 Demo: huggingface.co/spaces/rayli/…
(4/4) Generic part-segmented 3D models. Generic 3D part segmentations rarely correspond to object articulation and motion. We mix them in the training data as a pre-training step for aligning text prompts with 3D parts.
When combined with an off-the-shelf 3D generator, Instruct-Particulate enables image-conditioned articulated 3D asset generation!
Check out more results on our project page: instruct-particulate.github.….
You can also test out our model today on 🤗: huggingface.co/spaces/rayli/….
Work done with the dream team @RetnRosa @Mattzh1314 @ChuanxiaZ Christian Joan @elliottszwu and Andrea!
🚀 Introducing Instruct-Particulate, our new model for inferring articulated structures from static 3D meshes, with significantly improved generalization to novel object categories and support for kinematic prompting.
To achieve this, we scaled our training data 40× and redesigned the model to follow kinematic prompts.
The result: diverse, realistic, simulator-compatible articulated 3D assets can now be generated directly from real-world images!
🔗 Project page: instruct-particulate.github.…
🤗 Demo: huggingface.co/spaces/rayli/…
(2/4) Articulated 3D assets generated by a coding agent.
xitter.cf/RayLi234/status/205534…
🚀 Introducing Articraft, a coding agent for articulated 3D asset creation.
Articraft writes code, executes it, receives validation feedback, and refines the result into simulation-ready 3D assets with parts, joints, and motion.
We’re also releasing Articraft-10K: 10,000+ articulated objects across 250 categories, unlocking large-scale interactive scenes for robotics simulation and physical AI.
🔗 Project page: articraft3d.github.io/
💻 Code: github.com/mattzh72/articraf…
(1/4) Existing articulated 3D datasets. This is the highest quality data source, but it is limited in quantity and category coverage.
While articulated 3D models are widely used in gaming and robotics simulation, they are scarce on the Internet due to the cost of designing and annotating them manually. In developing Instruct-Particulate, we curate a massive dataset consisting of the following components:
RuiningLi retweeted
CAMEL-AI Live Talk this Friday 🚨
"Articraft: An Agentic System for Scalable Articulated 3D Asset Generation" by Matt Zhou @Mattzh1314 (Visiting Researcher at the University of Cambridge)
Articraft is a system for generating 3D articulated assets at scale. It leverages a custom agentic harness with geometric validation built in, and a LLM-friendly 3D design library to achieve this. Tuning for a blend of both realism and cost-effectiveness, Matt and his team use Articraft to create a large 10k scale dataset of articulated objects, and study using this data to improve existing models and unlock downstream applications in robotic simulation and VR.
Paper link: arxiv.org/html/2605.15187v1
🗓 May 29 · 8:00 PT / 16:00 BST
🔗 Register for Live Talk: forms.gle/K1drb9ao5QQmhB6z8
RuiningLi retweeted
This is an amazing project led by @Mattzh1314. I thought LLMs could already code 3D, but I did not expect such realistic, complex, and fine-grained articulation generation. It completely changed my view of 3D agents — this could be a major step toward real industrial agents.
🚀 Introducing Articraft, a coding agent for articulated 3D asset creation.
Articraft writes code, executes it, receives validation feedback, and refines the result into simulation-ready 3D assets with parts, joints, and motion.
We’re also releasing Articraft-10K: 10,000+ articulated objects across 250 categories, unlocking large-scale interactive scenes for robotics simulation and physical AI.
🔗 Project page: articraft3d.github.io/
💻 Code: github.com/mattzh72/articraf…
When we started this project, the goal was to augment training data for Particulate (ruiningli.com/particulate).
Then @Mattzh1314 kept adding magic to the agent, and at some point we were debating: do we still need neural generators for 3d design, or should we go all in on agents?
We realized last year that we couldn’t train the models we wanted to train without the right type of data…so we made the data.
Had a wonderful time with the folks from Cambridge/Oxford @RayLi234 @XiaoyangLyu22 and @elliottszwu! And shoutout @Remotion for helping make the video :)
RuiningLi retweeted
Check out Ariticraft 🦾 - a highly efficient agentic system that generates articulated 3D assets fully automatically at scale!
🚀 articraft3d.github.io/
🚀 Introducing Articraft, a coding agent for articulated 3D asset creation.
Articraft writes code, executes it, receives validation feedback, and refines the result into simulation-ready 3D assets with parts, joints, and motion.
We’re also releasing Articraft-10K: 10,000+ articulated objects across 250 categories, unlocking large-scale interactive scenes for robotics simulation and physical AI.
🔗 Project page: articraft3d.github.io/
💻 Code: github.com/mattzh72/articraf…
🚀 Introducing Articraft, a coding agent for articulated 3D asset creation.
Articraft writes code, executes it, receives validation feedback, and refines the result into simulation-ready 3D assets with parts, joints, and motion.
We’re also releasing Articraft-10K: 10,000+ articulated objects across 250 categories, unlocking large-scale interactive scenes for robotics simulation and physical AI.
🔗 Project page: articraft3d.github.io/
💻 Code: github.com/mattzh72/articraf…
Check out our project page (articraft3d.github.io/) for more examples!
The code is open-sourced with Apache-2.0 at github.com/mattzh72/articraf….
The project is led by @Mattzh1314 (I learned so much on coding agents from him), with incredible contributions from @XiaoyangLyu22, Matt Song, @ZheningHuang and an amazing advisor team @ChuanxiaZ, Christian, Andrea and @elliottszwu.
🔊We are welcoming data contribution from the community!
You can share the assets you generated with everyone by simply creating a pull request (check github.com/mattzh72/articraf…).
We will actively maintain the dataset and can't wait to see what people build with the assets!
