3d vision / head avatars CS PhD @UniofBath
United Kingdom
Joined June 2022
- Tweets137
- Following228
- Followers16
- Likes305
Avirup Dey retweeted
Tomorrow, I start a new full-time position as VP Digital Human Research at Epic Games. (And no, this isn’t an April Fools’ joke.)
Today, I’m taking early retirement from Max Planck and will become an Emeritus Director.
As an Emeritus Director, I will continue to supervise my remaining students, oversee ongoing projects, and wind down my department over the next couple of years, which is a normal process when a director retires.
Being a Max Planck director is the best academic job in the world and it has been my great honor to co-found and help build the Max Planck Institute for Intelligent Systems. I love the institute deeply.
But my statutory retirement age was looming and I’m not done yet. I’ve been working on capturing and modeling human movement for 30 years and it has gone from a fringe topic to something that works robustly and has immense potential.
With AI today, scale matters, and achieving that scale increasingly requires industry. In particular, digital humans are moving from research prototypes to foundational technology across industries. At this point in my career, I want to get this technology into the hands of millions of users, while pushing the frontier of digital humans.
I know that this is a challenging time for the games industry. Such times are precisely when people get creative, the industry is open to change, and real innovation can take root.
Epic Games is the ideal place for this. The talent is deep, there is a compelling vision for how games will evolve, and the commitment is clear. I’m excited to join them and the rest of the Meshcapade team.
There is no good way to express on social media the depth of my gratitude to the Max Planck Society, my co-directors, students, post docs, and staff. It has been an amazing 15-year journey because of you. Thank you.
Avirup Dey retweeted
Quite literally all of 3D vision
This includes VGGT, Dust3r and friends which are trained on COLMAP-generated data
Also gaussian splatting and NERFs in most cases use COLMAP-generated poses
Avirup Dey retweeted
Good news: Soham Parekh, a PhD student in our group, will finally have the bandwidth to focus on his academic career!
We just awarded him another scholarship that allows him to work remote.
doing god's work 🙏
🤖 🗞️ Free AI Generated Digital Humans Newsletter -> lovable.dev/projects/6849fc0…
Like everyone else, I've been struggling to keep up with the sheer volume of papers and news in the Digital Human space. Over the past few weeks, I've developed an agentic (ish) AI pipeline to find and summarise the latest daily papers and relevant news articles and deliver them as a newsletter to my inbox.
Now I want to share this with anyone interested. It will be a daily newsletter with up to three relevant papers (each with a summary) and up to three news items. I've been finding it very useful as an entry point (before doing my own further research). If you think this could be valuable, please sign up for free!
I'll still keep posting, and I will not just copy the LLM summaries.
Just a disclaimer, this is done by LLMs, so information may be inaccurate, and the news articles may seem irrelevant/promotional.
Avirup Dey retweeted
Ever wish you could turn your video generator into a controllable physics simulator?
We're thrilled to introduce Force Prompting! Animate any image with physical forces and get fine-grained control, without needing any physics simulator or 3D assets at inference. 🧵(1/n)
Ashamed of Mr. Tharoor for speaking up for his country instead of pandering his "fans". No progressive liberal should condemn terrorists and their enablers.
Avirup Dey retweeted
IM-Portrait: Learning 3D-aware Video Diffusion for Photorealistic Talking Heads from Monocular Videos
TLDR: A diffusion-based method is used to generate 3D renderable talking heads from a single image, using a single denoising step for real-time. The model can learn from monocular videos, leading to sharper results and improved convergence.
📽️ Project Page: y-u-a-n-l-i.github.io/projec…
📄 Paper: arxiv.org/pdf/2504.19165v1
One hell of a thread!
The original architects of American global power did something very clever that no other empire had ever done before: they deliberately hid the instruments of their power.
Specifically, they institutionalized the hard power of the post-WW2 American military into a "rules-based international order" and the organizations needed to run it.
These include the UN, IMF, World Bank, NATO, and numerous philanthropic NGOs like (as has been in the news recently) USAID.
The reason they did this is because repeated use of hard military power is fragile and self-defeating: it engenders resentment and breeds defiance. The British learned this and used prototypical methods of institutionalization in the declining years of their empire, but their American successors perfected it.
(If you don't understand how this works, I will link a post in the replies explaining how one example works - NATO)
The more sophisticated rivals of the US obviously know what's up, so they try to oppose or circumvent these institutions, but obviously the institutions are backed by hard power in the end. It sounds fair enough to say "if you don't abide by the 'rules,' we will invade you." It works well enough because it it sounds more fair to say that than "if you don't do what we tell you, we will invade you." Rival governments aren't fooled, but a lot of their ordinary citizens are, and combined with media dominance and control of the reserve currency (economic dominance), it's enough to keep everyone in line.
An unanticipated problem seems to have arisen:
It turns out that if you hide the levers of power, your own successors may have trouble understanding them. Especially if you failed to educate them, or let various cultural forces undermine the indoctrination of your elites.
With that happening, once the new generation of elites gains power, they don't recognize that the complicated weird control panel you built that doesn't seem to do anything but costs $10 billion a year to maintain is actually how you're controlling everything around the world and they take it down to save money. All because you did too good a job hiding the levers of power.
Soft power isn't "soft." It's real power. It's just soft because it's hidden.
I don't know how to solve this problem because I think hidden levers of power are definitely better, but you have to train a priesthood generation after generation to understand them, and that kind of thing corrupts itself too. Open power is much more honest, but no one likes it and it's hard to hold on to.
Avirup Dey retweeted
After hacking GPT-4o's frontend, I made amazing discoveries:
💡The line-by-line image generation effect users see is just a browser-side animation (pure frontend trick)
🔦OpenAI's server sends only 5 intermediate images per generation, captured at different stages
🎾Patch size=8
Avirup Dey retweeted
No... the fear isn't irrelevance, but that's a comforting thing for us technologists to tell ourselves.
The fear is degradation. You can sense where this leads even if you can't put your finger on it, which is why so many feel vaguely uneasy. They say they hate it but they don't know why...
No... They love it. They really do enjoy all of the images, and that's exactly the problem. Because the ghiblified images are just short of being ghibli. They're like 85% maybe 90% of the way there. And Ghbili is gorgeous--and 90% of gorgeousness is still beautiful, so of course you like it!
But then it's the unease creeping in. Like all magicks it comes at a cost. "What price will we pay?" you find yourself asking. And the answer might not be obvious but when you think about it it's obviously true.
Artists do and did spend thousands and thousands of hours over lifetime developing and refining this, and all artistic styles. That's what it takes. So what happens when you can reach 90% of perfection with .001% of the effort?
Perfection vanishes. There's no place for it anymore. At best you're left with cultural ossification, which was already the trend before AI by the way. Endless repeats and rehashes on the same formula, never *quite* as good as the original, but never terrible enough to justify the incredible expense and risk of doing something new.
At worst you're left with constant degradation. Everything is 90% as good as what came before, because each round of AI facsimile comprises the majority of the training data for the next round of AI generation.
Slop begets slop, and if we're really lucky, it accelerates into a sort of reverse cultural singularity where finally it gets bad enough for people to finally invest in skills and originality again to create something new.
We all want beautiful art that moves the soul and conveys something new to us. The quintessential human experience is transformation by way of art. But the market wants to make money. The market wants speed. AI generated art serves the market, which is a beautiful thing, but it doesn't serve human flourishing. At least, not directly.
And so we goodhart ourselves into oblivion. Perpetually missing the mark. And that is the definition of sin.
This quoted post is unavailable.
Avirup Dey retweeted
DisentTalk: Cross-lingual Talking Face Generation via Semantic Disentangled Diffusion Model
TLDR: A hierarchical diffusion-based speech-to-3DMM generator that decomposes FLAME parameters into lip, eye and global regions for better quality. It uses PiRender to get the final videos.
📽️ Project Page: lnkd.in/eNsgddME
📜 Paper: lnkd.in/ec-wTRed
Avirup Dey retweeted
Pippo: High-Resolution Multi-View Humans from a Single Image
TLDR: High-resolution and Multi-View but static human Generation from one image. Uses a DiT to generate the images directly (e.g. no 3D representation), with a control MLP.
Imagine this distilled onto one of Meta's Gaussian Avatars!
📽️ Project Page: yashkant.github.io/pippo/
📜 Paper: yashkant.github.io/pippo/pip…
💻 Code: github.com/facebookresearch/…
The code doesn't yet contain the pre-trained models.
Avirup Dey retweeted
Decentralized Diffusion Models power stronger models trained on more accessible infrastructure.
DDMs mitigate the networking bottleneck that locks training into expensive and power-hungry centralized clusters. They scale gracefully to billions of parameters and generate photorealistic images with just a week of training on eight independent GPU nodes. They’re easy to implement, adopt DiT hyperparameters directly and outperform standard models FLOP-for-FLOP.
Avirup Dey retweeted
📢📢📢 "𝐑𝐚𝐝𝐢𝐚𝐧𝐭 𝐅𝐨𝐚𝐦: Real-Time Differentiable Ray Tracing", a mesh-based 3D represention.
radfoam.github.io
arxiv.org/abs/2502.01157
Co-lead by my PhD students Shrisudhan Govindarajan and Daniel Rebain, and w/ @kwangmoo_yi
Exciting opportunity for those working on avatars. Do sign up!
📢 Working on Digital Humans/Avatars? Register for the BMVA Symposium on Digital Humans on the 28th of May in London! We're also looking for recent work to be presented on the day. Not sure if your work fits? Send me a DM.
Register/submit here 👉 bmva.org/meetings/25-05-28-D…
Many thanks to our co-organisers @avideypi, @vinaypn and @SanyalSoubhik, as well as our exciting keynote speakers Tadas Baltrusaitis, Abhijeet Ghosh, Thu Nguyen-Phuoc and Dimitris Tzionas.
