machine learning phd student @ UBC 🏞️ - currently doing a research internship @ google deepmind
Joined March 2013
- Tweets14
- Following124
- Followers101
- Likes669
Will Harvey retweeted
🎥🪄 What should happen when you remove an object from a video?
Example 1:
A domino chain is falling → remove the middle blocks → the last block should remain standing
Example 2:
Two cars are about to crash → remove one car → the other should drive away 🚙
Current video object removal models fail at these dynamic scenarios.
We introduce VOID: a model that removes objects and updates the scene as if they were never there.
🏆 Preferred 64.8% of the time vs Runway Aleph, Gen-Omnimatte, ProPainter, and more.
🌐 Project page: void-model.github.io
💻 GitHub: github.com/Netflix/void-mode…
🤗 Demo: huggingface.co/spaces/sam-mo…
📄 arXiv: arxiv.org/abs/2604.02296
w/ @willarvey @ZhuoningYuan @ChengTim0708 and collaborators at @NetflixResearch and @INSAITinstitute
Will Harvey retweeted
It is @NeurIPS time again! I am excited to present our trans-dimensional jump diffusion work with @AndrewC_ML @willarvey @ValentinDeBort1 @tom_rainforth and @ArnaudDoucet1 ! Come over on Thursday 2nd poster session, neurips.cc/virtual/2023/post…. arxiv.org/abs/2305.16261 #NeurIPS2023
This was a lot of fun to work on! And works well with test-time guidance: we can train on varying-length RoboDesk videos and then, at test-time, fix the first and last frames and automatically figure out how far apart they are - i.e. how long the robot needs to move between them!
ALT Sampled video based on the RoboDesk dataset. A robot arm slides a door from let to right over the course of 15 frames.
How can we apply diffusion models to data with varying dimensionality? We use jump diffusions to simultaneously generate the size and state values for varying size data e.g. molecules
arxiv.org/abs/2305.16261
w/ @willarvey @wh1lo @ValentinDeBort1 @tom_rainforth @ArnaudDoucet1
Will Harvey retweeted
This paper is a goldmine for anyone training diffusion models, carefully picking apart theory and practice and showing which choices really matter.
I was quite excited to see the authors of the StyleGAN series of papers tackle this topic, and boy do they deliver!
Elucidating the Design Space of Diffusion-Based Generative Models
abs: arxiv.org/abs/2206.00364
improve efficiency and quality obtainable with pre-trained score networks from previous work, including improving the FID of an existing ImageNet-64 model from 2.07 to near-SOTA 1.55
Thanks for the shout out @frankdonaldwood - the videos still have occasional glitches but are much better after scaling from training on 1 GPU to 4 GPUs. Simply scaling further might be the right direction to take
I think, much more than large language models, this work might be the first glimpse of what the foundation model for vision-based planning for embodied real-world AGI might look like. @sama, @demishassabis, @ylecun who is going to scale this first?
cs-plai-2019.sites.olt.ubc.c…
Will Harvey retweeted
Flexible Diffusion Modeling of Long Videos
abs: arxiv.org/abs/2205.11495
demonstrate improved video modeling over prior work on a number of datasets and sample temporally coherent videos over 25 minutes in length
Excited to announce our work (arxiv.org/abs/2102.12037v1) with hierarchical variational autoencoders - we found that they're ideal for making into realistic image completion models (with @saeidnaderip and @frankdonaldwood)
