Machine learning researcher
Joined April 2018
- Tweets182
- Following1.2K
- Followers2.4K
- Likes949
Pinned Tweet
Ever wanted to train CIFAR10 to 94% in 26 SECONDS on a single-GPU?!
In the final post of our ResNet series, we open a bag of tricks and drive training time ever closer to zero...
Colab: colab.research.google.com/gi…
Blog: myrtle.ai/how-to-train-your-…
David Page retweeted
I am thrilled to share the news that Fractile's mission to build chips and systems that unlock the next generation of AI scaling has been bolstered, with a $220M funding round led by Accel, Factorial Funds, and Founders Fund, alongside some incredible backers old and new.
AI inference is driving the defining infrastructure buildout of the 21st Century. We've written a bit about where we think capabilities must go, and how Fractile is working to bring this about: fractile.ai/news/fractile-ra…
It has been a privilege working on one of the hardest but most rewarding technical challenges of our time for over three years, with the most brilliant, kind and driven people I could have ever hoped to work alongside.
We are still just getting started. There is a lot to be done to deliver on our goals, but we are grateful to have the support of so many people in chasing these down every minute of every day. Thanks, all, for being part of the Fractile mission! 🚀
@fractile_ai
David Page retweeted
Excited to announce our new work, a unified theory towards explaining 3 black magics in deep learning: (1) ensemble, (2) knowledge distillation, and (3) self-distillation. An accessible blog post is below.
Microsoft and CMU researchers begin to unravel 3 mysteries in deep learning related to ensemble, knowledge distillation & self-distillation. Discover how their work leads to the first theoretical proof with empirical evidence for ensemble in deep learning: aka.ms/AAavp1k
David Page retweeted
Big accuracy update coming in the next version of Bonito 🚀 v0.3.0 combines everything we have learned with structured and unstructured approaches - @dcpage3, Tim and myself are working hard on the finished touches this week - watch this space 👀
First day of new job @nanopore where I get to apply ML to a bunch of fun science and engineering problems. Pretty excited!
David Page retweeted
Bonito v0.2.2
- SAM output
- Sequence and alignment tsv summaries
- Grab bag of training improvements from @dcpage3
github.com/nanoporetech/boni…
David Page retweeted
Preparing a short course on neural nets can be fun. Below is one of the fast Resnets by @dcpage3 on CIFAR10.
Would have been nice to track a UMAP-like representation of some internal layer, but have not found a reasonably fast/stable way to do so. Any idea? @NikolayOskolkov
Great study of training efficiency at large scale + nice results on compression for inference!
Not everyone can afford to train huge neural models. So, we typically *reduce* model size to train/test faster.
However, you should actually *increase* model size to speed up training and inference for transformers.
Why? [1/6] 👇
bair.berkeley.edu/blog/2020/…
arxiv.org/abs/2002.11794
Undertraining a large model is a good way to speed things up on toy problems myrtle.ai/how-to-train-your-… but it was far from clear this should extend to large scale.
Simple setup + attention to details -> sota self-supervised reps!
LARS -> large batches -> no need for memory bank of -ve examples
Random crops + color aug (to prevent hist cheating) -> no need for special arch
Projn head for contrastive loss -> hidden reps preserve info
Introducing SimCLR: a Simple framework for Contrastive Learning of Representations. SimCLR advances previous SOTA in self-supervised and semi-supervised learning on ImageNet by 7-10% (see next).
arxiv.org/abs/2002.05709
Joint work with @skornblith @mo_norouzi @geoffreyhinton.
David Page retweeted
The problem though with "you can always add those tricks to get the numbers up" is that *very* often I see papers that don't do data aug, or don't tune hyper-params, etc, then claim their new idea helps. But then I find it's actually just a poor proxy for the things they skipped
David Page retweeted
If you're interested in training small accurate nets efficiently, then the best in the world is @dcpage3, and he told all his secrets in this amazing series:
myrtle.ai/how-to-train-your-…
David Page retweeted
Fun fact: More data can hurt in linear regression. arxiv.org/abs/1912.07242
(aside: this is about the 3rd time that studying deep learning has taught me something about linear regression).
David Page retweeted
I've implemented a simple tool for analyzing the full loss Hessian spectrum here: github.com/LeviViana/torches…, I checked the spectrum changes after removing BatchNorm layers for a ResNet-18 architecture.
This looks promising! Just tried a simple version - backprop the top 40% of each batch by loss - and it knocks 10% off a highly tuned CIFAR10 training time (94% in 24s on 1 GPU as of now!)
Accelerating Deep Learning by Focusing on the Biggest Losers
deepai.org/publication/accel…
by Angela H. Jiang et al. including @zacharylipton
#Backpropagation #NeuralNetwork
David Page retweeted
While the BN paper didn't define ICS formally, as long as one understands the proposed mechanism, it's easy to plug in a precise definition of ICS that makes the argument work out (as David did in the blog post).
David Page retweeted
Actually, there were a few 1998 papers by Nicol Schraudolph on various variable centering schemes for multlilayer nets
nic.schraudolph.org/bib2html…
David Page retweeted
Precisely.
(I've since been told by my random matrix theory colleagues at Courant that the distribution of eigenvalues of a random covariance matrix can be obtained in a much simpler manner than with the replica symmetry breaking calculations used for this paper).
There was theory too!
For a single layer, centering improves conditioning of the Hessian of the loss, enabling higher learning rates.
(@ylecun et al ’90) papers.nips.cc/paper/314-sec…
David Page retweeted
This is the best distillation of recent (and old!) research on batchnorm I've seen.
There is so much to learn about training mechanics by studying this thread and the links it contains.
The paper that introduced Batch Norm arxiv.org/abs/1502.03167 combines clear intuition with compelling experiments (14x speedup on ImageNet!!)
So why has 'internal covariate shift' remained controversial to this day?
Thread 👇
The paper that introduced Batch Norm arxiv.org/abs/1502.03167 combines clear intuition with compelling experiments (14x speedup on ImageNet!!)
So why has 'internal covariate shift' remained controversial to this day?
Thread 👇
Thanks to @jeremyphoward for encouraging me to write this up and feedback on an early draft!