Machine learning researcher
Joined April 2018
- Tweets182
- Following1.2K
- Followers2.4K
- Likes949
David Page retweeted
I am thrilled to share the news that Fractile's mission to build chips and systems that unlock the next generation of AI scaling has been bolstered, with a $220M funding round led by Accel, Factorial Funds, and Founders Fund, alongside some incredible backers old and new.
AI inference is driving the defining infrastructure buildout of the 21st Century. We've written a bit about where we think capabilities must go, and how Fractile is working to bring this about: fractile.ai/news/fractile-ra…
It has been a privilege working on one of the hardest but most rewarding technical challenges of our time for over three years, with the most brilliant, kind and driven people I could have ever hoped to work alongside.
We are still just getting started. There is a lot to be done to deliver on our goals, but we are grateful to have the support of so many people in chasing these down every minute of every day. Thanks, all, for being part of the Fractile mission! 🚀
@fractile_ai
David Page retweeted
Excited to announce our new work, a unified theory towards explaining 3 black magics in deep learning: (1) ensemble, (2) knowledge distillation, and (3) self-distillation. An accessible blog post is below.
Microsoft and CMU researchers begin to unravel 3 mysteries in deep learning related to ensemble, knowledge distillation & self-distillation. Discover how their work leads to the first theoretical proof with empirical evidence for ensemble in deep learning: aka.ms/AAavp1k
David Page retweeted
Big accuracy update coming in the next version of Bonito 🚀 v0.3.0 combines everything we have learned with structured and unstructured approaches - @dcpage3, Tim and myself are working hard on the finished touches this week - watch this space 👀
David Page retweeted
Bonito v0.2.2
- SAM output
- Sequence and alignment tsv summaries
- Grab bag of training improvements from @dcpage3
github.com/nanoporetech/boni…
David Page retweeted
Preparing a short course on neural nets can be fun. Below is one of the fast Resnets by @dcpage3 on CIFAR10.
Would have been nice to track a UMAP-like representation of some internal layer, but have not found a reasonably fast/stable way to do so. Any idea? @NikolayOskolkov
David Page retweeted
The problem though with "you can always add those tricks to get the numbers up" is that *very* often I see papers that don't do data aug, or don't tune hyper-params, etc, then claim their new idea helps. But then I find it's actually just a poor proxy for the things they skipped
David Page retweeted
If you're interested in training small accurate nets efficiently, then the best in the world is @dcpage3, and he told all his secrets in this amazing series:
myrtle.ai/how-to-train-your-…
David Page retweeted
Fun fact: More data can hurt in linear regression. arxiv.org/abs/1912.07242
(aside: this is about the 3rd time that studying deep learning has taught me something about linear regression).
David Page retweeted
I've implemented a simple tool for analyzing the full loss Hessian spectrum here: github.com/LeviViana/torches…, I checked the spectrum changes after removing BatchNorm layers for a ResNet-18 architecture.
David Page retweeted
While the BN paper didn't define ICS formally, as long as one understands the proposed mechanism, it's easy to plug in a precise definition of ICS that makes the argument work out (as David did in the blog post).
David Page retweeted
Actually, there were a few 1998 papers by Nicol Schraudolph on various variable centering schemes for multlilayer nets
nic.schraudolph.org/bib2html…
David Page retweeted
Precisely.
(I've since been told by my random matrix theory colleagues at Courant that the distribution of eigenvalues of a random covariance matrix can be obtained in a much simpler manner than with the replica symmetry breaking calculations used for this paper).
There was theory too!
For a single layer, centering improves conditioning of the Hessian of the loss, enabling higher learning rates.
(@ylecun et al ’90) papers.nips.cc/paper/314-sec…
David Page retweeted
This is the best distillation of recent (and old!) research on batchnorm I've seen.
There is so much to learn about training mechanics by studying this thread and the links it contains.
The paper that introduced Batch Norm arxiv.org/abs/1502.03167 combines clear intuition with compelling experiments (14x speedup on ImageNet!!)
So why has 'internal covariate shift' remained controversial to this day?
Thread 👇
David Page retweeted
Greg you need to read the research you're commenting on *much* more closely. You've profoundly misunderstood the basic premise of it, let alone the details. Hopefully @dcpage3 will be able to explain! :)
David Page retweeted
Another cool analysis by @dcpage3: myrtle.ai/how-to-train-your-…
Experimentally, #BatchNorm increases the robustness of parameters wrt. adversarial perturbations and hence improves the conditioning of the Hessian significantly
David Page retweeted
Amazing analysis! We tried to reduce LR along the top eigenvectors of the Hessian in a network without BN in (link: arxiv.org/abs/1807.05031) arxiv.org/abs/1807.05031. Very interesting to hear this is somewhat related to what BN does :)