Our mission is to follow and contribute to the advancement of AI research, aiming to characterise the computational requirements of machine intelligence.
United Kingdom
Joined January 2024
- Tweets190
- Following158
- Followers404
- Likes48
Graphcore Research retweeted
The implicit segmentation in vision encoder heads can also be used for self supervised learning
Graphcore Research retweeted
The slides from the presentation on u-µP we gave at MIT & Harvard last week, for anyone interested: github.com/graphcore-researc…
Graphcore Research retweeted
New in Neural Network Parametrization Technique: Introducing Unit-Scaled Maximal Update Parametrization (u-μP). In partnership with @GCResearchTeam, u-μP merges μP and Unit Scaling to boost training stability & hyperparameter transfer across model sizes. Read more about the theory & practical tips for LLM training in our blog.
🔗aleph-alpha.com/in-awe-at-th…
#writtenbyalephalpha
Graphcore Research retweeted
I'll be at MIT & Harvard next week presenting our u-µP work with my colleague from Aleph Alpha (Constantin Eichenberg).
We're keen to meet up with anyone working on related stuff - training dynamics, low-precision & general efficiency. Please reach out! 👋
Graphcore Research retweeted
Happy to share our recent work on speeding up long-context LLM generation in llama.cpp! ✨ If you’re interested in inference and implementing efficient attention in C++, check it out here:
graphcore-research.github.io…
Graphcore Research retweeted
Sure matplotlib is cool, but what if I want to load my loss curves into the 2006 hit Flash game LineRider?
Graphcore Research retweeted
My colleague @douglasahorr has just written up a very satisfying little blog post on "how to unit scale an op"
We've been arguing for a while that people should re-design ops to produce σ=1 outputs, given σ=1 (normal) inputs - this is how!
graphcore-research.github.io…
Graphcore Research retweeted
Our u-µP paper hit arXiv this morning! I'm so proud of this one — and grateful for a wonderful team who put so much into it 🥰
We add lots of good things to µP. Better sweeping, transfer, simple FP8. Already @cloneofsimo has a great thread on it, which I highly recommend
babe wake up, new muP paper dropped
arxiv.org/abs/2407.17465
And holy smokes does this look promising!
Graphcore Research retweeted
Scalify
scale propagation for efficient low-precision LLM training
Low-precision formats such as float8 have been introduced in machine learning accelerated hardware to improve computational efficiency for large language models training and inference. Nevertheless, adoption
Graphcore Research retweeted
babe wake up, new muP paper dropped
arxiv.org/abs/2407.17465
And holy smokes does this look promising!
Graphcore Research retweeted
Excited to present our SparQ Attention paper tomorrow at @icmlconf !
If you're not around to chat to us in person, check out the recent blog graphcore-research.github.io… written by Luke explaining our method for speeding up long-sequence transformer inference!
Graphcore Research retweeted
Today I gave a talk @graphcoreai called "Neurons, Norms and Number Systems", about precision and scale in deep learning. I'm sharing my slides below...
My talk gave a very visual "neural circuits" perspective on Modula. We formally define two ways of building modules:
(1/6)
Graphcore Research retweeted
For all the µP and sparsity fans out there, here's an (opinionated) summary of the recent Cerebras SµPar paper
🧵 1/n
(1/n) Paper drop: arxiv.org/abs/2405.15743
TLDR: We introduce the sparse maximal update parameterization (SμPar), which ensures optimal HPs remain the same for any width or sparsity level. This dramatically reduces HP tuning costs, allowing SμPar to achieve superior losses. 🧵 👇
Graphcore Research retweeted
Congratulations to Graphcore's Simon Knowles and @Awfidius (Dr Andrew Fitzgibbon) who have been named as Fellows of the @royalsociety. graphcore.ai/posts/royal-soc…
Graphcore Research retweeted
Check out the @GCResearchTeam's review of February's most important and interesting AI papers of the month. graphcore.ai/posts/longer-ro…
Graphcore Research retweeted
It's the greatest AI paper review you'll listen to this month... guaranteed!
Get your quick hit of the latest AI research as we review the Papers of the Month with Graphcore's Charlie Blake. buzzsprout.com/2277524/14427…
Graphcore Research retweeted
You can now follow my old team, Graphcore Research, directly. Lots of great stuff coming out of this group, so do consider giving them a follow.
Graphcore Research is now on X! Follow us for announcements of new machine learning papers, blogs and paper summaries 🚀🔲
As a taster of what we'll be sharing, here's some highlights of our research from last year:
(1/n)
ALT A visualisation of the computational graph of a language model running on our IPU accelerator.
Graphcore Research retweeted
What do we do when there's not enough data to satisfy the insatiable appetite of new model training? In the latest episode of @DistillationPod
we're talking synthetic data with @florianhoenicke
of @JinaAI_ distillationpod.buzzsprout.c…
