AI for scientific discovery. Head of Science, Biohub. Founder and scientific director of the ESM project.
Brooklyn, NY
Joined April 2009
- Tweets400
- Following899
- Followers16.4K
- Likes3.3K
Pinned Tweet
Today we're announcing ESMFold2, an open scientific engine to power prediction, design, and discovery across protein biology.
The new model delivers state of the art performance on protein interactions, especially antibodies, a critical modality for therapeutics.
We have designed and validated miniprotein binders and single chain antibodies across five therapeutic targets that are important in cancer and immunology. We are seeing very high success rates, and affinities at levels consistent with therapeutic activity.
We’re also releasing an atlas of 6.8 billion proteins, and 1.1 billion predicted structures.
ESMFold2 is built on a state of the art language model that has been trained on billions of protein sequences.
A world model of protein biology emerges through language modeling.
We’ve used the techniques of mechanistic interpretability developed to understand large language models to understand the concepts ESM uses to represent proteins.
The model’s representation space has a compositional organization of features across scales, levels of complexity, and abstraction, that reflects and mirrors the understanding of protein biology developed through a century of empirical science.
This understanding emerges without prior knowledge, just from language modeling of protein sequences.
Language models are becoming a powerful substrate to understand and program biology.
The design of protein interactions is one of the most fundamental problems in biophysics, and has critical implications for the discovery of new medicines. A simple gradient based search with the model was able to discover high-affinity protein binders.
I'm excited by the potential this has to accelerate basic science and the understanding of proteins. And especially for the new avenues it opens up for therapeutic design and medicine.
A universal predictive model of the cell would dramatically accelerate science, allowing biologists to perform experiments digitally. Today we’re announcing a major partnership with the DOE and NIH. The goal of this partnership is to generate the data that will be needed to build an accurate model of the cell with artificial intelligence.
Isomorphic, Google DeepMind, and Meta are joining us as founding partners in the Virtual Biology Initiative. This is the beginning of a large scale coordinated effort to map cellular biology to power the development of digital models of life. Other leading scientific institutions and consortia including the Allen Institute, Broad Institute, Gladstone Institutes, the Human Cell Atlas, the Human Protein Atlas, the Wellcome Sanger Institute, as well as NVIDIA, are working together as part of this international scientific project.
Creating a predictive model of the cell is one of the most important challenges for the next decade of science. The insights that come from this could unlock a far greater understanding of disease, and open new paths for cures.
We invite the worldwide scientific community and other funders to join us in this effort.
Together with UC Berkeley we are announcing the laser phase plate - a breakthrough in atomic resolution imaging. This is the brightest continuous wave laser in the world, 100 million times the intensity of the surface of the sun.
Phase contrast plays an important role in microscopy, but it was thought close to impossible for electron microscopy, where it would require interfering with an electron beam. Holger Mueller and Robert Glaeser proposed exactly this using a standing wave laser. It has taken over 15 years to make this a reality. Biohub partnered with UC Berkeley and Mueller to support this work and to engineer and build the technology.
Contrast has been the critical barrier to achieving atomic resolution imaging of the cell. In cryo-electron tomography, a cellular imaging technology that uses electron microscopy, the low contrast makes it impossible to resolve anything but the largest proteins within their cellular context. The laser phase plate removes that barrier.
With advances in AI this breakthrough in contrast will start to open up a new frontier in structural biology, that will allow us to see the molecular machines of the cell, and how they assemble into far more complex and dynamic systems, and understand how they work.
Great talking with RJ and Brandon on the Latent Space AI for Science podcast, about protein language models, scaling in biology, and the new ESMFold2.
🆕Biohub’s Protein World Model: ESMC-6B, ESMFold2, 6.8B proteins, 1.1B structures, antibody design, SAEs, & the bitter lesson for biology latent.space/p/esmfold2
@biohub Head of Science @alexrives explains why biology may scale like language modeling, how metagenomics unlocked the next ESM scaling curve, why protein LMs can learn structure/function from sequence alone, how sparse autoencoders reveal biology inside the model, why ESMFold2 can beat specialized systems on antibody-antigen prediction, and how Biohub’s $500M Virtual Biology Initiative aims to build predictive models of cells, disease, and eventually physiology.
Scaling laws are powering AI. It’s time to scale biology.
Today we’re launching the Virtual Biology Initiative to generate the data to unlock scaling laws in biology and build accurate predictive models of the cell.
Digital representations of proteins are already expanding our understanding of life at the molecular level, and accelerating the design of molecules and medicines. Accurate digital representations of the cell could reveal the mechanisms that are responsible for disease, and show how to reverse them.
The protein data bank, and worldwide repositories of protein sequence biodiversity were created through decades of work by the scientific community. The advances in artificial intelligence for proteins would not have been possible without them.
The cell is orders of magnitude more complex, and we will need to create the data in just a few years rather than decades.
This will require a coordinated global effort. We're partnering with Broad, Wellcome Sanger, Arc, Allen, Human Cell Atlas, Human Protein Atlas, NVIDIA, and Renaissance Philanthropy.
Biohub is contributing to this effort as both a funder and a builder. We are developing microscopy to observe millions of cells in living organisms, and cryo-ET to resolve the cell in atomic detail. We're building instruments that expand the range of modalities and parameters that can be simultaneously measured. We’re developing molecular, cellular, and tissue engineering to create models of disease and design interventions.
The data we generate will be available to the worldwide scientific community.
We’re also committing $100M over the next five years to support work beyond Biohub.
We invite other scientific teams and funders to join.
Link: biohub.org/news/virtual-biol…
Sequence is all you need. Protein language models set a new state of the art for predicting the clinical effect of genetic variations.
xitter.cf/vntranos/status/203901…
To me, this is a very satisfying result from a theoretical perspective, demonstrating in a sense that "sequence is all you need" for variant effect prediction. The model we introduce, #VESM (Variant-effect ESM), is entirely derived from within the ESM family without using any external information.
Today CZI is announcing an unprecedented new scientific initiative to build the future of AI-powered biology. I am joining CZI to lead this initiative as Head of Science, and the EvolutionaryScale team is joining forces with Biohub.
This is the first large scale scientific effort to combine frontier AI and frontier biology.
I feel an incredible sense of optimism in this moment. There is a path to build predictive models of life that can fundamentally accelerate science, and unlock a new understanding of disease.
biohub.org/blog/frontier-ai-…
We’re also announcing an ESM3 compute grant pilot program for applications at the scientific frontier. ESM3 can enable creative applications of protein engineering from drug design, to green chemistry, and materials science. Please apply on Forge if your research could benefit from more compute with ESM3.
We're thrilled to present ESM3 in @ScienceMagazine. ESM3 is a generative language model that reasons over the three fundamental properties of proteins: sequence, structure, and function. Today we're making ESM3 available free to researchers worldwide via the public beta of an API for biological intelligence.
Trained with over a trillion teraflops of compute, this is the first time a model of this scale has been trained for biology, pushing the frontier of AI for biological discovery and engineering.
ESM3 learns to represent the immense complexity of protein biology, learning from billions of natural proteins. From this training it developed the capability to design proteins, responding to complex prompts combining atomic level details and high level instructions to generate new proteins.
ESM3 can explore protein space far beyond natural evolution. We prompted ESM3 to generate a fluorescent protein at a far distance from any known fluorescent proteins, searching an unknown region of protein space, to discover a new fluorescent protein.
We estimate this is equivalent to simulating five hundred million years of evolution.
We're thrilled to present ESM3 in @ScienceMagazine. ESM3 is a generative language model that reasons over the three fundamental properties of proteins: sequence, structure, and function. Today we're making ESM3 available free to researchers worldwide via the public beta of an API for biological intelligence.
Trained with over a trillion teraflops of compute, this is the first time a model of this scale has been trained for biology, pushing the frontier of AI for biological discovery and engineering.
ESM3 learns to represent the immense complexity of protein biology, learning from billions of natural proteins. From this training it developed the capability to design proteins, responding to complex prompts combining atomic level details and high level instructions to generate new proteins.
ESM3 can explore protein space far beyond natural evolution. We prompted ESM3 to generate a fluorescent protein at a far distance from any known fluorescent proteins, searching an unknown region of protein space, to discover a new fluorescent protein.
We estimate this is equivalent to simulating five hundred million years of evolution.
Read the paper: science.org/doi/10.1126/scie…
Our blog post: evolutionaryscale.ai/blog/es…
Introducing ESM Cambrian.
Unsupervised learning can invert biology at scale to reveal the hidden structure of the natural world.
We’ve scaled up compute and data to train a new generation of protein language models. ESM C defines a new state of the art for protein representation learning.
ESM C models establish a frontier of performance as a function of parameter scale. We see large improvements across all parameter scales over previous state of the art models.
Read more: evolutionaryscale.ai/blog/es…
An exciting day for our field at a time with so much possibility ahead of us. Congratulations David, Demis, and John for your incredible scientific achievements!!
xitter.cf/NobelPrize/status/1843…
BREAKING NEWS
The Royal Swedish Academy of Sciences has decided to award the 2024 #NobelPrize in Chemistry with one half to David Baker “for computational protein design” and the other half jointly to Demis Hassabis and John M. Jumper “for protein structure prediction.”
42
4,333
Exciting new work from @rohitsingh8080 and colleagues. An autoencoder in the ESM embedding space preserves structure and function while generating new proteins.
xitter.cf/rohitsingh8080/status/…
Fascinating exploration of the ESM2 latent space and its compressibility!
1/ 🧬 Excited to share CHEAP, our new work on compressed protein embeddings. We characterize the joint distribution of p(sequence, structure) in ESMFold's latent space, and find cool tidbits on compressibility, tokenizability, and pathologies:
biorxiv.org/content/10.1101/…
🧵
After we trained ESM3, we were excited to see it could respond to complex prompts, which greatly improved with scale.
While scaling has been driving advances across AI—in biology it has been largely underestimated.
Article by @ewencallaway for @Nature: nature.com/articles/d41586-0…
We have trained ESM3 and we're excited to introduce EvolutionaryScale.
ESM3 is a generative language model for programming biology. In experiments, we found ESM3 can simulate 500M years of evolution to generate new fluorescent proteins.
Read more: evolutionaryscale.ai/blog/es…
We prompted ESM3 to generate fluorescent proteins with a chain of thought.
In the first plate, shown below, we were intrigued to find B8. While very dim, 50x dimmer than natural GFPs, it was far from any known GFPs -- 43% of its sequence differs from the closest natural protein.
Continuing the chain of thought from B8 on the second plate below, ESM3 found C10 which is similarly bright to natural fluorescent proteins.
AI for biology has the potential to help solve some of the most important problems facing our society. Scientific openness will be critical for our field to progress. As scientists engaged in this work, with more than 90 signatories across the world, we are advancing a framework and commitments for responsible development:
responsiblebiodesign.ai
Metagenomic proteins are some of the least understood proteins on earth. Now with AI it is becoming possible to see deep into the structures of proteins and the incredible complexity of the natural world.
Published in today’s issue of @sciencemagazine. science.org/doi/10.1126/scie…
💻🧬🧪 This was a great collaboration between the @MetaAI Protein Team and @UWproteindesign @BasileWicky @LFMilles @JustasDauparas and @sokrypton
Language models understand natural proteins. But can they generalize beyond, to design completely new proteins from scratch?
New preprint: biorxiv.org/content/10.1101/…
A 🧵
In two new papers we have found that the ESM2 language model generalizes beyond natural proteins, and enables programmable generation of complex and modular protein structures.
ESM2 learns the design principles of proteins. With @uwproteindesign we experimentally validated 152 ESM2 designs, including de novo generations outside the space of natural proteins (<20% sequence identity to known proteins).
📄Read the paper here: biorxiv.org/content/10.1101/…
ALT Generated proteins are distinct from natural proteins
ESMFold and ESM models are on @huggingface. Easy to use for folding, fine-tuning, and new tasks!
Try it out here: colab.research.google.com/gi…
Thank you to the @huggingface team, @carrigmat and @GuggerSylvain for making ESM available in the transformers library.
xitter.cf/ClementDelangue/status…
It is so cool to see ESM-2 & ESMFold models from @MetaAI, new state-of-the-art Transformer protein language and folding models, available in the @huggingface hub & in transformers thanks to @carrigmat @GuggerSylvain & team!
huggingface.co/models?other=…
1
4