founder in stealth. solving for compute infra. excited for the present x future :)
Bengaluru
Joined March 2024
- Tweets566
- Following1.2K
- Followers146
- Likes878
shan. retweeted
Your job, if you wanna be great, is to dedicate your life to something and to continue the pursuit of it when everyone else is trend jumps.
shan. retweeted
“you can’t do critical research without the compute scale of the top frontier labs”
Nick: hold my beer
interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had
with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works
already-clean.goodfire.com/
shan. retweeted
.@colossusjeremy says Mark Zuckerberg’s obsession with the Roman Empire goes back to when he was 15 years old, and traces of it are still everywhere at Meta:
“He has this long-standing interest in the Romans and in Augustus in particular. Augustus was the first emperor of Rome and turned it from a republic into an empire.”
“He used to end meetings by shouting, ‘Domination,’ and Sean Parker, the early president of Facebook, told him he had to stop doing it because it could be cited one day in an antitrust suit.”
“Courses he took in Latin in high school, where they had to read and memorize verses from Virgil's Aeneid, he still remembers now at the age of 42 and can recite them.”
“Just recently at their keynote, he brought out his personal Muse agent, and refers to it as Agrippa. Agrippa was one of the two key early lieutenants for Augustus.”
“I remember at some point Elon compared himself to Napoleon. I think a lot of these world-straddling figures at that level, they do think in these historical parallels.”
Jeremy Stern (@colossusjeremy) writes the world’s best profiles on the elite entrepreneurs and investors of our time.
In this episode, we talk about what he’s learned from spending time with Mark Zuckerberg, Palmer Luckey, Josh Kushner, Neil Mehta, Scott Wu, and more.
0:00 Jeremy’s Zuckerberg profile
6:39 Why Zuckerberg identifies with Augustus
13:03 The competitive drive people still underestimate
18:49 Why Zuckerberg won’t walk away
21:11 How Palmer Luckey became Jeremy’s first profile
35:05 Bringing literary ambition to business writing
39:20 Why Scott Wu stood out among AI founders
48:29 Josh Kushner’s family history and the American dream
56:36 Asking Rick Rubin: What if I lose the magic?
1:01:44 The mentor who taught Jeremy to use his gifts
Includes paid partnerships.
shan. retweeted
The most important thing we can apply AI to is improving medicine and human health - very excited about this partnership with the CZI and the Virtual Biology Initiative!
Today, Isomorphic Labs joins the Virtual Biology Initiative (VBI) as a founding member.
We are helping build an open resource for the global research community to accelerate how we understand and treat disease.
Read more here: isomorphiclabs.com/articles/…
shan. retweeted
⠀ 🇫🇷 🇺🇸
⠀ 🤝
thicc open models
Meet Mistral Large 4, aka Le Chonk.
• 1T parameters, natively multimodal. 49B active.
It is the best open weights model from US or Europe on aggregated benchmarks.
• State-of-the-art on critical workloads, including cyber defense, manufacturing and finance and it surpasses closed frontier models on visual grounding.
• Forged in Europe end-to-end and is deployable from Europe via our own Mistral Cloud infrastructure.
• Available to all via API today. Working with cybersecurity partners privately.
Open weights release end of October.
thankyou for making me fall in love with the sport and for letting me understand what greatness in life looks like;
work hard, stay humble, choose your family and hone your craft so well that the craft is defined by you.
mucho gracias #messi
there will never be anyone like you.
shan. retweeted
Very happy to announce that our team @GoogleDeepmind has pushed the boundaries of generative biology, achieving the successful synthesis of AI-designed proteins that are both functional and watermarked.
This proof-of-concept watermarking of the building blocks of life is enabled by SynthID Bio, our new protein watermarking method. It is designed to safeguard the new era of AI-powered generative biology and strengthen global biosecurity.
You can read my thoughts here on why watermarking AI-designed proteins is an important research breakthrough: xitter.cf/pushmeet/status/210531…
shan. retweeted
AMD just paid $8.2 billion for the researcher who accidentally made Nvidia the most valuable company on Earth.
Back in 2007, Fei-Fei Li was a junior professor with an idea her mentors warned would end her career. She wanted to label millions of internet photos so machines could learn to see. Funding agencies passed. Senior colleagues told her the field advanced on algorithms and that data collection was grunt work beneath a Princeton PhD.
She built it anyway. Roughly 49,000 workers across 167 countries spent two and a half years labeling images. Then she released the whole dataset, ImageNet, for free.
In 2012, a Toronto grad student named Alex Krizhevsky trained a neural network on her free dataset using two $500 Nvidia gaming cards. AlexNet nearly cut the error rate in half, and that one result is why every AI lab on the planet started buying GPUs. Nvidia's data center business was a rounding error in 2012. Today it does over $100 billion a year, and the whole company is worth more than the GDP of most countries.
So the woman who created the demand for AI chips now works for the chip company that came in second.
AMD's logic is the underrated part. You can no longer design a chip without knowing where models will be in three years, because a chip taped out today ships into a model landscape nobody has seen yet. Nvidia solves this by being the default everyone builds on. AMD just solved it by buying one of the few people who decides where models go next, and making her chief scientist reporting directly to Lisa Su.
Li immigrated from China at 15 speaking almost no English and spent her Princeton years running her parents' dry cleaning shop on weekends.
She gave the dataset away for free. Getting her cost $8.2 billion.
shan. retweeted
The economics of a Neolab.
A neolab is loosely defined as a startup of AI researchers who raises a lot of money pre-production to be able to finance GPU compute to take on a large AI problem.
To buy 1000 GB300s or ~14 NVL72 racks will set you back $125-150M for 3yrs with 15-30% upfront. That’s about ~2-2.5MW.
Thats about enough to do 10^25 flops a quarter and get to a GPT-4 level model which is 1-2 OOMs off frontier for pretraining.
If you post-train on a great open source model, you have a better chance of getting to frontier. The risks are a) you need to spend millions on RL environments too and b) being lapped by another model release while being tied to a base model.
For this to payback, you need to give your customers a better and ideally cheaper inference service than a base model and serve them for long enough to recoup your large investment. Even at 50% margin on inference, to recoup $10M in training means serving ~10T tokens (!) if you price like Fable / Astra given a standard cache read / input / output split ($2/M blended). And you have to justify being better than a release like Opus 5.5 which is even cheaper. Often, you end up charging your customers a huge premium in terms of platform fees and compute fees on top of pure inference.
Meanwhile, every hour you’re not utilizing your GPUs you are burning money so you typically resell this compute back to a broker or run inference for open models / resell spot instances. At below a ~60% utilization on spot, you will still lose money.
Add to that insane cost of talent.
So what can you do with the compute?
- Not play the model game at all.
- Play an entirely different model game (Jev, World Labs) that if big labs played, would either a) cannibalize their business or b) be incrementally not significant revenue c) would cause too much distraction from the main main thing
- Acquire a proprietary data set (Peridodic Labs) in enough volume in a domain of usefulness to eclipse frontier quality. Often happens in robotics, biology, chemistry.
If you do overcome the challenge of building a model that is useful and well priced beyond big labs models, given the huge price of compute, you still need to play in an area where the revenue / compute ratio is signficant and market demand is large enough to payback your compute spend.
It is a difficult game.
shan. retweeted
everybody talks about the economics of the ai data center build, the biggest capex buildout in human history
not nearly enough people are stepping back to just marvel at the technology itself
i just met amin vahdat, leader of google’s ai data center buildout, whose team delivered this beautiful cluster. it is insane the elite level of technology and engineering excellence that goes into every component of the modern data center, from the accelerators to the model codesign to the racks to the networking to the manufacturing and assembly. the chips are the most insane feat of all, but every component of the whole system is a beast.
goosebumps. we didn’t get to see the great pyramids get built, but this is even better.
amin also predicted for me what this photo might look like for frontier clusters a decade from now…. training data episode drops soon :)
The first Vera Rubin clusters are here!
Yesterday, @IneffableLabs took delivery of their Vera Rubin NVL72 cluster from @googlecloud @nvidia
The AI frontier jumps forward by yet another generation of hardware. Acceleration continues.
