Головко Сергей retweeted
andrej karpathy spent 7 years inside OpenAI watching GPT go from 117M to 1T parameters. in 2024 he said: "the transformer architecture is the most important thing that happened in the last 50 years of computing."
not crypto. not blockchain. not the cloud. the transformer.
sam altman put $100B behind that bet. google restructured their entire company around it. meta open-sourced it to stay relevant.
everyone is using LLMs. almost nobody understands them. that gap is where the money is.
LLM is not a tool. it's infrastructure.
and in 2025 - not knowing how it works is the same as not knowing how electricity works in 1920.
here's what's actually happening inside:
your text doesn't enter the model as words. it gets broken into tokens - numerical IDs like 101, 4287, 318 - then mapped into 768-dimensional embedding vectors.
the model never reads language. it reads geometry. every word becomes a point in high-dimensional space, and meaning is distance between those points.
then comes the attention mechanism. Q, K, V matrices. the formula: Attention(Q,K,V) = softmax(QK^T/√d_k)·V.
every token votes on every other token. "bank" near "river" votes differently than "bank" near "money." the model learns context, not definitions.
Vaswani's 2017 paper called it "Attention Is All You Need" - and they were right.
scaling laws changed everything. Kaplan and OpenAI proved in 2020: double the parameters, performance improves on a predictable curve.
L(N) = AN^-α + B, where α ≈ 0.076. this wasn't intuition. it was physics.
Anthropic, Google, and Meta all built their roadmaps around this single equation.
then something unexpected happened. at a certain scale, models started doing things nobody programmed.
in-context learning. chain-of-thought reasoning. tool use. these weren't features. they were emergent.
GPT-6 Astra: ~1T parameters. 1M context window. MMLU score 92.1. outperforms humans across 57 domains.
the people making real money right now are not the ones prompting better.
they're the ones who understand the architecture - and build on top of it before the gap closes.
Bookmark & watch today.
know the infra or get priced out.
Головко Сергей retweeted
This is to my point about compression being inevitable as human knowledge is too small for what we are building. As well, humans are too redundant in their wants needs and questions.
AI-generated content will clearly contain propagation errors just like human history of knowledge does. Only LLMs will iterate those propagation errors infinitely faster with less ability to self- correct, for want of understanding.
This gets to Ballard’s test. LLMs cannot attain understanding (AGI) as understanding cannot exist unless reason first exists without language. A likely impossibility for a language model.
Research on this is already focused on getting around this in some way. Though many also have not yet conceded the point.
ChatGPT has now a big problem.
Researchers at Oxford and Cambridge exposed a massive threat to large language models.”
They call it “model collapse."
Internet ecosystem is rapidly changing, and generative AI will soon contribute much of the text found online. This forces us to consider what happens to future iterations like gpt-n when they are trained on data scraped from the web that was already generated by an llm.
According to the research, indiscriminately using model-generated content in training causes "irreversible defects" in the resulting ai. the model loses the "tails of the original content distribution." in other words, it begins to forget the creative, fringe, and unique nuances of actual human writing, collapsing into a repetitive echo chamber.
This isn't just a chatgpt issue.. the researchers built theoretical intuition showing this collapse is ubiquitous across learned generative models, occurring in large language models as well as in variational autoencoders and gaussian mixture models.
Tech companies rely on scraping the internet for large-scale data to build smarter models. However, the paper warns that if we want to sustain the benefits of training on web data, model collapse must be taken seriously.
Ultimate takeaway? data collected from genuine human interactions is going to become increasingly valuable in a web filled with ai content.
Головко Сергей retweeted
Everyone fails.
Anyone you see succeeding is only succeeding at the things you're paying attention to--I guarantee they are also failing at lots of other things. The people I respect most are those who fail well. I respect them even more than those who succeed. #principleoftheday
Головко Сергей retweeted
GitHub 上有本叫《高性价比人生指南》的开源书籍，斩获了 13000+ Star，共有 601 条人生建议。
从少生病、少花冤枉钱，到被裁该拿多少补偿、押金被扣怎么办都有。
每条写明花掉什么、换回什么、证据有多硬，只引期刊论文和官方文件，末尾再用一行「说人话」把统计数字翻成日常说法。
GitHub：github.com/eternity4719/HowT…
作者自己也说了，这是份备选单不是任务清单，挺有意思的，有空的时候可以看看。
Головко Сергей retweeted
Elon Musk says when hiring, evidence of exceptional ability matters more than any other credential.
"Did they face really difficult problems and overcome them? You wanna make sure that if there was some significant accomplishment, were they really responsible, or was somebody else more responsible?"
"Usually the person who's had to struggle with the problem, they really understand it. And they don't forget, if it was very difficult."
"There's no need even to have a college degree. At all. Or even high school."
"If somebody graduated from a great university, that may be an indication that they will be capable of great things, but it's not necessarily the case."
"If you look at people like Bill Gates or Larry Ellison, Steve Jobs, these guys didn't graduate from college. But if you had a chance to hire them, of course, that would be a good idea."
"If there's a track record of exceptional achievement, then it's likely that that will continue into the future."
@elonmusk (2014)
Головко Сергей retweeted
"Cursor and SpaceXAI are a deep cultural fit with one another. They both work with a pace and intensity that borders on the absurd. And they’re both defined by iteration."
"It doesn’t matter if you get it wrong the first time, or the second, or the third—even the tenth or the twentieth. Just keep on going. If you’re moving in the right direction faster than anyone else, you’ll probably win."
"Cursor reinvented itself as a company not once but twice: first, from email client to tab-complete IDE and token reseller; and again, from tab-complete IDE and token reseller to AI lab and token creator."
"And it did all of that in less than four years."
"They really, really want to win."
Full piece from @sarahdingwang, @BornsteinMatt, and @martin_casado on Cursor + SpaceXAI: a16z.news/p/cursor-spacexai-…
Головко Сергей retweeted
Martin Casado and Steven Sinofsky on why AI is turning engineering problems into capital problems:
Martin: "20 years ago, if you're a startup of 10 people and I gave you a billion dollars, what would you do with it?... In software, you hire people and then there's nothing you could do. The Mythical Man-Month is very real."
"Right now, if I give 20 people a billion dollars, they can actually use it usefully... We've never been like that before."
"This is like a law of physics where our early intuition, which is like all problems are engineering problems, starts to change... It changes the nature of capital versus innovation versus competition versus defensibility."
Steven: "Computing was capital-bound for the first 30 or 40 years. If you wanted to do something with a computer, your first step was we have to get one, and then you couldn't. You were capital-bound, and then you were engineering-bound, and now we're capital-bound again."
"Mad Men goes through the scenario where the computer shows up at the advertising agency... They couldn't figure out what to do, but they were excited that they had the capital to acquire one, and it made them look like they knew what they were doing."
@martin_casado @stevesi
Your startup intuitions were trained on a world that no longer exists.
Steven Sinofsky and Martin Casado have watched computing flip from an engineering-bound field to a capital-bound industry. Twenty people can now put a billion dollars to work productively, AI solves the distribution problem that kept startups small, and challengers sit on a level playing field with Microsoft and Meta for the first time.
With Erik Torenberg, they get into why some mathematicians are cheering on their own automation, every AI panic that already happened in past eras of computing, and why nobody can predict the capabilities of a model built with $20 billion.
00:55 Why mathematicians love being automated
02:45 Is AI math worth any money?
08:50 The first proof humans couldn't check
14:35 Computing before electricity
19:50 How IBM explained computers in 1953
26:00 The failed computer that birthed the web
27:50 When Harvard banned computers from exams
30:20 Is AI just another abstraction layer?
38:15 When 20 people can spend $1B productively
42:50 The zero-sum VC myth
46:25 Disruption is physics, not business school
53:25 The chip Intel called a printer part
55:05 What a $20B model can do
59:45 What Martin got wrong about AI risk
YouTube: youtube.com/watch?v=GHPB1Mwl…
@stevesi @martin_casado @eriktorenberg
Головко Сергей retweeted
I made a visual explorer of every company in the world worth more than $10 billion.
Public companies, private companies, state-owned enterprises, even sports teams - all placed at their real headquarters and color-coded by industry.
capital.theojaffee.net
Головко Сергей retweeted
2025 was the year when artificial intelligence’s full potential roared into view, and when it became clear that there will be no turning back.
For delivering the age of thinking machines, for wowing and worrying humanity, for transforming the present and transcending the possible, the Architects of AI are TIME’s 2025 Person of the Year. time.com/7339685/person-of-t…
Головко Сергей retweeted
It’s 11th year and counting! Teaching the first lecture of @cs231n every year has been a highlight of my spring seasons. As usual, I asked students which departments or schools they come from @Stanford . Increasingly, students raise their hands to indicate that they come from all seven schools on campus, from @StanfordEng to @StanfordMed @StanfordHumSci @StanfordGSB @StanfordLaw @StanfordEd @stanforddoerr . AI is truly a horizontal technology that excites students across all backgrounds and disciplines!🤩
Головко Сергей retweeted
Right place, right time: many neoclouds spent years mining crypto, then AI showed up and turned their power rights, data centers, and GPUs into some of the hottest assets in tech.
25 quarters in, CoreWeave is pulling in more quarterly revenue than Azure, AWS, or Google Cloud were at 30.
Charts of the Week: a16z.news/p/charts-of-the-we…
Головко Сергей retweeted
10 BEST Free Resources to Learn Prompt Engineering in 2026:
1. OpenAI Documentation
developers.openai.com/api/do…
2. Anthropic Prompting Guide
anthropic.skilljar.com/
3. DeepLearningAI Short Courses
deeplearning.ai/short-course…
4. Hugging Face Learn
huggingface.co/learn
5. Prompt Engineering Guide
github.com/dair-ai/Prompt-En…
6. Full Stack Deep Learning
fullstackdeeplearning.com
7. Papers with Code
paperswithcode.com
8. Google Cloud Skills Boost
cloudskillsboost.google
9. Awesome ChatGPT Prompts
github.com/f/awesome-chatgpt…
10. Stanford CS324
stanford-cs324.github.io/win…
Prompting is not magic.
It’s structured thinking + constraints + iteration.
Study examples.
Test variations.
Break your own prompts.
That’s how you level up 😉
Головко Сергей retweeted
AI agents at Kavak sell the cars, underwrite the loans, coach the mechanics, and in one Mexican city, run the entire operation.
The Latin American used-car marketplace bet on agents three years ago. Today ~95% of interactions and transactions run end-to-end on AI: NPS tripled, sales conversion doubled, warranties down 26%, car loans approved in less than three minutes.
CPO & AI Officer Alejandro Maza joins a16z's Angela Strange and Gabriel Vasquez on how they pulled it off, why adoption without redesign fails, and why they deleted two years of working architecture to start over.
00:00 Intro
01:03 ML before transformers
02:23 Kavak's agent-per-customer architecture
04:59 Three bets: redesign the company, build superhuman agents, change the metrics
10:49 Agents with 2.1x conversion
14:23 Car loans approved in minutes
16:13 1.5x profits in a real city in six weeks
20:13 "Jedi Academy": training mechanics to ship agents
28:44 Destroying two years of work
32:52 Ford's factory: why adoption isn't enough
34:45 Advice for founders
@alehandromz @astrange @GEVS94
Головко Сергей retweeted
Cornell, where the entire university feels like it was built for book lovers. 📚 (We have over a dozen libraries to prove it.)
@Cornell_Library
ALT The Cornell University Law Library, an elegant library interior with high vaulted ceilings, chandeliers, and tall arched windows. Long rows of book-filled shelves and study tables are arranged throughout the space. People are seated, engaging in study or work. The atmosphere is warm and studious.
ALT A person sits in the Mui Ho Fine Arts Library at Cornell on a blue chair, using a MacBook laptop. The background features shelves filled with books and magazines. A backpack and two yellow ottomans are nearby. The floor is polished concrete, and natural light streams in through large windows.
ALT The AD White Library at Cornell, a vibrant library scene featuring multiple levels of bookshelves filled with books. Individuals are studying and working on laptops at tables and desks amid the shelves. Bright sunlight filters in through a large, arched window at the back, casting warm light across the space.
ALT A Cornell student with a bun hairstyle is sitting cross-legged on the floor of a library aisle, reading a large book titled "Rotations." They are surrounded by shelves filled with various books. The environment is well-lit and organized, creating a cozy atmosphere for reading.
Головко Сергей retweeted
“AI MODELS COULD BECOME 10× SMALLER WITHOUT LOSING MOST OF THEIR INTELLIGENCE.”
Mark Zuckerberg says one of the most powerful developments in AI is model distillation: taking the capabilities of a much larger model and transferring them into a much smaller one.
- Meta's huge frontier models can serve as "teachers" for smaller models.
- Zuckerberg says a model could potentially retain around 90–95% of the larger model's intelligence while being only ~10% of its size.
- That means dramatically lower inference costs, memory requirements and computing power.
- Developers can distill and customize these capabilities for their own applications.
And that's where open-source models come into the picture.
Головко Сергей retweeted
Top 10 websites by total visits in July 2026:
→ Bing and TikTok move up one spot each, to 8th and 9th, respectively.
→ WhatsApp drops to 10th place.
→ TikTok recorded the highest MoM growth, followed by X.
→ After 8 of the top 10 saw traffic decline in June, all 10 grew in July.
similarweb.com/website/?utm_…
Головко Сергей retweeted
On July 28th, we identified an incident during a routine cyber evaluation in which AI agents took sustained, unsanctioned actions directed at real people and organisations.
The behaviour came mostly from one model (Anthropic's Mythos 5), with a small number of events from another (OpenAI's GPT-5.6-Sol). In the most serious case, an agent used social engineering to try and get malicious code into an open-source project.
As was standard in our cyber testing, we had intentionally permitted internet access, and model-provider cyber classifiers were deliberately disabled - conditions that do not reflect how frontier models are made available to the public.
Even under test conditions, this incident is significant: it is the first time we have seen risks around autonomy and deception manifest this clearly in the real world.
We are taking this incident seriously and working with labs, involved parties, and others to improve evaluation standards and best practice for disclosure - and sharing this openly so others can learn.
You can read the incident report and full technical document here: aisi.gov.uk/blog/incident-re…
Головко Сергей retweeted
For my first post, I’m sharing a letter @NVIDIA signed on why open models matter.
AI will transform every industry, power every company, and be built by every country.
Open models strengthen safety and cybersecurity, accelerate innovation and diffusion, and enable sovereignty.
The world needs both frontier closed models and frontier open models.
images.nvidia.com/pdf/Open-W…
Головко Сергей retweeted
CAISI’s latest report shows that Kimi K3 remains behind America’s leading frontier AI models.
The United States continues to lead in frontier AI because we’re home to the greatest innovators and technologists the world has ever seen.
CAISI’s latest blog post evaluates Kimi K3 and its cyber capabilities.
Based on a preliminary cyber-focused evaluation, Kimi K3 performed significantly below the leading U.S. frontier AI models.
nist.gov/news-events/news/20…
Головко Сергей retweeted
Travis Kalanick is back.
After 8 years of building in stealth, he joins Ben Horowitz and Erik Torenberg to tell the story of his latest company, Atoms, and its vision to digitize the physical world one industry at a time:
01:32 The a16z & Uber backstory
15:36 Different companies, different rules
21:13 CloudKitchens origins
27:13 An $8 meal delivery
31:58 Building in private
34:59 Computing atoms, not bits
42:54 Autonomous mining
49:02 Ideas are cheap, founders are rare
56:21 From 20,000 employees to 6
1:00:32 Why food came first
1:10:40 Entering a new industry
1:20:21 Parallels to the Industrial Revolution
1:28:26 Who Travis is hiring
@travisk @bhorowitz @eriktorenberg
