Aggressively building software with AI. Sharing what works (and what doesn't). Keeping the agents on track with @ToughCrowdDev.
San Francisco and Tahoe
Joined July 2009
- Tweets253
- Following928
- Followers253
- Likes471
Pinned Tweet
Can a solo engineer merge 500 pull requests a week with zero production incidents?
I don't know, but I'm going to try to get there. And I'll publish my results along the way.
The 1/500/0 Challenge:
toughcrowd.dev/1-500-0
Eric Bouck retweeted
something that keeps bothering me is how some people see slop as something that is happening to them, like they're some kind of helpless bystander powerless to do anything about it
excuse me but you are a software engineer, you literally learned how to make sand think and make computers do your bidding?! and now you can't do anything about it? please
the 2 things you should know to cut through all the noise:
1. you need to give your agent a way to verify its own work (i just posted about this so won't rehash)
2. your codebase is memory
the 2nd is not new either, we have always had lint rules, compilers with good diagnostics and strong type systems, LSPs, etc to help us write better code. there is no difference when agents write the code instead, you just need more of them, and stronger ones.
forget rules and skills. you can add those later.
yes, your codebase is a form of memory to agents! agents love extending what already exists. so every nit and every annoyance you've ever seen, every mistake and bad pattern you observe from agents, needs to turn into a constraint, or the agents will just keep repeating those same mistakes like a virus in your code. then you keep observing and repeating this process until your codebase is locked down enough that even agents with little context can one shot a high quality PR.
an even better direction is to refactor/rewrite your code such that agent anti-patterns become categorically impossible. use type systems to your advantage, use libraries/frameworks that reduce the architectural search space, or heck, create your own framework like we did for grok bot and design high quality primitives and well thought out APIs
Eric Bouck retweeted
the slop conversation is so one sided, it's not just that you can produce more code
you can also observe, debug, and refactor more code
even if the code is slop you have such good tools to deal with it. even if you write code manually, it's still 100x easier to maintain it
Eric Bouck retweeted
Dusting off this tweet from April because this gap in shared understanding of LLM capability is *widening*. It's now less "two groups of people speaking past each other" and more a sharp funnel.
- Napkin math somewhere around 6B people (~75% of the population) have barely come in contact with LLMs at all.
- Around 1-2B (~20%) are casual and infrequent users of free-tier ChatGPT-like products. This group sees derpy chatbots and treats them a bit like a better Google search, a writing aid, or etc. Maybe an agent tries to book you a flight. I have non-tech friends who (reasonably, imo) say they have not much use for it at all. Even many professionals outside of math&code are in this tier. For example, execs and many other functions spend a lot of their time talking to other people, so while they understand what is happening intellectually, it is still second-hand and a bit abstract.
- Now we get to professional use of frontier-grade LLMs in math&code. Somewhere around 20M people (0.2%) see first-hand that large, complex projects that used to take them weeks/months can now be completed by agents with a prompt. Building apps, copying apps, translating apps, decompiling apps from binaries... This has all happened very quickly and recently - less than 1 year ago, I was writing code manually by hand, typing memorized computer code commands into a code editor character by character, occasionally pressing Tab to autocomplete a little chunk of code.
- And finally we get to the ~5,000 people (~0.00006%) with access to frontier-grade systems internally. The external world has seen the preview. It looks like swarms of thousands of agents collaborating over weeks on software mega projects: minting zero days, running cyber attacks and defenses at machine speeds, discovering new science, advancing the frontier of mathematics. Things that would have taken top professionals in the industry years of work. Meanwhile, human review and comprehension are starting to fall behind. For example, people are still involved in the "archeology" of the OpenAI-HF incident from many months ago. Mathematicians may be poring over the 722 manuscripts on frontier mathematics for a while.
The funnel is driven by a combination of factors:
1. The impact scales with ambition, problem size, and horizon. A question with a paragraph answer barely stresses the system. You need a reservoir of big, difficult problems that you really care about. This aspect drives the consumer / professional dimension of the funnel.
2. The jaggedness of the system (which I have written about a lot separately). Capability peaks in domains that are digital, verifiable and economically valuable. This is because LLM capability emerges from reinforcement learning on verifiable rewards on a curated environment mixture driven by revenue potential. This aspect primarily drives the area (e.g. math&code) dimension of the funnel.
3. Access. Free-tier, paid-tier, internal.
So this is the weirdness of the moment. The general public has mostly not interacted with these systems. When they have, it looks like a derpy chatbot. The majority of professionals still see only a modest uplift. And a small sliver of professionals are experiencing the vertigo of the curve going vertical. And it is all happening at the same time.
Judging by my tl there is a growing gap in understanding of AI capability.
The first issue I think is around recency and tier of use. I think a lot of people tried the free tier of ChatGPT somewhere last year and allowed it to inform their views on AI a little too much. This is a group of reactions laughing at various quirks of the models, hallucinations, etc. Yes I also saw the viral videos of OpenAI's Advanced Voice mode fumbling simple queries like "should I drive or walk to the carwash". The thing is that these free and old/deprecated models don't reflect the capability in the latest round of state of the art agentic models of this year, especially OpenAI Codex and Claude Code.
But that brings me to the second issue. Even if people paid $200/month to use the state of the art models, a lot of the capabilities are relatively "peaky" in highly technical areas. Typical queries around search, writing, advice, etc. are *not* the domain that has made the most noticeable and dramatic strides in capability. Partly, this is due to the technical details of reinforcement learning and its use of verifiable rewards. But partly, it's also because these use cases are not sufficiently prioritized by the companies in their hillclimbing because they don't lead to as much $$$ value. The goldmines are elsewhere, and the focus comes along.
So that brings me to the second group of people, who *both* 1) pay for and use the state of the art frontier agentic models (OpenAI Codex / Claude Code) and 2) do so professionally in technical domains like programming, math and research. This group of people is subject to the highest amount of "AI Psychosis" because the recent improvements in these domains as of this year have been nothing short of staggering. When you hand a computer terminal to one of these models, you can now watch them melt programming problems that you'd normally expect to take days/weeks of work. It's this second group of people that assigns a much greater gravity to the capabilities, their slope, and various cyber-related repercussions.
TLDR the people in these two groups are speaking past each other. It really is simultaneously the case that OpenAI's free and I think slightly orphaned (?) "Advanced Voice Mode" will fumble the dumbest questions in your Instagram's reels and *at the same time*, OpenAI's highest-tier and paid Codex model will go off for 1 hour to coherently restructure an entire code base, or find and exploit vulnerabilities in computer systems. This part really works and has made dramatic strides because 2 properties: 1) these domains offer explicit reward functions that are verifiable meaning they are easily amenable to reinforcement learning training (e.g. unit tests passed yes or no, in contrast to writing, which is much harder to explicitly judge), but also 2) they are a lot more valuable in b2b settings, meaning that the biggest fraction of the team is focused on improving them. So here we are.
We ran CI 2,316 times in September. Our frontend regression tests caught 27 regressions where the existing test failed, the implementation was fixed, and the same test passed unchanged.
toughcrowd.dev/lessons/playw…
The frontend tests I rely on most: Playwright with a fully mocked backend.
* Easy to audit visually
* Cover happy paths, edge cases, network failures, etc.
* Run independently in parallel on every push
* Range from individual pages to full user journeys
toughcrowd.dev/lessons/playw…
Eric Bouck retweeted
It takes a while to build the confidence, but there's no future where you're manually reviewing every line of agent code. Not much acceleration in that. You need adversarial agent reviews, you need automated testing, and maybe you spot check. That's it. From prompt to production!
Eric Bouck retweeted
"People aren't ready, this is gonna change everything"
You know what people weren't ready for? Fucking mammoths. Tigers eating them. Monkeys ripping their arms off. The black plague. The ice age. Smallpox. Vikings pulling up on the beach. Drinking water that gives you diarrhea until you die. Getting a scratch and losing your whole leg. The Mongols. The Spanish Inquisition.
One volcano can send us back to the dark ages and these safety guys are gonna be shocked. Oh the robots didn't kill us? It was a volcano? Imagine their disappointment as the hot ash rains down on them.
A solar flare is gonna explode your phone in your pocket your legs are gonna be covered in battery acid and your bank account will stop working. You're afraid of a computer program doing math?
"Things are moving too fast" homie you live on a rock that periodically deletes everything. You are one the most luckiest entities in a 100 light year radius living during a golden era. Cheer up.
My ancestors crossed an ocean in a wooden box because they ran out of fucking potatoes.
"The social disruption will be enormous!"
Nigga World War 2? World War 2? Hop in the people carrier we're landing on the beach.
Are these people out of their fucking minds? Do they not know what we've survived already?
A couple decades of air conditioning and people suddenly think oh this is the natural condition of mankind. No. We used to have eight kids because seven didn't make it. You're gonna be fine. Man the fuck up.
Accelerate.
I think this is generally correct. But one thing that AI is really good at is refactoring. Another is analyzing code for bad patterns. So if you diligently revisit the code base you can clean up a lot of the slop that creeps in.
Feel this one in my bones for sure. I've been building software professionally for 20+ years. The one thing I can tell you with certainty? It's always been a battle against entropy.
Once you get more than 5-10 people on your team, it's a *constant* battle of constraining the complexity of your software. Unless you got really lucky and hired increcibly amazing folks who happen to be incredibly aligned in their software tastes. This actually should be your goal.
I could type a very long message here and rant for many paragraphs, but I think @zachdavis is very much onto something here.
Yes, today's AI-generated slop cannons are absolutely terrible and will crush productivity - but if you are able to find the handful of folks who know what they are doing, and you also have a systematic and aligned approach on how you are going to use AI to build software, it becomes *much* easier to build great things compared to the 'classical' times. AI can amplify your skills
Only caveat here, is that, if you get this wrong and allow an unskilled slop-cannon to impact your code-base, it is in fact going to be much worse. The failure mode pre-AI with those type of folks is that they just couldn't produce anything of value, and they were dead-weight and couldn't contribute. The failure mode now is that they can generate infinite amounts of code that is maybe plausible and it just distract syour good folks and consume their time and makes them jaded.
I guess the consistent aspect of all of this is: You still need to find good people. The inconsistent part is that finding not so good people has a different type of impact.
Eric Bouck retweeted
the right way to use model capabilities is not to ship 10x more features to prod
it's to spend more time understanding your users, trying experiments, building prototypes, learning about things you don't understand so that you can ship things that actually work
Eric Bouck retweeted
It is a tell that @sapinker, who authored Rationality, finds the extinction scenarios from the ‘rationalists’ of Effective Altruism preposterous.
Is AI really going to kill us all? The scenarios are preposterous, and the presumption of inevitability encourages fatalism, panic, and distraction from the more mundane and realistic safety challenges. By @clairlemon theaustralian.com.au/inquire…
Eric Bouck retweeted
Is AI really going to kill us all? The scenarios are preposterous, and the presumption of inevitability encourages fatalism, panic, and distraction from the more mundane and realistic safety challenges. By @clairlemon theaustralian.com.au/inquire…
Eric Bouck retweeted
There's been a ton of talk about the role of humans in code review, and when and how humans should be signing off on changes.
I believe that, long-term, humans have no role in routinely reviewing code.
Eric Bouck retweeted
You should follow @martin_casado for more sane takes on AI risk. His bio undersells the depth of his expertise - he did a CS Masters and PhD at Stanford, he’s worked at a National Lab and US Dept of Defense on networking and cybersecurity. His company got acquired by VMWare for $1.26B and he was their CTO for networking and security. If he isn’t worried, why are you?
Eric Bouck retweeted
Sequel to AI Pacing
I have now spent more time talking to people who run AI labs, Open Source projects and those in government and infrastructure.
I am beginning to feel the NINJA move could backfire.
I understand the pressure to come out and share where AI is "unmanageable ", and constantly share examples where it runs rogue. This fits in the category of "self-reporting" and an attempt to limit liability. Even the bleeding edge research examples are being cast in a negative light.
Here are the consequences of the NINJA move.
1. They have successfully encouraged every law maker around the world to have an opinion, and in cases a poorly un-informed one.
2. By proposing pacing - they have introduced uncertainty in the AI infrastructure trade, because we really don't know when "un-pacing" will begin or what those conditions will be.
3. The notion that a few leaders can collaborate and self pace is illusory, it takes one breaking ranks to start the race again, it could be a player in a different country or open source or an AI Lab itself.
4. There will be a regulatory body created as a consequence of this and it will be impossible to balance every stakeholder in this process.
5. No remedies, tools, solutions are being proposed other than "compute spent on safety", there is no mention of security (which will hamper adoption)
The recommendation is to fix alignment - Alignment is a hard problem - to fix alignment one shouldn't have trained models with "negative behavior". Alignment is a combination of moral standards, right and wrong and guardrails. Whose sensibilities will we align to? Alignment edge cases are hard. I look forward to learning more on this.
Guardrailing attempts post training have not shown precision. Distillation makes it worse, so AI will continue down it's path of getting smarter, while we will be chasing it to ensure alignment and building guardrails.
The AI labs have gone from being research projects to wanting to be the largest businesses in the world if they want to continue their progress. Transition from research winners to improving the lot of humanity and partnering with enterprises. Act like the largest companies in the world:
- Demonstrate the positive impacts of AI and how all of us benefit.
- Show enterprises how you can collaborate to solve real world problems and progress innovation safely and securely. Post an example every day how you helped.
- Solve the problem with a few players and demonstrate leadership so others can follow.
It's time to rebuild the brand of AI - any marketing expert will tell you, this Ninja move has done more to harm the brand of AI and will take a while to rebuild. #letsbepositive in our actions and our narrative.
I ran GPT-5.6 Sol with @addyosmani's code-review-and-quality skill on branches Greptile had reviewed.
I was surprised to see it catch more bugs, and more serious ones.
Anecdotal for now, but very interesting!
xitter.cf/ericbouck/status/20988…
Eric Bouck retweeted
Been going pretty deep into voice AI over the past few months and ended up building Midnight, a voice advisor for founders. It helps you talk through decisions, research things as you go, and practice important conversations.
Take a look at: askmidnight.com
PR review -> autofix cycle is live in @ToughCrowdDev! DM me if you want to give it a try.
xitter.cf/ToughCrowdDev/status/2…
Tough Crowd can now review your PRs, fix findings automatically, help you keep track of issues, and more. See the latest in our changelog for September 14th.
toughcrowd.dev/changelog