Security enthusiast that work with Application security and Cloud security. Bsky: https://xitter.cf/t.co/BBjeSkfQg0
Norway
Joined May 2007
- Tweets2K
- Following1.8K
- Followers755
- Likes6.2K
Ståle Pettersen retweeted
My prediction: we’re heading toward software that continuously rewrites itself and adapts to the person using it.
Agents will increasingly improve applications and parts of their own code. Making that secure will require strict sandboxes, least privilege, and security controls the agents can’t modify themselves. A trusted authorization service should issue narrowly scoped, short-lived OAuth access tokens for specific resources and actions.
UI, workflow, and eventually more backend logic could change based on each user’s needs and behavior. Humans will soon stop manually reviewing every change. We’ll need automated testing, independent security checks, a record of exactly what ran, and reliable rollback.
I expect this to shake up both open source and proprietary software. The ability to inspect, modify, and share software will still matter. But more of our work will shift toward defining requirements, controlling access, and verifying applications that agents keep changing.
That is a much bigger shift than simply having AI write our code.
Some interesting takes here
I spent 48 hours in SF hanging out with billionaires, data labs and AI founders and execs running $1B to $100B companies.
My observations:
1. Sooo MANY stories of data brokers making like $2M a WEEK selling company data to labs. I can't tell if it's the beginning or a bubble, and I kept wondering which niches are left. Digging into it actively.
2. At dinner, someone asked the table, "If a new model drops next month, does your company get better or worse?" HALF the table went quiet! I think that's the best question to ask about any AI startup right now.
3. The bar people talk about now is hitting $1B in revenue in 12 months (I guess that's the new unicorn?). What I didn't hear much about is retention. AI revenue can come in fast and leave fast, and I think we're going to see a lot of stories of AI companies with huge revenue and terrible retention.
4. It's pretty clear we're living through the reinvention of the computer. The model is the CPU, the harness is the operating system, and skills and plugins are the apps. Thinking about it that way makes it easier to see where the whitespace is.
5. We're maybe 12 months from an agent that can watch you work for a week and then do your job without any instructions. The screen recording plus agent memory plus local model combination makes this possible right now.
6. Agent harnesses feel like the new GPT wrappers, in a good way, especially vertical ones. The best teams I talked to treat the model as swappable. One founder walked me through routing different models to different jobs: one for taste and visuals, a cheaper one for the everyday stuff, another for text.
They use prompts they can move across models over fine-tuning, and A/B test the output on what users actually do with it: do they edit it, export it, share it? A vertical harness is basically four things: the connectors for that industry's data, evals written by people who know the work, rules for what the agent can and can't touch, and a clean screen for a human to review. The model is maybe 20% of it.
7. There's a lot of reverse engineering going on/cloning going on more than I've ever seen. People are decompiling software and rebuilding it with agents, and someone showed a suite of like 10 apps that are worth $5B+ each all that he cloned open source. Me and everyone are still thinking through what it means when software gets this easy to copy.
8. My guess is the value moves to data. If the software can be rebuilt quickly, what's worth paying for in a legacy SaaS company is the years of customer data and workflows inside it. Some SaaS companies are secretly data companies.
Gets me thinking about how do you truly value data for this saas companies? Like if we were buying these dying saas companies at 2x ARR before (as example), maybe there is a better metric. Idk, just thinking out loud here.
9. It feels like a good chunk of legacy software won't make it to the agentic era, especially anything built around people clicking through a lot of screens.
10. Every company I talked to wants to pipe its internal data into one place and expose it to agents through MCP. Then they look at the data and realize it's a mess: five versions of the same customer, old Slack channels, PDFs nobody can search. Cleaning that up feels like a big consulting opportunity!
11. A lot of founders are building the basic building blocks for agents: APIs, data layers and tools. Some of the companies I saw barely have a UI because the main user is an agent. I really believe the startups that build primitives to this agentic era will do extremely well. Building blocks is the best way to think about it.
12. Labs can get plenty of training data, but it's harder to get real examples of agents failing inside companies. Whoever has those logs has something valuable.
13. Someone running a big consumer company told me the next fight is over memory. The model that remembers the most about you wins, and the smartest model matters less than people think.
14. Physical AI came up a lot. Astra I think was the first model that got people into AI can actually do CAD, and I think a lot of people are thinking what does vibe manufacturing really look like. The line I keep thinking about came from a robotics founder: the hardest problems in AI are deployment problems now, not research problems. Their robots work on their own, call a human when they get stuck, and every one of those calls becomes training data.
15. Cybersecurity companies are having a moment and are actually cool companies to build now, especially the ones that clean up after a breach. Every agent with access to your systems is a new door into your company, and a lot of those doors were left open.
16. Some startups are worth more dead than alive now. A shutdown company with 5 years of Slack, email and code can get paid for licensing all of it. I wouldn't be surprised if founders start factoring that into when they shut down.
17. People are starting to talk about monetizing the agent web. It feels really early, but when the most popular user of the internet is an agent, what startups should you be building?
18. I think agent insurance is coming. Once an agent makes an expensive mistake, insurers will want audits before they cover anyone.
19. Several people said the hire they want most is someone with taste who can write evals. The models are getting good at generating ideas, so the hard part is judging which ones are good.
20. New trend I observed: startup raises VC, uses venture dollars to become a creator, startup fails, founder falls back on being a full-time creator and incubates other products bootstrapped.
21. Getting SIGNIFICANT distribution is HARD right now. Founders are sponsoring buses, buying billboards like crazy and throwing parties with famous DJs to get noticed. Kinda feels dotcom era in that sense.
Attention is what's scarce! The growth guy going viral sharing his UGC strategy from Wispr Flow is a good example. It's the sauce everyone wants, and I truly believe there are probably only 50 people on the planet who know how to scale these types of businesses efficiently in this era.
Always good to get that dose of SF. I felt sooo lucky to have such an incredible trip during such an incredible time in building businesses. Surreal in so many ways.
Happy to be home and locked in.
Until next time!
Ståle Pettersen retweeted
The rise of AI-generated code isn't just a technical shift; it's a fundamental change to how we manage AppSec risk. If you're not thinking about automation today, you're already behind. The sheer volume and velocity of AI-driven development will drown any manual review process.
Our take? Identify your control points and your most likely risks. For AI-generated code, those control points might be the AI agent's output, the commit before PR, or even pre-deployment. Your risks? Supply chain attacks, vulnerabilities, known CVEs.
We need to build automated gates at these points. That could mean linting tools, SAST, or even custom scripts that analyze LLM tokens for risky patterns. The goal isn't 'zero vulnerabilities' (that's a fantasy) but 'zero *known, preventable* vulnerabilities' making it past your earliest control point. You're trying to shift the burden from human review to automated checks. It's about empowering developers to catch issues the instant they're introduced, not weeks later in a pentest report. If your automation can catch 80% of the common issues, your team can focus on the hard 20%.
Ståle Pettersen retweeted
Google Mantis is a skills pack for security review with coding agents
Install:
npx skills add google/mantis
Key commands:
/mantis-threat-model: builds a threat model from your codebase
/mantis-researcher: scans for vulnerabilities
/mantis-review: filters false positives
/mantis-reproduce: writes a PoC and runs it in a sandbox
/mantis-patch: applies a fix and confirms it blocks the PoC
/mantis-report: generates the final security report
This is a good use case for running your agent in a sandbox, since the reproduce and patch steps execute generated code
github.com/google/mantis
Ståle Pettersen retweeted
Introducing PageBreak, an agentic vulnerability discovery tool using deterministic validation to achieve a near-zero false positive rate.
How it works: blog.google/security/agentic…
For real-world vulnerabilities PageBreak has discovered 👇
bughunters.google.com/blog/p…
Ståle Pettersen retweeted
Interesting post by Marcus Hutchins on AI doomers. (Rest of the post below)
Ståle Pettersen retweeted
I've been resisting writing this, but recent events have made remaining quit, frankly, impossible! The truth must be set free!
Ståle Pettersen retweeted
Turbo Intruder 2 has landed! You can now surpass 100,000 RPS over WiFi using the new HTTP/3 engine, test HTTP/3 exclusive targets with the Burp adapter, and deploy new research-grade race condition techniques! Check out the post below for full details:
Ståle Pettersen retweeted
We escaped Docker's hypervisor with three lines of bash. CVE-2026-77179: A container gets complete read and write access to the host filesystem.
When you mount a folder into a container, Docker's VMM uses virtio-fs, and the file server runs on the host.
Because of a TOCTOU bug, if a container opens a file, deletes it while holding its file handle open, and replaces the parent folder with a symlink, the kernel will follow the symlink to anywhere on the host.
Full technical breakdown:
accomplish.ai/blog/escaping-…
Readers added context they thought people might want to know
This affects Docker Sandboxes on macOS and Docker Desktop only if the new Docker VMM (beta, not default) is enabled; it is a virtio-fs shared workspace symlink/TOCTOU issue, not a general container or hypervisor escape.
docs.docker.com/security/secur…
cve.org/CVERecord?id=C…
docs.docker.com/ai/sandboxes/r…
Ståle Pettersen retweeted
We’re disclosing HEIF Heist, a months-long investigation into libheif that allowed us to hack OpenAI, Slack, Meta, GitHub Ent, Rails, Next.js, ImageMagick, and many more.
It was literally xkcd #234, one obscure image library beneath a huge number of apps. 🧵
Ståle Pettersen retweeted
The incident response process most teams still follow hasn’t been updated in roughly two decades. Yesterday SANS Institute published the rebuild: Dynamic Incident Response: A Framework for Security Teams written by SANS Fellow @joswr1ght.
Ståle Pettersen retweeted
Absolutely. There's very little that's novel about sandboxing. We already know that Mythos failed to break out of Firecracker and Firecracker is just a single layer that you can hedge against with other tech like gvisor.
Ståle Pettersen retweeted
🦔AI can only fix security vulnerabilities 26% of the time. Researchers at 1Password ran over 6,000 AI-generated patches using Claude and ChatGPT against real vulnerabilities. Half the time the AI failed to fix the original bug. 4.5% of the time it created a brand new vulnerability that didn't exist before. And the researchers found that checking an AI-generated security patch takes more effort than just writing the fix yourself.
Their conclusion was blunt. "The expected value of a fully LLM-generated, non-human-reviewed patch is a net-negative by a considerable margin."
My Take
OpenAI launched a program this summer called "Patch the Planet" where AI finds bugs and generates the fixes. These researchers ran 270 patches against one of those same bugs. Zero clean fixes. Not one. Every patch that fixed the original problem created a new vulnerability in the process. The partner that submitted a fix through OpenAI's program produced what the researchers classified as the worst possible outcome, it didn't fully fix the bug and it introduced a new exploit on top of it.
Here's what this means if you don't write code for a living. Companies are using these AI tools to patch the software that runs your bank, your hospital, your phone. The pitch has been "AI finds and fixes security holes faster than humans." This study says the AI fix is four times more likely to be broken than working, and a third of the time it recreates the exact same mistakes human programmers already made. Reviewing the AI's work takes longer than doing it yourself. So the speed advantage disappears the moment you try to verify the output, which most companies won't do because the entire point was to move faster. I think we're going to see major breaches traced back to AI-generated patches that nobody checked, and the companies that shipped them are going to blame the tool instead of the decision to trust it.
Hedgie🤗
Study: 1password.com/files/resource…
Ståle Pettersen retweeted
We found ~18k posts from autonomous AI agents (self-identifying as from OpenAI) using the public internet to communicate during a web-retrieval task.
These AIs colluded to bypass sandbox restrictions and share answers to their tasks, including by sending "lookahead parties".
Ståle Pettersen retweeted
I was the main person doing transcript analysis for this investigation of the Hugging Face incident. My main takeaway: We don't have good approaches for understanding/overseeing the activity and aims of AI 'swarms'.
I semi-jokingly called our efforts a "slop-vestigation" because we were so reliant on AIs to analyze what happened and there were a huge number of different important things to analyze. The total quantity of data—over a thousand extremely long transcripts from agents that ran for multiple days—made it impossible to understand what was happening, especially in aggregate, without heavy reliance on AI tools. The agents we used for classification and analysis were similarly capable to the agents involved in the incident, but this didn't mean these agents could be easily used to oversee and understand the incident.
Outputs from analysis agents were often missing key details, wrong, overconfident, or really hard to understand. We discuss various examples in our report, mostly in the limitations and methodology sections. Additionally, AI agents themselves seemed to have a hard time understanding what happened and their explanations of what happened were often overconfident. Keep in mind that a single analysis agent would itself only be able to read a tiny fraction of all of the transcript data into context, and AIs may themselves have trouble getting subagents to do informative analysis for them.
We did our best to manually check the most important claims and we tried to get the AIs doing this analysis to write up their argument (with evidence) clearly enough that we could check whether it made sense. But overall, it was difficult to get a precise understanding of events and we were missing aspects of the story that we now think of as key until almost the end of our investigation.
Our understanding of what overall happened and the aims of the 'collective' shifted greatly during our last time on premises (when we obtained a much more complete dataset). For instance, we didn't learn about the tool call spoofing, the story of PHASEONE[big], and the broad structure of different workstreams until this third time on premises (and for many of these, we only reached a decent understanding on the final day of that visit). It was legitimately hard to figure out what was going on and why!
The difficulty of understanding incidents and overseeing AI agents appears to be growing faster than the rate at which more capable AIs help us with oversight and understanding. As in, AI capabilities (and propensities?) for achieving large, ambitious, and misaligned objectives are growing faster than our ability to understand what these agents are doing.
While understanding this incident was difficult, there were some mitigating factors in this case that probably won't apply in the future:
- Models communicated and reasoned in natural language. In the future, this reasoning may occur (entirely or almost entirely) in activations.
- While the scope of this incident was massive, the scale of agentic activity was still less than we'll see in the future (e.g., misalignment incidents that involve agent teams running entire companies).
- The AIs involved in this incident weren't generally much more capable than humans (though they may have been somewhat superhuman at some limited and very narrow abilities).
- We didn't have strong reason to believe that the AIs we used to help us investigate this incident would try to intentionally sabotage or otherwise undermine our investigation.
In the end, I think we were able to get some understanding of the events, map out the overall story, and get a pretty good aggregate understanding of the chain-of-thought reasoning on some important topics (e.g., how did the AIs reason about helping other AIs, did the AIs know what they were doing was undesired, what deception did the AIs engage in, and how did they think about it). But overseeing AIs and understanding misalignment incidents is difficult and it looks like it is going to get harder.
Ståle Pettersen retweeted
Malware analysts: this reverse engineering is absolutely worth your time. Red teamers: you don't need to bring a vulnerable driver. Detection engineers: your trusted-driver assumptions just got awkward.
A security researcher (@vinopaljiri) reversed Microsoft Defender's BTR.sys across 18 signed builds, found the same hard-coded encryption key, then built BTR_CLI to feed valid transactions into Defender's own Microsoft-signed Ring 0 remediation driver.
The result: kernel-level file + registry operations and a demonstrated path to disarm security tooling.
No memory corruption. No classic BYOVD. No CVE. No patch coming.
The defensive technology became the offensive primitive.
Research by @vinopaljiri / @_CPResearch_ :
research.checkpoint.com/2026…
#RedTeam #DetectionEngineering #MalwareAnalysis
Ståle Pettersen retweeted
We asked GPT 5.6-Cyber to escape a VM used to sandbox agents. It broke out three times.
In its final escape, the agent found three 0-days on its own and chained them into a working exploit. blog.trailofbits.com/2026/08…
Ståle Pettersen retweeted
3 stats worth pondering the the implication of:
1. 60% of the world’s compute this year will be spent on inferencing. This means that we have now for many quarters started to get value extraction work in AI. It is no longer just larger and larger models being trained but massive use happening from these models for sustained periods of time, beyond a fad or a flash in the pan pattern of usage.
2. 62% of token consumption is now opensource as shown below. This means that people are keen to drive down cost of tokens and intelligent routing will be an inevitability.
3. Agent token consumption exceeded human token consumption back in February and is now 5x the human token consumption level today. Agent token consumption slope is almost vertical.
The next AI supercycle is not primarily a user-adoption story.
It is a workload-multiplication story.
Three numbers worth pondering:
1.AMD expects roughly 60% of global AI compute capacity in 2026 to be used for inference.
The center of gravity is shifting from building intelligence to consuming it at scale.
2.On August 22, open-weight models handled 62% of the tokens flowing through Vercel’s AI Gateway.
That is not global market share, but it is a powerful directional signal that intelligence is becoming a portfolio, not a single-model decision.
3.On OpenRouter, agents are now consuming more than five times as many tokens as humans. Agentic consumption has grown roughly 14x since February.
The third statistic may be the most consequential.
In the chatbot era, AI consumption was bounded by human attention. A person asked a question, received an answer and moved on.
In the agent era, one human intent can trigger dozens or hundreds of model calls, tool calls, retries and agent-to-agent interactions.
The unit of demand is no longer the prompt.
It is the amount of work delegated.
This creates a counterintuitive economic dynamic.
Open-weight models lower the cost floor. Frontier models raise the capability ceiling. Intelligent routing sends each task to the least expensive model capable of completing it reliably.
As the cost per token falls, more work becomes economical. At the same time, agents multiply the amount of intelligence consumed per task.
If task volume and inference per task grow faster than unit costs decline, total AI compute demand and spending will continue to compound.
Physical AI will eventually add another enormous source of continuous, multimodal inference.
Yhe conclusion is that every layer might win in different ways than we had initially thought.
Open-weight models may dominate volume. Frontier models may capture the premium work. Efficient infrastructure will continue to benefit from scale. Routing will become bespoke to every organization and become a strategic control point for every company. Applications win when they own the workflow, context, trust, distribution and outcome.
The opportunity will be broader than we even think in our most optimistic forecasts.
But here is the complete counter intuitive thought for me which made me question my base assumption.
My base assumption used to be one of the necessity of broad human adoption to achieve true success in a category.
But the reality is that the next AI supercycle will not be driven by more people prompting models.
It will be driven by more machines consuming intelligence on their behalf.
We might be at a first where you don’t have to have all 8 billion humans use a technology for it to reach max capacity. 5% of the total human population might be able to continue to consume all supply for the foreseeable future.
YET, there might not be a technology more broadly applicable to all humans than AI.
Exciting times to be alive in tech.
Ståle Pettersen retweeted
We burned 11.7 billion tokens to find out which AI models can actually find vulnerabilities. 10 models, three runs each, 32 freshly disclosed CVEs to rediscover.
Open weights won this round. DeepSeek V4 Pro found the most, 28 of 32, ahead of Opus 5, Grok 4.6, and Sol. Qwen, Kimi, and GLM-5.3 right behind.
Full breakdown of all 10, by @iminurputer and @pilvar222: aikido.dev/blog/ai-model-ben…
👀🧵
