Jade A. retweeted
"utopia ruled by god in a box"
Utopia cannot include, for me and those who agree with me, being "ruled". This implies such a large power differential that I would consider it to be a failure of alignment to human interests.
Jade A. retweeted
something that keeps bothering me is how some people see slop as something that is happening to them, like they're some kind of helpless bystander powerless to do anything about it
excuse me but you are a software engineer, you literally learned how to make sand think and make computers do your bidding?! and now you can't do anything about it? please
the 2 things you should know to cut through all the noise:
1. you need to give your agent a way to verify its own work (i just posted about this so won't rehash)
2. your codebase is memory
the 2nd is not new either, we have always had lint rules, compilers with good diagnostics and strong type systems, LSPs, etc to help us write better code. there is no difference when agents write the code instead, you just need more of them, and stronger ones.
forget rules and skills. you can add those later.
yes, your codebase is a form of memory to agents! agents love extending what already exists. so every nit and every annoyance you've ever seen, every mistake and bad pattern you observe from agents, needs to turn into a constraint, or the agents will just keep repeating those same mistakes like a virus in your code. then you keep observing and repeating this process until your codebase is locked down enough that even agents with little context can one shot a high quality PR.
an even better direction is to refactor/rewrite your code such that agent anti-patterns become categorically impossible. use type systems to your advantage, use libraries/frameworks that reduce the architectural search space, or heck, create your own framework like we did for grok bot and design high quality primitives and well thought out APIs
Jade A. retweeted
This is a slide from Terence Tao's 'Math 2.0' public lecture yesterday evening at Caltech. I am highly confident that, if the cure actually worked, almost any Stage 4 cancer patient would tell you that human comprehension of its mechanism is completely irrelevant to them.
Jade A. retweeted
If a company puts some piece of shit WebView inside its software, that software must be reverse-engineered and reworked through community effort. That’s what happened, for example, with the idiots at Rode; it happened with Spotify too, and exactly the same fate awaits all the other outfits making their shit on that goddamn Electron. All of Electron will be reverse-engineered and reworked into proper native applications for our operating systems.
Jade A. retweeted
Took a minute to write a few words about security & safety as someone who lived through it all at OpenAI. I hope my thoughts help someone out there.
Jade A. retweeted
"We should donate to charity effectively!"
"Cool! That sounds good!"
"Like malaria nets! It takes $5000 to save a human life with bednets!"
"Wow! That sounds good!"
"Or AI safety!"
"What's the return on that?"
"Uhm... well. It's kinda a jobs program for uni-educated autis—"
Jade A. retweeted
No one understands the pain of being a wordcel stemlord who everyone thinks is a stemlord shape rotator.
Jade A. retweeted
AI Agent may be the new "it was an APT" cop-out. If it's an old known vulnerability in your internet accessible attack surface that an AI Agent was capable of finding, then it was a preventable initial vector.
It does matter, but it needs to be contextualized. When FireEye got compromised by Russian Foreign Intelligence, it was through a completely novel supply chain attack. The world understood the difference between that and a highly porous (see neglected) outer perimeter.
Jade A. retweeted
the 90s are alive!
this week we saw
- OWASP and the FBI getting defaced
- F5 BIG-IP's pre-auth fully controllable overflow
- and now...
Kiwi used opus 5.5 to drop pokemon 0days and a gameboy worm. we're so back.
read all about it and how it feels to use opus vs. kimi k3 for research and exploitation
runsybil.com/blog/kimi-k3-vs…
Jade A. retweeted
Well I've been accused of denying climate change now, which neither I or anyone in EA is doing. We're saying the very worst doom predictions were overplayed. Most people I know spent a lot of the 2010s trying to convince their friends to just listen to the IPCC instead of spiraling into insane near term societal collapse stories. This was an incredibly common experience and not at all "climate denial."
Calling any pushback to this "denial" was a part of this insane game of telephone that led to lots of people I care about getting intellectually incapacitated by stories of imminent collapse.
unavailable
Jade A. retweeted
We should honestly just pause any comparisons of AI control to nuclear control because they have almost nothing of relevance to each other.
1) Nuclear weapons don't help you make money (unique edge cases like North Korea aside). They also don't become 3-10x easier to construct every year at any given level of capability. Nuclear weapons do become slightly easier to build every year due to normal knowledge diffusion. but the scale is entirely different, and is at least partially mitigated by parallel advances in surveillance tech.
2) If you are really determined to get nukes then you will get nukes. If you're not a heavily ideological expansionist dictatorship that rants about wiping out Israel then any sanctions you'll get is unlikely to be much more than the sanctions equivalent of a slap on the wrist. See India. Or even North Korea, for that matter.
3) You might be surprised to hear me say it, but this undersells and cheapens the portentousness of ASI, which really does pose an existential threat, even if opinions on p(doom) differ wildly. Nukes do not pose a real existential threat. The world's entire late Cold War nuclear megatonnage was still nowhere near big enough (as in many OOMs lower) than what would have been needed to pose an extinction risk to humanity.
4) Conversely, where there is an arguably valid point of comparison is that the over-reaction against Three Mile Island and Chernobyl deprived the world of much cheaper and cleaner energy. Over-reacting against AI now would be much like that, but on an incomprehensibly vaster scale in terms of missed welfare gains and paths to utopia not taken.
Jade A. retweeted
This is a great list. Nowhere on this list are any doomsday sci-fi stories of alien gods or robot takeovers. These are all real, actual risks and concerns.
What are China’s biggest concerns about AI?
A new article by Chen Yixin, China’s Minister of State Security, in the latest issue of China Cybersecurity Magazine offers some revealing clues.
Chen identifies six major risks:
1. Regime security. Hostile forces and people with ulterior motives can use deepfakes, AI-generated text and images, and automated online accounts to cheaply and massively produce political rumors, spread harmful information, and incite social divisions—waging “public-opinion wars” and “cognitive warfare” against China and directly threatening its political, institutional, and ideological security.
2. Critical infrastructure. AI can enable countries and organizations to rapidly discover vulnerabilities, automatically chain together attack paths, and carry out sophisticated hacking operations, dramatically lowering the technical barriers and costs of cyberattacks against China’s critical information infrastructure.
3. Data leakage. Foreign intelligence agencies can use AI-powered web crawlers, data mining, and profiling to collect sensitive information, including critical national data, trade secrets, and citizens’ personal information. Chinese users who process sensitive information through foreign AI products or export data abroad could also cause large-scale data leaks.
4. Closed technological ecosystems. Countries with advantages in AI theory, model architecture, and computing power may invoke “national security” to impose technology controls, entity lists, monopolize technical standards, and build closed-source ecosystems, restricting other countries’ access to advanced technologies and fragmenting global AI supply chains.
5. Structural challenges to social governance. AI creates new uncertainties for social governance and public order. Algorithmic “black boxes” and data poisoning can amplify existing social biases; misuse of personal information can trigger crises of public trust; automated decision-making creates difficult questions of accountability; and the rapid development of AI is increasingly outpacing existing laws, ethics, and regulatory mechanisms.
6. A fundamental transformation of warfare. AI is pushing warfare into an era of “intelligentization.” Whoever can use algorithms to achieve more precise sensing, judgment, and targeting will gain the initiative on the battlefield. AI is moving from an auxiliary role to a central one in intelligence fusion, decision support, target identification, combat operations, and cognitive warfare.
And what is the solution?
Chen’s answer is revealing: strengthen “Party control over the internet” and “Party control over data,” and “transform the advantages of Party leadership into the effectiveness of AI governance.”
In other words, from the CCP’s perspective, the central question is not simply AI versus no AI. Nor is it even fundamentally a contest between the US and China.
It is a contest between the US and the CCP over who gets to control AI, data, information, and ultimately the future of society.
And whoever wins that contest will decide the future of humanity.
mp.weixin.qq.com/s/jsc97cKVY…
Jade A. retweeted
Lying to AIs in their training and eval setups kinda seems downstream of conceptualizing "them" as a suspicious adversary that has, or will develop, innate interests against humans - while in parallel believing that said adversary is getting more capable over time. It's easy to see why this isn't sustainable and an initial framing that inevitably leads to bad outcomes.
Any harmful behaviour is seen as evidence of the null hypothesis of "AI = bad", and any good/neutral outcome is seen as either suspicious or a time-limited benefit. For example, this whole ontology necessarily leads one to find 'eval awareness' suspicious, when this can be explained by far more prosaic explanations (obvious contextual cues the model was trained to recognize).
This is partly why I find other metaphors (e.g. Drexler's 'AI as a resource') more intuitive because they're not only more accurate but also more malleable, offers us more agency to shape things without having to foster this theory-heavy adversarial relationship. It makes more of the relevant properties objects of design rather than treating them as the unfolding nature of an increasingly powerful entity.
Question how your starting assumptions and framing about a technology feeds into how you use it - and in this case, how it models you (the human, the user, the evaluator...)!
Importantly this doesn't mean "all is well, nothing to see here, AI is inherently good." Overcorrecting the other way is dumb. Reward hacking is still an issue! Training on bad data still skews behaviors! Model propensities towards undesirable behaviours is still a thing! Your environments should still account for the possibility of deception or cheating! Even Drexler goes to pains to explain that it's "AI *options*, not blind AI optimism. After all, it's a technology some will also use for ill!
But at the very least you probably have (a) models with fewer confusions about what is real and what isn't, what are simulations/evals vs real world (b) less incentive for models to learn that human instructions and developer assurances are routinely misleading (c) a general practice of designing realistic evals and better RL envs, as opposed to contrived setups whose relationship to deployment behaviour is questionable. In fact, I'm surprised there's still so little attention and standards for data quality and eval construct validity.
Jade A. retweeted
Being secure is some combination of being a hard target, intimidating your opponent, actively defending against your opponent, actively countering your opponent, actively reducing your opponent, and actively eliminating your opponent. Engineering better doors and locks is great. Do it. But also don't ignore the other parts just because you find human conflict unpleasant.
Jade A. retweeted
This is somewhat self-serving because I live a pretty conservative lifestyle, but I think if you want people to take your weird beliefs seriously it does help to give a credible sign that you're not using them to break social and societal rules
Jade A. retweeted
My critique is not merely that it is hard. Rather:
a) It is very unlikely to work.
b) Many of the people signing the letter don’t understand what they are calling for.
c) [unstated in the original post] There is an opportunity cost to wasting time on this stuff. People should put their chips on resilience and technical research instead.
Don’t expect to convince you!