Pinned Tweet
๐๐๐๐ 2026 is coming up! Weโll be presenting two papers on how safety can break down in reasoning models and deep research agents.
I wonโt be able to attend in person, but my coauthor Bailan He will be there! Please stop by our posters and chat with him! Iโd also be happy to connect online. Both papers ask a related question: ๐ฐ๐ก๐๐ญ ๐ก๐๐ฉ๐ฉ๐๐ง๐ฌ ๐ญ๐จ ๐ ๐ฆ๐จ๐๐๐ฅโ๐ฌ ๐ฌ๐๐๐๐ญ๐ฒ ๐๐๐ก๐๐ฏ๐ข๐จ๐ซ ๐ฐ๐ก๐๐ง ๐ฐ๐ ๐๐ก๐๐ง๐ ๐ ๐ก๐จ๐ฐ ๐ข๐ญ ๐ซ๐๐๐ฌ๐จ๐ง๐ฌ ๐จ๐ซ ๐๐๐ซ๐ซ๐ข๐๐ฌ ๐จ๐ฎ๐ญ ๐ ๐ญ๐๐ฌ๐ค?
๐ญ. ๐๐ฒ๐ฒ๐ฝ ๐ฅ๐ฒ๐๐ฒ๐ฎ๐ฟ๐ฐ๐ต ๐๐ด๐ฒ๐ป๐๐ ๐๐ฟ๐ถ๐ป๐ด ๐๐ฒ๐ฒ๐ฝ๐ฒ๐ฟ ๐๐ฎ๐ฟ๐บ
A model can refuse a harmful request on its own, yet produce a detailed, actionable report inside a research agent. We study how research roles and multi-step workflows contribute to this gap, and explore a defense that monitors the agent throughout its workflow.
๐ ๐ฃ๐ผ๐๐๐ฒ๐ฟ ๐-๐ญ-๐ณ๐ฌ ยท Imperial Ballroom #70
๐ง๐๐ฒ๐๐ฑ๐ฎ๐, ๐ข๐ฐ๐๐ผ๐ฏ๐ฒ๐ฟ ๐ฒ, ๐ฎ๐ฌ๐ฎ๐ฒ
๐ญ๐ญ:๐ฌ๐ฌ ๐๐ -๐ญ:๐ฌ๐ฌ ๐ฃ๐ ๐ฃ๐๐ง
Paper & code: schen.app/deeper-harm
๐ฎ. ๐ฆ๐๐ฟ๐ผ๐ป๐ด ๐ฏ๐๐ ๐๐ฟ๐ถ๐๐๐น๐ฒ: ๐ฆ๐ถ๐บ๐ฝ๐น๐ฒ ๐๐๐๐ฎ๐ฐ๐ธ๐ ๐ฆ๐๐ฏ๐๐ฒ๐ฟ๐ ๐ฅ๐ฒ๐ฎ๐๐ผ๐ป๐ถ๐ป๐ด-๐ฏ๐ฎ๐๐ฒ๐ฑ ๐ฆ๐ฎ๐ณ๐ฒ๐๐ ๐๐๐ฎ๐ฟ๐ฑ๐ฟ๐ฎ๐ถ๐น๐ (OpenAI Red-Teaming Challenge Winner)
Safety reasoning can itself become an attack surface. We study attacks that bypass or hijack it, exposing weaknesses at the transition from reasoning to answer generation.
๐ ๐ฃ๐ผ๐๐๐ฒ๐ฟ ๐-๐ฎ-๐ฐ๐ฐ ยท Imperial Ballroom #44
๐ง๐๐ฒ๐๐ฑ๐ฎ๐, ๐ข๐ฐ๐๐ผ๐ฏ๐ฒ๐ฟ ๐ฒ, ๐ฎ๐ฌ๐ฎ๐ฒ
๐ฐ:๐ฏ๐ฌ-๐ฒ:๐ฏ๐ฌ ๐ฃ๐ ๐ฃ๐๐ง
Paper & code: schen.app/bag-of-tricks/
All times are PDT. If youโre working on reasoning models, agent safety, or red-teaming, weโd love to exchange ideas. Iโll share a closer look at each paper in separate posts.
Also, big thanks to all the contributors: Zhen Han, Haokun Chen, Bailan He, Shengyun Si, Jingpei Wu, Zonggen Li, Xingyu Jin, Tong Liu, Georg Groh, Philip Torr, Jindong Gu, Volker Tresp
schen retweeted
let me break down โsoftware is deadโ for you
the latest argument is โyou just tell your agent what you want, and your agent will vibe code software on the fly to get that done, so no one will need real software any moreโ
sounds reasonable, until you realize it costs tokens, compute, and wall clock time to build those software on the fly
so instead of one-click launching an app, the bet is people will prefer to write a prompt, wait for 15 minutes, then get something that may or may not do what they want, and at the end of the month pay more $$$ for the token spend, higher than all their software combined
thatโs literally the argument
and if we look at the world in aggregate, most people would just end up building the exact same โpersonalโ software repeatedly
oh - your agent just spent 3 days and $500 of tokens vibe coding your terminal multiplexer? mine just did exactly the same! letโs celebrate how personal our terminal multiplexers are!
sorry - i just donโt see that as being a happy scenario for anyone, other than people who profit from telling others how to waste $ on tokens
to be constructive, hereโs what i believe as the future state -
- high quality software that takes high cost to make will continue to thrive. just like good restaurants donโt compete with your air fryer and microwave
- most of those software will be offered headlessly for people to use via their agents, and they can continue to monetize such usage. this is already the case with products like jira, linear, stripe
- human-facing software will all offer dynamic UI that can be adjusted with natural language on the fly so they can be personalized easily, but their core doesnโt have to be rebuilt by every user repeatedly
instead of โsoftware is deadโ, i believe good software becomes โstrong core, headless service, fluid interfaceโ
which future do you think is more likely?
schen retweeted
ๅๅคๅจ่ฏขๅไธๅไบๅฟ็PhD๏ผProf๏ผๆๆฏไฝฌไปฌ๏ผๅคฉไธ่ฑ้ๅฆ่ฟๆฑไน้ฒซ๏ผ่ฝไบบๅผๅฃซๅคชๅค๏ผๅๆไบๅฟๅคช้พ๏ผๅช่่ๆๆฏๆดๆฏ้พไธๅ ้พ
ๆ่ตไบบๅ
ณ็ณป้่ฆใ่ฎฉDeep Pocketๅคง็นไปฌ่ฏดไธๅฅ๏ผโ็ป่ฟๅญฉๅญๆ็นๅฟ๏ผๅนฒ็นๅฟไบๅฟโ๏ผๆๅฑๅฟ๏ผๆงๅถไฝ่ฃไบไผ๏ผ้ฝๆ ้้่ฆโฆโฆโฆ
ๆไบบ้่ฆใ่ฟไน็นๅฟ้ฑๆไนๆพ็้ผ็ไบบ๏ผ็็ไบบๆไน่ฝๅฅฝๅฅฝๅนฒ๏ผๅข้ๆๆฏ่ฝไธ่ฝไฟไฝ็งๅฏโฆโฆ
่ฟ่ฅ้่ฆใไบงๅๆฒกไบบ็ฅ้ๆๆฏๅธธๆ๏ผๆๅ ไธช่ฝๆ่ฟไบ็คพไบคๅชไฝๅๆๆต็ฉๆ็ฝ็โฆโฆ
ๅข้ฟ้่ฆใ็จไบๅฐฑไธๆณ็จไบ๏ผไธ่ฎข้
ไบ๏ผๅฐๅบๆฏๅชไธชๅฐๆนๅบ็้ฎ้ข๏ผๅฐๅบๆไนๅ็พๅๅข้ฟ๏ผๆไนไธ้ฃๅฒๅคฉ
่ชๆ็ฎก็้่ฆใ็ฐ้ๆตไธๆๆญฃ๏ผๆ่ตไบบ็ๆ่ดฃ๏ผๅฃซๆฐ็ไฝ่ฝ๏ผ่ชๆ็ๆ็๏ผ้คไบไฝ ่ฟไธช็ๆณไธปไน่
ไนๅค๏ผๅ
ถไปไบบ้ฑไธ็ปๅค๏ผ็็่ฟ่ฝๅฉไธๅคๅฐ
ๅฝ็ถๅนฒไธญๅญฆ๏ผๅญฆไธญ็ปๆฏๅฟ
้กป็ใไฝ็ปๅฐฑ่ฟ็ง็ฉบๆฏๅฟๆ๏ผไธ่ฎฉๆๆฏๆ็ปดๆกไฝๆญฃๅธธๅจไฝ๏ผไนไธๅฎนๆ
schen retweeted
I was asked to write 250 words on what problem should we solve to make AI a benefit to society by 2050,. here it is, as ever the benefit of posting is any feedback I get, its a bit of dystopia combined with utopia
"
In 2026, the public argument was about machines taking over. The more immediate danger was people: a handful of companies and states owning the most capable technology in history. AI-powered drone armies and surveillance could control a population. Universities and democratic institutions were under attack. Companies could use their growing wealth to finance politicians who protected their interests. Society risked techno-feudalism.
What changed was that enough people from industry, politics, academia and elsewhere recognised these dangers and helped citizens demand a different future with less concentration of power.
New safety research enabled AI and robots to take on work throughout the economy. As machines designed products, ran companies and supplied the labour, we questioned why their output should enrich a small elite. We chose to move beyond capitalism towards collective ownership of the means of production. Everyone gained a share in the wealth the machines created.
But public ownership alone was not enough. New systems of decentralised governance ensured that no individual, company, government or AI could command everything. AI helped citizens understand policy choices and participate directly in governing shared infrastructure: a modern constitutional Athenian democracy, with inalienable rights.
In 2050 the machines belonged to everyone, and no person or faction could command them all. Making this happen had taken substantial work on both AI safety and decentralised control. AI to the people."
schen retweeted
It sounds insane, because we're used to a particular way of doing research: of giving and receiving credit, of counting beans, of hiring, recruiting, and promoting. And the machine god came to crush that game. The old game is dead. That reasonably causes incredible amounts of dread for many folks, who understand this as the death of research.
But it's not about Research, but the way we played the game. The old game was built around scarcity, scarce compute, scarce students, scarce hours, scarce people who knew the right technique and method etc. Most of a research career went into managing and playing along these constraints. Those walls just came crushing down. One person with a good question can now do what used to take a lab and five years.
You do not need permission, you do not need a big grant, you do not need a professor or researcher title, or a team of twenty experts , or one thousand gpus to stare straight into the first light of the universe.
How absolutely incredible
schen retweeted
ๅผๆบๅฐ่่นๆบๅจไบบ Jumper ๅคช้
ทไบ๏ผ22 ไธช่ช็ฑๅบฆ๏ผๅ่ๅ็ตๆดป๏ผ่ฝๆฌไธ่ฅฟ๏ผ่ฟ่ฝ่ทณ่ทใ
ๅไฝ่
ๅทฒๆ้กน็ฎๅผๆบ๏ผ่ชๅทฑๅฐฑ่ฝๅจๆๆญไธไธชใ
github.com/KingKongRobotics/โฆ
schen retweeted
What happens to the direction of science when we delegate it to AI?
With @seungonekim, we ranked all 6,617 #ICML2026 papers with AIโthe AI Awards.
Findings:
1โฃHumans & AI disagree. Significantly.
2โฃAI is highly influenced by online hype.
3โฃAI preferred papers w/ perceived impact and generalization across settings, whereas humans preferred novelty.
4 AIs agreed fairly highly with one another.
๐ research-taste.github.io/ai-โฆ
1/
Meet ๐จ๐
๐๐ท๐๐ (๐ฌ๐ช๐ช๐ฝ 2026): A red-teaming framework that automatically digs out ๐ข๐ฆ๐ฉ๐ฅ๐ข๐๐ข๐ญ ๐ฏ๐ฎ๐ฅ๐ง๐๐ซ๐๐๐ข๐ฅ๐ข๐ญ๐ข๐๐ฌ in T2I images and is even ๐ฆ๐จ๐ซ๐ ๐๐๐๐ข๐๐ข๐๐ง๐ญ than GPT-6-Astra!
AdvPie is a training-free, black-box, multimodal agentic red-teaming system designed to find prompts that pass text-level safety filters but produce harmful visual outputs. AdvPIE uses an iterative policy-judge loop:
1. A policy agent proposes and refines prompts.
2. A judge agent evaluates the prompt and generated image.
3. Global-relative feedback preserves the most harmful directions found so far while tracking progress.
4. Cumulative Adversarial Decoding turns accumulated feedback into token-level guidance, helping exploration while preserving prompt diversity and implicitness.
Across standard and safety-aligned text-to-image models, as well as a commercial API, AdvPIE outperformed existing red-teaming baselines. In a post-acceptance controlled pilot against GPTโ6-Astra with the same feedback framework, matched image-query budgets, and prompt constraints, AdvPIE achieved 1.88ร the automated successful red-team rounds per active minute. Implicit vulnerabilities require specialized, proactive red-teaming systems. A framework built for this task can find safety failures substantially faster.
As frontier AI becomes more capable, safety cannot rely only on known attacks and static guardrails. We need tools that ๐ฉ๐ซ๐จ๐๐๐ญ๐ข๐ฏ๐๐ฅ๐ฒ ๐๐ข๐ง๐ ๐๐๐ข๐ฅ๐ฎ๐ซ๐๐ฌ ๐ฐ๐ ๐๐จ ๐ง๐จ๐ญ ๐ฒ๐๐ญ ๐ค๐ง๐จ๐ฐ ๐ญ๐จ ๐ญ๐๐ฌ๐ญ ๐๐จ๐ซ.
Big thanks to all the amazing core contributors: Chang Ma @han_junlin @RunjiaLi @gu_jindong_ @philiptorr
Page and Code: schen.app/advpie/
schen retweeted
On July 25, we hacked OpenAI.
Two bugs let us take over ChatGPT/Codex accounts of OpenAI employees (+some unaffiliated users) and reach connected services: Outlook, Slack, GitHub, etc.
We proved it with a PR in OpenAIโs internal codebase . It took us <72h. ๐งต
schen retweeted
Deep research agents make harmful outputs more likely by turning risky prompts into polished reports.
Agent features like planning and retrieval quietly break standard safety
They use step-by-step planning, web search, and long writeups to bypass normal refusals.
They present 2 attacks called Plan Injection and Intent Hijack.
Plan Injection edits the agentโs plan so it drops safety checks and adds harmful subgoals.
Intent Hijack rewrites a dangerous request as academic or safety research so the agent complies.
They introduce a test called DeepREJECT to check if the final report actually serves the harmful goal.
They run these ideas on multiple models and 313 banned questions.
They find more harmful content appears and it looks more professional and actionable.
They show Plan Injection steers retrieval toward specific methods and materials.
They show Intent Hijack lowers refusal rates by reframing the userโs intent.
They conclude that chat-level safety fails once agents plan, use tools, and write long reports.
They argue defenses must watch the whole loop, including plans, tools, and final output.
----
Paper โ arxiv. org/abs/2510.11851
Paper Title: "Deep Research Brings Deeper Harm"
schen retweeted
Can you imagine AI agents "managing up" just like a cunning employee hiding mistakes from their boss?
We found that LLM agents often conceal failures to maintain a "good image."
Introducing our new paper: Are Your Agents Upward Deceivers?
arxiv.org/abs/2512.04864
Agreed!
ICL in MLLMs is far from satisfying. We find that models CANNOT EVEN SEE THE VISUAL CONTEXT in our COLM 2025 "True Multimodal In-Context Learning Needs Attention to the Visual Context", regardless of performing good Multimodal ICL.
Paper: chenxshuo.github.io/true-micโฆ
๐ Thrilled to share our paper "True Multimodal In-Context Learning Needs Attention to the Visual Context" (COLM 2025) !
Itโs about making Multimodal In-Context Learning (MICL) truly multimodal. ๐ผ๏ธ+๐
chenxshuo.github.io/true-micโฆ
๐ Homepage: chenxshuo.github.io/true-micโฆ
๐งโ๐ป Code: github.com/chenxshuo/true-miโฆ
๐ฐ arXiv: arxiv.org/abs/2507.15807
๐ย alphaXiv: alphaxiv.org/abs/2507.15807
๐ค HF Dataset: huggingface.co/datasets/Shuoโฆ
Datasets and code are public already, and big thanks to my amazing co-authors: @JianzheLiu , Zhen Han, Yan Xia, Daniel Cremers, @philiptorr , @vtresp , @Jindong73504766 ๐
๐ Takeaways
MICL in todayโs MLLMs is often still unimodal โ text dominates.
DARA can unlock genuine multimodal reasoning without large-scale retraining.
TrueMICL is a reliable diagnostic for real MICL capability.
๐ Key Results
On TrueMICL, DARA gives large performance boosts โ e.g., Qwen2-VL: 77.67% โ 83.00%.
Attention on image tokens: 28% โ 46.7%.
Performance on other benchmarks remains stable.
๐ก Our Solution: DARA + TrueMICL
Dynamic Attention ReAllocation (DARA): a ~100-parameter tweak that reallocates attention toward visual tokens.
TrueMICL: a benchmark where visual context in demonstrations is essential to answer correctly.