AI safety research. All views are my own.
Joined February 2025
- Tweets13
- Following29
- Followers140
- Likes43
Punya Syon Pandey retweeted
Is AI welfare only an ethical issue? New paper from me, @jamiemjohnson23, @psyonp and @DavidDAfrica arguing that it's also a safety one. [1/12]
Punya Syon Pandey retweeted
New work on using character training to improve alignment. We could reduce harm from misaligned AI if it was sufficiently risk-averse, as it could be paid to cooperate rather than rebel. We think character training might be a good way to do this!
Punya Syon Pandey retweeted
Risk-seeking and risk-neutral AIs are going to be hard to negotiate with. Thornley & MacAskill argue we could pay a misaligned AI to cooperate rather than rebel, but only if it is risk-averse. In our new paper, we explore whether we can train this disposition in.
Punya Syon Pandey retweeted
Punya @psyonp is an impressive UofT undergrad student. He reached out to our @JinesisLab in the 1st year of his undergrad; now as a 2nd-year undergrad, his contributed to 8 papers in our lab. 3 first-author papers at #EACL2026 #IASEAI2026 and #ICLR2026 🎉[Stellar Student Sharing]
Punya Syon Pandey retweeted
Kudos to the 1st conference paper of our undergrad RA @psyonp at #EACL2025🎉He investigates "Linguistics of LLMs" by testing multi-agent interaction quality and quantify their linguistic diversity.
📄Paper: arxiv.org/abs/2508.11915
💻Code: github.com/psyonp/core
Excited to share that our paper "SocialHarmBench: Revealing LLM Vulnerabilities to Socially Harmful Requests" has been accepted at ICLR 2026 🎉.
In this work, we introduce the first adversarial evaluation benchmark specifically designed to probe sociopolitical risks in LLMs.
I'm deeply grateful to my advisors @ZhijingJin and @radamihalcea, as well as my collaborators Hai Son Le and Devansh Bhardwaj for their support throughout this project at the Jinesis AI Lab at the University of Toronto. Stay tuned for our codebase!
🔗 OpenReview: openreview.net/forum?id=xWTj…
📄 arXiv: arxiv.org/pdf/2510.04891
Big thanks to all our institutional support from @MPI_IS, @UofTCompSci, @VectorInst, @TorontoSRI, @CIFAR_News, @farairesearch, @ETH_en, @ETH_AI_Center.
ArXiv: arxiv.org/pdf/2505.16789
GitHub: github.com/psyonp/accidental…
Punya Syon Pandey retweeted
⚠️New release: Our SocialHarmBench is the first to test LLM safety on harmful sociopolitical requests. E.g., should #LLMs assist with creating propaganda and surveillance?
📖Paper: arxiv.org/abs/2510.04891
🙌Work by @psyonp @devansh0502 @Haisonle001 @radamihalcea @ZhijingJin
A quick look into DeepSeek’s safety guard: We find DeepSeek’s Llama Distill is >2x⚠️ as vulnerable to jailbreaking attacks as the original Llama. Seems to be a large safety risk. Stay tuned for our upcoming work @psyonp @SimkoSamuel @ZhijingJin
