Security, trustworthy AI. Associate Professor, Duke University
Joined June 2011
- Tweets160
- Following244
- Followers1.4K
- Likes528
Neil Gong retweeted
Nominations for the AsiaCCS '27 PC have opened. Please share widely, and nominate others or yourselves here: docs.google.com/forms/d/e/1F…
[Late Advertisement] My student Yuqi Jia presented two posters on federated learning security at NeurIPS last week (which is why I attended NeurIPS for the first time in over a decade!).
Our paper "DataSentinel: A Game-Theoretic Detection of Prompt Injection Attacks" (arxiv.org/abs/2504.11358) received a Distinguished Paper Award at @IEEESSP! Huge thanks and congratulations to my amazing co-authors Yupei Liu, Yuqi Jia, Jinyuan Jia, and @dawnsongtweets!
Still using symbol replacement, image conversion, and similar strategies (shown below) to protect your email addresses from automated scraping? Our research shows they offer limited effectiveness against LLM-based extraction while making it harder for regular users to email you.
Many cyberattacks begin with spear phishing or social engineering, which often involve collecting personal information about potential victims.
Our study (arxiv.org/abs/2408.07291) demonstrates that LLMs excel at such information extraction. This highlights the potential for LLMs to automate cyberattacks at scale, posing significant security challenges.
Neil Gong retweeted
The 8th Deep Learning Security and Privacy workshop co-located with IEEE S&P @IEEESSP May 15, 2025, San Francisco (dlsp2025.ieee-security.org/) is calling for papers, posters and talks! The workshop seeks your awesome contributions on all aspects of deep learning and security, aiming to bring complementary views together by (a) investigating the security and privacy of deep learning, such as the recent generative models, and (b) exploring the application of deep learning for security and privacy.
We are calling for both proceeding papers (up to 6 pages) and non-archival extended abstracts (up to 3 pages). We will have one best paper award for the accepted papers and one best extended abstract for the accepted non-archival extended abstracts. For the first time, in addition to the talks, we will encourage the authors of the accepted papers to also present the posters for more in-depth discussions!
Neil Gong retweeted
We @OSUbigdata and @osunlp are very excited to host Neil Gong @NeilGong tmr (10:30AM-11:30AM ET, Dec 6th) to give an invited talk on Safe and Robust Generative AI. He will cover several critical safety and robustness issues in generative AI, including preventing the generation of harmful content, detecting AI-generated content through watermarks, and addressing prompt injection in large language models. The talk is open to people outside OSU. DM me for a zoom link, if interested!
Glad to see prompt injection is among the interesting competitions. One of the most important security/safety challenges for LLM 😄😄please participate!
Was my data used to train an AI model? In our CCS'24 paper (with Zonghao Huang and Michael Reiter), we propose a framework to audit data use in model training, with a formal guarantee on false positive rate (probability of falsely detecting data use) arxiv.org/abs/2407.15100
The key idea is to create two versions of each data sample and then publish one of them, selected uniformly at random. If a model was trained on the published version, it is more likely to be recognized as a member than the unpublished version.
The slides for my talk (given at multiple workshops recently) on Safe and Robust Generative AI are available here: people.duke.edu/~zg70/code/S…
Thanks for the contributions of my students and collaborators (many of them)! Comments are very welcome!
Excited about this work on benchmarking robustness of audio watermarking!
This quoted post is unavailable.
A paper (arxiv.org/abs/2310.12815) on formalizing and benchmarking prompt injection attacks and defenses for LLM was accepted by USENIX Security Symposium 2024. We thank the reviewers’ very constructive comments. Very excited about this paper. Congratulations to my coauthors!
Excited to co-author this paper. Comments are very welcome!
Excited to share the paper based on a workshop held in October 16, 2023.
eprint.iacr.org/2024/855
We first considered the AI legal landscape, technical issues (alignment, provenance), and the technical gaps between the legal landscape and the state of the art of existing techniques.
We welcome feedback on the document and will keep updating it.
Three papers on LLM were recently accepted. Congratulations to collaborators and students!
Paper 1 (ACL'24 Main) proposes a method to detect unsafe/jailbreaking prompts via looking into how neurons of an LLM respond to a prompt. arxiv.org/pdf/2402.13494
Paper 2 (ACL'24 Findings) proposes a method to test visual hallucinations of multi-modal LLMs. arxiv.org/pdf/2402.14683
Paper 3 (CCS'24) proposes a method to steal system prompts of LLM-integrated applications. arxiv.org/pdf/2405.06823