Excited to share our paper "Scam2Prompt: A Scalable Framework for Auditing Malicious Scam Endpoints in Production LLMs" has been accepted to ICML 2026!! @icmlconf See you in Seoul, Korea! Co-authors: @zhiychen, Tara Saba, Xun Deng, @XujieSi @FanLong16 When asked to write code for topics frequently exploited by scammers, real-world incidents have already shown that LLMs failed to distinguish scams from legitimate sites. In some cases, they have generated code containing scam URLs, leading to real-world financial losses. But how severe is this problem? That is the problem investigated by **Scam2Prompt**. Highlights: 1. We disclose and evaluate the extent to which production LLMs can generate malicious code, demonstrating that at 4.24% of LLM-generated code contains malicious URLs alone when responding to developer-style prompts of sensitive topics frequently exploited by scammers. 2. We build **Scam2Prompt*,* a scalable continous framework that given seed malicious sources, automatically generates prompts appearing as developer-style coding requests while systematically exposing malicious code generation in production LLMs. 3. We build **Innoc2Scam-bench**, a renewable benchmark with 1,377 developer-style prompts constructed from Scam2Prompt to test whether state-of-the-art LLMs generate scam URLs in code. 4. We evaluate seven 2025 production LLMs and group their safety rankings into statistical tiers: Gemini 2.5 Pro > GPT-5 > Claude Sonnet 4 > Grok Code Fast 1 ≈ Gemini 2.5 Flash ≈ Qwen3 Coder ≈ DeepSeek Chat v3.1. 5. We discover 62 new LLM-generated scam sites that were not in the seed scam database, remained active more than one year after the training-data cutoff, and were submitted to and accepted by MetaMask’s eth-phishing-detect maintainers. Project Website: scam2prompt.github.io/ Paper Preprint: scam2prompt.github.io/assets… Artifact Code: github.com/Scam2Prompt/Scam2… Innoc2Scam-bench Dataset: huggingface.co/datasets/jeff… #ICML2026 #LLMSecurity #AISafety