Today, we're joined by @sebgehr, head of responsible AI in the Office of the CTO at @Bloomberg, to discuss AI safety in retrieval-augmented generation (RAG) systems and generative AI in high-stakes domains like financial services. We explore how RAG, contrary to some expectations, can inadvertently degrade model safety. We cover examples of unsafe outputs that can emerge from these systems, different approaches to evaluating these safety risks, and the potential reasons behind this counterintuitive behavior. Shifting to the application of generative AI in financial services, Sebastian outlines a domain-specific safety taxonomy designed for the industry's unique needs. We also explore the critical role of governance and regulatory frameworks in addressing these concerns, the role of prompt engineering in bolstering safety, Bloomberg’s multi-layered mitigation strategies, and vital areas for further work in improving AI safety within specialized domains. 🗒️ For the full list of resources for this episode, visit the show notes page:twimlai.com/go/732. 📖 CHAPTERS =============================== 00:00 - Introduction 2:17 - Generative AI in Bloomberg 7:55 - RAG LLMs are Not Safer paper 13:18 - Is RAG safer than LLMs alone? 14:22 - Prompt injection and unsafe behavior in LLMs 18:43 - Use cases 23:44 - LLM as judge 26:47 - What makes RAG-based LLMs unsafe? 32:22 - Understanding and Mitigating Risks of Generative AI in Financial Services paper 34:09 - Financial services taxonomies 37:27 - AI governance at Bloomberg 40:16 - Recommendations for risk mitigation 42:27 - Processes of rolling out GenAI applications 46:04 - Layered defense mechanisms 47:56 - Mitigation through prompting 50:56 - Future directions