7/8 Bonus findings: • CluE preserves strengths when starting from a stronger seed • Transfers to Gemini-3-Flash backend • Single-step gains carry over to continual memory settings • Produces clean, structured taxonomies, not bloated rule lists

6/8 Results on BEHEMOTH (starting from a minimal seed prompt with Qwen3-32B): CluE: +9.04% overall relative gain, and positive gains across all three categories. ✅ Best among prior self-evolving methods.

5/8 Our solution: CluE (Cluster-based Evolution) 🌱 Each round: 1️⃣ Summarize each example by extraction scenario (not dataset) 2️⃣ Cluster examples with similar scenarios 3️⃣ Analyze each cluster independently 4️⃣ Synthesize cross-cluster insights into an updated prompt

4/8 Findings: 1. No static prompt dominates: • Category-specific prompts win in their target domain but regress elsewhere • Taxonomy-based ones are balanced but mediocre 2. Existing self-evolving frameworks were built for homogeneous distributions, and face a dilemma under heterogeneity: • Update too often → overfit to recent batch • Update too rarely → signals get averaged out

3/8 So we built BEHEMOTH (Benchmark for Extracting HElpful Memory On Tasks with Heterogeneity) 🦣. 18 datasets across 3 categories: 🧍 Personalization (5) 🧮 Problem-Solving (7) 🤖 Agentic (6) Each example: a source conversation plus a target query. Extract memory from the source, see if it helps answer the target. That's the signal.

2/8 The problem: Existing memory systems are built and evaluated under homogeneous task distributions. Personalization work targets preferences and facts; agentic work targets strategies and lessons. But a single assistant handles all of this: casual chat, math, debugging, agent tasks. This heterogeneous setting has no benchmark, and hand-writing one prompt that covers it all is tricky.

🧵 1/8 What should an LLM assistant remember across conversations? Existing memory work studies this one task at a time. But real-world assistants see all kinds of conversations, and that changes the problem. Introducing BEHEMOTH 🦣 + CluE 🌱: a benchmark & self-evolving method for heterogeneous memory extraction. 📄 Paper: arxiv.org/abs/2604.11610

🧵5/7 Can we teach models to retract better? Yes✅. Simple supervised fine-tuning (SFT) makes models learn better internal beliefs and thus significantly boosts retraction performance. Even better: belief directions before fine-tuning still steer model behavior after.

🧵3/7 We then steer the model’s belief via activation intervention: - Steering the model to believe an answer is wrong sharply increases retraction; - Steering it to believe an answer is correct suppresses retraction. 🌟This suggests that belief causally controls retraction.

🧵2/7 🚀We find that retraction is tightly linked to the model’s internal belief. Using belief probes trained on a universal truthfulness dataset, we show: Models fail to retract when they internally “believe” a wrong answer is correct.