Even vs strong fine-tuned models like GoLLIE-34B, DEER (with GPT-4o) is competitive — all without training 💪 Training-free, label-aware, and robust. LLMs can do NER — they just need better demonstrations. 8/8

📊 On 5 datasets × 4 LLMs: ✅ DEER consistently beats all ICL baselines. ✅ Error reflection further boosts performance. ✅ DEER handles unseen entities better than KATE — showing real generalization. 7/8

Here’s how it works ⚙️ 1️⃣ Split training tokens into Entity, Context, and Other. 2️⃣ Compute label-based token weights. 3️⃣ Retrieve demos focusing on Entity and Context tokens via weighted token embeddings. 4️⃣ Apply error reflection to refine predictions at token level. → A full ICL pipeline tailored for NER. 6/8

We ran a sanity check 🔍 For test entities that also exist in the training data — how often do their counterparts get retrieved as demos? ❗Result: many overlapping entities are missing (~40% on OntoNotes). → Evidence that ICL retrieval often ignores crucial token-level signals. 3/8

Most ICL methods (e.g., KATE) retrieve demonstrations by sentence-level similarity. But for NER, this leads to a big gap vs fine-tuning methods. So, we asked: 🧩 Is the limitation from the LLM itself, or from retrieval? 2/8

🤔 LLMs can ace Olympiad math, yet struggle with something as “simple” as NER — even with in-context learning (ICL)? 💡 Our #EMNLP2025 paper answers why: “LLMs are Better Than You Think: Label-Guided In-Context Learning for Named Entity Recognition.” @mdredze @jhuclsp 👉 TL;DR: The issue isn’t the LLM — it’s how we pick demonstrations. 📄 arxiv.org/pdf/2505.23722v2 1/8

3. Scaling synthetic data narrows the gap with human-generated data, though the label quality of "hard-to-learn" examples remains the central bottleneck, which may be addressed through selective annotation.

2. Instruction-tuned LLMs make generating "hard-to-learn" examples surprisingly straightforward (simply tell the model what that data should look like). However, those examples may introduce label quality problems due to their challenging nature.

Glad to see my first project at JHU has been accepted to #ml4h2024. Here are a few key takeaways: 1. Naive prompting often produces "easy-to-learn" examples, and methods that promote syntactic diversity in LLM output don't address this fundamental issue.

Furthermore, we explore the potential of extracting from table images by using multi-modal models, such as GPT-4V, for table OCR, which significantly expands the set of documents our approach could be applied to.