Account based inGermany
About this account
- Account based in
- Germany
- Connected via
- Web
Account-level information from X, not a live location or the device used for a specific post.
AI Professor at Hasso Plattner Institute + University of Potsdam Prev. Rutgers (USA), Tsinghua (Beijing) https://nitter.cf/t.co/L2V1BquztY https://nitter.cf/t.co/cl6BMzsT6m
- Tweets2.1K
- Following1.2K
- Followers2.8K
- Likes7.4K
ALT AI@HPI Conference logo
ALT Group photo in front of a university building
ALT Language models encode substantial factual knowledge in their parameters, which can lead to unreliable behavior when this knowledge is outdated, incomplete, or misaligned with the provided context. In this work, we study whether modifying the pretraining signal can systematically shift models away from parametric recall and toward evidence-grounded reasoning. We introduce Knowledge–”Less” Language Models (KLLMs), a fundamentally different epistemic training paradigm for LLMs, which are pretrained on corpora in which named entities are anonymized, thereby removing a primary channel for entity-linked factual supervision. This intervention substantially reduces closed-book factual recall, while often improving performance on tasks where relevant information is provided as context. Across multiple model scales, KLLMs consistently outperform matched baselines on contextual question answering, fact verification, and hallucination detection benchmarks. Crucially, in retrieval-grounded setting
ALT Token Distillation: Attention-Aware Input Embeddings for New Tokens Abstract: Current language models rely on static vocabularies determined at pretraining time, which can lead to decreased performance and increased computational cost for domains underrepresented in the original vocabulary. New tokens can be added to solve this problem, when coupled with a good initialization for their new embeddings. However, existing embedding initialization methods require expensive further training or pretraining of additional modules. In this paper, we propose Token Distillation and show that by distilling representations obtained using the original tokenization, we can quickly learn high-quality input embeddings for new tokens. Experimental results with a wide range of open-weight models show that Token Distillation outperforms even strong baselines.
ALT panelists on stage
ALT panelists on stage