jwilles retweeted
This awesome project (led by @LouisKirschAI) was the result of an internship filled with exciting, surprising, and often very confusing results. Come for the thorough evaluation of in-context learning, stay for the super weird training dynamics!
Emergent in-context learning with Transformers is exciting! But what is necessary to make neural nets implement general-purpose in-context learning? 2^14 tasks, a large model + memory, and initial memorization to aid generalization.
Full paper arxiv.org/abs/2212.04458
🧵👇(1/9)
