Serge Thill

ALT The top says: “Embers of Autoregression: Understanding Large Language Models Through the Problem They are Trained to Solve. By R. Thomas McCoy, Shunyu Yao, Dan Friedman, Matthew Hardy, and Thomas L. Griffiths.” The bottom left shows a ClipArt image of fire. The top of the fire is labeled “Sparks of AGI,” and the bottom is labeled “Embers of autoregression”. The bottom right shows a box labeled “Shift ciphers” with two examples of GPT-4 responses. First, when asked to shift each letter in a message back by 13, GPT-4 gets the correct answer: “I think everyone has their own path, and they can make it happen.” But when it has to shift back by 8, now GPT-4 incorrectly answers “I think therefore I am the best, and they can come at me with all their might.” The bottom says: “GPT-4 is much better at shifting back 13 letters (accuracy 0.51) than 8 letters (accuracy: 0.00). Explanation: In natural corpora, shifting by 13 is about 400x more common than shifting by 8.”