Introducing Muse Wonder, our new family of frontier models with state of the art performance for personal and professional agentic work.

Muse Spark 1.3 is rolling out today with frontier performance almost too cheap to meter. This is the biggest jump we've made so far on coding and agentic work. Try it in Muse Code and our API. Next up 🍉 and Muse Spark open weights releases coming soon.

Still surprised this doesn’t start reward hacking

Ten years ago in 2015 we published a paper called End-to-End Memory Networks (arxiv.org/abs/1503.08895). Looking back, this paper had many of the ingredients of current LLMs. Our model was the first language model that completely replaced RNN with attention. It had dot-product soft-attention with key-value projections. We stacked multi-layers of attention so the model can attend to different parts of input. We introduced position embeddings to remedy order-invariance in attention. It was an improvement of the Memory Networks paper (arxiv.org/abs/1410.3916) in 2014, the year before, which had introduced stacked hard attention over multiple layers – concurrently while soft attention was introduced on a single layer by Bahdanau et al. (arxiv.org/abs/1409.0473), see Dzmitry’s note about the history (xitter.cf/karpathy/status/186402…). End-to-End Memory Networks put these two ideas together and showed that multiple layers of soft attention induced complex reasoning abilities – which is one of the most important aspects of architectures today (note: Transformers are from 2017). Given its 10 year anniversary, I thought it would be fun and maybe inspiring to share a little story about how this paper came about. It started around spring of 2014 with my advisor Rob urging me to do research about memory. While now it’s obvious, back then I didn’t even understand what memory meant. This was a world dominated by recurrents networks and convolutions. Memory wasn’t as big a catchphrase like in today’s research. But the groundwork was done already. As said above, a year earlier, Jason and colleagues published a paper that outlined this crazy new architecture called Memory Networks. They also published a set of tasks called bAbI that made recurrent models fail miserably. These tasks required looking up multiple facts in out-of-order fashion, which of course was a deal-breaker for RNNs. Despite all this, I was still clueless. And I was starting my internship at FAIR as a second-year PhD. The project initially attracted a lot of attention, with a lot of people attending the weekly meeting. But things didn’t go straightforwardly. Besides the general theme of memory, we had little concrete ideas. This lasted for months and people left the project one by one, probably frustrated by lack of any progress. Eventually we started working on Memory Networks with the goal of making it learn where to attend instead of requiring given labels. Maybe because RL was a hot topic back then (like today I guess), or maybe because I like complex ideas, we decided to use RL training to teach memory where to attend. Fast forward to winter of 2014-2015 and we had actually implemented the RL code and were ready to compare against baselines on LM tasks. One obvious choice was soft-attention used in Bahdanau’s seminal work – but in our work stacked in multiple layers, which was not done before. So we implemented it as a baseline, but changed a few things like using dot-product instead of a small MLP to compute attention. Immediately our MemNet with this soft-attention started working so effortlessly that it became obvious to us that this was the way to go. Things started moving fast after this. We started using the bAbI tasks as a benchmark, thanks to Arthur's insistence. We developed several new techniques such as using different projections for keys and values etc. We also needed something to resolve order-invariance of attention, so we added time embeddings (now known as position embeddings). Jason suggested adding random noise to these time values to reduce overfitting. Finally, we decided to do a language modeling task, which was at the time not popular. To our surprise, we actually beat LSTM using attention without any recurrence over time (in the paper, we use the word “recurrence” to describe repeated layers, i.e. sharing weights like universal transformers). We wrote most of the paper on the last day of Neurips submission, with everyone scrambling. Thus our End-to-End Memory Networks paper came together. Fun fact, it was initially called “Weakly Supervised Memory Networks” because it required less supervision. Anyways, that period was a golden age of new architectures, with new papers coming up such as Neural GPUs, Stack RNN, and Neural Turing Machine. Looking back after 10 years and the current state of LLMs, I think we got several things correct with our paper, even though it is overshadowed by the 2017 Transformer paper. Our model was the first attention-based language model that didn’t rely on recurrence. We did stacking multiple layers of attention so that the model can attend to different parts of context before outputting the next token. We also used position embeddings, even a relative one, which now has become an obvious thing for LLMs. Of course, Transformer made important improvements, like using previous hidden states as memory for the next layer. And there are FF layers, multi-headed attention, and so on. Still – 10 years later – I think there is more work to do to improve architectures! Just a few days ago we released Multi-token attention (MTA) (arxiv.org/abs/2504.00927), so I’m still working on the same things! When Jason mentioned that attention operated at only token-level, it became obvious what we needed to do.  MTA conditions attention over multiple queries, keys and heads, and it improves over standard soft attention in many metrics. Particularly, it is able to solve long-context e.g. the needle tasks much better. That’s somewhat amusing that the conclusion of the Memory Network paper in 2015 mentioned this as future work: “smooth lookups may not scale well to the case where a larger memory is required”, which is something the field is still working on today. For the future, I’m sure there’s much more to come, of course maybe something much different to attention – let’s keep innovating and working together as a community – I’m excited about what further research lies ahead!