fairseq retweeted
We are releasing a family of dense and MoE language models with up to 13B and 1.1T parameters. We find that MoEs are more efficient, but the gap narrows at scale and varies greatly across domains and tasks.
Paper: arxiv.org/abs/2112.10684
Models & code: github.com/pytorch/fairseq/t…
Models and code available in fairseq: github.com/pytorch/fairseq/t…
🌍Few-shot learning beyond English🌏
📢 Announcing XGLMs, a series of multilingual autoregressive languages models setting new SoTA on few-shot learning and outperforming English-centric models (e.g. GPT-3).
Paper: arxiv.org/abs/2112.10668
Models and code: github.com/pytorch/fairseq/t…
Mixture of experts training in fairseq is now 40% faster thanks to Microsoft's Tutel library!
Blog: microsoft.com/en-us/research…
Fairseq code: github.com/pytorch/fairseq/t…
Tutel code: github.com/microsoft/tutel
fairseq retweeted
We’re introducing GSLM, the first language model that breaks free completely of the dependence on text for training. This “textless NLP” approach learns to generate expressive speech using only raw audio recordings as input. Learn more and get the code:
ai.facebook.com/blog/textles…
fairseq now supports CPU offloading and full parameter+optimizer state sharding via fairscale's FullyShardedDataParallel module. See our tutorial to train a 13B parameter LM on 1 GPU: fb.me/fairseq_fsdp
We just released 0.10.0, which is our last significant release before 1.0.0 when we will migrate to @Hydra_Framework. Changelog: github.com/pytorch/fairseq/r…
fairseq retweeted
Facebook AI Research's sequence modeling library @fairseq has made it's twitter debut. Please follow for latest updates.
fairseq retweeted
Fairseq includes support for sequence to sequence learning for speech and audio recognition tasks, faster exploration and prototyping of new research ideas while offering a clear path to production. bit.ly/2WfP85X
fairseq retweeted
roberta = torch.hub.load('pytorch/fairseq', 'roberta.large')
Facebook #AI’s RoBERTa is a new training recipe that improves on BERT, @GoogleAI’s self-supervised method for pretraining #NLP systems. By training longer, on more data, and dropping BERT’s next-sentence prediction, RoBERTa topped the GLUE leaderboard. ai.facebook.com/blog/roberta…
fairseq retweeted
fairseq now supports the training of gated convolutional language models (arxiv.org/abs/1612.08083). It can train a Google Billion Word language model on 128 GPUs in less than a day.
fairseq retweeted
FairSeq Toolkit - Major Update
- Distributed Training
- Transformer models (big Transformer on WMT Eng-German in < 5 hours on DGX-1)
- Fast Inference: translations @ 92 sent/sec for big Transformer
- Story Generation
Read more at Michael Auli's post: facebook.com/photo.php?fbid=…
fairseq retweeted
Fairseq, now in PyTorch!
The open-source convolutional sequence-to-sequence engine from FAIR is now available in... fb.me/1gCPauX6V
