fairseq retweeted
We are releasing a family of dense and MoE language models with up to 13B and 1.1T parameters. We find that MoEs are more efficient, but the gap narrows at scale and varies greatly across domains and tasks.
Paper: arxiv.org/abs/2112.10684
Models & code: github.com/pytorch/fairseq/t…
fairseq retweeted
We’re introducing GSLM, the first language model that breaks free completely of the dependence on text for training. This “textless NLP” approach learns to generate expressive speech using only raw audio recordings as input. Learn more and get the code:
ai.facebook.com/blog/textles…
fairseq retweeted
Facebook AI Research's sequence modeling library @fairseq has made it's twitter debut. Please follow for latest updates.
fairseq retweeted
Fairseq includes support for sequence to sequence learning for speech and audio recognition tasks, faster exploration and prototyping of new research ideas while offering a clear path to production. bit.ly/2WfP85X
fairseq retweeted
roberta = torch.hub.load('pytorch/fairseq', 'roberta.large')
Facebook #AI’s RoBERTa is a new training recipe that improves on BERT, @GoogleAI’s self-supervised method for pretraining #NLP systems. By training longer, on more data, and dropping BERT’s next-sentence prediction, RoBERTa topped the GLUE leaderboard. ai.facebook.com/blog/roberta…
fairseq retweeted
fairseq now supports the training of gated convolutional language models (arxiv.org/abs/1612.08083). It can train a Google Billion Word language model on 128 GPUs in less than a day.
fairseq retweeted
FairSeq Toolkit - Major Update
- Distributed Training
- Transformer models (big Transformer on WMT Eng-German in < 5 hours on DGX-1)
- Fast Inference: translations @ 92 sent/sec for big Transformer
- Story Generation
Read more at Michael Auli's post: facebook.com/photo.php?fbid=…
fairseq retweeted
Fairseq, now in PyTorch!
The open-source convolutional sequence-to-sequence engine from FAIR is now available in... fb.me/1gCPauX6V
