Gareth

A LEAKED GOOGLE DOCUMENT SUGGESTS GOOGLE & OPENAI FEAR THE PACE OF OPEN-SOURCE INNOVATION A leaked internal document from a Google researcher claims that neither Google nor OpenAI is positioned to win the AI race, as open-source AI is rapidly advancing and surpassing their capabilities in areas such as LLMs on a phone, scalable personal AI, responsible release, and multimodality. The document suggests that Google should prioritize enabling third-party integrations, consider where their value add really is, and focus on making small model variants more than an afterthought. The document acknowledges that Google's models still hold a slight edge in quality but suggests that the gap is closing quickly. The document is only the opinion of one Google employee, but the authenticity of the leaked document has been verified. The open source community has recently made significant progress in the development of open language model (LLM) technology, largely due to the introduction of the low-rank adaptation (LoRA) technique, which allows for efficient fine-tuning of models. This has led to a flurry of innovation, with many people contributing ideas and improvements to the technology. The success of LoRA highlights the importance of avoiding the costly process of retraining models from scratch, and instead focusing on iterative improvements to existing models. The document suggests that more attention should be paid to these open source developments, as they have the potential to solve problems that larger organizations are still struggling with. The document argues that focusing on maintaining some of the largest models puts Google at a disadvantage because it is more efficient to iterate faster on small models. Data quality is also more important than data size, and smaller, highly curated datasets are becoming the standard for training. The document suggests that directly competing with open source is a losing proposition and that individuals are not constrained by licenses to the same degree as corporations. The author also believes that owning the ecosystem and establishing itself as a leader in the open source community would benefit Google. This would mean relinquishing some control over its models, but the compromise is inevitable to drive innovation. The leaked document talks about the launch of LLaMA, an open-source AI language model by Meta, the leaks that followed, and the various developments that happened in the field of AI language models within a short period. Though the leaked doucment acknowledges that OpenAI, which has a closed policy, has attracted a steady flow of senior researchers from the open-source community, it states that OpenAI's inability to maintain an edge is in question. The document provides a timeline of events starting from the launch of LLaMA on Feb 24, 2023, to the release of a dataset for Alignment via RLHF by Open Assistant on April 15, 2023. Within this period, several models, such as Alpaca, Vicuna, GPT4All, and Koala, were released, and several techniques, such as fine-tuning, multimodal training, and RLHF, were introduced. The models were trained using different datasets, ranging from freely available data to restricted ones, and at different costs, ranging from a few hundred dollars to training from scratch. The document concludes that the open-source alternatives will eclipse closed models like OpenAI, which will need to change their stance to maintain an edge. IMAGE: GPT-4 grades LLM outputs [Source: lmsys.org/blog/2023-03-30-vi…] @AdrianDittmann @TurmericBae @MissMillion06 @MazeLove14 @nelsonepega @PsyopsRabbit