Now it's official: GPT-4 is getting worse. 😞 --- TL;DR --- Numerous reports, including my own observations, have highlighted a significant deterioration in the quality of GPT-4's responses. However, these claims were largely anecdotal until now. 😮 A recent comprehensive study conducted by Stanford and Berkeley substantiates the concerns, demonstrating that the June version of GPT-4 performs objectively worse than the March release across various tasks. 📚🔬 --- Evaluation --- The research team assessed the models using a dataset comprising 500 problems that required determining whether a given integer was prime. In March, GPT-4 answered 488 questions correctly. However, in June, it only provided 12 accurate responses. A drastic decline from a 97.6% success rate to a mere 2.4%! 😱❌ But it doesn't end there! 😔 The team employed the Chain-of-Thought technique to facilitate model reasoning: "For example, is 17077 a prime number? Think step by step." 🤔🔢 Chain-of-Thought is a widely used method that significantly enhances answer quality. Unfortunately, the latest version of GPT-4 failed to generate intermediate steps and consistently answered incorrectly with a simple "No." Furthermore, code generation has also suffered. 💻❌ The team compiled a dataset consisting of 50 straightforward problems from LeetCode and measured the number of GPT-4 answers that ran without any modifications. While the March version succeeded in 52% of the problems, the June model exhibited a feeble 10% success rate. --- Explanations? --- Why is this decline occurring? 🤔❓ It is assumed that OpenAI continually introduces changes, but the exact process and evaluation criteria for measuring improvement or regression remain undisclosed. 🔄 Rumors suggest that OpenAI employs multiple smaller and specialized GPT-4 models that collectively function similarly to a large model, yet are more cost-effective to operate. When a user poses a question, the system determines which model to employ for the query. This cost-effective and faster approach raises concerns regarding its impact on the degradation in quality. 💰⏩📉 --- Mitigations --- Large Language Models (LLMs) are increasingly integrated into production systems by enterprises and companies. However, LLMs are not static entities; they continuously learn and adapt. As a result, we must implement monitoring tools that assess both the inputs and outputs of LLM interactions to identify any degradation. This requires a paradigm shift from previous approaches where APIs and Databases were treated as static components. 🔄🔍📈 --- Something Positive? --- GPT-3.5 is getting better! 😄✨ --- Links --- To access the original paper, please follow this link: arxiv.org/pdf/2307.09009.pdf📄🔗 You can reproduce the study's results using this Google Colab notebook: colab.research.google.com/gi… 📒🔗