LLM evaluation is a confusing mess. Standardized eval frameworks have helped, but there's a long way to go. I think we should promote evaluation as a specialized activity carried out by impartial third parties who are not also developers. Of course developers will evaluate their own models and agents, but third-party evaluations should be considered more credible. For example I think HELM results should be much more prominent in model comparisons. (I have no affiliation with HELM.) Journalists reporting on AI releases should prioritize third party evals over the developer's claims when possible. Or they should explicitly note that third party evals are not yet available. Separating testing & evaluation from building is the norm in most engineering fields. It's time for AI to grow up.