Erick Friis

Underrated fact about @MistralAI's new mistral-large model - it's awesome at extracting details from inputs thanks to their new function calling abilities! 📏 tl;dr: see the benchmarks at the end. I ran it against my @langchain 🦜🔗 LangSmith extraction dataset, which uses a model to extract fields and content from real-world emails. mistral-large outperformed all but GPT-4! (Claude 2 technically outscores it, but has a much higher error rate since it lacks a way to reliably specify an output schema via something like function calling) However, this might not even tell the whole story - the evaluator uses GPT-4 to score out of 10 against the reference output, which was itself hand curated from a GPT-4 base. Some of mistral-large's deviations from that reference seemed to be more subjective, like a more concise set of extracted action items from the email vs. more verbose, rather than more "wrong". Try it out for yourself! And congrats to the Mistral team on the launch! Dataset and full rankings: smith.langchain.com/public/3… Repo for running over the dataset: github.com/jacoblee93/oss-mo… Structured output guides via LangChain: Python: python.langchain.com/docs/gu… JS: