Open-weights models are closing in on TypeSafe's Jev As @_rchaves_ pointed out on LinkedIn, Jev's lead as a decision model is already under pressure: several open-weight models are close behind and ahead on some tasks. I recreated Rogerio's LangWatch benchmark (11 tasks, same public datasets) on my M5 Pro Mac mini and ran all 15 open models through it. The best was Shisa DE-1, about 6 points behind Jev's published average, and level with or ahead of it on several tasks. The 4B-class models were only a few points further back, running locally at around 120 ms per decision. I then ran Shisa through the 9 use cases from @MatthewBerman's video, against a simple local reproduction of Jev's approach on a 4B model. The results were almost identical (5 ties, 2 wins each), with Shisa at a little over half the speed. Under two weeks after Jev's launch, that's very exciting.

What a crazy week for open-weight LLMs DeepSeek-V4-Pro is a 1.4T model so not something I'm going to be running locally but having the choice, and the ability to literally own it, is where open-weight models win.