Sam Artioli

How many years until open source models take over? Is it never? I've been manually testing 20-30 different models all claiming impressive scores on benchmarks against OpenAI and Anthropic. What I've found: * Super tiny models are becoming insanely good * Medium models are starting to beat GPT-4 with prompt tuning and on specific tasks * GPT-4 still has a huge lead if you only want to use one model * Claude-instant is being slept on * OSS models have a pretty big deployment problem 1. Tiny models will definitely win: StableLM-Zephyr, at 3B (2.9 GB on disk and 50 t/s on most hardware) is punching way above its weight and providing completely working, coherent responses - all the way from writing to SQL generation. For some tasks it's indistinguishable from 3.5-turbo. At this size you solve some of the economies of scale problem by just embedding it in your applications and running it client-side. 2. GPT-4 is not the best every time: Cybertron-v2 (at 13B) did better than GPT-4 for writing and creative tasks - it's subjective but the fact that I thought so is insane. We're looking at a future where specific models are used for specific tasks, and any meaningfully complex AI application will be multi-model. 3. GPT-4 is still the best on average, and there's no real competition: If you're looking for a model to plug into every task and get reasonable output, use GPT-4. Maybe Claude-2, but you'll need more time prompt engineering. 4. Claude-instant is underrated: at 10-15x the speed of GPT, and way cheaper costs, it's performance is pretty close to GPT-4 and well above 3.5 in my tests. Speed and cost make a huge difference - it's becoming my go-to model for ancillary tasks (fixing, labelling, etc). 5. There are no reasons left to use gpt-3.5-turbo. It performed abysmally in my testing, Claude-instant is cheaper, GPT-4 was actually faster, and it's become pretty bad at following multiple instructions. 6. OSS models still have a deployment problem: I've tried nearly every provider (Replicate, Vertex, Modal), and the cost, cold boot, time to first token, and generation speed are all pretty far behind what you can get from the big providers. It's likely that none of them have the economies of scale the big guys do on one or two model flavors. When you can't saturate H200s on a single model, and are forced to serve multiple finetunes or run arbitrary code of off-the-shelf cloud offerings, you likely have huge inefficiencies that may never be surpassable. This really is the big problem for OSS models above 3B-7B. How do you deploy them? Even with custom finetunes, how do you justify the cost when deploying to customers - both in running the model, and in the devops time managing things yourself? When you can plug in and API key and get sub-second time to first token on GPT-4/Claude-instant, why do that? Imagine if you (or your provider) needed to spin up a separate VM for every edge function invocation on Vercel. This doesn't apply to the 3-7B models I think. There are huge privacy, speed and cost implications to having weights and models embedded in local apps (through the work of people like @vatsal_manot), for things like natural-language-to-X, semantic search, conversational interfaces, tool usage, etc. WDYT?