Having lots of fun with local audio models…
Voxtral 2 realtime coming to MLX-Audio thanks @KarnikShreyas and @ochafik
New release this coming week once we land the PR.
I believe we can get to 100-200ms range. Perfect for iOS!
github.com/Blaizzy/mlx-audio…
Olivier Chafik retweeted
You can now run apps within Claude.ai, powered by MCP Apps.
Analyze data, edit tickets, draft messages, generate diagrams and more with Claude's interactive connectors. Grateful to our team, the open source community, and our partners for this launch 🚀
Had so much fun co-authoring the MCP Apps spec and helping launch it in claude.ai! 🎉
🧵 About some demo apps you may wanna check out as user or developer
Pdf (github.com/modelcontextproto… ; also available in claude.ai Connectors) will show any arxiv PDF and allow you to ask Claude about any page / selection you need details about
Say (github.com/modelcontextproto…) uses the amazing Pocket-TTS to let Claude speak out loud to you, live!
Wanna disable thinking in llama.cpp? Try the new `--reasoning-budget 0` flag
github.com/ggml-org/llama.cp…
Should work w/ Qwen3, QwQ, DeepSeek R1 distills, Command R7B; please report any issues!
(Upcoming per-request behaviour discussed on github.com/ggml-org/llama.cp… @ngxson)
#llamacpp
Preview above wasn't for the PR that introduces --reasoning-budget, here it is again:
github.com/ggml-org/llama.cp…
Really slick integration! MCP all the (cool) things!
You really can just do things! Use *any* Hugging Face space as a MCP server along with your Local Models! 🔥
Here in we use Qwen 3 30B A3B with @ggml_org llama.cpp and @huggingface tiny agents to create images via FLUX powered by ZeroGPU ⚡
It's quite a bit crazy to see local models be capable of so much and just be able to understand/ infer from tool description!
There's a lot of potential here in automating video generation workflows, content curation and a lot more..
Bonus: you can plug any other Inference Provider if you don't want to run locally too!
npx @ huggingface/tiny-agents run [TASK]
oh, and we provide both typescript and python client! 🐐
Olivier Chafik retweeted
Months of @ochafik's work finally paid off, we have MCP w/ llama.cpp 😁😁
llama.cpp streaming support for tool calling & thoughts was just merged: please test & report any issues 😅
github.com/ggml-org/llama.cp…
#llamacpp
Olivier Chafik retweeted
Docker is embracing the ggml/llama.cpp on-device future
Who is going to be next?
WIP: streaming support for llama.cpp's tool calling + normalization of streamed <think> tags (for QwQ, DeepSeek R1 & Cohere Command R7B)
Still strongly conforming to tool schemas (grammar-constrained).
Testing / feedback welcome :-)
github.com/ggml-org/llama.cp…
Olivier Chafik retweeted
Day-zero Gemma 3 support in llama.cpp 🤯
👉 4 model sizes: 1B, 4B, 12B, 27B
👉 Vision capability (except for 1B) with bi-direction attention
👉 Context size: 32k (1B) and 128k (4B, 12B, 27B)
👉 +140 languages support (except for 1B)
👉 Day-zero support on many frameworks 🚀
Loving this... hope OSS catches up quickly 🤗
Also, conforming to JSON schemas in a diffusion context is going to be an interesting challenge (schema / grammar compliance gradients anyone?)
Olivier Chafik retweeted
You're seeing a minor miracle: Phi-4 3.8B doing agentic tool calls.
Doing that reliably, locally, even with 8B, has been a fools errand until @ochafik's opus of a patch series to llama.cpp (PR coming for phi-4)