building fast local inference @usemagnitude
San Francisco
Joined April 2014
- Tweets813
- Following220
- Followers493
- Likes2.5K
Anders Lie retweeted
Magnitude has hit #1 on HN! Lots of great discussion from the community so far
TLDR is that Magnitude runs open models as fast as possible on your hardware. Up to 2x faster than llama.cpp in our benchmarking
It works on Mac, Linux, and Windows, on any hardware. Macbooks, NVIDIA and AMD GPUs, DGX Sparks, Strix Halos, or just a laptop CPU
Our kernels are tuned on your actual device for maximum performance. And it's built for agents, where sessions are long, several run at once, and you still want to use your computer for other things
Feel free to read more and join the discussion on HN :)
Anders Lie retweeted
The easiest way to find out which models you can run on your computer:
Magnitude is a 100% free, open-source desktop app that profiles your machine and ranks the models across:
- Speed
- Accuracy
- Intelligence
- Memory required
Then it downloads the one you pick and tunes it for your machine.
Finally, connect your favorite harness (Pi, OpenCode, Claude Code, Codex, etc.) in one click.
Get started here: github.com/magnitudedev/magn…
(don't forget to star 🌟)
My co-founder wrote a detailed article on running your favorite harnesses with local models.
The article is quoted below.
Anders Lie retweeted
The easiest way to run your agent harnesses using local models:
Whether you're using Claude Code, Codex, OpenCode, or Pi, the local setup itself isn't really the hard part.
The harder problem is figuring out which local model your machine can actually handle well.
You have to think about RAM, model size, quantization, context length, KV cache, speed, accuracy, and a bunch of other trade-offs before you even start.
Magnitude is an open-source solution that removes all of that guesswork for you.
It profiles your machine, benchmarks what it can realistically run, recommends the best models for your hardware, and then lets you connect them to your preferred agent harness.
The entire setup takes just two commands.
In this video, I walk through the full process from hardware profiling to running a harness on a local model.
Chapters:
00:00 Intro
00:17 What Magnitude is
00:45 Setup in two commands
01:29 Why picking a local model gets confusing fast
02:11 Hardware profiling + model recommendations
02:59 Choosing your harness
03:22 Connecting a harness that's already running
04:18 Outro
Get started: github.com/magnitudedev/magn…
(don't forget to star 🌟)
Anders Lie retweeted
Doing gods work.
Which models can your machine actually run?
Magnitude now answers that. The new model catalog:
- Profiles your hardware automatically
- Estimates tok/s for every model before you download
- Recommends the best models for your machine
Pick one and Magnitude handles the rest:
- Downloads the model and quant from Hugging Face
- Loads it into the built-in inference engine
- Configures speculative decoding (MTP, DFlash, etc.)
- Sets concurrency based on your memory
Try it on your hardware:
npm i -g @magnitudedev/cli
Anders Lie retweeted
Interesting to see what kinds of local models can run on my maxxed out Mac Studio Ultra
Which models can your machine actually run?
Magnitude now answers that. The new model catalog:
- Profiles your hardware automatically
- Estimates tok/s for every model before you download
- Recommends the best models for your machine
Pick one and Magnitude handles the rest:
- Downloads the model and quant from Hugging Face
- Loads it into the built-in inference engine
- Configures speculative decoding (MTP, DFlash, etc.)
- Sets concurrency based on your memory
Try it on your hardware:
npm i -g @magnitudedev/cli
Anders Lie retweeted
I feel that we're on the cusp of this type of thing really taking off.
The open source models that can actually run on local hardware are ALMOST good enough for most dev tasks (for those who are still heavily in the loop with their agents).
Introducing Magnitude: your actually local agent
100% private and offline. No token costs, no API keys. Open source.
Today's agents are local. The model isn't. Every prompt, every file, every secret gets sent straight to Anthropic and OpenAI.
Magnitude is built around local models and runs the whole stack itself. The inference engine is part of the agent, so the models run inside it, right on your computer.
It lives in your terminal, and setup is one command. Magnitude profiles your hardware and shows you which models fit, with the trade-offs between quality, speed, and memory. Pick one and start working. No painful config or server to babysit.
Out of the box, it can use your shell, edit files, and run scripts. Add skills and it can work with Excel, PowerPoint, PDFs or Chrome.
Use it for everyday work:
- Analyze sensitive data
- Manage private notes
- Review code and logs
- Search and organize files
- Build docs or slides
npm i -g @magnitudedev/cli
GitHub: github.com/magnitudedev/magn…
Anders Lie retweeted
My good friends are making the ultimate harness for local models that just works out of the box. They built their own inference engine in rust, profile hardware, and recommend best fit models.
If you want to get the most out of open source models, highly recommend following Tom
Introducing Magnitude: your actually local agent
100% private and offline. No token costs, no API keys. Open source.
Today's agents are local. The model isn't. Every prompt, every file, every secret gets sent straight to Anthropic and OpenAI.
Magnitude is built around local models and runs the whole stack itself. The inference engine is part of the agent, so the models run inside it, right on your computer.
It lives in your terminal, and setup is one command. Magnitude profiles your hardware and shows you which models fit, with the trade-offs between quality, speed, and memory. Pick one and start working. No painful config or server to babysit.
Out of the box, it can use your shell, edit files, and run scripts. Add skills and it can work with Excel, PowerPoint, PDFs or Chrome.
Use it for everyday work:
- Analyze sensitive data
- Manage private notes
- Review code and logs
- Search and organize files
- Build docs or slides
npm i -g @magnitudedev/cli
GitHub: github.com/magnitudedev/magn…
Anders Lie retweeted
"Local agent" has been one of the more glossed over phrases in AI. Your harness may run locally. but usually the model doesn't. shoutout to @usemagnitude for shipping the full stack on your machine
Introducing Magnitude: your actually local agent
100% private and offline. No token costs, no API keys. Open source.
Today's agents are local. The model isn't. Every prompt, every file, every secret gets sent straight to Anthropic and OpenAI.
Magnitude is built around local models and runs the whole stack itself. The inference engine is part of the agent, so the models run inside it, right on your computer.
It lives in your terminal, and setup is one command. Magnitude profiles your hardware and shows you which models fit, with the trade-offs between quality, speed, and memory. Pick one and start working. No painful config or server to babysit.
Out of the box, it can use your shell, edit files, and run scripts. Add skills and it can work with Excel, PowerPoint, PDFs or Chrome.
Use it for everyday work:
- Analyze sensitive data
- Manage private notes
- Review code and logs
- Search and organize files
- Build docs or slides
npm i -g @magnitudedev/cli
GitHub: github.com/magnitudedev/magn…
Anders Lie retweeted
dope work from @tomgreenwald @anderslie - fully locally offline coding agent on your mac
Introducing Magnitude: your actually local agent
100% private and offline. No token costs, no API keys. Open source.
Today's agents are local. The model isn't. Every prompt, every file, every secret gets sent straight to Anthropic and OpenAI.
Magnitude is built around local models and runs the whole stack itself. The inference engine is part of the agent, so the models run inside it, right on your computer.
It lives in your terminal, and setup is one command. Magnitude profiles your hardware and shows you which models fit, with the trade-offs between quality, speed, and memory. Pick one and start working. No painful config or server to babysit.
Out of the box, it can use your shell, edit files, and run scripts. Add skills and it can work with Excel, PowerPoint, PDFs or Chrome.
Use it for everyday work:
- Analyze sensitive data
- Manage private notes
- Review code and logs
- Search and organize files
- Build docs or slides
npm i -g @magnitudedev/cli
GitHub: github.com/magnitudedev/magn…
Anders Lie retweeted
Magnitude scores 75.5% on Terminal-Bench 2.1, making it the top coding agent for GLM 5.2.
npm i -g @magnitudedev/cli
See how we did it (and kept it fair) below ⬇️
Anders Lie retweeted
All these GLM 5.2 providers advertise 200+ tok/s. Yet you try them and get 50 tok/s
The top speeds are super impressive. And I love how much activity there is in the ecosystem. But it makes it really hard to get a true read of provider inference speeds under normal conditions
My guess is they allocate a ton of compute to their serverless model APIs right around launch, then slowly pull it back once their score has been established. It's benchmaxxing but for providers
You can see it in the Artificial Analysis chart of speeds over time. They peak right on launch day, then gradually or sometimes rapidly decline over the next few days
I'm very excited for when one of these reported 200+ tok/s scores holds for 7+ days
You may have heard that GLM-5.2 at 328 token/s is cool,
How about 392?
Databricks is now #1 in inference speed for GLM-5.2 on Artificial Analysis. It's a great model, and we did a lot of optimizations.
Anders Lie retweeted
anthropic inference is very costly vs performance
Introducing Magnitude.
It's a coding agent that runs entirely on open models.
It costs 60% less than Claude Code with no drop in performance.
Try it now: npm i -g @magnitudedev/cli
Here's how it works 👇
Anders Lie retweeted
clever!
Introducing Magnitude.
It's a coding agent that runs entirely on open models.
It costs 60% less than Claude Code with no drop in performance.
Try it now: npm i -g @magnitudedev/cli
Here's how it works 👇
Anders Lie retweeted
it turns out if you do a lot of really good context engineering, you can get frontier intelligence for like half the price, and in some cases even better than a single frontier model
Introducing Magnitude.
It's a coding agent that runs entirely on open models.
It costs 60% less than Claude Code with no drop in performance.
Try it now: npm i -g @magnitudedev/cli
Here's how it works 👇
Anders Lie retweeted
Excited for @wafer_ai to partner with @usemagnitude to power their coding agent with powerful open source models!
Open source models are the closest to frontier LLMs they've ever been, so it's now feasible to reduce costs by 60% while maintaining quality across the board.
Introducing Magnitude.
It's a coding agent that runs entirely on open models.
It costs 60% less than Claude Code with no drop in performance.
Try it now: npm i -g @magnitudedev/cli
Here's how it works 👇
Anders Lie retweeted
My friends @tomgreenwald & @anderslie made a harness designed for open source models.
If you’re working with OSS models, don’t use anything but magnitude. They squeezed every last drop of performance out of their harness.
Introducing Magnitude.
It's a coding agent that runs entirely on open models.
It costs 60% less than Claude Code with no drop in performance.
Try it now: npm i -g @magnitudedev/cli
Here's how it works 👇
Anders Lie retweeted
🦄 ai that works: Event driven agents with @anderslie xitter.cf/i/broadcasts/1lDxLBVRE…
Anders Lie retweeted
"OpenAI just killed so many startups", "n8n is dead", "95% of YC cos are now irrelevant". Lot of hot takes around the OpenAI agent builder. But they're mostly stupid and wrong.
People are overlooking 2 major things:
1. Agents vs. automations
An agent is given a high-level task and decides what tools to call, in what order, and with what data. This is fundamentally nondeterministic. You have zero guarantee on how your data gets used. This is still a no-go for real production data. The models aren't good enough for people to trust them with meaningful use cases, even with "guardrails".
An automation is a deterministic workflow that takes the same actions, across the same apps, every time. This has been the gold standard (think Zapier). Now people are sprinkling in AI at specific points to make workflows more dynamic. Usually to summarize or analyze data, sometimes to make small decisions ("is this email a bug report?").
Using AI in traditional automations still has massive opportunity. People trust these with production data because they control what data is used, when, and how much they delegate to AI.
People incorrectly group these together. Yes, as you add more AI decision points, a workflow starts resembling an "agent". But the way to get there is gradual. Layer in more AI over time, don't jump to a completely probabilistic agent immediately.
2. Building for developers vs. non-technical users
This is an even bigger difference. Building for developers could not be more different than building for non-technical users. Implying agent builder could be used by non-technical users is technical malpractice.
Building for non-technical users means abstracting away ALL complexity. They shouldn't need to know what an API key, webhook, or JSON is. Agent builder is extremely far from that. Even with a chat interface to help construct the agent, too much needs abstracting for non-technical users.
It makes zero sense to compare agent builder to platforms targeting non-technical users. The two don't even converge (like agents and automations eventually do). You can't serve both audiences in the same platform. Developers need control, flexibility, transparency. Non-technical users need convenience, full-service, ease of use. Building for both is basically impossible (and dumb).
There's so much room for startups to build differentiated experiences for non-technical users (like Sagekit 😉)
Overall: There's so much FUD surrounding agent builder. I get it, fear plays well in the algorithm. But posting hot takes just for the sake of it makes you seem dumb. Which I know many of these people aren't.
There's still so much to build!
Anders Lie retweeted
had lunch with @anderslie & his cofounder today (they're building sagekit, a chat-based automation platform)
gathered that they've evolved through three stages in their engineering practices:
1. writing code manually
2. using AI to write code while staying in the loop
3. not even spending time in the loop anymore - just writing markdown files for agents
they're spending most their time now building automation infrastructure. this means writing markdown files that codify patterns in their codebase and workflows they want to repeat
so when they need to build a new integration that follows an existing pattern, they use a /slash-command to call a specific workflow to build it. then another /test-scenario workflow with a new agent that comes up with scenarios for that integration
what i'm talking about here is seriously scratching the surface of what they're up to, but the point is they've abstracted from manual coding to AI coding to only writing natural language files for agents
this is the tip of the spear, and I've been experimenting with this in @cline for my own needs (largely around repetitive internal reporting & things like docs)
here's what you should be doing:
at the end of a task which can be repeated, ask cline (or whatever agent you use) to build a workflow for you and codify what was once a manual task to make it an automation and save your future self the agent hand-holding
turn your coding agent into your intelligent taskmaster
Anders Lie retweeted
This is probably one of the best AI tools recently launched! Sidekick - Lovable for automations.
Watch how @tomgreenwald builds a full n8n-style workflow just by chatting with the AI agent using
@joinsidekickhq.
0:00 – Intro: Meet Sidekick, the chat-first automation
1:34 Sidekick's Unique Features
2:26 Experience with Y Combinator
3:02 Use Cases and Demonstrations
11:14 Building a Workflow from Scratch using Chat
18:39 Future Plans and Connectors
21:21 Scheduling Workflows and Final Thoughts