Making coding agents better at inference optimization and hardware acceleration for edge devices like Nvidia Orin/Thor. Backed by Y Combinator
Joined June 2024
- Tweets12
- Following2
- Followers33
- Likes6
RunLocal AI retweeted
"Edge inference is not optional" - We hear this from everyone deploying truly autonomous robots in the wild (especially at larger scale)
After this, we became convinced that edge inference is not optional for home robots.
We realized that this problem only became visible once three things were true at the same time:
1. Memo was fully autonomous.
2. We were evaluating it in real, unseen homes.
3. The policy was reliable enough that network jitter became a dominant failure mode.
Before that, you do not even get to work on the real blockers to mainstream market adoption.
RunLocal AI retweeted
🔥 VLMs on mobile devices with world-facing cameras key for proactive, intelligent computing. Local/on-device inference key for real-time, private experiences.
Great to see an emphasis on smaller VLMs. Excited to see where @huggingface, @moondreamai, etc. take things 🚀
Holy shit! Did we just open-source the smallest video-LM in the world? SmolVLM2 is runnning natively on your iPhone 🚀 huggingface.co/blog/smolvlm2
RunLocal AI retweeted
Snap's announcement about their on-device text-to-image model seems to have slipped under the radar…
Apparently, it generates 1024x1024 images with quality that's comparable to cloud-oriented models like Stable Diffusion XL.
But it can do that locally on an iPhone 16 Pro Max in <1.5 seconds! 🤯
Snap are planning to ship it soon to their ~450m daily active users, and I wouldn't be surprised if it's free.
I wonder how all these subscription-driven, cloud-based image generation apps will respond…
Announcement: newsroom.snap.com/ai-text-to…
Paper: arxiv.org/abs/2412.09619
RunLocal AI retweeted
Short but sweet talk about the WebNN API: youtube.com/watch?v=FoYBWzXC…
Def worth checking out the YouTube playlist from @jason_mayes WebAI Summit last year. It's packed with great talks!
Looking forward to the next summit!
RunLocal AI retweeted
Awesome work from @soldni and team at @allen_ai!
If you're interested in shipping on-device AI language features in your app, I highly recommend checking out their demo app to get a sense of what's possible these days on an iPhone: apps.apple.com/us/app/ai2-ol…
We took our most efficient model and made an open-source iOS app📱but why?
As phones get faster, more AI will happen on device. With OLMoE, researchers, developers, and users can get a feel for this future: fully private LLMs, available anytime.
Learn more from @soldni👇
RunLocal AI retweeted
creativestrategies.com/resea…
Stable Diffusion power consumption between M3 and X Elite:
"M3 MacBook Air, 8-core CPU 10-core GPU with 16GB RAM spec, we see averages of 87.63 Joules used per image generated.
On the Snapdragon X Elite system, we used a prototype Surface Laptop 15-inch with 16GB RAM, with the X1E78100 SKU of Snapdragon X Elite. We see averages of 41.23 Joules used per image generated"
RunLocal AI retweeted
Bitesize Benchmark (iPhone 15 Pro Max)
WhisperKit (v0.6) vs Whisper.cpp (v1.6):
- Accuracy: Both perfect for this short clip
- Speed: WhisperKit at 45 w/s. Whisper.cpp at 30 w/s. (average over 3 runs)
- Peak RAM (note: physical footprint only): WhisperKit at 180Mb, Whisper.cpp at 680Mb
RunLocal AI retweeted
Always benchmark your on-device models!
Tried to run whisper-tiny on MacOS, using computeUnits.all in CoreML.
Produced worse inference/load times than only using ANE, which is surprising because you'd expect CoreML to pick the optimal config.
Why CoreML did this 👇
RunLocal AI retweeted
Just converted @bria_ai_’s awesome background removal model to CoreML for inference on iOS/MacOS.
There's a 3x inference speedup when using the NPU (ANE) vs CPU/GPU on an iPhone 15 Pro Max.
Shout out to @yairadato and team for their work (HF repo: briaai/RMBG-1.4).