Open source agentic coding harness, optimized for difficulty aware multi-model execution layer workflows. A @morganlinton invention 💡
Joined April 2024
- Tweets608
- Following39
- Followers835
- Likes727
Albatross retweeted
I am going to start to share the leaderboard at VulcanBench more regularly.
VulcanBench is the only benchmark I know that benchmarks across effort levels, on an eval suite of 100% real engineering tasks.
Getting ready to sunset eval suite 3 and eval suite 4 is almost ready to go.
Here's the current leaderboard for Eval Suite 3, Grok 4.5 High is the highest scoring model still with Fable 5 Low, yes Low, right behind it.
What I've been able to uncover with VulcanBench is that way too many people are running models at Max effort, thinking that buys them more accuracy, when really it just costs more and uses more tokens so takes more time.
If you're still in the mode of, tell my agent to do something then go get coffee, you're probably still living in the past. You can move faster, with higher accuracy, the key is not thinking you need Max effort all the time.
Time to share the leaderboard for VulcanBench, Eval Suite 3.
As a reminder, these are 100% real engineering tasks, from real repos. No puzzles, no math problems, no exotic architecture.
Our leaderboard takes the data we have from our full effort sweeps, every effort level, and reports the effort level that gave the highest accuracy.
Right now, so many engineering teams just run a model on High or Max, assuming that's what they need to get the best accuracy.
But for most regular, routine, engineering tasks, the stuff we do day in and day out, that's overkill. And it means the company you work for is paying a LOT more money for tokens, all to get the same accuracy, just a lot slower.
Currently Grok 4.5 High still tops the chart at 90, with Fable 5 Low right behind it at 89 and DeepSeek V4-Flash Max right behind that at 88.
If you're using Grok 4.6, Medium is actually the sweet spot for effort, and Opus 5, which everyone seems to be complaining about these days...well you're probably using it on High, which actually doesn't perform as well as low.
Effort matters.
Live long and benchmark 🖖
Albatross retweeted
If you are benchmarking LLMs and not taking effort level and token use into account, you are only telling a part of the story.
I’m not saying benchmarks that miss this are wrong, but they are incomplete.
If one model w/low effort, is faster, cheaper, and more accurate than another model w/high effort, I would rather use the first one.
A lot of benchmarks just look at models at high or max effort, but have evals with tasks that don’t need this thinking depth.
Not saying I have it perfect, but it is something I’ve been thinking about a lot as I built @VulcanBench
Albatross retweeted
Qwen3.6 397b will not be open source. This pretty much seals it. A real tragedy.
📢 Official Announcement: Qwen Partners with Fireworks AI to Accelerate Access to Qwen Family Models
We are pleased to announce a strategic partnership between Qwen and Fireworks AI to deliver optimized, production-ready deployment of Qwen's closed weights models via the Fireworks Platform. @FireworksAI_HQ
This collaboration empowers developers and enterprises to:
✅ Deploy Qwen models with lower latency and reduced fine tuning and inference costs
✅ Leverage enterprise-grade reliability, security, and scalability
✅ Integrate seamlessly into modern AI workflows
🔹 Get started with Qwen on Fireworks: app.fireworks.ai/models/fire…
#Qwen #FireworksAI #OpenSourceAI #LLM #AIInfrastructure #ResponsibleAI #DeveloperCommunity
Albatross retweeted
Finally decided to dive into the world of fine tuning llms, to hopefully contribute in some way to the local model community that has inspired me so much lately.
My tools for the job are:
- @Dell Alienware Aurora R11 (five years old)
- @nvidia RTX 3080 (also from five years ago)
- @UnslothAI
- and of course, if I get anything I'm not totally embarrassed by, it will go up on @huggingface.
The journey begins.
Albatross retweeted
this was one of the biggest weeks in AI because claude opus 4.6 and gpt-5.3 codex dropped basically at the SAME time.
they solve the same problem in VERY different ways.
- opus spins up agent teams and disappears for a while.
- codex stays with you and ships ridiculously fast.
i tested them both for you in this episode of @startupideaspod (on yt/spot/apple/X) with my pal @morganlinton
we set them up live, tuned them properly, and had each one build a polymarket clone from scratch.
you’ll see where each one shines, where it struggles, and which approach actually fits how you work.
if you’re trying to decide what to use next, this episode will save you a lot of trial and error.
Albatross retweeted
We are hiring a Head of Preparedness. This is a critical role at an important time; models are improving quickly and are now capable of many great things, but they are also starting to present some real challenges. The potential impact of models on mental health was something we saw a preview of in 2025; we are just now seeing models get so good at computer security they are beginning to find critical vulnerabilities.
We have a strong foundation of measuring growing capabilities, but we are entering a world where we need more nuanced understanding and measurement of how those capabilities could be abused, and how we can limit those downsides both in our products and in the world, in a way that lets us all enjoy the tremendous benefits. These questions are hard and there is little precedent; a lot of ideas that sound good have some real edge cases.
If you want to help the world figure out how to enable cybersecurity defenders with cutting edge capabilities while ensuring attackers can't use them for harm, ideally by making all systems more secure, and similarly for how we release biological capabilities and even gain confidence in the safety of running systems that can self-improve, please consider applying.
This will be a stressful job and you'll jump into the deep end pretty much immediately.
openai.com/careers/head-of-p…
Albatross retweeted
Curious to learn how YC startups are using AI? Come meet the teams actually shipping.
We're kicking off a monthly meetup on Dec 10 for founders and builders — quick talks, demos, and plenty of time to hang with the people building the future.
RSVP at events.ycombinator.com/found….
Albatross retweeted
I decided it's time for a Founders Summit here in Tahoe ❄️🏔️ With so many talented founders between here and SF, feels like it's time to get a group of us together at least once a year.
This is going to be more of an unconference than anything else - here's the tentative plan.
- Founders get together for a little snowshoe adventure on Friday afternoon, then all go out to dinner together at a restaurant in Incline Village
- Saturday, wake up and ski Diamond Peak, have lunch together at the top of the mountain, dinner together at my house, or someone's house, but not at a restaurant, so we can have more focused time together and really go deep
- Sunday, wake up and either ski, snowshoe, or just chill by the lake
No sponsors, no talks, no fee, just focused time together with other founders. I want to keep the group small, likely ~10 - 15 people.
If this sounds interesting to you, you can apply at the link in the first comment below.
Albatross retweeted
1 agent → n agents
the interface adapts
(would you actually use this?)
idea:
what if @cursor_ai starts as simple as chatting with one smart Agent, and when you need more power, blooms into a full-blown IDE?
simple up front, limitless when it counts
Albatross retweeted
my weekend project to learn about bluetooth mesh networks, relays and store and forward models, message encryption models, and a few other things.
bitchat: bluetooth mesh chat...IRC vibes.
TestFlight: testflight.apple.com/join/Qw…
GitHub: github.com/jackjackbits/bitc…
