@AlphaSignalAI
Account based inUnited States
About this account
- Account based in
- United States
- Connected via
- Web
Account-level information from X, not a live location or the device used for a specific post.
We help you track, rank, and understand the entire AI industry in real time. Used by 300,000+ developers. 5-min daily AI digest: https://nitter.cf/t.co/ZGatS9i5eD
Build your feed
Joined February 2010
- Tweets1.4K
- Following348
- Followers16.7K
- Likes2.9K
Pinned Tweet
Kimi K3 is getting called Fable/Sol level, and it's 7th in our tests.
Arena Frontend Code: #1 at 1679 points.
Artificial Analysis: #3 at Intelligence Index of 57.
We ran it the next day on our coding-agent repair harness against GPT-5.6 Sol, Fable 5, Grok 4.5, Opus 4.8, GLM-5.2, and Gemini 3.1 Pro.
Results:
> Last of 7 models
> 53 of 67 attempts (79%)
> $0.186 per successful fix
> 702s average wall time
Sol hit 100% (70/70) on the same suite. Grok sat at 99% and 46s.
So why does the internet sound so sure K3 is crushing coding agents, if our tests have it at the bottom?
-----
> Full write-up: alphasignal.ai/news/arena-1-…
> 5-min daily signals: alphasignal.ai/newsletter
Kimi K3 scores 57 on the Artificial Analysis Intelligence Index. Its intelligence is comparable to Opus 4.8 and GPT-5.5 but remains behind Fable 5 and GPT-5.6 Sol. Moonshot AI has expressed plans to release the 2.8T parameter model's weights, which would make it the leading open weights model
Key results:
➤ Strong agentic task performance: @Kimi_Moonshot's Kimi K3 reaches an Elo rating of 1668 on GDPval v2. This is a marked improvement over K2.6’s 1190, surpassing GLM-5.2 (1514), GPT-5.5 (1494), and Claude Opus 4.8 (1600). However, it still lags behind Claude Fable 5 (1760). Kimi K3 also scores an impressive 53% and takes the #1 position on AutomationBench-AA, our implementation of Zapier’s Agentic SaaS workflow evaluation.
➤ Second-highest performance on AA-Briefcase (agentic knowledge work): On our private long-horizon knowledge work evaluation, Kimi K3 reaches an overall Elo of 1547, +732 points from Kimi K2.6 and behind only Claude Fable 5. It is well-rounded: its rubric scoring and analytical quality almost reach Claude Fable 5’s scores, while GPT-5.6 Sol continues to outperform other leading models on presentation quality.
➤ Set to lead open weights models once weights are released: Moonshot AI has not yet released the weights but expressed plans to do so. Once available, Kimi K3 would clearly lead other open weights models including GLM-5.2 (51) and DeepSeek v4 Pro (44). However, at 2.8T parameters, it is significantly larger than its open weights peers (eg. GLM-5.2 at 753B params and DeepSeek V4 Pro at 1.6T), as well as the Kimi K2 to K2.6 models (1T params).
➤ Cost per task ($0.94) is similar to GPT-5.6 Sol ($1.04), ~1/2 the price of Opus 4.8 ($1.80) and higher than open weights peers: Moonshot AI’s pricing for K3 is significantly higher than their K2 pricing (K3’s output token price is $15/1M tokens while K2.6 was $4). This positions the model as cheaper on a cost per task basis than Opus 4.8, similar to GPT-5.6 Sol ($1.04) and more expensive than open weights peers, GLM-5.2 ($0.32) and DeepSeek V4 Pro ($0.04)
➤ Improved token efficiency alongside higher intelligence: Kimi K3’s token usage on the Artificial Analysis Intelligence Index decreased significantly, using 21% fewer output tokens than K2.6. The new model used approximately 132M output tokens to complete all nine evaluations, compared to approximately 166M for K2.6, while achieving higher scores.
➤ Native multimodal capabilities: Kimi K3, like K2.6, is released with native image and text multimodal input. If weights are released, this will position Kimi K3 as one of the leading open weights models with multimodal input capabilities
Other model details:
Context window: 1M
Size: 2.8T total parameters
Pricing: The first-party API is priced at $3.00/$15.00 per 1M input/output tokens, with cached input discounted 90% to $0.30 per 1M tokens.
Modality: Native multimodal input supports text and images, and the model remains text-only for output.
Accessibility: Accessible at launch through Moonshot’s first party API. Model weights are not yet released but Moonshot AI has expressed plans to do so.
A support ticket comes back as billing, sales, or technical. The return is that label, with a probability on it.
You pass the state, the questions, and the allowed answers. That exchange is one step.
TypeSafe's Jev does that in 70 to 500 ms. Input is $0.042 per million tokens, and output tokens are not metered.
> 40x to 200x faster on the comparison table, for System One shaped queries
> 444.6x cheaper only as the workflow-eval high end
A negative balance stays in code. The model is for the call you cannot separate with a rule.
> Route the ticket
> Score the urgency
> Pick the next tool
The vocabulary is the list you typed. A fraud ticket that only offers billing, sales, and technical comes back as one of those three. A spam score of 0.52 is not a 0.998
> Laya, 421 million parameters, about 33 ms for one question on a Tesla T4
> That 33 ms is a local pass, not the hosted 70 to 500 ms
> CLM, up to 9x faster zero-shot when actions repeat or the list is long
> Fine-tuned CLM, 81.6% of 38 DeepSWE tasks and 87.6% of 30 Terminal-Bench 2.1 tasks
> Jev on that judge test, 71.1% and 83.1%, under the one-sample line
Which real ticket has no label in your list?
Full breakdown↓↓
Your audio can score well on DNSMOS and still hurt your AI model.
That’s the problem we’re digging into with @Voices at our upcoming virtual workshop.
We’ll look at the audio quality issues traditional scoring can miss, why reverb matters for TTS, voice cloning & ASR, and how to tell when audio is actually training-ready.
Plus, you’ll hear the difference live.
Join us → luma.com/6w9brzb3
People keep lining Qwen3.8-27B up with Opus 4.6, the coding frontier from a few months ago.
So, can that class of model sit on a consumer Mac?
Yes, but you still need to build the stack around it:
> an engine that quotes peak RAM
> a harness with a stop
> and tests you own
If you already use Claude Code or Codex, this is a private loop beside that bill.
Qwen3.8-27B fits a Mac as a 17.6 GB file. But MTPLX still peaks at 23.6 GB once the draft-ahead head is on.
A speed taken on a 128 GB Mac with thinking turned off is a different machine, so leave it out of a 24 GB average.
Two ways this goes wrong:
> You stop at the download, and 24 GB of RAM only loads it tight. A 4-bit copy there is still perfect for most tasks. 32 GB is the first real agent. 48 GB can hold a /goal that a machine can check. 16 GB is not this model.
> Or you stop at the first hour. Ollama is a fine look, but after that you still pick an engine, a harness, and a stop.
Local tokens have no invoice, so a runaway does not get expensive, and you still own the merge. Tests are the stop.
We put the pulls, the install, the skill files, and the git-config check on an interactive HTML tutorial.
Which file to pick if you already live in MLX, and what to refuse to train this week, are a longer argument.
Full breakdown below
Full breakdown: alphasignal.ai/news/how-to-r…
Your AI agents are generating a ton of operational data.
Prompts. Tool calls. Latency. Tokens. Errors. Retries. Cost. Outcomes.
The hard part is turning all of that into something you can actually monitor.
In our upcoming workshop with @TigerDatabase, we’ll show how to build an agent monitoring system on Postgres + time-series data, then scale it for teams running hundreds or thousands of agents.
If AI agents are moving into production, this is the infrastructure layer you’ll need.
Join us → luma.com/j1s2bxf9
METR’s vibe-coded dashboard exposed an agent through an authentication flaw.
An attacker prompted it to reveal its API key.
Over three weeks, the attacker consumed roughly $600K in credits. The provider supplied them free; METR reported no financial loss.
A successful login doesn’t test what happens without one.
Even an HTTP 401 can hide an unauthorized action. If the backend queues work before authentication, a worker can still execute it.
Check the response, queue, and task database together.
Then trace which credentials the worker can read. A separate inference service can hold the provider key and enforce access and usage limits.
Before you deploy:
> Test that rejected requests create no jobs
> Check access across different user accounts
> Keep provider keys outside code-execution workers
> Restrict worker permissions
> Test credential revocation and recovery
The article applies these checks to an illustrative agent dashboard.
Full breakdown ↓↓alphasignal.ai/news/how-to-r…
Grok 4.7 just landed on @mercor's APEX leaderboards:
> #5 on APEX-SWE at 53.6%
> #13 on APEX-Agents at 54.6%
The notable part is the bracket, which sits in the same cost and latency tier as Gemini 3.7 Flash and GPT-5.6 Luna while leading on SWE.
APEX updates with every major model release, so you can see how the newest models stack up, compare token costs, and explore results from third-party benchmarks like BrowseComp and Terminal-Bench.
See the full leaderboard at mercor.com/apex/
*partnerpost
Grok 4.7 is now live on the APEX leaderboards.
APEX-SWE: 53.6% Pass@1 (#5)
APEX-Agents: 54.6% Pass@1 (#13)
Compared to other models with similar cost and latency profiles, it’s a strong model for agentic coding tasks.
Congrats to the @SpaceXAI team.
A good DNSMOS score doesn’t always mean your audio is ready for AI training.
Join AlphaSignal + @voices for a technical deep dive into what DNSMOS misses, including the “grey zone” where reverb can quietly degrade TTS, voice cloning & ASR models.
We’ll cover:
→ Where DNSMOS falls short
→ How to spot problematic audio
→ What training-ready audio actually sounds like
→ Live audio comparisons
Register → luma.com/6w9brzb3
Same model bills 5x more in a different harness
HarnessTax tested 21 model-harness combos across 60 tasks and cost moved far more than success did
> $1.33 against $0.67 per attempt
> 97.8% versus 96.7% success rate
> 9 of 12 beat vendor default
Claude Code and Pi averaged 15 turns, yet the heavy setup carried more than 10 times the initial context because longer instructions and larger tool definitions ride along on every call.
Study co-author @melissapan told us many harness choices run on tribal knowledge and word of mouth, so it is very unclear which setup wins on accuracy or cost for your workload.
We read this as a buying test and the lean setup still holds the cost-success frontier on four tools, read, write, edit and bash.
> When does the heavier harness earn its higher price?
> Which failures hide when two setups post the same score?
> What would your own tasks show if you priced model and harness as one system?
Answers below ↓↓
Does your Claude model really need Claude Code…? 🤔
We evaluate 7 models on Claude Code, Codex, and Pi. Three surprising findings emerge:
1️⃣Harness choice has little effect on task success rate, but can significantly affect the cost
2️⃣A simple harness can be competitive
3️⃣The native harness isn’t always the best.
Millions of people are using coding agents, but the impact of harness choice remains unclear.
(1/n) More details in the thread. 🧵
Full breakdown by @bendee983: alphasignal.ai/news/understa…
Taste answer: the lean 4-tool setup held the frontier on both benchmarks, so start from the lean setup and add weight only when a failure or a needed feature earns it.
The full check, the 9-of-12 vendor-default split, and when to pay more stay in the full breakdown.
not if you're subscribed to @AlphaSignalAI
One AI agent is easy to monitor. 1,000 agents running across your org? Different problem.
Join AlphaSignal + @TigerDatabase for a live technical workshop on building enterprise-grade AI agent monitoring.
We’ll cover:
→ Tracking tool calls, latency, errors & cost
→ Modeling agent activity as time-series data
→ Real-time dashboards for live agent activity
→ Finding trends & failure modes
→ What changes when you scale from 1 agent to 1,000+
If agents are becoming part of your infrastructure, you need to know what they’re actually doing.
Register → luma.com/j1s2bxf9
AlphaSignal retweeted
Snap just entered the agentic race.
They just introduced SPECS Intelligence, a copilot that spans Mac, iPhone, and AR glasses.
The hard part isn’t connecting email, calendar, and notes.
It’s turning that firehose into a continuously updated model of your life.
The system has to:
• resolve identities across apps
• map events to the right project, trip, or relationship
• track what changed and what’s stale
• isolate work and personal context
• rank what matters now
• decide when to interrupt
• sync state across devices
• gate actions behind approval
Snap’s answer is a structured context layer built around three primitives:
> Portrait: who you are, who matters, and your patterns.
> Corners: scoped contexts for work, family, travel, health, and relationships.
> Goals: persistent objectives that give individual events longer-term meaning.
Instead of treating memory as one giant retrieval pool, @specs can organize context by domain, update it as new events arrive, and surface the relevant slice when needed.
And Snap has one unusual advantage, they own a device in your field of view.
If agent quality depends on context, glasses could eventually give Snap a source of physical-world context that software-only agents don’t have.
DeepSeek's new bigger flash model but uses 4x less cache
It is almost twice as large as its predecessor, and KV storage fell from 3,514 bytes per token to 890
While the model grew, long agent sessions got cheaper to hold.
Active parameters, cache hits, and the scoreboard all moved too:
> 8B active on input
> 16B active on output
> Cache hits $0.006 per million
> Scores 40 versus Gemini 41
Then a million-token session holds under a gigabyte of global KV, so the predecessor needed about 3.5 GB for the same window.
Also, they no longer persist local window state to disk, and persistent SSD cache falls to about one-eighth.
We care about how they split reading from writing, because that is what lets input tokens skip half the stack.
So, how does a bigger model use less memory? What has to stay in GPU memory versus disk? And why does parameter count now say less about your bill?
Answers below ↓↓
How does a bigger model use less memory?
Prefill only runs the first 20 layers, so each input token activates 8B parameters instead of the 16B decode path.
alphasignal.ai/news/what-dee…
Google's new paper proved your skill file matters least in training
Hand-writing agent instructions is a losing game. Files rot the moment tasks shift.
Skill evolution rewrites them from rollout evidence instead. But most loops forget every failure.
Their WikiSkill setup keeps a private notebook behind the skills. So the coach learns from history while the actor stays blind.
> Blind the rollout actor
> Log every rejected edit
> Quarantine every transferred skill
The notebook lifted accuracy 48.7 to 63.7 Then peeking dropped it back to 60.9
And transferred skills beat self-made ones 70.2 to 63.4, Yet one crashed a stronger model 50.5 to 18.1
Small models can discover skills that bigger ones execute. EvoSkill's repo is public under Apache 2.0
So gate every skill like a model release
Full breakdown ↓↓
The appendix builds the loop in an hour
The full article, with the runnable scaffold and the transfer gate, lives here: alphasignal.ai/news/self-imp…