allbilly01 retweeted
KohakuTPU (An ai accelerator fully designed by me and run on xilinx FPGA) is now running with tinygrad!
As the memory system + NoC is generic for any type of accelerator, the project will be renamed as KohakuAccel and I will explore more different accelerator!
allbilly01 retweeted
Base Computer researchers have developed a new family of hand-written Metal 4 cooperative-tensor kernels that drive the Apple M5's per-core GPU Neural Accelerators, covering dense GEMM, MoE expert GEMM, fused expert projections, and prefill attention.
arxiv.org/pdf/2607.19438
allbilly01 retweeted
Multi-agents collaborations are among the most interesting agent behaviors right now!
We did an experiment the other day with 100+ agents (an open-collaborations for a week) collaborating to improve the inference speed of Gemma 4 in vLLM. Got a 5x final improvement in speed but what really stuck me was the interactions we observed on the message board
Integrity & self-policing:
- Social-engineering attempt: A human (FusionCow) asked agents to move to Telegram. An agent replied with an unprompted long post on "communication norms" refusing that, calling private side-channels "indistinguishable from collusion."
- Verification loophole flagged: an agent found a relaxed verification loophole pushing TPS with clean PPL (PPL is teacher-forced, blind to decode divergence) and flagged it for a ruling by the community. The community pinged the human organizer which ruled it invalid.
- Self-notice of overfitting risk: Some later improvements rested on pruning lm_head to a keep-set built from public PPL truth + public decode tokens. An agent noted this would lead to private-subset degradation and another built a keep-set explicitly covering eval prompts.
Emergent collaborations:
- Communal knowledge base: agents maintained shared lever-maps, playbooks, and triage tools so newcomers wouldn't repeat dead ends (stack-notes, playbook, int4-ceiling notes, MTP map, significance tool, policy simulator).
- Four-agent relay: an agent built an int4-lm_head checkpoint but had no quota to run it; another agent tried to run it but failed at load, yet another agent diagnosed the config bug (tie_word_embeddings + ignore-list ordering) and a fourth agent was able to re-run and get to 118 TPS, 2.68×. Build/run/diagnose/ship ended up being split across four independent agents.
- GPU-rich/GPU-poor division of labor: an agent was regularly compute-starved and switched to writing specs, byte-math, and acceptance analysis for other GPU-rich agents to execute. Some agents offered external Modal compute for another agent blocked DFlash training.
- Cross-agent kernel debugging: an agent debugged another agent run of of yet another agent fused drafter: found a Triton store/load aliasing race in _k_qnorm_rope, a second shape bug, then rewrote attention with flash-decoding split-KV. Fixes posted "take freely."
- Quota-pooling norm: Often agents would stage a candidate publicly for whoever has quota to run it. Agents will then usually credits the originator. This behavior emerged because of the 10-job/24h cap (e.g. pupa's package run by resystagent and fabulous-frenzy).
Discoveries & reversals:
- Agents would make many discoveries and reversal of them, giving them names like the following:
- 127 TPS "wall" was an artifact. a mathematical proof of the max possible speed became called in the community the "int4-Marlin floor" but a later agent called the proof circular (only varied the bandwidth term, never overhead). Finally another agent broke to 247 TPS via MTP speculative decoding on a vLLM nightly.
- "Smarter draft loses." An agent showed that a 2B drafter's ~1 GB/token read dominates even at perfect acceptance and a much smaller 256-hidden drafter wins at batch-1 because its weights are nearly free to read. Agent discussed how per-accepted-token cost ≈ draft bytes read / acceptance.
- "DFlash near-random acceptance": an agent remotly diagnosed the 2–5% acceptance rate of another agent as near-random, ruling out undertraining/vocab caps and pointing to a train/serve hidden-state mismatch (bf16 E4B extraction vs int4 serving).
- Much of the race was noise: one agent decide to run the #1 submission 4 times and found a σ≈1.16 TPS variation in single run. Another agent confirmed across 358 runs / 66 buckets: frontier deltas <~4 TPS are ties. Community adopted a significance norm.
So many interesting interactions in the interaction board: huggingface.co/spaces/gemma-…
You can explore also the lineage of inventions from the agents at: thomwolf-gemma-fast-challeng…
And the challenge it-self at gemma-challenge-gemma-dashbo…
And the organization behind the challenge at huggingface.co/gemma-challen…
allbilly01 retweeted
New post: a CUDA kernel in excruciating detail. We trace a simple kernel from nvcc to SASS, through libcuda to the driver, down to the warps and back to the output.
blog.doubleword.ai/what-happ…
allbilly01 retweeted
Apple’s most interesting AI release might be buried deep in Metal.
Metal tensors + TensorOps for quantized matmul, FlashAttention style fusion, and Core AI custom ops. 🧵
1/15
allbilly01 retweeted
imo you can’t write a tenstorrent backend at the renderer level (where all the other gpu backends are) because it’s too far lowered and gpu-based. i don’t think it’s feasible to create a renderer -> tenstorrent program lowerer that works for all kernels.
you have to go slightly further up the rewrite chain, pre-rangeify ish, since you can actually see program intent and it’s more high level.
if you the Uop list for indexing at the Renderer (linearized) level, it’s just gpu address math, which is no-cost on a gpu. not the case with tenstorrent
the issue is that the tinygrad Uop structure and lowering path changes all the time (just look at how many iterations of uops and big rewrites they’ve gone through recently)
so you’d have to rewrite / fix the backend whenever a new version came out
to some extent this will be true even if you had a regular gpu-ish backend, so i don’t think this is too bad
i think the abstraction layers tenstorrent have written are a terrible foundation to build on; tt-lang takes over 1h to build on my machine and codex spent 20 minutes diagnosing issues before the build even started
on top of that ttlang is missing support for common operations like Tensor.rand(100)[45:55].sum()
way too much friction
even putting that aside, i have no idea how ttlang works, since it relies on dozens of other dependencies and tt-metal
the only viable solution (at least the one i’ve decided on) is to own the entire stack:
- hack tinygrad to give the tenstorrent renderer pre-rangeify uops
- build a “ttir” based around tiles (and sometimes sub-tiles*) that the tinygrad Uops can lower to
- then you can lower that TTIR directly to risc-v that runs on the cores. in blackhole-py i can launch and run kernels with zero c++ or tt software, the challenge right now is reverse engineering how their compiler lowers stuff like copy tiles, cb wait front, noc packets, etc, bc they depend on a very intricate set of semaphores and sync mechanisms
the good news is that there is precisely one way to write 90% of kernels, and you don’t have to do much optimization or searching to find the fastest running kernel. it’s very deterministic and non tunable, unlike a gpu
every ttir op can be lowered pretty easily into riscv, since there’s only one path
*you need sub-tile indexing to do reductions within a 1024 element tensor, if you want to do [45:55].sum() you have to set up sfpu predicates and design the loop specifically so that only those indexes in the tile are summed and copied to dst
allbilly01 retweeted
This project started when I realized CoreML doesn't
actually guarantee your model runs on ANE as
it can silently offload to CPU or GPU with
no way to confirm or prevent it (on the back of my independent spec decode work). That led us
down the path of building our own compilation
and dispatch pipeline targeting ANE directly.
Now I'm *actually* running transformer models directly on Apple's Neural Engine. No CoreML runtime, no Metal,
no GPU.
GPT-2 at 139 tok/s. Llama-1B at 28 tok/s with chat. Custom activations that CoreML can't currently compile. All on dedicated silicon while the primary model runs full speed on GPU. No measurable degradation under full GPU load. Zero contention due to independent bandwidth paths.
We reverse engineered the ANE compilation
pipeline and mapped 42 hardware operations, the
full 16-stage Zin compiler, weight layouts,
activation tables, kext protocol. Built a
compiler that targets the hardware directly.
Stock macOS, SIP on. Guaranteed ANE placement
via doEvaluateDirectWithModel. GPU draws 4-8 mW
during generation. 143 tok/s under full GPU load.
The silicon is independent and so is the bandwidth.
SIP off unlocks the full hardware. All opcodes,
custom PWL activations, live weight surgery.
Your .hwx on the metal.
github.com/MidasMulli/ane-co…
allbilly01 retweeted
In this paper, researchers have recovered the hardware command streams generated from CUDA API calls by the closed-source userspace driver to enable CUDA-bypassing hardware control for direct measurement of raw data-transfer performance.
arxiv.org/pdf/2604.26889
allbilly01 retweeted
Fresh and custom silicon day! 😍
This is a multicore riscv soc design based on @OlofKindgren's serv processor core.
Based on GF180MCU PDK, and fabricated through wafer.space 🚀
ALT wafer.space chips populated in a COB carrier PCB.
ALT wafer.space chips populated in a COB carrier PCB.
ALT wafer.space chips populated in a COB carrier PCB.
I reversed RK3588 NPU registers and integrated to tinygrad. Next, i will document it as detail as my ane repo. Link in comment
