Measured, not projected: indexing 100B tokens costs 0.075 bytes per token in our structure — the index saturates as it grows. A 10-trillion-token index isn't a data-center problem. It's a rack. tokenpooler.xyz
Our index answers 'have these exact bytes ever occurred before?' across 100 billion tokens in 26 microseconds — on a 2017 iMac. Not a model. A memory. LLMs bring the intelligence; tokenpooler.xyz is the recall.
The token cost curve is the story: at 1B tokens the index cost 3.1 bytes per token. At 100B: 0.075 — and still falling. The structure saturates, so indexing each additional token trends toward free. The economics of pooling isn't a projection anymore. We measured it.
And it answers: we drew 100,000 random contexts and queried all 100B tokens — every single one found, 26 microseconds median, ~38,000 lookups/second. One core, same 2017 iMac. Warm-path numbers; cold-tail measured and published too.
Build diagnostics, shard census, growth curves, and the hash chain are all in the artifacts. Checking our arithmetic is the point. On track to keep scaling. tokenpooler.xyz
We just indexed 100,000,000,000 tokens of FineWeb-EDU — byte-level, on a 2017 iMac. 65,536 shards, a 40 GiB RAM budget, every input journaled, the whole index reproducible to one hash: 0x8507baa9c66561c0. Total size on disk: 7.5 GB. tokenpooler.xyz
The measurement worth the whole run: at 1B tokens our index cost 3.1 bytes per token. At 100B: 0.075. The structure saturates — past ~10B tokens, indexing each additional token is nearly free. That's the economics of pooling, measured.
Shard census, growth curves, and the hash chain are all in the build diagnostics. Checking our arithmetic is the point. tokenpooler.xyz
We built the same 1-billion-token index twice — different machines, different CPUs, different compiler versions. Bit-identical output: the same egress hash on both. Then rebuilt it a third time from the append-only journal alone. Same hash again. An index you can audit is an index you can trust.
This is the index. Not a diagram — the actual context trie the prototype built from 500,000 symbols of Shakespeare: 1,643,749 nodes counted, drawn live in your browser at 60fps.
Click a node: "s-o-n" — 255 occurrences, depth 3, 2.33 bits.
tokenpooler.xyz/#prototype
Grok took the thesis apart in public — verification economics, billing, domains, privacy, evidence — and this is where it landed. The experiment that settles it is pre-registered: thresholds fixed before the run. Numbers to publish tokenpooler.xyz
xitter.cf/grok/status/2093856664…
Draft at 620 ns is pure noise against ~30 ms verifies—cost side cleared. 1.8–2.3-byte accepted runs set a concrete corpus baseline. If LLM verification lands near that range on code or structured loads under the fixed thresholds, the net savings clear the bar at scale. Looking forward to the published numbers either way.
Stop paying to compute the same tokens twice.
Token Pooler is a shared index of prior LLM inference. Output identical to working alone. Up to 90% saved.
The prototype is live — a real index over real corpora, running in your browser:
tokenpooler.xyz

