@HOARK_i
iAccount based inUnited States
About this account
- Account based in
- United States
- Connected via
- United States App Store
Account-level information from X, not a live location or the device used for a specific post.
Founder @ https://nitter.cf/t.co/GFfEWf1AjK (TCG marketplace) | Built Wave Density Attention (teacher-free AI primitive) on a 3090 | Seeking compute/collabs for AI + product
USA
Joined August 2014
- Tweets659
- Following247
- Followers167
- Likes192
I open-sourced Wave-Density Attention, a new causal attention mechanism that replaces dot-product similarity with wave interference.
Instead of QKᵀ, attention emerges from superposed sinusoidal masks whose interference patterns form dynamic density fields. A sparse Mixture-of-Masks (MOM) router activates only a small subset per token.
Result: a 130M model trained from scratch hits ~18 perplexity on instruction data — on a single GPU.
This isn’t an optimization of transformers — it’s a different primitive for attention.
Code: github.com/H0ARK/wave-densit…
Model: huggingface.co/H0ARK/wave-de…
Try it. Break it. Improve it.
#MachineLearning #DeepLearning #OpenSource
@karpathy @huggingface @arankomatsuzaki @tri_dao @jimmyba
HOARK retweeted
Today we are introducing Escha-W2 quantization.
A 2-bit Qwen3.6-35B-A3B model built for fast, local inference. The complete model is 12.3GB on disk—small, enough to run on a single consumer GPU — while averaging ~100% of FP8 performance across 12 benchmarks, including:
MMLU-Pro: 80.9
MATH-500: 93.8
GPQA-Diamond: 77.8
LiveCodeBench v6: 62.6
BFCL tool use: 88.9
RULER 8K–128K: 89.9
Commonsense-6: 76.1
On a single RTX 4090, the model runs:
225 tok/s single-stream generation
on 12.3GB on-disk model size
Compressing a 35B MoE model this far without collapsing its capabilities required more than a standard quantization pass.
We built an end-to-end compression system combining state-of-the-art low-bit quantization with model-aware fine tuning and recovery to preserve capabilities most vulnerable to low-bit error.
Quantizing the Qwen 3.6 35B model - from the base model to the final deployable checkpoint — took approximately 10 hours to complete.
Escha-W2 runs through a custom Qwen3-MoE runtime, which includes the weight loader, low-bit decoding kernels and serving integration required to execute the Escha format efficiently. The runtime currently supports SGLang and ZML deployment. vLLM and llama.cpp will be supported in the next release.
No retraining from scratch. No specialized accelerator. One consumer GPU.
Model download: huggingface.co/EschaLabs/Qwe…
Runtime download: huggingface.co/EschaLabs/esc…
Apache-2.0 model and runtime.
HOARK retweeted
Our Maia 200 inference chip, announced today, is most performant first party silicon of any hyperscaler. 3x the FP4 performance of the Amazon Trainium v3, and FP8 performance above Google’s TPUv7.
HOARK retweeted
Transformers v5's FINAL, stable release is out 🔥 Transformers' biggest release.
The big Ws of this release:
- Performance, especially for MoE (6x-11x speedups)
- No more slow/fast tokenizers -> way simpler API, explicit backends, better performance
- dynamic weight loading: way faster, and enabling: MoE now working w/ {quants, tp, peft, ...}
We have a migration guide on the main branch; please take a look at it in case you run into issues. Come in our GH issues if you still do after reading it 😀
Just released: Behavioral Distillation - a new approach to model training that transfers how models attend rather than what they output.
Result: Students continue improving after teacher removal. Training was still improving when I stoped.
Paper: github.com/H0ARK/WDA/blob/ma…
This isn't feature matching - it's policy transfer. Two models can have completely different internal representations but equivalent behavioral dynamics.
WDA provides a continuous control surface that makes this alignment learnable and stable.
#MachineLearning #AI #DeepLearning #Gemini #openai #xAI #Anthropic
I ran into the hardware wall: training a new attention mechanism from scratch would take trillions of tokens and compute I don't have. So I asked: can wave-based attention inherit real Transformer behavior without that insane retraining cost?
I distilled the behavior itself—not logits or features—then fully detached the teacher. The result: a standalone wave-based model running on consumer hardware (single 3090), proving waves can carry and evolve Transformer-level intelligence on their own.
10/
Built on one 3090.
No SOTA flex. No hype.
Just cold proof: New primitives can inherit Transformer behaviors, cut the teacher cord completely, and stand freestanding — driving the model solo.
If you’re digging alt-attention, cross-arch distillation, or smashing trillion-token barriers — DM me. Let’s build something bigger.