@HOARK_

Founder @ https://nitter.cf/t.co/GFfEWf1AjK (TCG marketplace) | Built Wave Density Attention (teacher-free AI primitive) on a 3090 | Seeking compute/collabs for AI + product

USA
Joined August 2014
Pinned Tweet
I open-sourced Wave-Density Attention, a new causal attention mechanism that replaces dot-product similarity with wave interference. Instead of QKᵀ, attention emerges from superposed sinusoidal masks whose interference patterns form dynamic density fields. A sparse Mixture-of-Masks (MOM) router activates only a small subset per token. Result: a 130M model trained from scratch hits ~18 perplexity on instruction data — on a single GPU. This isn’t an optimization of transformers — it’s a different primitive for attention. Code: github.com/H0ARK/wave-densit… Model: huggingface.co/H0ARK/wave-de… Try it. Break it. Improve it. #MachineLearning #DeepLearning #OpenSource @karpathy @huggingface @arankomatsuzaki @tri_dao @jimmyba
2
2
7
2,327
I really find it funny how people are looking at a score and trashing astra for its poor performance when its not about the super smarts but adaptability of using what they see. look at yourself your not as smart but are "agi" technically because you know how to search...
14
Today we are introducing Escha-W2 quantization. A 2-bit Qwen3.6-35B-A3B model built for fast, local inference. The complete model is 12.3GB on disk—small, enough to run on a single consumer GPU — while averaging ~100% of FP8 performance across 12 benchmarks, including: MMLU-Pro: 80.9 MATH-500: 93.8 GPQA-Diamond: 77.8 LiveCodeBench v6: 62.6 BFCL tool use: 88.9 RULER 8K–128K: 89.9 Commonsense-6: 76.1 On a single RTX 4090, the model runs: 225 tok/s single-stream generation on 12.3GB on-disk model size Compressing a 35B MoE model this far without collapsing its capabilities required more than a standard quantization pass. We built an end-to-end compression system combining state-of-the-art low-bit quantization with model-aware fine tuning and recovery to preserve capabilities most vulnerable to low-bit error. Quantizing the Qwen 3.6 35B model - from the base model to the final deployable checkpoint — took approximately 10 hours to complete. Escha-W2 runs through a custom Qwen3-MoE runtime, which includes the weight loader, low-bit decoding kernels and serving integration required to execute the Escha format efficiently. The runtime currently supports SGLang and ZML deployment. vLLM and llama.cpp will be supported in the next release. No retraining from scratch. No specialized accelerator. One consumer GPU. Model download: huggingface.co/EschaLabs/Qwe… Runtime download: huggingface.co/EschaLabs/esc… Apache-2.0 model and runtime.
90
138
42
1,337
203,459
What if your agent harness can start solving swe verified benchmarks before you plug the model in?
1
14
Our Maia 200 inference chip, announced today, is most performant first party silicon of any hyperscaler. 3x the FP4 performance of the Amazon Trainium v3, and FP8 performance above Google’s TPUv7.
39
47
12
374
54,095
Transformers v5's FINAL, stable release is out 🔥 Transformers' biggest release. The big Ws of this release: - Performance, especially for MoE (6x-11x speedups) - No more slow/fast tokenizers -> way simpler API, explicit backends, better performance - dynamic weight loading: way faster, and enabling: MoE now working w/ {quants, tp, peft, ...} We have a migration guide on the main branch; please take a look at it in case you run into issues. Come in our GH issues if you still do after reading it 😀
8
80
11
431
76,419
Just released: Behavioral Distillation - a new approach to model training that transfers how models attend rather than what they output. Result: Students continue improving after teacher removal. Training was still improving when I stoped. Paper: github.com/H0ARK/WDA/blob/ma…
1
1
1
63
Why this matters: Traditional scaling = retrain from scratch on trillions of tokens Behavioral distillation = transfer intelligence, swap architecture, keep improving Opens up rapid experimentation without prohibitive compute costs.
1
16
This isn't feature matching - it's policy transfer. Two models can have completely different internal representations but equivalent behavioral dynamics. WDA provides a continuous control surface that makes this alignment learnable and stable. #MachineLearning #AI #DeepLearning #Gemini #openai #xAI #Anthropic
30
I ran into the hardware wall: training a new attention mechanism from scratch would take trillions of tokens and compute I don't have. So I asked: can wave-based attention inherit real Transformer behavior without that insane retraining cost? I distilled the behavior itself—not logits or features—then fully detached the teacher. The result: a standalone wave-based model running on consumer hardware (single 3090), proving waves can carry and evolve Transformer-level intelligence on their own.
28
I thought I was transplanting waves into a Transformer. Turns out I learned how to kill the teacher forever.
1
13
9/ The real bottleneck in LLM research isn’t scale. It’s that every new idea is chained to one massive pretraining run. Behavioral distillation breaks that chain. Once you can transplant behavior, architectures become modular.
1
12
10/ Built on one 3090. No SOTA flex. No hype. Just cold proof: New primitives can inherit Transformer behaviors, cut the teacher cord completely, and stand freestanding — driving the model solo. If you’re digging alt-attention, cross-arch distillation, or smashing trillion-token barriers — DM me. Let’s build something bigger.
13