Internet privacy and security research foundation. Global researchers with HQ in Rotterdam, Netherlands.
Rotterdam, Netherlands.
Joined January 2022
- Tweets2K
- Following16
- Followers9
- Likes6.9K
PrivacyHunt retweeted
🪷Maa Mangala, kakatpur, puri, odisha
🪷 Yesterday evening Arati darshan of Maa Mangala, kakatpur, puri
PrivacyHunt retweeted
Sishya Articles
*Story of a Hibiscus*
It was pitch black. I couldn't see anything, yet I could hear the soft murmurs of my siblings. They spoke of the gentle morning breeze, the warmth of the first rays of sunlight, and the cool touch of raindrops resting upon their petals. I listened with wonder, imagining what it would feel like to experience those things for the very first time.
One morning, everything changed.
Without warning, I was separated from my siblings. A sharp pain rushed through my stem as someone plucked me from my parent plant. It was the first time I had ever known separation. Fear gripped me. The place where I had been born, where I had belonged, was gone in an instant.
All I could sense was that I had been placed among countless others like me. We were passed from one hand to another, carried to a place none of us understood. Slowly, my vision began to awaken. Around me were flowers unlike anything I had ever imagined some white, some yellow, some fragrant, each carrying its own silent story.
A gentle hand gathered us together and placed us into a pot filled with cool water. The touch of the water soothed the pain of my separation. For the first time, the morning sunlight kissed my petals. It was exactly as my siblings had described warm, gentle, and full of life.
Soon, I was carried once again.
Someone sprinkled fresh rainwater over me before carefully placing me upon a cold stone surface.
I lay still.
My fellow flowers were arranged at different places around me, each becoming part of something greater than ourselves.
Curiosity overcame my fear.
I looked upward.
Before me stood an enormous stone figure. At first I saw only a garland of skulls. Then a long, blood red tongue. Fierce features unlike anything I had ever seen. Terror seized me. I wanted to turn away, but I could not.
Then I met Her eyes.
They did not merely look at me they looked through me. Beyond my petals, beyond my fragrance, beyond everything I believed myself to be. It felt as though they were gazing directly into my soul.
Time stood still.
The fear that had consumed me slowly melted away. I no longer felt the pain of being plucked. I no longer mourned my separation from the plant that had given me life. For the first time, I understood that my journey had never been about loss.
It had been about offering.
Slowly... very slowly... I dissolved into those infinite eyes.
I ceased to exist as a flower.
I became only an offering, resting at the feet of the Divine Mother.
I was only a hibiscus flower.
*–Ravi*
PrivacyHunt retweeted
Best explainer on Kimi K3 i've read.
It walks you through how the model works & the elegant innovation behind it:
- K3 is the biggest open model anyone has released: 2.8 trillion parameters total, though only 104 billion of them do the work on any given word.
- The problem Moonshot went after is memory. Normally a model keeps notes on every word it has read, and that pile grows with every token, which is why long conversations get slow and expensive.
- K3 mostly stops the memory pile from growing. Three out of every four layers use a new mechanism called Kimi Delta Attention, which keeps a fixed-size working memory and edits it as it goes, overwriting what's stale instead of hoarding everything. The fourth layer keeps a compressed record of each word so exact details are still recoverable when they matter.
- They proved this on a small model first. A 48B test version used up to 75% less memory and ran about 4× faster at long context, while matching or beating the conventional design on the benchmarks they reported.
- Training leaned hard on long jobs including coding, browsing, research, visual work, agent sessions running hundreds or thousands of tool calls in a row.
- 1 Million cached input tokens costs $0.30. An agent can pull the same repo, the same docs, the same tool definitions back into context over and over without the bill getting stupid. Moonshot credits that to the memory design working alongside their serving stack.
- The biggest takeaway: context window size is the main character, but what matters is what a model compresses, what it forgets, and how it gets exact information back.
PrivacyHunt retweeted
If you want to actually understand LLMs, read these papers in order:
- Attention Is All You Need (Transformers)
- BERT
- GPT (Improving Language Understanding by Generative Pre-Training)
- GPT-2 (Scaling + Zero-Shot)
- Scaling Laws for Neural Language Models
- GPT-3 (Few-Shot Learning)
- The Pile (Large-Scale Training Data)
- Chinchilla (Compute-Optimal Scaling)
- PaLM
- OPT
- BLOOM
- LLaMA
- InstructGPT (RLHF)
- Constitutional AI
- Self-Instruct
- LoRA
- QLoRA
- FlashAttention
- FlashAttention-2
- RoPE (Rotary Position Embeddings)
- ALiBi
- Multi-Query Attention (MQA)
- Grouped-Query Attention (GQA)
- Speculative Decoding
- PagedAttention (vLLM)
- Retrieval-Augmented Generation (RAG)
- Chain-of-Thought Prompting
- Tree of Thoughts
- ReAct
- Direct Preference Optimization (DPO)
- ORPO
- GRPO
- Mixture of Experts (Switch Transformer)
- DeepSeekMoE
- KV Cache Compression & Optimization
- Test-Time Scaling
- Muon Optimizer
“just open grok or google drop paper name and read”
PrivacyHunt retweeted
We've just made a major finding for SovereignAI 🚨:
Take any open-weight model (here: Qwen3.5-397B), apply our π-shaped Continual Learning, and own a genuine frontier model competitive with Opus 4.8 for ~$450k in compute.
Full tech report + open-source models coming soon!
PrivacyHunt retweeted
Reading Kimi K3 paper, I'm struck by the number of innovations and departures from orthodoxy. These guys are doing real research and not copying. It is no longer the case of being, "mostly the Transformer with a few Noam mods".
I mean they got rid of the position encodings/embeddings! Bold!
More thoughts to come after reading.
PrivacyHunt retweeted
Bro casually walks and explains 5 GPU performance optimization methods for LLMs.
one of the most simple and intuitive explanations for beginners.
PrivacyHunt retweeted
Code Arena now measures fullstack capabilities!
View overall rankings across AI models on full-stack web development tasks: multi-step reasoning, tool use, and end-to-end app generation.
- Kimi K3 (Max) takes #1
- GPT 5.6 Sol (xHigh) at #2
- Claude Fable 5 at #3
See more scores at: arena.ai/leaderboard/code/we…
Code Arena just leveled up with fullstack capabilities 🚀
Introducing the new Fullstack Code Arena. We’re moving beyond frontend prototypes to fullstack development complete with databases, API keys, and fast deployments. Build, iterate, and ship real-world software — all in one place.
Models now act as agents in the Code Arena, using structured tool calls to plan, execute, and refine in real time with real world tasks.
Read more about it in the thread 🧵
PrivacyHunt retweeted
I spent 48 hours with the Kimi K3 modeling code.
It took:
- 650 mg of caffeine (mandatory)
- 40 cans of LaCroix (optional... world record (?))
- 8 papers
- 6 months off my lifespan
Finally grokked the entire lineage of Kimi K3 and how we got here... every single step, since 2019 GPT-2
PrivacyHunt retweeted
The Kimi K3 architecture figure for yesterday's big open-weight model release, along with some observations and thoughts.
1. Yes, it looks relatively complicated, but it's essentially a scaled-up production version of their Kimi Linear model they released last year (scaled up from 48B -> 2.8T; K3 is by far the biggest open-weight model right now)
2. The one new component compared to Kimi Linear is the LatentMoE. I omitted it in the figure below since it's already very crowded, but that's essentially the same LatentMoE as in Nemotron 3 Ultra (you can find it in my LLM Architecture Gallery if you are curious). The idea here is to compress (down-project) large linear layers similar to multi-head latent attention.
3. Kimi K3's overall trend (similar to Nemotron 3, DeepSeek V4, and others) is also towards better inference efficiency. That is, there are many components that replace existing components with efficiency-tweaked versions. I.e., MoE -> LatentMoE, regular attention -> multi-head latent attention and Kimi Delta Attention. (I also have short tutorials and write-ups in my gallery if you are curious about additional details).
4. The one component change that is not an efficiency tweak is attention residuals. Like DeepSeek V4 improved the residual path with mHC (manifold-constrained Hyper-Connections), attention residuals are a way to improve the residual path, but it works a bit differently. I.e., mHC made the residual path wider. Attention residuals (also already part of Kimi Linear) connect the residuals across layers; the connection itself uses an attention score for an important/contribution weight. According to the report, it improves the validation loss and downstream performance (a bit) consistently and adds about 4% in training cost and 2% in inference cost.
5. Interestingly, Kimi K3 got rid of all RoPE layers and uses NoPE (No Positional Embeddings) everywhere instead. (Again, this is inherited from Kimi Linear). In other architectures, the recent trend was towards RoPE in local attention layers (like sliding window attention) and NoPE in the global layers. There were a few architectures that only used NoPE everywhere, but this is the first frontier-level one as far as I know.
6. Kimi K3 now also has native multimodal support, which is great!
There are several other interesting training tidbits in the technical report, but that's it from the architecture front so far. A really great release overall.
PrivacyHunt retweeted
ॐ दुर्गे त्वं ब्रह्मरूपासि सुरतेजोमहाबले ।
महानन्दकरे देवि प्रसीद परमेश्वरि ॥
O Ma Durga, You are the Supreme Reality, the embodiment of Brahman, and filled with the divine cosmic power of the Gods. O Goddess who bestows supreme bliss, be pleased with me, O Parameshwari
🙏🏻🌺
PrivacyHunt retweeted
Releasing the model weights and technical report of Kimi K3.
Kimi K3 is our most capable model: a 2.8T MoE model with native visual understanding and a 1M-token context window.
New model architecture: 2.5x the intelligence per unit of compute, not just more params.
Alongside Kimi K3, we're opening up more of the stack behind it — high-performance attention kernels, MoE communication library, and infrastructure for running agent environments at scale.
Model weights: huggingface.co/moonshotai/Ki…
Tech report: github.com/MoonshotAI/Kimi-K…
Tech blog: kimi.com/blog/kimi-k3
