This is exactly what I meant last month! as more of the work of formalizing and checking proofs becomes routine, mathematicians will climb higher, towards new abstractions, objects and conjectures. Perhaps that’s where the genius goes!

GNNs are not neural networks. They are graph algos wearing a neuralnetwork costume. The intelligence is in the message passing structure, not the weights! which is exactly why they generalize where MLPs memorize.

The DeepSeek mHC paper is a real breakthrough, and the reason is geometric, not architectural. Early neural networks were just repeated matrix multiplications: x <- W x. Depth was unstable. ResNets changed one line: x <- x + F(x) which linearizes to x <- (I + W)x. That single identity term is what made deep learning scale. Hyper-Connections broke this by replacing identity with a learned matrix, turning depth back into unconstrained matrix products. mHC fixes this in a principled way. Instead of identity or an arbitrary matrix, mHC uses a doubly stochastic one. Doubly stochastic matrices form the Birkhoff polytope. They are convex combinations of permutations. Geometrically, the residual stream undergoes conservative transport and mixing, not amplification or decay. Identity is just one extreme point of this space. Under composition, stability is preserved. mHC does not abandon identity. It generalizes it into a stable geometric object. This is not an engineering trick. It is linear algebra and geometry doing the real work

Most ML models are taught symmetries. This paper shows something deeper: symmetries can be discovered. LieGAN treats symmetry as a geometric property of the data distribution. If a transformation moves data along the same manifold without changing the distribution, it is a symmetry. My take: this is learning as quotient geometry. Gradient descent is not learning rules, it is collapsing representations along group orbits.

After a year of mulling geometry in deep learning, I suspected something was afoot. Then I found this baffling paper. It removes attention completely, replaces it with low-rank subspace flows, and the model still reasons. Turns out, geometry was doing the thinking all along!!

Deep learning sequence models can be seen as associative memory systems that decide what to store, what to forget, and how to retrieve information efficiently. The MIRAS framework from Google Research makes this explicit by defining four core design choices: memory architecture, attentional bias (internal objective), retention/forgetting, and memory learning algorithm.

The brilliant Google Reseach paper "Nested Learning: The Illusion of Deep Learning Architecture" proposes a striking idea: Deep learning is not a stack of layers. It is a hierarchy of nested optimization problems, each acting as an associative memory with its own time scale. Backprop, momentum, Adam, linear attention, Titans, CMS… All become instances of the same mathematical object.

The paper “On the Edge of Memorization in Diffusion Models” uncovers something deeper than it explicitly says. It shows that diffusion models have a built in data compression limit. Generalization happens only while the model is too small to store all training samples. Once capacity crosses a threshold, diffusion models stop learning structure and begin remembering images pixel by pixel.

New paper drops a reality check: even trillion-parameter LLMs hit hard mathematical ceilings. - diagonalization means hallucinations are inevitable, - finite info means long-tail facts stay fragile, - and attention geometry means context isn’t really context. Scaling helps, but theory says limits remain.

Most people think Vector Databases are a recent invention driven by AI. But the story actually started in 1998. Piotr Indyk & Rajeev Motwani(of page rank fame) changed how we think about search itself … and paved the way for modern VectorDBs like Pinecone, Qdrant, and FAISS. Here’s how it all evolved: