Pinned Tweet
"Sharpness of the loss surface tells us about generalization... except when it doesn't (like in transformers)."
But what does sharpness mean? You might say
"it's how much the function changes within a little ball".
Ok, so in the simple picture shown below, at which of the two points is the loss surface sharper?
To find out
* why this is a tricky question, and
* why our answer to it does allow sharpness to tell us about generalization, even in transformers(!!),
come to our #ICML2025 #spotlight poster E-2001 on Wednesday at 11:00am-1:30pm PDT
Work by Marvin F. da Silva and Felix Dangel
@dalfcs @VectorInst
[1/🧵] ✨ Hide & Seek: Transformer Symmetries Obscure Sharpness & Riemannian Geometry Finds It ✨
Super excited to announce our paper on factoring out parameter symmetries to better predict generalization in transformers (accepted as #ICML25 spotlight! 🎉)
Amazing work by Marvin da Silva (@marvinfsilva) and Felix Dangel (@f_dangel).
Symmetries hide sharpness — Riemannian geometry reveals it👇
one issue with some "conversations" i hear about #creativity and #AI is that, i believe, some of the people talking don't understand Cohen's point of view in this video. i don't at all think one needs to agree with him, but if one doesn't *understand where he's coming from*, that indicates such a lack of depth about the topic that it simply undermines their subsequent credibility about it. maybe like someone opining about imaginary numbers who doesn't know what a polynomial is.
facebook.com/share/r/1931ctA…
I actually do find it quite surprising, because I wouldn't have thought that [7.1,7.3] is a popular range picked either by humans or by LLMs when asked to pick a number between 1 and 10. Also, I don't think the interval [5.5,5.6] is more popular for numbers than some of the ones in [5.9,6.1].
AI alignment is hard to begin with, but I imagine it's extra challenging for people to work on it within the moral and ethical vacuums of many research environments, often both structural and otherwise. Just, ya know, cognitive dissonance and all that... 🤪
sageev retweeted
there is a difference between "i am ok with models being trained on my data and released publicly later" and "i am ok with internal models trained on my very recent efforts, being used to race the solution before me, after rumors i am close to it". really not the same thing.
"sharing" is the load-bearing word in that first sentence.
We’re sharing a solution to the Navier-Stokes Millennium Prize Problem, one of the deepest problems at the frontier of mathematics.
The proof was produced by a group of agents, using an OpenAI next-generation model significantly more capable than GPT-6 Astra.
The problem concerns whether the description of smooth three-dimensional fluid motion modeled by the Navier-Stokes equations can break down. It has remained unresolved for roughly 90 years.
puppetry and piano and dysfunctional relationship dynamics and more.... all at once!
instagram.com/reels/DcpaWn0P…
the biggest AI4science revolution will happen once these systems become legit better than PI's at framing their contributions.
But for all the new AI consciousness experts out there, just a rnd q: Say you had 20 models, and each of them was a little bit conscious. Now you combine them (maybe a mixture of experts with a little gating network, or whatever frankenstein architecture you like). How many consciousnesses does your new bigger model have, and what happened to the old 20?
Wrong answers only :)
tantrums are fairly often understood as situations where the child has "bigger feelings/reactions" than what they have the emotional tools to cope with. so this behaviour proves that it has feelings, too.
/jk: in case not obvious: 1st sentence true, 2nd sentence false conclusion
"fast & sloppy & fuck it"
See this Instagram post by @osageev instagram.com/reel/DcO06StQI…