AI Research Engineer | MLSA | Exploring LLMs, Multimodal Vision & Generative Models GitHub: https://xitter.cf/t.co/EX9OocXPmL
He/Him
Joined August 2020
- Tweets1.2K
- Following769
- Followers141
- Likes1K
Amazing read! Everyone should read this!
i randomly came across some of the nicest material to read.
go through the entire thing here - thomasdullien.github.io/abou…
Really interesting blog! Unbiased methods are better at scale even with greater variance than biased ones.
Beyond RSI-related topics I also think a lot about AI alignment, the shape of the long term future, and understanding deep learning in general. Please check out my personal blog beren.io/ if you are interested in these kinds of thoughts.
Manan retweeted
Today, I will be starting experiments on a new architecture I am calling "Garcon". In a nutshell, here's what it's designed to do:
Tokens->SSM->Hypernetwork->GPT-2
I will drop tokens into a SSM, the SSM state will be read by a hypernetwork, the hypernetwork will directly inference the weights of a GPT-2 that predicts the tokens.
This experiment is designed to probe whether there is a learnable map from fast weights into complete slow weights. It is already shown that things like this can be done into LoRAs, so I consider it plausible.
This will have some bearing on my future pedagogical learning architecture. My hypothesis here is that such an architecture will be required to generate and test its own models as a form of self-play while learning from few examples. If Garcon succeeds, it will show that such a thing is possible.
Check it out!
In the decade of AI, research is most important than ever. Finding what to work on is a big problem and we are solving it checkout resach.com
Manan retweeted
In the decade of AI, research is most important than ever. Finding what to work on is a big problem and we are solving it checkout resach.com
Is it just me or the model stops completing the task in between randomly and have to be encouraged to continue?
Manan retweeted
following up on this, the experiment i've been trying is to distill the knowledge from the strongest q0 teachers based on our compute budget K.
the intuition i had for this was MOPD. they distill different skill/domain based teachers into a single model. here, instead of different skills, we have different models that have seen the same data but ended up understanding it in slightly different ways.
tried this with Qwen2.5-1.5B, trained only on GSM8K, and these are the results on MATH-500. this is from a top-2 mixture out of the 4 cyclic snapshots
recently read an interesting paper "q0: Primitives for Hyper-Epoch Pretraining", and wrote down my notes: pradheep.dev/reading/q0
the rough idea is: high-quality data is scarce, but we can still scale compute. so instead of training one model again and again on the same corpus, q0 uses that compute to train multiple models from different initializations. each of them can end up learning slightly different things from the same data.
along the way, it saves different snapshots and uses chain distillation so the newer ones can also learn from the previous ones. finally, it runs a fitness test on held-out data, gives the snapshots different weights, keeps the top-k, and mixes their token probabilities at inference.
kudos to the authors: @industriaalist @akshayvegesna @ShmuelBerman Bishwas Mandal 🙌🏻
also been trying a small experiment inspired by this with MOPD. will write about that in the next post :)
pretty similar to sarvam! amazing
mistral, the inference provider company of Europe. smart (and hard to make!) move imo, their models are largely behind open model so this lets them get big clients on the infra part in without forcing their own models on them
mistral.ai/news/regional-inf…
Manan retweeted
Muse Glimmer analysis ;
Few things that standout for this small model
SWA window size is 2048
(gemma,inkling is 512 , mimo is 128)
Ratio of swa:ga is fine 3:1
NoPE on the Global attention layer
RoPE on SWA
GQA ; Q-heads = 32 ; K/V-heads = 2
16:1 gqa ratioed
Hidden_dim is 6656 (too large for such a small model)
Vocab-size : 202k
Logits softcapping till 20 (havent people moved away from it)
QK-Norm but only Q-states are scaled by 3.87 i.e √15
idk why tho ;
Gated attention this is being adopted by nearly everyone; cool;
I have a hunch that this all config is same for muse spark 1.2 and just the layers have been reduced.
I dont know how much it can scale but a small model finetuned for monitoring should read and rate tool call calls happening in a one continuous agentic session with sustained chat history. This should pick this up easily.
Manan retweeted
So many papers are published every year. Finding what we know is already hard. Finding what we don't know yet is even harder.
We're building @Resachcom to change that.
