MF FOOM (@MF_FOOM) header
MF FOOM (@MF_FOOM) profile picture

masked attention head

Joined July 2023
Photo by MF FOOM (@MF_FOOM)Photo by MF FOOM (@MF_FOOM)Photo by MF FOOM (@MF_FOOM)Photo by MF FOOM (@MF_FOOM)Photo by MF FOOM (@MF_FOOM)Photo by MF FOOM (@MF_FOOM)Photo by MF FOOM (@MF_FOOM)Photo by MF FOOM (@MF_FOOM)Photo by MF FOOM (@MF_FOOM)Photo by MF FOOM (@MF_FOOM)
mf trained a simple model to translate ada-002 embeddings back to text and found something interesting: sentence embeddings have arithmetic properties the classic "king - man + woman = queen" still works just like w/ token embeddings — but can go a few steps beyond that as well
50
157
51
1,408
568,339
he can't keep getting away with it
FlashAttention is widely used to accelerate Transformers, already making attention 4-8x faster, but has yet to take advantage of modern GPUs. We’re releasing FlashAttention-3: 1.5-2x faster on FP16, up to 740 TFLOPS on H100 (75% util), and FP8 gets close to 1.2 PFLOPS! 1/
8
724
there was a good thread recently estimating the current upper bound of tokens in the world (including Gmail and other private repositories), but I can't find it can anyone point me to it?
2
5
888
Why not just train your embedding model to embed queries directly into answer space? Feels like an inefficient use of flops.
i find HyDE (hypothetical document embeddings) to be such an elegant and intuitive idea - like to improve your semantic search, you ask an LLM to generate a hypothetical answer to your query and then you use this query for embedding search. idea is the (embedding of) hypothetical query is going to be closer in latent/embedding space than your actual natural language query. it's funny how you are literally using a hallucinated answer to get better results.
3
15
3,131
Contrast pairs are overpowered. Once you have them, you can use them to generate control vectors, and to initialize classifiers, and to do RL/DPO, and probably more
Replying to @AnthropicAI
To make the probes, we track how the model’s internal state changes between “Yes” vs “No” answers to questions like "Are you doing something dangerous?" We use this info to detect when a sleeper agent is about to misbehave (e.g. insert a code vulnerability). It works quite well:
2
5
81
11,944
This is the most in-depth technical report we’ve gotten about a frontier-class model in a while!
Replying to @RekaAILabs
Along with Core, we have published a technical report detailing the training, architecture, data, and evaluation for the Reka models. publications.reka.ai/reka-co…
2
2
45
4,419
Yes, and get SOTA by doing so. xitter.cf/vaibhav_adlakha/status…
could you turn a pre-trained decoder into an encoder by just removing the attention mask and finetuning on encoder-like tasks?
1
1
11
1,233
‼️⚠️bad day to be a LLM⚠️‼️ @haizelabs took one of our favorite adversarial attack algorithms, GCG, and made it *38x* faster
2
14
3
65
12,357
assuming it's not just encouraged via prompt to make it sound natural: uh/ums could be when it's waiting on either the LLM or TTS model stuttering could be from them feeding the results of a speculative decoding draft model into the TTS and part of the draft getting rejected?
how did it learn to use filler words like "uh" and stammer under pressure? what
2
9
1,353
i really think it's just encouraged via prompt tho, don't see when would you have these out of sync issues if you're using internal models with reserved hardware as they're almost certainly doing
1
4
194
When a student sadly tells me that the idea we've been working on for weeks was just arXived, I say: "Great! We've just gotten *strong* confirmation that our thinking was in the right direction. We've had the initial work done for us. Lets figure out how to make this 10x better"
6
12
1
229
19,259
new blog post! played around w/ representation engineering, and released a new library for training control vectors in <60s! i train some useful control vectors, some… out there… control vectors, test them for jailbreaking, test them *against* jailbreaking, and more!
39
32
21
442
170,853
New blog post: how to make LLMs go fast! Want to understand how people are making LLMs go brrrrr? This post is a survey of lots of different LLM inference optimizations, ranging from "everyone uses this in prod" to "I cooked this up last week (but it seems to work)"
12
149
11
1,001
176,928
so who's gonna extract Gemini Nano from their Pixel and put it on huggingface first?
8
3
1
148
21,440
btw do we have reason to believe the architecture of Gemini Nano is the same as Pro and Ultra?
1
10
1,302
Talk: Inverting Language Models (by @jxmnop) youtube.com/watch?v=lguThumF… Techniques for extracting text from vector databases and prompts from LLM APIs.
2
21
2
202
108,522
TIL that while A100 SXM & PCIe both use an identically binned GA100, H100 PCIe uses a GH100 with much heavier binning than H100 SXM
1
9
791
and seemingly not just because PCIe Gen 5 can only drive up to 600W of power (the H100 SXM exceeds it but only slightly) looks like its explicitly targeted at buyers who want a more power efficient gpu
351