David Gill retweeted
Rumors are swirling about the deprecation of VMware Cloud on AWS.
I'm proposing a helpful update for the VMC marketing page:
David Gill retweeted
LLaMA-3 is a prime example of why training a good LLM is almost entirely about data quality…
TL;DR. Meta released LLaMA-3-8B/70B today and 95% of the technical info we have so far is related to data quality:
- 15T tokens of pretraining data
- More code during pretraining (leads to better reasoning capabilities)
- More efficient tokenizer with larger vocabulary
- Super sophisticated (including LLM components) data quality filtering
- Extensive empirical analysis of data mixture
- Focus on quality filtering of post training data (for SFT/RLHF/DPO)
All of the cool stuff in this report is related to how to curate data effectively for pre/post-training! This really shows that data curation/filtering is the most difficult and impactful aspect of training foundation models.
(1) Model architecture: Only 5 sentences are provided about the model architecture, which simply state that LLaMa-3 uses a standard decoder-only architecture with grouped query attention to improve inference efficiency (and a longer 8K context). It’s pretty clear that model architectures are becoming standardized, and most of the research focus is going into constructing datasets. In fact, the main architecture modification made by LLaMA-3 is a more efficient tokenizer!
“Llama 3 uses a tokenizer with a vocabulary of 128K tokens that encodes language much more efficiently, which leads to substantially improved model performance.” - from LLaMA-3 blog
(2) Better tokenizer: LLaMA-3 comes with a custom tokenizer with a vocabulary of 128K tokens (LLaMA-2 had a vocabulary of 32K tokens). This tokenizer is more token efficient (i.e., fewer tokens are necessary to encode the same piece of text relative to LLaMA-2), which makes inference more efficient. Authors also note that the new tokenizer improves performance! In other words, making sure that we are encoding the model’s input data correctly is super important.
(3) Massive pretraining corpus: LLaMa-3 is pretrained over 15T tokens of text (5% non-English), which is a 7X improvement over LLaMA-2 and even larger than the 12T pretraining corpus of DBRX. The pretraining corpus also has 4X more code relative to LLaMA-2 (this was a big criticism of LLaMA-2). With this in mind, it’s not a surprise that LLaMA-3 has strong reasoning/code capabilities—several papers have correlated pretraining on code to better downstream reasoning in LLMs.
“We found that previous generations of Llama are surprisingly good at identifying high-quality data, hence we used Llama 2 to generate the training data for the text-quality classifiers that are powering Llama 3.” - from LLaMA-3 blog
(4) FIltering pretraining data: Few concrete details are provided on the filtering process for the pretraining corpus of LLaMA-3, but it’s clear that a lot of filtering is done. These filters include heuristic filters, NSFW filters, semantic deduplication, and text classifiers to predict data quality. Plus, authors note that LLaMA-2 is very good at detecting text quality, so they use these models in the filtering process (see above). Authors also mention that they do extensive empirical analysis to figure out the correct data mixture (DBRX also mentions this is hugely important).
(5) Overtraining: Chinchilla proposed the compute optimal training regime for LLMs, but recent work indicates that pretty much everyone overtrains their LLMs relative to the compute-optimal ratio. LLaMA-3 is pretrained on two orders of magnitude more data (for the 8B model) beyond the compute-optimal ratio, and we still see log-linear improvements. Sure, we could train a larger model on fewer tokens and achieve similar performance while spending less on training compute. But, this doesn’t consider inference costs! We almost always will pay for more training compute if it means we can deploy a smaller model with the same performance.
“The quality of the prompts that are used in SFT and the preference rankings that are used in PPO and DPO has an outsized influence on the performance of aligned models.” - from LLaMA-3 blog
(6) Post training data quality: Even beyond pretraining, data quality is pivotal for LLaMA-3! The model is aligned with a combination of SFT, rejection sampling, PPO, and DPO. During alignment, authors claim that the quality of supervised/preference data is super important. In fact, the biggest quality improvements in LLaMA-3 came from curating this data and performing multiple rounds of quality assurance on humans annotations!
David Gill retweeted
Replying to @Steve8708
Any discussion of interesting browser APIs cannot be complete without mentioning HTMLMediaElement.canPlayType(), which can return the values: "probably", "maybe", and ""
David Gill retweeted
Attaching styles directly to elements strikes people as “wrong” in web dev because of the history of the platform.
But in basically every ecosystem actually designed for UI programming from the ground up it’s absolutely normal, and no one would consider any other approach.
David Gill retweeted
Today we found a PR trying to poison our GitHub project 😨. This `chalk-next` parody package is malicious and will try to collect your config and delete your local files 👎. Beware! #opensource
github.com/toeverything/AFFi…
I have been saying for years that football would be more interesting with firepits and other hazard zones inspired by robot wars.
Glad to see they're taking my feedback onboard.
1
144
David Gill retweeted
DUDE WTF engraved.blog/building-a-vir…
Don't worry folks, I'm working on a Twitter 2.0
Right now it's a Google Sheet with 2 columns. To tweet you simply insert a row to the top - say what you want in column A and your name in column B.
Verified badges coming soon.
4
David Gill retweeted
The State of the Shopify App Store Report is HERE!
A report to help merchants, developers, & marketers get a better understanding of the ecosystem and benchmarks they can be working towards. 📂
Like, RT, and comment “👋” below if you would like the report via DM.
David Gill retweeted
BMW has begun selling heated seat subscriptions for $18/month. It is currently available in digital stores in the UK, Germany, New Zealand, and South America.
The future is bright for malware developers.
Just found out that AWS Lambda Functions now have their own invoke URLs without integrating API Gateway.
Now if they can just add rate limiting, authorizers and WAF support that would be great👌
Arranging life insurance always feels like a con
1) I don't plan on dying and have an excellent track record so far.
2) If I fuck that up then I'll be ded, so what will I care.
1
2
It's not much to look at right now but potentially this is a sister project to Slantt.co
2
2
What a great demonstration of Slantt.co by a fellow cat lover @robertodigital_ 😁
1
























