chaos wrangler | managed to be more interesting on the Internet | opinions mine 🇺🇸🇹🇼
Beach
Joined September 2008
- Tweets9.8K
- Following402
- Followers428
- Likes8.3K
lin classon retweeted
We are often so focused on being right that we miss the advantage of simply avoiding stupidity.
lin classon retweeted
Don’t ask whose fault this was. Plenty of time for second-guessing and recriminations. Ask instead, what can you do? For my part, that means telling the truth as I see it, as long as I can. The media will be under a lot of pressure to toe the line; don’t capitulate in advance
Is there a support group for NYT Connections players where we could commiserate over "omg how dumb is purple today?"
I like to imagine that social media managers have this group chat where they egg each other on and also high five each other
Just found out I have been paying Consumer Reports digital membership over a decade. My last login was in 2011. 🤡💀
lin classon retweeted
In the latest article about Kamala Harris’ staff turnover in the Washington Post, former staffers complain that she expects her team to be just as prepared as she is, and to be able to explain everything they put in her briefings and schedule.
How Telegram Became a Sanctuary for Domestic Terrorists propublica.org/article/teleg…
lin classon retweeted
OK, here is my best guess on the state of LLMs:
- The scale increase between gpt-3 and gpt-4 was 100x
- Doing that for the next model is going to be very hard
- We're nearly out of general language tokens. So let's say we can 2x that. And perhaps get more proprietary tokens and get to 3-4x. And do a lot of data cleaning and get to 6-7x.
- A 100x training run also requires a Gigawatt datacenter which we don't have yet
- Synthetic data is great, but it's not clear how that can be used for general language. I suspect this is why both OAI and Anthropic are focusing on math and code which can be improved via various "synthetic" compute methods (simulated data, or recursive self improvement of some sort)
- In the meantime, there is focus on getting more learnings from the same data. Perhaps there is a breakthrough there but I've not heard of it
- Planning can be pushed to inference in some domains (e.g. coding) which we're starting to hear about. But again, not clear how much this buys.
- Moronic policies like SB 1047 are threatening to slow all this down.
So tl;dr I don't see where the 100x jump will come from for general language reasoning. This is why we're seeing a focus on math and code. I'm glad teams are working hard at new algorithmic unlocks.
(btw, this is pure speculation, would love to know where I'm wrong!)
Is there a support group where we can go and complain about the NYT Connections game?
#Purplefirst
A lot of food for thought here 👇
OK, here is my best guess on the state of LLMs:
- The scale increase between gpt-3 and gpt-4 was 100x
- Doing that for the next model is going to be very hard
- We're nearly out of general language tokens. So let's say we can 2x that. And perhaps get more proprietary tokens and get to 3-4x. And do a lot of data cleaning and get to 6-7x.
- A 100x training run also requires a Gigawatt datacenter which we don't have yet
- Synthetic data is great, but it's not clear how that can be used for general language. I suspect this is why both OAI and Anthropic are focusing on math and code which can be improved via various "synthetic" compute methods (simulated data, or recursive self improvement of some sort)
- In the meantime, there is focus on getting more learnings from the same data. Perhaps there is a breakthrough there but I've not heard of it
- Planning can be pushed to inference in some domains (e.g. coding) which we're starting to hear about. But again, not clear how much this buys.
- Moronic policies like SB 1047 are threatening to slow all this down.
So tl;dr I don't see where the 100x jump will come from for general language reasoning. This is why we're seeing a focus on math and code. I'm glad teams are working hard at new algorithmic unlocks.
(btw, this is pure speculation, would love to know where I'm wrong!)
Raise your hand ✋ if this is what you have in your mind when you see the headline
Family on blackberry hunt stumble across python bbc.in/3AJ30M4
