machine learning • search • abstraction • cog sci
Texas
Joined December 2021
- Tweets668
- Following621
- Followers137
- Likes4.8K
This is a nice analysis, as expected, from Hamel and Antaripa. Good illustration of more traditional analysis in the newer domains of LLM evals and decision models.
For critique I would say it's pushing the boundaries of decision models by using a very broad prompt and single decision vs breaking the trace eval into several smaller decisions.
Really appreciate the ongoing work from @HamelHusain and @doesdatmaksense. They are so helpful for advancing this eval field.
hamel.dev/blog/posts/decisio…
Patterns in System One space:
. Zero shot general classifier
. Zero shot general slot filling
...
What's next?
one thing that made this click for me — with decision models, you have to define categories and it returns probabilities that the input matches those categories.
but so much of the data we interact with is completely new to us. how can you define categories for something you've never read?
with clairic, you can extract values directly from the inputs, without any premeditated categories. the pipeline becomes clairic to extract values → clef to make decisions with those values → LLMs to take actions on those decisions. all of these models run on Workers AI.
This space of possible models was alluded to by @CompleteSkeptic on the @latentspacepod recently. It's been on my mind a lot. It's going to grow faster than the community is able to orient and integrate the ideas. Very interesting area of research right now.
Using the word Taste when talking about AI seems like sidestepping the hard work of describing more specific skills which are a value add.
Phrases that need to see increased use given some techniques growing in popularity:
. Black box optimization
. Bayesian optimization
. Derivative free optimization
. Stochastic hill climbing
A huge part of leveraging AI is being able to evaluate and encapsulate risk.
High risk? Slow down and be deliberate.
Low risk? Just rip and lfg
Building custom local tools is a crazy unlock. It's so useful for analysis, research, and development in data centered domains.
I was doing this at a startup before code gen with AI took off. It was a useful tactic for me at the time. Looking back, it was so slow and primitive compared to what you can do now.
Codex with training wheels?
Cursor in learning mode?
In coding agents, I would like to see a mode that only allows creating one function at a time.
The early days of coding agents were only that good. Guide it a function at a time. Carefully arranging modules and then populating them. It was an interesting experience. The cursor tab model was similar.
Having a new mode could orient engineers to learn good composition practices while also still getting some advantages of code gen.
Imagine putting junior engineers, or any engineer entering a new domain, on codex with training wheels and encouraging more learning and synthesis.
Cool framings for Jev and System One models that I've run across:
Zero shot general classifier
Evidence based judgements
Common sense decisions
I've been tinkering with the chat agent for my custom Hacker News index. The mechanics are getting a lot easier these days, and it's mostly about the prompt, tool design, and tool decsriptions.
chat.orangewords.com?x=2026-…
Recently I spent time exploring Jev and their System One Model concepts. If you only read one thing about system one models, this is one of the best for engineers:
docs.typesafe.ai/concepts/ho…
When you are trying to optimize a developer workflow, put friction in the paths of the agents not the humans.