๐๏ธ Founder of Craiyon ๐ฅ Author of dalle-mini
Joined February 2012
- Tweets2.3K
- Following359
- Followers14.3K
- Likes4.6K
Looks cool but why is it so expensive compared to Jev?
I guess purpose-built model is the reason of such a low price for Jev.
introducing ๐๐๐๐: our first models trained by @cloudflare's workers ai team.
today, we're releasing two fast and accurate decision models that top the benchmarks for quality and latency. use them hosted on workers ai or grab the weights from @huggingface, because we open-sourced it too.
blog.cloudflare.com/clef-decโฆ
Somehow I find myself always cheering for the underdogs, who change every month in AI these days.
Anyway right now can't wait to try the new Gemini model!
Boris Dayma ๐๏ธ retweeted
Jev can generate game levels in real time.
Faster and cheaper structured output could be a big deal for game dev!
Sharing some experiments on optimizers: PSGD Kron
Muon is slowly replacing Adam, now time to work on a better optimizer.
One thing that helped Muon vs Shampoo is its simplicity.
Shampoo was easy for me because I use TPUโs and there was an official Google repo for it, well optimized by top researchers (@_arohan_ + JAX team). It contained a lot of things I didnโt understand but as a user it just worked great, in particular when needing complex sharding on TPU pods.
For people on Pytorch, it seems there was never a good enough port. Also the many settings required either deterred people or lead to sub-optimized performance.
I hope there will be more people doing research on this optimizer and try to simplify it, making it performant and easy to use on both Pytorch and JAX (bringing shampoo hacks).
There may even be a simpler Muon/Shampoo equivalent for Kron that works well enough and is more simple.
Many thanks to:
- @evaninwords for the optimizer port to JAX and leading these experiments
- @HessianFree for all the optimizer ideas
- @pcuenq for running experiments
- @eraznafre for the crazy JAX optimizations
- Xilin for the official PSGD repo
- @GoogleAI TRC program for the compute
- @wandb for the training infra
Full report here: api.wandb.ai/links/optim-teaโฆ
Getting a bit annoyed that me and Codex/Claude donโt have the same standards in code.
For example they refuse committing code with failing tests while Iโm ok pushing that to prodโฆ Need to add "Donโt argue and just do what I ask"
This is quite a big deal!
Usually for smart & cheap Google was unbeatable with Gemini 2 flash lite series (2.0/2.5). They later increased price a lot (unit + mandatory thinking) making cheap devs like me stick to previous versions.
Gpt-nano series were bad so time for a new test
Claude/Codex have helped me update codebases very fast, but for new repos/scripts/logic, not completely sure of the positives.
1/ it can often one-shot a working solution, but itโs often over-designed, lots of abstractions, tests, security paranoia where not needed, etc
2/ I used to develop everything through Jupyter notebooks, explore, test, etc. It was actually quite fast, great for exploring deeply my workflow, edge cases, etc
3/ now, maybe because of laziness, maybe thinking it will be faster/better/more efficient, I resort to Claude/Codex. Itโs likely ok for quick or non-important features (personal admin dashboard/monitoring), but Iโm not convinced of the long term value
4/ "but what about karpathy auto-research loop workflow"?
I think it has its use cases, but you cannot prompt forever "make it better, here is the target" on every type of problem. Also on any specialized project, I donโt think LLM can reach the insight/creativity of humans, and those are better triggered through step-by-step development / visualization of the full pipeline vs just a top level overview