Leading Product for CoreML Group @yelp Prev: ML Researcher @McMasterU, ML EM @etsy | Investing and Building at the Frontier @frontier_fund
Toronto
Joined May 2010
- Tweets4.9K
- Following3.9K
- Followers464
- Likes4.1K
Using grokbot as an orchestrator for claude code + codex
Persistent VM and UI is pretty nice for driving multiple agents
Rajvinder (Robin) retweeted
how did NLP researchers deal with their entire field being solved by LLMs?
perhaps we should look into that as inspiration for other fields now being replaced by LLMs?
Rajvinder (Robin) retweeted
I think the biggest counter example to "agents do random search" is how insanely efficient Idea Descent is in the space of Experiments/Ideas, i.e. within O(100) steps they arrive at Sota artifacts that entire communities could not find before
Fork-and-Flush: help your agents escape Idea Basins!!
I love thinking of AutoResearch as Idea Descent: a metric of interest goes down, as a function of ideas/experiments, where the agents get pulled towards one idea or another. But we've seen very often an agent develop this weird tunnel vision, and honing in just one type of "idea", kinda like GD stuck in a local minimum.
We studied this and seems we can fix it with Fork-and-Flush, a super simple and kinda cute idea. Every so often, fork the agent into T copies in the same workspace but empty the context, let them run for a bit, and continue from the best one.
If you like the GD analogy, it's basically basin hopping en.wikipedia.org/wiki/Basin-….
BTW we found a silly way of also plotting "idea basins": an LLM rewrites every experiment as a spec and answers a few thousand binary queries "is Experiment A closer to B or to C?". Then we embed this NXN matrix into 2D and indeed see in many cases that each run saturates into a small region and keeps polishing it.
We call these idea basins!
At equal compute Fork and Flush beats single runs and best-of-N on 12/13 long-horizon tasks (some 4+ days), and in my own research projects it's been working really well!
BTW we only use it at test time, but it could also be part of training or RL.
Led by @jackcai1206 @christos_ziakas during Jack's internships with us, along with @vkontonis @ShivamGarg91462 @Tea_Pearce Sid Sen and Akshay Krishnamurthy.
Codex notifications are really bad - there’s no reason to receive a notification click into it and not have anything to review
Great way to get blocked from the notifs in the first place
Rajvinder (Robin) retweeted
Canada claiming a spot at the forefront of AI sounds like a joke until you realize the entire modern AI industry came out of one Toronto lab.
Geoff Hinton spent the AI winter at the University of Toronto, funded by a Canadian government program that kept backing neural networks when every US funder had written them off. In 2012 his students built AlexNet, the model that started the deep learning era. One of those students was Ilya Sutskever.
Then the drain began.
Google bought Hinton's three-person startup for $44 million in 2013. Ilya went with it, then co-founded OpenAI, which is now fielding investor offers at $1.2 trillion. Aidan Gomez, a University of Toronto undergrad, co-wrote the 2017 transformer paper that every frontier model runs on, the same architecture underneath Anthropic, whose investors expect a $2 trillion IPO next month.
Add it up and Canadians built the foundational architecture, the breakthrough model, and helped start the labs racing to be the first $2 trillion AI company. Almost all of that value sits in San Francisco.
What Canada kept is Cohere, Gomez's lab in Toronto, reportedly raising at $20 billion right now with Ottawa itself in the round. Put $20 billion next to $1.2 trillion and $2 trillion. The country that invented the field kept about 1% of it.
So Carney's line is funny in a way he didn't intend. Canada has been at the forefront of AI for twenty years. It just kept exporting the forefront.
Rajvinder (Robin) retweeted
It’s easy to dismiss Jev it as “just a classifier”.
But people (me included) who have been training encoder-style models for classification for many years know they were usually special-purpose and limited in some way.
The breakthrough of Jev is that it generalizes well.
And I’d say the secret sauce is probably more in the data than in the training algorithm. (Plus a nice API design on top of it.)
poor guy claim to have built Jev a year ago but no one cared, and now Jev stole all the thunder
many people are saying “you gotta tell your story” or “marketing is important”, and they just completely missed what actually made the difference here
i just looked into this laya model laya.convaiinnovations.com/ and:
- it only supports 512-1k context… a lot of use cases won’t fit at all
- evaluating the model directly shows its accuracy is as good as a coin flip. in order to get good results, you need to first fine tune it
i’m sorry, but that’s not Jev
there’s a massive gap between an interesting research and a useful product
you can “tell your story” all you like, but you can’t blame Jev for stealing your thunder when Jev did all the work to make a well-packaged solution anyone can just grab and go
Jev is not completely new from an academic sense, just like how ChatGPT was not the first LLM
don’t underestimate the effort and value in putting together something that’s actually good enough for adoption - it makes all the difference
Rajvinder (Robin) retweeted
hey my guys if you are interested in learning more about synthetic data and how to generate it to great effect I got just the right stuff for you fresh out of the oven
pretty packed interview + do check out all the open stuff the huggingface team made for you:
on the menu:
- 0:00 : synthetic data in pre-training LLM
- 3:36 : Joel Niklau ML Research Engineer at Hugging Face
- 4:50 : how to approach research problems at the edge?
- 7:26 : the label here is still wrong is should be “benchmarking work overview”
- 9:10 : what are the first few steps to get started building a private benchmark.
- 11:18 : synthetic data pipeline and it’s impact in benchmark making
- 13:19 : Synthetic Data Playbook Presentation Overview
- 19:30 : what is rephrasing in synthetic data?
- 21:05 : why are large scale ablation studies on data pipeline becoming feasible now?
- 24:00 : how is rephrasing done?
- 25:42 : experimental results
- 27:58 : why does synthetic restructuring hurt common-sense knowledge?
- 30:10 : what’s the value prop behind doing rephrasing?
- 32:16 : is math a type of rephrasing?
- 38:02 : what’s up with closeness between generator vs the student?
- 39:40 : synthetic data + mixing real data
- 41:20 : what drive this real-to-synthetic ratio?
- 43:50 : what other structured formats could be explored for synthetic data?
- 45:20 : is the order of datasets used for pretraining also affecting the distribution?
- 46:40 : would adding multilinguality be considered a form of rephrasing for LLM?
- 50:40 : which non-synthetic dataset should you mix in?
- 1:03:20 : do model-specific writing habits survive the rephrasing process? + sawing off the edges?
- 1:08:30 : would their be a gain to have a purpose built 270M model for being a generator?
- 1:14:30 : does generator diversity still help as the student model scales?
- 1:26:30 : ❤️❤️❤️
- 1:27:15 : useful of synthetic data for post-training?
- 1:28:50 : what’s next for this direction and for the field?
Synthetic data derived from production user data of consumer AI tools is used for training. I’ve heard this rumour from both large labs’ employees.
In particular, if you’re doing something “interesting” like working on complex math/business/software/bio problems you’re dramatically more likely to get trained on because they filter/up-weight towards those usecases where the model has the most to learn.
Even in ZDR and “we won’t train on you” regimes, derivative data is usually carved out. The promise is only not to train on exactly the data you put in, rewritten data is fair game.
Rajvinder (Robin) retweeted
Synthetic data derived from production user data of consumer AI tools is used for training. I’ve heard this rumour from both large labs’ employees.
In particular, if you’re doing something “interesting” like working on complex math/business/software/bio problems you’re dramatically more likely to get trained on because they filter/up-weight towards those usecases where the model has the most to learn.
Even in ZDR and “we won’t train on you” regimes, derivative data is usually carved out. The promise is only not to train on exactly the data you put in, rewritten data is fair game.
We congratulate Levent Alpöge and Tristan Buckmaster on their remarkable mathematical work.
We (the researchers and the agents) did not see any of their work through any means until they released it publicly — in particular, no specific user data was accessed in order to solve this problem.
While unlikely, we cannot rule out that de-identified data derived from their usage of our products helped improve our models.
However, our proofs differ significantly and even the precise results proved are different in the Euler case (forced vs. unforced).
Rajvinder (Robin) retweeted
Of course that’s your contention. You’re a first-year San Francisco MTS. You just got finished readin’ some scaling-laws paper, Jared Kaplan, probably. You’re convinced data and compute are the only constraints, whoever trains the next frontier model wins, everybody else is just buildin’ wrappers.
You’re gonna be convinced of that ’til next month when you get to Alex Karp, and then you’re gonna be talkin’ about how open-source models are drivin’ the cost of intelligence to zero and the real value’s in the ontology of the firm. Takin’ those models, givin’ ’em proprietary context, reinforcement learnin’ on the company’s actual workflows. Specialized models for every business function. Deployed. Integrated. Suddenly the model’s a commodity, the business is the training environment, and you’re an organizational theorist.
That’s gonna last until next year- you’re gonna be in here regurgitating Ray Kurzweil, talkin’ about, you know, the post-human corporation and the enduring economic value of human connection. Suddenly the singularity’s inevitable, intelligence is abundant, and the last real competitive advantage is knowin’ how to get three people to agree on a meeting time.
“Well, actually, I won’t, because Kurzweil fundamentally underestimates the importance of proprietary context, especially the accumulation of-”
firm-specific decision traces that can’t be reconstructed from the final state of the database. Yeah. I listened to the podcast too.
Rajvinder (Robin) retweeted
Steve Kerr says Roger Federer taught his Warriors team the secret to sustained success wasn’t outworking everyone, it was building a life around your craft you never want to escape from
“I’ll tell you a great story about Roger Federer. We were playing in China with the Warriors in 2017. Roger was in Shanghai for the Masters tournament, so we invited him to come speak to the team.”
“Draymond Green asked him, ‘How do you sustain success? How have you managed to win majors 20 years after you first won one?’ Everyone was expecting him to say, ‘I work harder than everybody,’ the will, all the clichés.”
“Roger Federer says, ‘I get up every morning and I make breakfast for my kids. Then I take them to school and drop them off. Then I go practice tennis for about two hours, and I’ve figured out a really good routine where I don’t destroy my body, but I can get all the work in I need.”
“‘Then I go have lunch with my wife. In the evening, we cook dinner together. The kids are all home from school. We ask them about school. There’s so much joy in the house. We put them to bed, and then I put my head on the pillow and go, “Man, that was just a great day.” And I’ve been doing that for 20 years.’”
“And it was like, yes. That’s it. That’s the formula. That’s what leads to sustained success. He loves tennis. He loves his family. He loves life. It’s not outworking everybody and banging your head against the wall, ‘I’m going to be better than everybody.’ It’s allowing your natural talent to shine through with a work ethic, with a great family life, with perspective, with peace, with mindfulness.”
“It’s perspective. It’s joy. It’s passion. It’s mindfulness. It’s an awareness that we are truly lucky. To get the most out of ourselves, it’s our daily rituals and experiences and love and joy.”
Rajvinder (Robin) retweeted
I think we should revisit outrageously small neural nets.
I needed a 10k tok/s CPU model for data processing. So I gave Anthropic claude code a pile of tokens to build one.
It made three interesting discoveries: huggingface.co/gdiamos/amx-r…
Rajvinder (Robin) retweeted
Edward, the creator of LoRA (fine-tuning), wrote a blog post on how to post-train open-source models.
I strongly encourage checking this out for anyone who's post-training.
