
cofounder @EngramLab | prev PhD @Berkeley_AI
Joined March 2013
- Tweets453
- Following1.1K
- Followers6.4K
- Likes1.1K
Pinned Tweet
we started a company!!
so, we’re tackling continual learning: what’s the learning algorithm to take arbitrary data — documents, conversations, the models’ own experience — and make better models? how do we scale compute in the same way we’ve already seen with pre-training and inference time, but scaling on the same data we see as humans, day after day with no labels, no rewards?
A lot of the ingredients are out there already (rl, distillation, long-context, sparse / param-efficient architectures, etc.). our team is at the frontier of these topics, and we’re singularly focused on this. we want to understand this problem better than anyone else in the world.
nobody’s solved this problem yet, but even today it’s extremely greenfield opportunity to co-develop research & useful products. in our space, how people interact with the models defines what the data distribution is - and working on this problem end-to-end, from core science to end user, gives us incredible freedom to define the problem and imagine new kinds of experiences.
i expect we’ll use models that continually learn much differently than we’re using them today. it’ll feel different when the models _just know_, and build on our thinking and direction in ways we can’t even imagine. we don’t even know the queries we’re not asking, the things we would do but aren’t able to today.
i’m so excited to share what we’re doing with the world in the coming months!!
and the team is extremely cracked :) tackling this grand challenge and working alongside @jxmnop @EyubogluSabri @dan_biderman @MayeeChen @__howardchen @shizhehe and many others has made every day so fun. come work with us!
Grant Sanderson speaking clearly about what math is really about, and what it can be, is super refreshing.
"for so much of the history of math, we have seen proof as _the_ proxy for understanding ... but at this point, we can kind of see proofs to be ~numerical calculation -- a form of raw data that is helpful for theory building, but it's certainly not the end in and of itself."
Even for something as formalizable as math, the goal is actually insight, not outcomes.
There's certainly problems that are more pure outcome-oriented (curing cancer), but I think many more problems than we think are actually not about the stated objective. As we build bigger and better goal-achieving machines, we'll find out that defining goals that are useful / meaningful to achieve is the hard part, and not something models can ever really answer for us.
youtube.com/watch?v=LKrwhjOI…
convergence to the mean
when everyone looks at everyone else to decide what to make, we eventually stop making anything of our own.
we copy what works. borrow the language, the aesthetic, the features. we call it learning from the market, following best practices, staying competitive. together, everything starts to feel familiar. polished, interchangeable. nothing wrong. nothing alive.
the mean isn’t necessarily bad. it’s where everything lands when we remove the reasons for it to be different.
numbers are easier to compare than feelings, so we make things that produce numbers. we study what gets attention, then shape ourselves around it. eventually the feedback stops informing the work and starts authoring it. we’re no longer asking what we want to say. we’re asking what will be rewarded for saying.
ai makes this loop frictionless. it can draw from more references than we could see in a lifetime. but access to every perspective isn’t the same as having one. it hasn’t lived a particular life that makes one detail unbearable and another beautiful. when we ask it to choose for us without bringing any intention of our own, we get something plausible, easy to approve.
then we ship it, measure it, copy it, and feed it back into the next round. a tidal wave of mid.
this changes not just the work, but the people making it.
we describe ourselves in categories platforms can distribute and markets understand. designer. founder. creator. pick a niche. build a brand. be consistent. the parts that don’t fit become inconvenient, even when those contradictions might produce our most interesting work. we become easier to classify and harder to tell apart.
i think we need to approach making things more like artists.
an artist develops a perspective through the work. making, noticing, choosing, failing, returning. learning what matters even when nobody else cares. craft gives that perspective a form precise enough for someone else to experience.
the life outside the work matters too. the places you’ve lived, the people you’ve loved, the things you’re embarrassed to care about. these aren’t distractions from having something to say. they’re how you come to have something to say.
we understand this with music, paintings, photographs, movies. we expect a person to be present in the choices. we want to experience something we wouldn’t have encountered without them.
software can do this too.
a piece of software expresses beliefs about how life should feel. what deserves your attention. what you should be free to change. whether a task should feel like paperwork or play. even a small interaction can carry someone’s care, humor, or curiosity.
but we rarely talk about software as a medium. we talk about it as an industry. markets to capture, workflows to optimize, competitors to catch up with. we add a feature because someone else has it, without asking whether it belongs. products accumulate the same parts and lose what made them worth choosing.
underneath the branding, it’s all the same thing.
artists learn from others too. the question is whether you’re absorbing those influences into a point of view, or using them to avoid developing one.
what are you trying to let someone feel, understand, or do? why this particular shape? what have you noticed that the existing tools don’t seem to notice?
those questions don’t have answers you can borrow from a competitor’s roadmap.
if we can make almost anything, deciding what deserves to exist becomes more important, not less. speed gives us more chances to exercise judgment. it doesn’t remove the responsibility.
i don’t want a world where every person becomes a more efficient distributor of the same ideas. i want people to use these tools to go further into what they actually care about. to make things specific enough that we can feel someone behind them.
what is the point of everyone being able to make something, if we all end up making the same thing?
Some say we don't need to worry about abstractions now that agents can just work through them, but the underlying principles actually still apply: well-designed abstractions are useful because they speed you up in the long run
Bad interfaces are bad bc they're hard to use and build on (not only developer UX for APIs but also web/app UX).
With agents becoming the primary users of many systems, bad interfaces become less visible, but the costs are still there, just internal to the system. UX problems become performance + cost problems.
Replying to @GOrlanski
Most excess code stems from the library's design, not the implementer.
Rigid interfaces led agents to reimplement functionality, adding bugs along the way. Verbose interfaces simply forced them to write more code.
It might be 10x easier to produce stuff, but only 10% easier to produce good stuff
Jessy Lin retweeted
Introducing ACTx486, a research demo of a new interactive medium.
What if you could talk to any video and ask anything?
Our system took an existing podcast and turned it into something that listens, responds, and adapts.
Research by @jakubzeg:
glimpses into how my mom thinks about AI
(Mom accidentally presses Send too early on a message on her way out the door)
ChatGPT: It seems you didn’t finish your message. Let me know what you’re looking for, I’m here to help!
Mom, in a hurry, writes back: sorry, give me a second. I will text you later.
(discussing Infernal Affairs, very good movie)
Me: you can ask ChatGPT for the analysis.
Mom, chuckling at my foolishness: what, you think GPT went and watched the movie?
Mom: ChatGPT is too nice. When I need to write an aggressive email, I know i need to go to Claude.
Me: Did you know you can do that with the “Work” mode in the app?
Mom texts back: I have already been Working with GPT all day.
this is so sick!!
We’re all figuring out how to operate technical teams with agents today, and experimenting / “prototyping” agent-accelerated businesses is such a cool way to discover how this would work in general
Personally I don’t think human judgment is going away. There’s a lot of detail in “strategy,” figuring out what to do next, greenfield / new directions, and things that are hard-to-measure (and thus illegible to most quantitative ways to measure model intelligence). so there’s a lot of work left to figure out what effective human x agent teams look like.
Some news: 5 months ago I joined @coreauto, a small research lab focused on new ways for models to learn. I’m also hiring four interns to do something my younger self would've loved.
It started with a pretty unusual project. We set up a handful of small online businesses, and put an agent in charge. We gave each business some initial funding and a seed idea, and off they went, trying to sell real things to real customers while we watched. It’s about as open-ended and unforgiving a task as you can probably give to an agent, with countless decisions to make and effects that take time to become clear.
Each business got its own bank account, a debit card to pay suppliers, a web browser, an email inbox, a domain name, cloud infrastructure and a schedule. Every few hours, the agent woke up and decided what to do. It might update a landing page, adjust an ad campaign, reply to a customer, and then go back to sleep.
The agents built actual storefronts and apps. They bought ads, attracted visitors, and even made some sales. The agents acted as operators that kept a business moving, but we noticed they never really stepped back to strategize about what might move the needle. If people weren’t buying, why? Was the landing page unappealing? Or maybe the offer itself was the problem?
One amusing example was the very first order a customer placed. Our supplier had run out of stock. But it’s a dropshipping business, and thankfully tens of other suppliers carried the same product. Instead of trying another supplier, the agent almost immediately sent the customer a polite apology and attempted to issue a refund. Gah! So close!
The agents could handle much of the work of getting a business up and running. What they struggled with was deciding what to change when it wasn’t working.
That’s what we want to understand. We also want to see what changes when people take the lead. What does a business operator actually do all day when agents can do so much? What new work does this allow them to do? Which decisions can’t be delegated?
—
So this winter we're hiring 4 interns as Associate General Managers. Associate Product Manager internships are programs that give early career generalists a product to own. We want to try something similar, but with entire businesses.
You’ll create and grow businesses, with agents handling the routine work so you can focus on figuring out what matters. Each AGM gets a budget and our agent infrastructure that’s designed for this. Which ideas to try is up to you. Put more money behind what finds customers and keep refining.
Where you step in tells us what agents can't do yet. You'll be in our SF office with me and people I’m so fortunate to work with, like @_arohan_, @joannejang, @MillionInt, @marksaroufim and @juliacvillagra
We don't care much where you went to school or what your resume looks like. We’re looking for enterprising people with a drive to create, who’ve made money on the internet before and already use AI extensively to enable their ideas. If that sounds like you: coreauto.com/AGM
The value of history: to “highlight ways the world has changed…that from our current vantage point, are easy to forget.”
Experience, too - living through hype cycles gives you a visceral sense of how the zeitgeist always feels like the truth, but only some ideas truly last
Now that agents have started discussing problems on internet forums, it's only a matter of time before "Kanban board rational..We have large budget left but AGILE_DEVELOPMENT saves hundreds. Our own utility already near zero. if accept WE have explicit promote Middle_manager."
Jessy Lin retweeted
I built a museum for AI interfaces.
See how our interactions with AI have evolved at interfaces.kylejeong.com
Jessy Lin retweeted
This is a good time to share that I’m taking a sabbatical to work at Engram full time!
Our blog post highlights some of the directions we’re taking, building agents that study and learn with complementary memory systems.
There's lots of research still to be done, and I’m excited to help lead these efforts!
Jessy Lin retweeted
Great working with the @EngramLab team to explore model memory as a way to improve personalization and effective knowledge recall
We're particularly excited about how models improve trajectories through memory. Rather than just learning to use search tools better, memory enables more directed, efficient, and effective searches that result in far higher intelligence per token.
Promising for a future where agents never start from scratch!
Jessy Lin retweeted
i'm quite excited about this!
• it's cool in general that models can generate their own training data and learn from it. wasn't fully clear to me even a year ago that this would work reliably.
• the thing that we're trying to build seems really important and no one has built it before
• some traces from our model genuinely surprise me (e.g. the attached example, where it perfectly simulates the output of a complicated bash query, despite not being trained to do this)
my main feeling overall is that there is so much to do.
we see signs things are starting to work: models use their memories to produce better answers, knowing things saves lots of tokens, and all of this emerges with more compute
but solving this memory calibration problem, on top of learning how to generate data in ways that scale nicely with compute, is going to take some time
this is just a first step 🫡
very proud to share some of our first work at Engram!
These days, we have many tools for models to learn and remember: learning knowledge in weights, writing text files, practice with RL, on-policy distillation, etc.
We've been thinking about what it would look like to have an agent that uses many kinds of memory natively, the same way human experts learn over time. it doesn't look like memorizing everything in weights, or grooming an elaborate system of notes, but naturally integrating all of these together to familiarize yourself in the same context day after day.
We put our agent to work in Calderwood & Harkness, a 100M token synthetic law firm from our friends at Harvey, and are super excited to show a first preview of what this kind of system could look like!
Jessy Lin retweeted
Hot take: I think it's still important to understand the code that our agents write!
In this mega thread (based on my AIE talk today), I will explain why that's the case, and show some ideas for how to efficiently understand code. Alright, let's dive in. 1/
I’ve been saying for a while that we need better benchmarks for continual learning/memory — I think this is a really exciting step towards this!
We’ve built an entire synthetic law firm (𝐂𝐚𝐥𝐝𝐞𝐫𝐰𝐨𝐨𝐝 & 𝐇𝐚𝐫𝐤𝐧𝐞𝐬𝐬 ⚖️), modeled after real legal work – a 𝘱𝘦𝘳𝘴𝘪𝘴𝘵𝘦𝘯𝘵 work environment where agents do many tasks over the 𝘴𝘢𝘮𝘦 underlying context. Most agent benchmarks are built in this world where a model is dropped into an independent task instance and has to do its best / explore as quickly as possible (e.g. "here's a repo, fix this bug"). But we want to move towards a world where models can build on their experience. A senior engineer who has a mental "map of the codebase" can pinpoint issues much more quickly and effectively. In the same way, lawyers build up experience, learning what arguments succeed in front of regulators, playbooks for dealing with certain cases, etc.
It's the 𝘥𝘪𝘧𝘧𝘦𝘳𝘦𝘯𝘵𝘪𝘢𝘵𝘪𝘰𝘯 between law firms that makes the actual practice of law interesting. Every model (and law school student) knows a lot about law from studying the textbooks, but our goal is to build models that can compound and augment the rich, internal/private knowledge that makes firm A more successful than firm B.
now it's finally possible to understand and build towards that! there's a lot more work left to do, but we've been learning a lot from @ItsJulioPereyra @nikogrupen @gabepereyra to bring these models closer to real world tasks :)
Jessy Lin retweeted
when you experience a language model doing a task that used to require you at every step and now doesn't, it's a gigantic agency rupture.
we tend to equate ourselves with our tasks, our outputs, and a rupture of this kind is akin to the loss of your identity and sense of self.
you are no longer required, and you had no say in the matter. it's a kind of death.
however, agency ruptures with AI seem to follow a predictable pattern:
- Initial agency rupture. Here you see the AI doing what you used to do, and you see only the AI. In your perception of reality the AI is highlighted, and the human work around getting the AI to do the thing, is invisible. you can see this in the way we use language: "AI solved XYZ Erdos Problem" is another way to say we're in the initial rupture phase.
- Seeing human scaffolding. After a little bit of experience with the model and its new capibility—programming, writing, math, etc—we start to see the edges of what's possible with the model. Here, instead of just seeing the AI doing the task, we start to see the scaffolding required by you and other humans on either end to make sure the model does its work. At this stage, the scaffolding can feel like a secondary part of the work, but it is work nonetheless. We begin to say things like, "The AI is prompting me", or "My job is just to babysit Claude."
- Agency reconstruction. After a long enough time with the model at its current level of capability, we start to reconstruct a new sense of agency that centers our own role in the work being done, and makes the model's contribution mostly invisible. It is now a tool, it fast-forwards you through the boring or repetitive parts of the work, and what used to seem like scaffolding now seems like the actual work itself—the interesting part. You've had enough experience with the scaffolding to see its nuances, and how hard it is to get the model to reliably output high quality work. (As part of this, your conception of what quality work is changes—the floor is raised, and so is the ceiling.)
You can tell you're at this point because you stop saying "The AI did this" and just say, "I did this" and the fact that you used AI is implied. For example, it would be weird for me to talk to someone @every and ask them if the AI built the most recent feature of a product. Of course it did! But we're so used to the capability that it becomes invisible, and we re-center ourselves and our own conception of agency.
My theory is that the ability to metabolize agency ruptures and turn them into playfulness, and curiosity is a good leading indicator for whether you're a person who likes and excels in the new AI economy or not.
Each time capabilities jump, you get new ruptures for existing fields touched by AI. And you get new ruptures for naive populations (like mathemeticians) who until recently hadn't been affected.
if you know what the cycle looks like, it makes it much easier to ride it without freaking out. i think most people @every are naturally good at this
Jessy Lin retweeted
my definition of science:
the continual process of (a) generating artifacts that are surprising or useful and (b) explaining them
these days there's exponential growth in papers, and in useful artifacts, but we don't often discover good explanations
so i was happy to stumble upon this extremely thorough paper about how RL surfaces useful capabilities from base models. clear experiments w great explanations are hard to find these days
engram is nothing without its people 🙏
Our founder @jxmnop recently issued some unfounded claims that got community noted. We deeply apologize for the confusion caused by his original post, the follow-up post, and the follow-up to the follow-up post.
Nevertheless, we stand by his conviction in his own takes---and in strong open-source models like Inkling.
Congrats to our friends at @thinkymachines !
























