product @elicitorg former: Product @Cruise, VP Product @Teespring. engineer at heart, love outdoors, and my @49ers
San Francisco
Joined December 2009
- Tweets1.1K
- Following693
- Followers1.1K
- Likes2.3K
Really excited to keep advancing Elicit's platform to help enable research backed decision-making. With Routines, Elicit's agent can now be proactive and make sure you're always updated!
More to come..
Decision-making doesn't take place in a static environment, and what was true today might be falsified tomorrow.
To address this, Elicit is launching Routines to help you keep pace with a world of evolving evidence. Ask our Research Agent to keep your work up-to-date, and we'll automatically revise everything from figures to meta-analyses, as soon as new evidence is available.
Read more about what you can do with Routines here: elicit.com/blog/introducing-…
We recognized that reasoning in complex decisions does not have a benchmark. That's why we created BioDecisionBench to measure agents on how well they do here. Excited to use this as a tool to drive not just us, but everyone in this space!
We believe that a few basic reasoning errors underlie many of the mistakes in drug development, example:
1. study design for preclinical studies that doesn’t meet our basic standards for randomized controlled trials
2. assuming that observational evidence shows causality
3. uncritically using performance on biomarkers and other surrogates as evidence of therapeutic benefit
So to try to correct these errors, we built a system that:
1. gets an initial answer from a model
2. has another model critique the answer, based on guidance (derived from state-of-the-art methods research) on how to avoid common biomedical reasoning errors
3. has the original model update its answer based on the critique
We find that this approach sets a new performance frontier on BioDecision (p < 0.05), and thus for reasoning through high-stakes decisions in life sciences.
Kadir Annamalai retweeted
years ago we talked about "verify process not outcomes". the idea was that for the most important decisions you can't really check outcomes, because they're too big and far off, and so you need to rely on checking the process instead. now we've built a reasoning checker and invested tens of thousands of dollars and hundreds of expert hours into creating an internal benchmark for decision quality to see if it helps
the answer is yes - we found that the process verifier often finds confounders, brittleness, and unaddressed sources of bias that made the final decisions worse. this makes it likely that in cases where we can't check the answers, where we have to purely rely on the process, applying this verifier also improves the answers
at high effort settings elicit's research agent now runs this verifier as an explicit step. for many everyday use cases it's fine to be a little wrong. but if you're trying to advance the frontier and understand things that others have not understood yet, or make decisions that lives depend on, noticing these errors is critical
this is only a start, most useful in bio & healthcare, and for checking fairly straightforward errors. our broader goal is to get to reasoning that's as trusted as the reasoning we see in math today, but for strategic decisions where we can't check the answers
AI has become a useful research partner. It can find information, summarize evidence, and suggest ideas. But can it help us think better? Can it help us navigate the complex nuances of high-stakes decisions?
Today, we’re taking the next step toward our mission of scaling up good reasoning for high-stakes decisions with the launch of the Elicit Research Agent, an AI environment built for high-stakes decisions.
You can use the Research Agent today at elicit.com or access it through API (docs.elicit.com) to bring its capabilities into your own workflows.
Kadir Annamalai retweeted
AI is gobbling up any task where the feedback loops are clear, abundant, and immediate. Math, coding, molecular prediction - tasks that are easy to verify.
But the most important decisions in life are not easy to verify. They play out over years and decades. They try to make sense of complex, unpredictable systems where the counterfactual is impossible to know. Outcomes are not available for tasks like these, which means the reasoning carries most of the weight.
Our mission is to radically increase good reasoning for high-stakes decisions. We started by automating research because good reasoning is evidence-based. We are now training Elicit to be explicitly good at weighing tradeoffs and catching reasoning failures that lead to bad, expensive decisions. We built a benchmark, BioDecisionBench, collecting data on such cases, and now release the world's first AI agent trained to reason well for high-stakes decisions about science.
AI has become a useful research partner. It can find information, summarize evidence, and suggest ideas. But can it help us think better? Can it help us navigate the complex nuances of high-stakes decisions?
Today, we’re taking the next step toward our mission of scaling up good reasoning for high-stakes decisions with the launch of the Elicit Research Agent, an AI environment built for high-stakes decisions.
You can use the Research Agent today at elicit.com or access it through API (docs.elicit.com) to bring its capabilities into your own workflows.
So excited to share what we've been building at Elicit! The first AI agent built to help you make good decisions. Elicit's agent is grounded in sources, transparent in its process, and designed with reasoning principles at the core.
AI has become a useful research partner. It can find information, summarize evidence, and suggest ideas. But can it help us think better? Can it help us navigate the complex nuances of high-stakes decisions?
Today, we’re taking the next step toward our mission of scaling up good reasoning for high-stakes decisions with the launch of the Elicit Research Agent, an AI environment built for high-stakes decisions.
You can use the Research Agent today at elicit.com or access it through API (docs.elicit.com) to bring its capabilities into your own workflows.
We built the Elicit Research Agent for these long feedback loops. When output verification is uncertain, you have to rely on:
1. Trustworthy sources that you can inspect
2. A transparent process that the agent went through
3. Good reasoning principles natively built in
So excited for the new wave of problems where AI can help us. Elicit is at the frontier of doing this for decisions. Try us out at elicit.com , and let me know what you think!
Just want to do a shoutout for the PM Career led by @shreyas . Just took it this weekend and really opened my eyes to how to think about evolving your product career. And spending live time with Shreyas is golden. High insights / minute.
Kadir Annamalai retweeted
I’ve been taking the driverless robotaxis around SF.
They’re safe and fun. Also slower and less efficient than Ubers, because they’re strict rule followers.
What’s wild is that we’re back to a world human-taxis are expensive and hard to find. Suddenly it’s easier and cheaper to call a robotaxi!
Last night I waited 30 minutes and two cancels from Uber before I called the Cruise and it came in 5.
It’s fascinating to watch a labor shortage get solved by robots.
Kadir Annamalai retweeted
Thrilled to announce that the California Public Utilities Commission just approved @Cruise for fared operation 24/7 across all of San Francisco!
It’s a huge milestone for the AV industry, but even more importantly a signal to the country that CA prioritizes progress over our tragic status quo.
We share the CPUC’s commitment to delivering safer, cleaner and more accessible transportation options, and remain committed to collaborating closely with regulators to push toward this critical goal.
Let’s Cruise!
Kadir Annamalai retweeted
Last year @Cruise AV's apparently drove 10x more driverless miles in California than any other company.
We grew about 50x year over year. Lots of challenging bottlenecks are now out of the way, but I'm sure there are more ahead.
Source: dmv.ca.gov/portal/news-and-m…
Kadir Annamalai retweeted
Feels like we got robbed of an epic NFC Championship game. Would've been nice to see these teams at their best
Kadir Annamalai retweeted
Speaking of AI breakthroughs, this is what scaling fully driverless cars—all powered by crazy cool ML—looks like.
We now regularly deploy 100+ fully driverless @Cruise cars (no driver, just passengers) concurrently on the streets of San Francisco, scaling faster every day.
Kadir Annamalai retweeted
A driverless @Cruise AV (named Lemur) was dangerously cut off by another driver but managed to avoid a collision.
Lemur went from 24mph to zero in ~10m with nearly instant reaction time. Based on the physics here, it's unlikely many of us could do the same.