Clinical AI Engineering @UpToDate
Greater Chicago Area
Joined April 2017
- Tweets618
- Following667
- Followers145
- Likes2.8K
Pinned Tweet
Three updates:
(1) I've been building @UpToDate Expert AI since last summer and it's now helping doctors across thousands of hospitals
(2) We are accelerating our release pace as the number of things that can meaningfully improve healthcare and can practically be built quickly and safely multiplies with every major AI capability jump (and I hope that doesn't stop)
(3) We have multiple full-stack software engineering openings on the team. If you are looking for a place where your hard work will still have a lasting impact, DMs are open
Happy to answer any questions below
Just learned that fully offline versions of @UpToDate
are orbiting Earth on the ISS and roaming the oceans. And they are overdue for a nice upgrade! Very exciting!
"Less is more" will be the ultimate outcome in everything, and the most entertaining one! We just need to build the most powerful tech ever to finally figure out how little we actually need :-)
Soon we'll see how boring it is to live in godmode. And maybe then we create a new game, again?
If you're not in medicine, you might not know @UpToDate . Almost every physician does, almost everywhere in the world, and directly or indirectly, all of us have benefited and continue to benefit from it.
The same is true of its 7,500+ contributors. Many started their careers relying on it and later wanted to give back.
Below is one example: an inside story from Dr. Vincent Rajkumar, one of the clinical contributors behind it.
Here is the inside story on how @UpToDate does what it does.
From my experience as an author/section editor for UpToDate for 20 years, here is how it works. It’s amazing!
Every day a team of people @UpToDate scour the top medical journals (and major meeting abstracts) for articles of importance. In my case they pertain to myeloma and related disorders. Any article that is felt to be important is sent to a expert MD medical editor at UpToDate who reviews the paper. If the information is felt to be important, the relevant chapter is updated with this content, summarized in a few sentences in Word track and then sent to the author of the chapter who is a top expert in the field.
As an author I have to review the update in detail and then make a decision whether to include the new information, and whether the way it has been incorporated is correct or needs to be edited.
Once I make a decision, it is then reviewed by an expert section editor. For 15 years for me, the section editor was Dr. Robert Kyle! (Currently I’m the section editor and we have about 6 experts serve as authors on over 40 chapters pertaining to myeloma and related disorders). The section editor independently decides whether the author made the right call and either approves or we have a back and forth till we agree.
The updated chapter is then reviewed again by the expert MD editor at UpToDate and then posted.
For important phase III trials or FDA approvals we usually make the update within a week. Sometimes by the next day! With so many important papers, every week there are 2-3 updates pertaining to my chapters that go through the above process! It’s hard work but it keeps me ridiculously current. And more importantly it keeps the content accurate and current. This is not random addition of new information. It’s vetted information that’s added.
Every UpToDate chapter is also peer reviewed annually. Each chapter also undergoes an annual author and editor review to make sure that the overall content and flow is good and to delete content that may be no longer important or accurate. The extraordinary process and rigor involved is why it’s such an invaluable irreplaceable resource. It is AMAZING and hard to replicate.
UpToDate is easy to navigate. But the newly launched UpToDate Expert AI takes it to the next level! It makes the entire content available to query just like you query a top ranked LLM. But it’s not searching the random internet or a stack of published articles. It’s primarily trained and derives vetted and adjudicated UpToDate content. It will not randomly hallucinate. It will not list 10 options for treatment newly diagnosed myeloma but will give you what experts currently feel is the best option based on the latest evidence. All answers are referenced and pointing to specific content sections in UpToDate (which also has links to source material). So you can immediately get all the background information if you need more detail.
Working with UpToDate has helped my career immensely. For the last 20 years it has been impossible for me to not know about every important study related to myeloma and related disorders as they get published! For 20 years, I have received a precise summary of every important article published in my field. More importantly I am forced to read them, review the source material, and make a judgment call. As a result I remember this stuff. A gift indeed!
Also many personal/patient stories like this one:
xitter.cf/UDarnTuten/status/2111…
Last week Ì ended up at the local trauma center (critical access hospital) with hives covering every part of my body. No known cause. I took 50mg of diphenhydramine when they started. It was fuel. So off to the ED we went. Next closest ED is 25 miles away. Staffed by a FP resident. We discussed idiopathic urticaria, and how to treat it. He pulls out his Ì phone and tells me he is going to look it up on Up To Date. I had zero issue with him telling me he needed to look it up. One shot of decadron later and an Rx for a few days of steroids Ì was out the door. I have zero cares if a doctor has to look something up. Please do.
Evgeniy (Gene) Padaliak retweeted
Last week Ì ended up at the local trauma center (critical access hospital) with hives covering every part of my body. No known cause. I took 50mg of diphenhydramine when they started. It was fuel. So off to the ED we went. Next closest ED is 25 miles away. Staffed by a FP resident. We discussed idiopathic urticaria, and how to treat it. He pulls out his Ì phone and tells me he is going to look it up on Up To Date. I had zero issue with him telling me he needed to look it up. One shot of decadron later and an Rx for a few days of steroids Ì was out the door. I have zero cares if a doctor has to look something up. Please do.
If you're not in medicine, you might not know @UpToDate . Almost every physician does, almost everywhere in the world, and directly or indirectly, all of us have benefited and continue to benefit from it.
The same is true of its 7,500+ contributors. Many started their careers relying on it and later wanted to give back.
Below is one example: an inside story from Dr. Vincent Rajkumar, one of the clinical contributors behind it.
Evgeniy (Gene) Padaliak retweeted
When I was writing Unreal Engine 1, my time was divided as follows:
- Around 30% went to seriously deep coding on hard problems: assembly language texture mappers, C++ BSP trees, data persistence, object oriented systems design, and networking.
- Around 70% was just typing and refactoring large bodies of code needed for a complete engine feature set. This was not hard, and while I was doing it my mind was mostly on the deeper coding topics, thinking and planning for what's next. It didn't feel much different from vibe coding today, except that it was just me typing code on an IBM Model M keyboard rather than prompting it and reviewing the results.
So, it has always seemed perfectly logical that we would rely on AI for that 70%, once the AI could do it with a similar quality standard as we could, which has never been perfection. It came as no surprise that, after a few years of AI model evolution, it was able to take the reigns on that 70% and free up programmers' time for more valuable tasks.
What's more of a surprise is the emergence of AI as a sort of insta-Carmack able to solve hard technical problems quite cleverly and often cleanly, and that it's able to reason about and work in complex 3D scenes both by operating the tools and by directly understanding the raw data files.
I abandoned assembly language in 1998 when out-of-order CPUs made much simpler C/C++ code faster, and never I missed it. If prompting through architectural documents, concept art, and design plans can transcend typing code and tweaking vertices, that will enable developers to spend their time on higher-level, higher value technical and creative tasks.
What matters is the end product, whether it's a game or an engine. Anything developers can do to try out more ideas and make better software and games faster is a win for everyone.
Evgeniy (Gene) Padaliak retweeted
A fact, not widely appreciated, is that the content of UpToDate (copyright protected) is not included in the data bases of commonly available open access AI formats (Claude, ChatGTP, Open Evidence, Gemini, Co-Pilot.
Any clinicians here using UpToDate Expert AI? I'm one of the engineers behind it. My teammates and I would love to hear from you.
Over the past few weeks we've rolled out v2 of UpToDate Expert AI to all our users in the US and Canada across thousands of hospitals.
Months of work went into clinical reasoning, more complex diagnosis and treatment questions, drug dosing and a long list of fixes based on your feedback. We're really happy with this one. It's our best release yet and we need your help making the next one even better.
If you've tried the new version (or this post gets you to!) tell us what you think. Any bugs, issues with the answers, features you're missing, that one thing that annoys you or just a question you'd like to ask. We'd love to know and we'll be here troubleshooting and fixing what we can in real time. For the parts that need expert clinical judgment, our staff clinicians will be helping too.
I'm relatively new to this product and don't have a wide user network here. If you know someone who uses UpToDate, I'd appreciate you passing this along.
Evgeniy (Gene) Padaliak retweeted
Dusting off this tweet from April because this gap in shared understanding of LLM capability is *widening*. It's now less "two groups of people speaking past each other" and more a sharp funnel.
- Napkin math somewhere around 6B people (~75% of the population) have barely come in contact with LLMs at all.
- Around 1-2B (~20%) are casual and infrequent users of free-tier ChatGPT-like products. This group sees derpy chatbots and treats them a bit like a better Google search, a writing aid, or etc. Maybe an agent tries to book you a flight. I have non-tech friends who (reasonably, imo) say they have not much use for it at all. Even many professionals outside of math&code are in this tier. For example, execs and many other functions spend a lot of their time talking to other people, so while they understand what is happening intellectually, it is still second-hand and a bit abstract.
- Now we get to professional use of frontier-grade LLMs in math&code. Somewhere around 20M people (0.2%) see first-hand that large, complex projects that used to take them weeks/months can now be completed by agents with a prompt. Building apps, copying apps, translating apps, decompiling apps from binaries... This has all happened very quickly and recently - less than 1 year ago, I was writing code manually by hand, typing memorized computer code commands into a code editor character by character, occasionally pressing Tab to autocomplete a little chunk of code.
- And finally we get to the ~5,000 people (~0.00006%) with access to frontier-grade systems internally. The external world has seen the preview. It looks like swarms of thousands of agents collaborating over weeks on software mega projects: minting zero days, running cyber attacks and defenses at machine speeds, discovering new science, advancing the frontier of mathematics. Things that would have taken top professionals in the industry years of work. Meanwhile, human review and comprehension are starting to fall behind. For example, people are still involved in the "archeology" of the OpenAI-HF incident from many months ago. Mathematicians may be poring over the 722 manuscripts on frontier mathematics for a while.
The funnel is driven by a combination of factors:
1. The impact scales with ambition, problem size, and horizon. A question with a paragraph answer barely stresses the system. You need a reservoir of big, difficult problems that you really care about. This aspect drives the consumer / professional dimension of the funnel.
2. The jaggedness of the system (which I have written about a lot separately). Capability peaks in domains that are digital, verifiable and economically valuable. This is because LLM capability emerges from reinforcement learning on verifiable rewards on a curated environment mixture driven by revenue potential. This aspect primarily drives the area (e.g. math&code) dimension of the funnel.
3. Access. Free-tier, paid-tier, internal.
So this is the weirdness of the moment. The general public has mostly not interacted with these systems. When they have, it looks like a derpy chatbot. The majority of professionals still see only a modest uplift. And a small sliver of professionals are experiencing the vertigo of the curve going vertical. And it is all happening at the same time.
Judging by my tl there is a growing gap in understanding of AI capability.
The first issue I think is around recency and tier of use. I think a lot of people tried the free tier of ChatGPT somewhere last year and allowed it to inform their views on AI a little too much. This is a group of reactions laughing at various quirks of the models, hallucinations, etc. Yes I also saw the viral videos of OpenAI's Advanced Voice mode fumbling simple queries like "should I drive or walk to the carwash". The thing is that these free and old/deprecated models don't reflect the capability in the latest round of state of the art agentic models of this year, especially OpenAI Codex and Claude Code.
But that brings me to the second issue. Even if people paid $200/month to use the state of the art models, a lot of the capabilities are relatively "peaky" in highly technical areas. Typical queries around search, writing, advice, etc. are *not* the domain that has made the most noticeable and dramatic strides in capability. Partly, this is due to the technical details of reinforcement learning and its use of verifiable rewards. But partly, it's also because these use cases are not sufficiently prioritized by the companies in their hillclimbing because they don't lead to as much $$$ value. The goldmines are elsewhere, and the focus comes along.
So that brings me to the second group of people, who *both* 1) pay for and use the state of the art frontier agentic models (OpenAI Codex / Claude Code) and 2) do so professionally in technical domains like programming, math and research. This group of people is subject to the highest amount of "AI Psychosis" because the recent improvements in these domains as of this year have been nothing short of staggering. When you hand a computer terminal to one of these models, you can now watch them melt programming problems that you'd normally expect to take days/weeks of work. It's this second group of people that assigns a much greater gravity to the capabilities, their slope, and various cyber-related repercussions.
TLDR the people in these two groups are speaking past each other. It really is simultaneously the case that OpenAI's free and I think slightly orphaned (?) "Advanced Voice Mode" will fumble the dumbest questions in your Instagram's reels and *at the same time*, OpenAI's highest-tier and paid Codex model will go off for 1 hour to coherently restructure an entire code base, or find and exploit vulnerabilities in computer systems. This part really works and has made dramatic strides because 2 properties: 1) these domains offer explicit reward functions that are verifiable meaning they are easily amenable to reinforcement learning training (e.g. unit tests passed yes or no, in contrast to writing, which is much harder to explicitly judge), but also 2) they are a lot more valuable in b2b settings, meaning that the biggest fraction of the team is focused on improving them. So here we are.
Evgeniy (Gene) Padaliak retweeted
I’m no longer sure what “intellectual property” means when intellect becomes free.
Evgeniy (Gene) Padaliak retweeted
Let’s be very clear: there is value in automating tedious and repetitive and dangerous tasks; there is value in delegating certain work to our machines that can do the work tirelessly and much faster.
But at the same time, let us remember that there is value in the activity of engaging one’s mind and body to discover, to understand, to create, and yes, even to struggle and overcome. Indeed, these are some of the things that make us human, that contribute to giving us meaning, that are part of the simple joy of experiencing being alive.
Evgeniy (Gene) Padaliak retweeted
IMO the only tests you should have in the age of coding agents are, in order of priority:
1. Full E2E tests. Nothing mocked out at all. Can even be something that runs in prod on test accounts via Playwright or equivalent.
2. Integration tests. Agents make more mistakes as things bleed between data/API boundaries where schemas can drift.
3. Golden tests. This helps ground the code with real examples of data, and can be used as regressions against edge cases.
Unit tests are bloat 99% of the time if you’re using a frontier model since they’re smart enough now to get the implementation (and subsequent iterations) right on the first shot, so they only add bloat at this point.
Evgeniy (Gene) Padaliak retweeted
People seem shocked that the author of Clean Code doesn't read code anymore. But that should not be surprising.
Why did I write Clean Code? What is the goal of keeping code clean? To get it out of the way of the real job -- thinking.
Evgeniy (Gene) Padaliak retweeted
In my career, compilers have always generated better code that I ever could have. I find this to be increasingly so with contemporary AIs.
But you know what neither can do better than me?
Knowing what software to write, what shape it should take, and what I want it to do.
The entire history of software engineering has been one of rising levels of abstraction.
This is how it was, how it is now, and how it shall be in the fullness of time.
