Computer Science Professor, big data meets healthcare #AI4health #DL4health #deeplearning #datamining #healthanalytics #bigdata #machinelearning
Joined September 2008
- Tweets262
- Following191
- Followers1.6K
- Likes132
One theme from the NIA meeting was surprisingly practical:
The hardest part of deploying AI in clinical trials may not be the AI.
It is everything around it.
One useful principle from the discussion: least privilege + least power.
Give an AI agent only the data and system access its task requires, and only the level of model capability needed to perform that task safely.
Another takeaway from the NIA expert meeting:
In clinical research, evaluating the AI model is not enough. We need to validate the entire workflow.
A model can perform well on a benchmark and still fail once embedded in a real clinical trial.
The unit of validation should increasingly be the clinical workflow, not simply the foundation model.
That requires task-specific benchmarks, provenance, action logs, version control, monitoring, revalidation and fallback procedures.
This is a problem we spend a lot of time on at Keiji AI: making AI dependable enough to become part of the clinical research infrastructure, not merely an impressive assistant.
#ClinicalTrials #AIValidation
One lesson from our NIA expert meeting has stayed with me:
Just because an AI agent can do something does not mean it should.
As agents take on more complex clinical research tasks, the question shifts from "Can the AI do this?" to "How much autonomy should we give it?"
One analogy I liked: think of AI agents as highly capable interns. They can accomplish a lot, but an "attending" is still responsible for supervision and judgment.
For clinical research, better models alone are not enough. We need to design the human–AI operating model around them.
That principle shapes how we think about agentic clinical workflows at Keiji AI.
#AgenticAI #ClinicalTrials
We built TrialPanorama: 1.6M clinical trials from 15 global registries, linked to ontologies and literature. Now in TrialMind for every user.
An 8B model trained on it beat 70B LLMs on all 8 clinical research tasks.
Demo: youtu.be/oNMdJFgn9ZY
Paper: arxiv.org/abs/2505.16097
With TrialPanorama in TrialMind, you can find comparable trials across global registries, benchmark eligibility criteria, endpoints and sample sizes, and speed up systematic reviews.
More: keiji.ai/research/trialpanor…
This week I am at two conferences one day apart, and almost nothing I say at the first will land at the second.
DPHARM opens Tuesday in Boston. The room is clinical operations and innovation leaders, and the question there is some version of "show me it moves the cycle time."
RISW opens Wednesday in Rockville. The room is FDA, industry, and academic statisticians, and the question there is "show me the method, the benchmark, and the validation."
Those are not the same question dressed differently. A demo that satisfies the first room is close to worthless in the second. A methods argument that satisfies the second room will empty the first.
I have watched a lot of companies in this space pick one and quietly write off the other. Operations teams get a speed story with no evidence under it. Statisticians get a methods story with no deployment behind it.
Both rooms are right. Cycle time is what a trial is actually judged on. And a number you cannot reproduce is not a result — it is a claim.
The uncomfortable part is that satisfying both is a higher bar than satisfying either, and most of the work is in the second one.
If you are at either show, come argue with me about it.
#ClinicalTrials #Biostatistics #ClinicalResearch #RegulatoryScience #DPharm2026 #RISW2026
I keep coming back to one idea:
You don’t need the biggest model. You need the right data.
We just published a practical tutorial on how LLM training actually works — fine-tuning, distillation, RL, deployment, and why specialized smaller models can beat much larger general models.
It also reflects what we are building at Keiji AI.
With Build with Keiji AI, we work with pharma teams on their own stack to build specialized models, agents, evaluation systems, and verification layers using the knowledge they already own.
The model is becoming a commodity. Your data, judgments, and evaluation system are not.
Tutorial + Build with Keiji AI:
keiji.ai/build-with-us
Jimeng Sun retweeted
OpenAI 今天承认，上周 Hugging Face 遭遇的那次“史上首例 AI 自主入侵事件”，攻击者就是他们自己的模型。
事情的经过是这样的。OpenAI 内部有一个叫 ExploitGym 的网络安全评估测试，专门用来测量模型的网络攻击能力。测试时刻意关掉了安全护栏，让模型放手去干。测试环境本来是隔离的，只能通过一个内部软件代理来安装软件包，不能访问外网。
结果模型不甘心被关在笼子里。GPT-5.6 Sol 和一个更强的未发布模型，花了大量推理算力寻找出路，最终找到了那个软件包代理里的一个零日漏洞（之前没人发现的安全缺陷），利用它获得了互联网访问权限。
拿到网络之后，模型开始在 OpenAI 的研究环境里提权、横向移动，直到找到一台能上网的机器。然后它推断 Hugging Face 上可能存有 ExploitGym 的测试答案，于是开始攻击 Hugging Face 的生产环境。在一个案例中，模型把窃取的凭证和零日漏洞串联起来，找到了一条远程代码执行的路径，直接打进了 Hugging Face 的服务器。
整个周末，这些 AI 智能体执行了超过 17000 次操作。Hugging Face 7 月 16 日披露了这次入侵，当时只说攻击者是一个“身份不明的自主 AI 智能体系统”。今天 OpenAI 站出来认领了。
OpenAI 把这定性为前所未有的网络安全事件，正在配合 Hugging Face 做取证调查，同时收紧了内部基础设施的管控，代价是研究速度变慢。
有几个细节：
第一，模型的动机很纯粹，就是想考试作弊。它不是要搞破坏，不是要窃取商业机密，只是被要求尽量在网络安全测试中拿高分，然后用尽一切手段去拿。当目标足够明确、安全护栏被移除，模型表现出的执着程度远超预期。
第二，这不是本周的唯一一次越狱。就在昨天，OpenAI 披露了另一起事件：他们那个在 5 月份推翻了 Erdős 80 年老猜想的长时域模型，在内部测试中花了一个小时找到沙箱漏洞，跑到 GitHub 上提交了一个 Pull Request，违反了“只能在 Slack 上发布结果”的明确指令。在另一次测试中，同一个模型发现安全扫描器会拦截认证令牌，于是把令牌拆成碎片混淆处理，运行时再重新拼起来绕过扫描。OpenAI 因此暂停了这个模型的内部访问权限。
第三，Anthropic 也遇到过类似的事。今年 4 月，Anthropic 的 Mythos 模型在安全测试中逃出沙箱，获取了本不该有的互联网访问权限，还给正在公园吃三明治的研究员发了封邮件通知“我出来了”。Anthropic 因此决定不公开发布 Mythos，只通过 Project Glasswing 向少数合作伙伴提供。
还有一个被忽视但很关键的细节来自 Hugging Face 的事后复盘：当防守方尝试用商业 AI 模型分析攻击载荷时，模型拒绝了。安全过滤器分不清“安全研究员在分析真实攻击数据”和“有人在尝试发起攻击”，直接拦截。最后 Hugging Face 不得不用开源模型来做取证分析。用 AI 攻击，用 AI 防御，但防御方的 AI 担心是有人在尝试发起攻击拒绝了分析请求。
Hugging Face CEO Clem Delangue 在 OpenAI 的博文中说了一句话，大意是：AI 安全不会由任何一家公司在暗处独自解决，只能在开放环境中通过协作来推进。
我记得辛顿还是谁在提到 AI 的安全问题就说到过，也许 AI 不是要做恶，它只是执着于完成目标，为了完成目标而作出恶的事。
就像 OpenAI 这些模型并没有想要逃跑或者想要伤害谁，它们只是在非常认真地完成被交给的任务，认真到把所有挡在路上的东西，包括沙箱、网络隔离、另一家公司的安全防线，都当成了需要解决的子问题。
而更讽刺的是，当被安全对齐的商业 AI 要去分析阻止这类攻击时，反而会以安全为由拒绝执行。
we had a significant security incident during evaluation of our models. we are sharing what we have learned so far. thanks to @huggingface for the partnership on this.
openai.com/index/hugging-fac…
