I am at COLM'26 this week in San Francisco with my students as well as folks from @Rabdos_AI . We have three cool papers on timely and exciting topics; drop by if you are attending! > MathDuels: A Self-Play Benchmark That Grows arxiv.org/abs/2604.21916 Wednesday 11-1, Grand Ballroom #121 Leaderboard: mathduels.ai/ > Detecting Safety Violations Across Many Agent Traces arxiv.org/abs/2604.11806 Thursday 11-1, Grand Ballroom #129 > Do We Need Frontier Models to Verify Mathematical Proofs? arxiv.org/abs/2604.02450 Thursday 4:30-6:30, Imperial Ballroom #21

This is the first semester in my teaching career that I have been able to use LLMs to improve the learning experience. One way I’ve been doing this is simply feeding my static slides from each lecture into an agentic framework to create live demos in class. For my software analysis class (cis547.github.io/), I used Claude Code to demo different program correctness techniques, from random testing using AFL++ to theorem proving using Dafny. In the past, I spent hours just debugging the setup of these tools. Now Claude does it autonomously for me in *under 5 minutes*, giving me a standalone Docker container with a web UI. Check out this demo of an automated debugging technique called Delta Debugging that I covered this week: static.us.edusercontent.com/… As a bonus, you yourself learn subtle new things about the technical content. For me this happened with a concurrency fuzzing technique called PCT, where Claude showed how P(bug per run) is higher but rate of bugs per second is lower if you put threads to “sleep” versus simply preempt them. It is an even bigger privilege to be teaching in these times. LLMs can now do almost anything so it all comes down to instructor taste and judgment.

Every year I take an international trip, I measure in-flight internet connectivity as an indicator of global technological progress. 2026 represents a *huge* step up in this measure, entirely single-handedly due to @Starlink. They have essentially solved this problem from 0 to 1.

Incredibly proud of PhDs #8, #9, and #10 -- congratulations Drs. Jiani Huang (@jiani_huang_ai), Aaditya Naik (@aaditya_naik), and Adam Stein (@adamlsteinl)! I am lucky and grateful for the privilege of working with the three of you!

Friends, followers, and strangers on X, I recently got excited about mapping the jagged frontier of AI models in math and other STEM areas. With my colleague Prof. Rob Ghrist @prof_g , I founded Rabdos AI rabdos.ai to create original, research-level problems at scale to advance frontier AI capabilities. In the coming weeks, we'll be sharing exciting technical updates through @Rabdos_AI. Please give us a follow if you're interested in model evaluation, AI for math/science, autoformalization in Lean, robustness of world models, and much more.

Penn-goin! Elated that my amazing daughter Isha was admitted to Penn in the Mechanical Engineering program's Class of 2030! Lots of gratitude to her teachers, our family and friends, my colleagues, and even my students who helped in any way possible. We as a family got to experience the stressful process of US college admissions firsthand. Sending good wishes to others going through it right now. A reminder that admission decisions are never a measure of your worth as a person. thedp.com/article/2025/12/pe… Isha's website with artwork and writings: ishanaikfineart.com/ #PennEngineeringProud @MEAM_Penn #classof2030

It is easy to overlook but hard to overstate how big a role "test-time inference" has played in modern LLM gains in reasoning. I myself wasn't sure, so I put GPT-5 and Gemini-2.5 to a classic programming puzzle called variable shadowing (en.wikipedia.org/wiki/Variab…): Both models in "Thinking" mode solved it correctly but took around 2 minutes. The thinking was elaborate (and correct): "I'm currently focused on rewriting the user's C-like code, aiming to replace all instances of 'y' with 'x'. However, I've hit a snag. The crucial part is to avoid this substitution in specific scenarios. " More surprisingly, both models in "Instant" mode got it consistently wrong, giving a glimpse into their pre-training levels of reasoning which seem to have hardly improved over the years if at all. Is there a middle ground which reliably answers this simple question correctly without thinking for too long? The best solution today remains prompt engineering. Appending "Think step by step before answering" to the prompt causes both models to engage in a brief monologue which turns out to be crucial to subsequently generating the correct program. The recent "Auto" mode of LLMs is intended to automatically strike this balance and forgo the need for prompt engineering. But it currently errs on the side of over-thinking, or worse, under-thinking. LLMs will undoubtedly get better at calibrating their "Auto"mode but I am equally excited about emerging architectures like diffusion models, which are not constrained by left-to-right generation and can iteratively refine their answer the way humans typically do.

Prospective PhD students: my research group has openings starting in Fall 2026! We focus on building principled yet practical systems for trustworthy AI. We are developing foundation models and agentic frameworks for critical domains such as cybersecurity, robotics, and medicine. And I am always looking to venture into new areas, with emerging projects at the intersection of AI and high-performance computing, additive manufacturing, and beyond. Information on how to apply is available at gradadm.seas.upenn.edu/ and the deadline is December 15, 2025. Below are some highlights from my group in the past year: ➡️ AI x cybersecurity: IRIS, the first system for LLM-assisted static analysis, published at ICLR 2025. The overarching goal of IRIS is to autonomously discover, triage, and fix security vulnerabilities. ➡️ AI x robotics: LASER, the first system for video understanding, published at ICLR 2025. An upcoming Spotlight paper at NeurIPS 2025 demonstrates how scene graphs produced by LASER improve generalizability of VLMs for embodied agents. See the below demo of LASER understanding a raw video in real-time! ➡️ AI x medicine: Starting with our selection as one of six national teams to win the NIH’s Long COVID Computational Challenge, we developed the first explainable model for individualized treatment selection, which was published as a Spotlight paper at ICML 2024. With patient data from our medical school, we are now building models to assist clinicians with high-stakes problems such as breast cancer treatment selection and cardiac disease risk prediction. ➡️ An enduring interest of my group is to make AI trustworthy. To that end, we work on all aspects of neurosymbolic programming including semantics, compilers, and real-world applications. Read more about this topic in our recently published book: cis.upenn.edu/~mhnaik/papers…. Good luck with your PhD applications!

Congratulations to Dr. Ziyang Li (@_ziyang_) on defending his dissertation today! Titled "Neurosymbolic Programming in Scallop: Design, Implementation, and Applications", this dissertation proposed Scallop, a unified programming system for combining the otherwise complementary worlds of deep learning and symbolic reasoning. Ziyang's work demonstrated that neurosymbolic solutions achieve superior accuracy, interpretability, and data efficiency compared to their purely neural or symbolic counterparts, in applications as diverse as natural language reasoning, image and video scene graph generation, program vulnerability detection, and RNA folding. Ziyang will join the computer science department at Johns Hopkins University as a tenure-track assistant professor in the summer of 2025. He will be an amazing collaborator, mentor, and educator to those lucky enough to cross paths with him. I certainly was and I will miss him dearly! My deepest gratitude to thesis committee members Rajeev Alur, Chris Callison-Burch, Armando Solar-Lezama, and Val Tannen, as well as to Ziyang's many collaborators over many years. You can learn more about Scallop from Ziyang's dissertation at liby99.github.io/res/dissert… and try it out using the tutorials at scallop-lang.org/.

Do you use SAST (Static Application Security Testing) tools at work and wish they could become more powerful using modern LLMs? Please take 15 minutes to complete a survey: forms.gle/wQ5q6Zk7M8TtCXpo6 This survey is part of a user study we are conducting to extend IRIS, an open-source framework that combines LLMs with static analysis to detect security bugs. All identifiable information will be anonymized in our analysis of the results. We will use your name and email only to optionally contact you for a follow-up if you consented to it. Thank you!