Finally benchmarked top US & Chinese LLMs on our internal OpenResty eval suite! 🚀 We tested two private benchmarks based on real, complex engineering tasks from our company: - Coding: Designing & implementing complex algorithms from scratch. - Bug-Hunting: Root-causing & fixing our hardest production bugs. Both were run directly using our coding agent harness. I'll let the results speak for themselves—attached are the quality rankings and avg latency charts for both benchmarks. TL;DR based on our coding benchmark: The top Chinese open-source models are currently ~6–8 months behind the leading US frontier models.

Not bad—the newly released GPT-6.1 Sol is already beating (or at least matching) GPT-6 Astra on two of our internal private benchmarks: from-scratch coding and bug localization/fixing. Insane value from OpenAI here. However, 6.1 Sol takes noticeably longer to complete tasks compared to previous Sol and Astra models. A trade-off, as always...

Anthropic just dropped Claude Opus 5.5 — supposedly smarter than Fable 5.1 and cheaper than Opus 5. Gotta upgrade ASAP lol. Funny how Anthropic is the loudest voice calling for slowing down AI development, yet they themselves aren't slowing down one bit 😂 Looking at the official benchmark charts, it's apparently outperforming GPT-6 Astra too.

Over the past couple of days, I benchmarked GPT-6 Astra, Claude Fable 5.1, Grok 4.6, and Gemini 3.8 Flash using our internal (harder) OpenResty BugHunt Evals suite. Claude still comes out on top, with GPT-6 Astra as a close second. This benchmark tests how well frontier models can locate and fix the hardest bugs in large, real-world, complex software projects, fully autonomously via a coding agent harness. Surprisingly, Grok 4.6 has made massive strides — it even beats GPT-5.6 Sol and other previous frontier models. Google's latest Pro model, on the other hand, is dead last by a wide margin. Their Gemini 3.8 Flash does much better though, landing around the middle of the leaderboard. Looks like Google is going all in on cheaper, smaller models 😄 On overall latency, Fable 5.1 does quite well — not because individual responses are particularly fast, but because it needs fewer trial-and-error steps to get to the fix. Details in the two charts below — quality scores on the left, latency scores on the right.

The cost-effectiveness of the Gemini 1.5 Flash model API is quite good, especially in batch processing mode (although its batch API is difficult to use). However, Google's cheapest Gemini 1.5 Flash 8b model is too weak—so weak that it sometimes doesn't follow instructions, causing many of our test cases to fail, and it's not easy to work around these issues. In my daily AI application development, I really enjoy using the 1.5 Flash model for testing and debugging—it's fast and affordable. The experimental Gemini 2.0 Flash model shows significantly better quality than 1.5 Flash. I hope the pricing won't increase after it becomes generally available (GA). #LLM

I recently noticed that the concurrency of large model batch processing tasks in each Google Cloud region has dropped from 4 to just 1. This is true for all Asia-Pacific and North American regions at least. The change happened just a few days ago. Perhaps they're preparing computing resources for the GA release of the Gemini 2.0 Flash new model? The official documentation states that the regional batch processing concurrency is 3. I'm looking forward to Gemini 2.0 GA. Currently, the API rate limits for the experimental model are too strict, and the computing power is too low - the 1500 RPD (requests per day) isn't enough to run our test suite. Running the test suite once requires at least tens of thousands of LLM requests. #Gemini #LLM

Many users ask me about the implementation principles and technology behind @OpenResty XRay, as it looks magical. • For those interested, you can refer to my previous article about dynamic tracing: blog.openresty.com/en/dynami… • Here's another series of articles I wrote introducing the ylang language: blog.openresty.com/en/ylang-… • And here's an article I wrote about the y++ language: blog.openresty.com/en/cpp-tr… Our recent advanced dynamic tracing capabilities for Java are implemented using our C++-compatible y++ language. Interestingly, our team has been using OpenResty XRay to analyze OpenResty XRay itself. However, this capability is disabled for our customers to prevent misuse. #DynamicTracing #DomainSpecificLanguages #eBPF

Some people speculate that our OpenResty XRay uses the ELF file's build ID to look up existing DWARF information in a database, which made me laugh so hard. We're actually targeting binaries that customers have modified and compiled without the -g flag or binary software packages downloaded from third parties without debug symbols. This is the most challenging and general problem to solve. For any existing binary programs and their debug symbols found online, our crawler system has already indexed them in our Pkg DB, which has accumulated hundreds of terabytes of data and is still growing daily. Why would we need such advanced machine learning technology for those cases?

Someone on the Internet asked: "What if you randomly grab a programmer from the street, and they can solve problems using your OpenResty XRay tools?" They seem to be concerned that only experts could use our products. My response is that their worry is unnecessary, and our technical experts will directly help clients solve problems - that's our guarantee. It doesn't matter if clients themselves don't understand technology or know how to use OpenResty XRay preciselyproducts. This is exactly why some non-technical business owners like our services. As the saying goes, "You make a wish, we make it happen!"

The open-source vector database Qdrant actually has a nasty bug that occasionally causes unexpected data loss. The official team finally confirmed it was a bug in the database index implementation. Alas. github.com/qdrant/qdrant/iss… There are so many bugs in the software world... a database that loses data is the scariest thing. Fortunately, it got fixed in the latest version.

One of my dreams for many years has been to turn all tech stacks and applications in all programming languages from black boxes into white boxes, eliminating mysterious issues in production environments. This is precisely the mission of our OpenResty XRay. Previously, we encountered a case where a hardware bug was discovered using OpenResty XRay on a customer's production Xeon CPU. Intel engineers initially didn't acknowledge it was a hardware issue and insisted on patching the Linux kernel to confirm the problem. Dramatically, OpenResty XRay then discovered bugs in their kernel patch. Finally, it was confirmed to be a CPU hardware issue. We still need to thank the Intel engineers for helping confirm the problem. There are so many bugs in the world, so OpenResty XRay aims to be the best bug hunter. Going further, hopefully, one day, it can even automatically apply binary patches to VMs or machine code in running processes to fix bugs or security vulnerabilities. This belongs to live patching. Everything can be live-patched from kernel to system software and up to business code without even restarting processes. We have already made some exciting experiments in this direction, and it's very promising. Advanced technology looks like magic 😁

Years ago, while working at Alibaba, I wrote online services using Erlang/OTP. However, for the same business complexity, the performance of #Erlang services was far behind my OpenResty [Grin]. Nevertheless, Erlang is still a comfortable, functional programming language to use, unlike Haskell, which requires an entirely different programming mindset. Now OpenResty XRay already supports dynamic tracing for Erlang, and in the future, we might add support for #Haskell applications in OpenResty XRay as well.

All the new shiny Erlang dynamic tracing tools are written in our Ylang universal debug language as always: blog.openresty.com/en/ylang-…

Some users asked whether our @OpenResty XRay dynamic tracing technology relies on JVM frame pointer preservation. Of course, we don't have such a requirement—we've always pursued a strict zero-modification, zero-cooperation approach. Preserving frame pointers has a significant performance impact. I know some third-party open-source solutions and even commercial solutions require frame pointer preservation, which is not suitable for production, obviously. #DynamicTracing #JVM

Support for OpenJDK 21 is also coming! These analyzers are implemented in our Y++ language toolchain: blog.openresty.com/en/cpp-tr…

When we develop new dynamic tracing analyzers in @OpenResty XRay, we usually start by directly copying the C/C++ code from the target programs into our Y/Y++ code, such as CPython, JVM, PHP, etc. Generally, we still need to make minor edits to minimize copying excessive runtime dependency code. Now, even these last bits of manual editing may be automated through #Claude 3 Opus. Just yesterday, I used Claude 3 Opus to rewrite a large C function, and surprisingly, the resulting code was nearly production-quality. The diff patches compared to the original code looked perfect—I was impressed. In contrast, when I assigned the same task to #ChatGPT’s GPT-4, the results were disappointing, with some obvious errors. It felt like comparing a professional’s work to that of an elementary school student. I also saw a post on Reddit where someone mentioned using Claude 3 Opus to refactor a large software project with many files, and he found it significantly better than GPT-4. He commented that ChatGPT often imagines things that don’t exist. #LLM #AI

I've been leading some engineers in our team to focus on developing #jvm deep analysis tools based on non-invasive #DynamicTracing technologies like #ebpf. We aim to release the first version this quarter. The tools are particularly suited for online production environments as they have minimal impact, unlike tools like BTrace and Alibaba's Arthas, which can significantly disrupt online #java applications and pose high risks. They also operate independently of the JVM Safepoints, providing a comprehensive analysis with no blind spots. When ready, @OpenResty XRay will integrate these tracing tools into our AI diagnostic reasoning engine and analysis report generator. The initial release will support OpenJDK, and there is potential for future support for Oracle's proprietary JDK. This approach avoids using mechanisms like JVM's Agent for code injection, enabling a completely read-only analysis of Java services already running online, akin to taking an X-ray. The full dynamic probes at Java methods' entry and exit points are just one of many tracing features, including JIT-compiled code. Additionally, the tool will offer CPU hotspot analysis, IO blocking and lock contention analysis, memory usage analysis, GC analysis, exception analysis, thread scheduling analysis, latency analysis, targeted smart packet capture for network traffic, and tracking of Java data structure states. We can also support older systems like CentOS/RHEL 7 with 3.10 kernels and even ancient versions of JVM, as our technology does not rely on any cooperation or assistance from the JVM itself. This is beneficial since the software used by enterprise customers tends to be quite old.

This opens the door to writing complex, nonintrusive dynamic tracing tools for big C++ applications like HotSpot JVM, NodeJS, and MySQL, with strictly 0 collaborations from the target process :D How can it be easier than writing C++ to trace C++? ;)

I just wrote a new blog post, "Use C++ to Dynamic-Trace C++ Applications": blog.openresty.com/en/cpp-tr…. It can compile real C++ code into noninvasive dynamic tracing analyzers for eBPF+, Stap+, GDB, and more! It has been a big leap since I was brought into this tracing world by @brendangregg 's blog posts more than a decade ago :) #eBPF #Stap

Recently I spent a day or two on #Android 14 and finally passed the full unit test suite of eBPF+. On Android 14, the ELF files compiled with the NDK toolchain have a difference of 0x1000, or 4096 bytes, between the relative memory address and the file offset, which is quite strange. Android's OS #kernel, compiled with #clang, also has more runtime checks, such as CFI. But the kernels of mainstream Linux distributions are almost all compiled with gcc, without such extra checks. To learn more about our eBPF+ and Ylang in OpenResty XRay: blog.openresty.com/en/ylang-… #Android #eBPF #DynamicTracing