Finally benchmarked top US & Chinese LLMs on our internal OpenResty eval suite! 🚀 We tested two private benchmarks based on real, complex engineering tasks from our company: - Coding: Designing & implementing complex algorithms from scratch. - Bug-Hunting: Root-causing & fixing our hardest production bugs. Both were run directly using our coding agent harness. I'll let the results speak for themselves—attached are the quality rankings and avg latency charts for both benchmarks. TL;DR based on our coding benchmark: The top Chinese open-source models are currently ~6–8 months behind the leading US frontier models.

Not bad—the newly released GPT-6.1 Sol is already beating (or at least matching) GPT-6 Astra on two of our internal private benchmarks: from-scratch coding and bug localization/fixing. Insane value from OpenAI here. However, 6.1 Sol takes noticeably longer to complete tasks compared to previous Sol and Astra models. A trade-off, as always...

Anthropic just dropped Claude Opus 5.5 — supposedly smarter than Fable 5.1 and cheaper than Opus 5. Gotta upgrade ASAP lol. Funny how Anthropic is the loudest voice calling for slowing down AI development, yet they themselves aren't slowing down one bit 😂 Looking at the official benchmark charts, it's apparently outperforming GPT-6 Astra too.

Over the past couple of days, I benchmarked GPT-6 Astra, Claude Fable 5.1, Grok 4.6, and Gemini 3.8 Flash using our internal (harder) OpenResty BugHunt Evals suite. Claude still comes out on top, with GPT-6 Astra as a close second. This benchmark tests how well frontier models can locate and fix the hardest bugs in large, real-world, complex software projects, fully autonomously via a coding agent harness. Surprisingly, Grok 4.6 has made massive strides — it even beats GPT-5.6 Sol and other previous frontier models. Google's latest Pro model, on the other hand, is dead last by a wide margin. Their Gemini 3.8 Flash does much better though, landing around the middle of the leaderboard. Looks like Google is going all in on cheaper, smaller models 😄 On overall latency, Fable 5.1 does quite well — not because individual responses are particularly fast, but because it needs fewer trial-and-error steps to get to the fix. Details in the two charts below — quality scores on the left, latency scores on the right.

Today, I tried a more complex test case that required multi-step reasoning. The newly released #Gemini 2.0 Flash Thinking model (01-21 version) was disappointing and significantly inferior to the o1-mini. The critical issue was that it couldn't even generate XML/HTML tags correctly - became ```foo>. Alas. Meanwhile, Claude 3.5 sonnet, through prompt engineering, can achieve results close to o1-mini's performance. After the holiday, I plan to try Deepseek R1 with a few H100/H200 GPUs. #LLM

Recently, I accidentally noticed that Google Gemini's large language model API quietly changed its JSON format for returning citation meta error messages, so I had to modify my own API client code accordingly. Such backward-incompatible API changes are pretty annoying. Of course, Google's citation error checking for references and copyrighted text is exceptionally frustrating in itself. I haven't encountered such issues with other large language models. #Gemini #LLM #API

The cost-effectiveness of the Gemini 1.5 Flash model API is quite good, especially in batch processing mode (although its batch API is difficult to use). However, Google's cheapest Gemini 1.5 Flash 8b model is too weak—so weak that it sometimes doesn't follow instructions, causing many of our test cases to fail, and it's not easy to work around these issues. In my daily AI application development, I really enjoy using the 1.5 Flash model for testing and debugging—it's fast and affordable. The experimental Gemini 2.0 Flash model shows significantly better quality than 1.5 Flash. I hope the pricing won't increase after it becomes generally available (GA). #LLM

I recently noticed that the concurrency of large model batch processing tasks in each Google Cloud region has dropped from 4 to just 1. This is true for all Asia-Pacific and North American regions at least. The change happened just a few days ago. Perhaps they're preparing computing resources for the GA release of the Gemini 2.0 Flash new model? The official documentation states that the regional batch processing concurrency is 3. I'm looking forward to Gemini 2.0 GA. Currently, the API rate limits for the experimental model are too strict, and the computing power is too low - the 1500 RPD (requests per day) isn't enough to run our test suite. Running the test suite once requires at least tens of thousands of LLM requests. #Gemini #LLM

Many users ask me about the implementation principles and technology behind @OpenResty XRay, as it looks magical. • For those interested, you can refer to my previous article about dynamic tracing: blog.openresty.com/en/dynami… • Here's another series of articles I wrote introducing the ylang language: blog.openresty.com/en/ylang-… • And here's an article I wrote about the y++ language: blog.openresty.com/en/cpp-tr… Our recent advanced dynamic tracing capabilities for Java are implemented using our C++-compatible y++ language. Interestingly, our team has been using OpenResty XRay to analyze OpenResty XRay itself. However, this capability is disabled for our customers to prevent misuse. #DynamicTracing #DomainSpecificLanguages #eBPF

Asahi Fedora Linux runs very well on my M2 Ultra Mac Studio. All devices including USB-C, HDMI, 10G network card, etc., work without any issues. It's very stable. In comparison, Linux support for Qualcomm Snapdragon X Elite devices is far behind. Thanks to the great work from the Asahi team. Previously, running Linux VMs through qemu HVF on macOS had too many problems and poor performance. Going forward, our team will rely on Asahi Linux for development work targeting the Aarch64/ARM64 architecture.