Thanks to all the judges and @DigitalJersey for a great night. Delighted @HeyHelixAI won the AI award 🏆☺️

Congrats @typesafeai team... tested Jev on some @HeyHelixAI core routing functions (for example "Is this a flight / hotel / restaurant / experience request?") against 100s of evals. Early test results: - Accuracy on par with, or slightly ahead of, the (already small, fast, cheap) LLMs we use today, on real user messages the models had never seen ✅ - Roughly 4x faster: about a quarter of a second per decision, versus 1 to 1.7 seconds (note some of this was network latency from Europe - US) ⚡ - A fraction of the cost per call 💷 - Confidence that means something: when it said it was confident (0.8+), it was right 96 to 100% of the time 🎯 - Matching remembered traveller preferences to hotels ("does this hotel have what I like?"): it found 97.6% of real matches, with zero false positives 🏨 Headline image from typesafe.ai - go check it out

What is intelligence? Memory? Just a bunch of facts used at the right time? I’ve spent a lot of time thinking about agent memory. Tulving’s 1972 work on episodic vs semantic memory is surpisingly still very relevant - how does human memory work - referenced by lots of much more recent papers on agentic memory. At @HeyHelixAI we use our own custom relational data store plus @zep_ai Graphiti + @falkordb - storage is the relatively easy bit; context is a bigger challenge. Retrieving the right memory, in the right context, at the right time, efficiently, and ensuring the agent has acces to retrieve more if necessary (tools etc.) If I ask for a hotel in the Le Marais neighbourhood twice, is that now a preference? Just for Paris? Does it decay? Should the agent ask before assuming? What would a human do? Such a fascinating challenge... @MillaJovovich thinks so too (didn't use @mempalaceai for this tho). Anybody else working on interesting memory challenges? Keen to share notes.

To WiFi or not to WiFi? @Starlink is fast becoming a differentiator for business travel... so we built this into @HeyHelixAI. Helix remembers all your preferences so flight results are always ranked uniquely for you; for example "must have Starlink flying 5h+ in daytime" Some people prefer to switch off when they're flying and that's totally fine, but for some (me included!) good WiFi is a differentiator, especially travelling long haul. The amount of work I can get done uninterrupted and focused has sometimes surprised me! I have backup approaches using @ollama and local LLMs for some tasks (or even write or code manually sometimes, crazy I know!). But Starlink is definitely a game changer. Do you insist on WiFi on planes? Whether you do or you don't, Helix might still be for you / your company - check it out: heyhelix.ai

Let's talk about evals... AI without evals is like having junior colleagues and never checking their work. Ever. Just hoping they get it right every time. Obviously not a great strategy! Running evals occasionally isn't much better. That's the equivalent of a 1:1 every few months vs actually looking at what someone produces day to day. Humans make mistakes. Without a framework to catch them, the business pays for it eventually. Agents are no different... except you can test agents constantly, 24/7, and they don't even mind, even when you give them 'feedback' 😅 @HeyHelixAI runs over 40 agents. Many of them have a narrow remit with one job. Narrow jobs are testable jobs. Every agent ships with its own golden dataset before it ships; we run those with the help of @pydantic 's Logfire (which we love for a bunch of reasons, evals being one of them, go check it out!). When do we run evals? 1. When a system prompt changes. Ours are assembled dynamically, so one edit can mean dozens of permutations to check. 2. When we're evaluating a model. New ones land every few weeks, and the cost/quality trade-off shifts every time (sometimes less thinking / reasoning can increase accuracy AND reduce cost!) 3. On every conversation. Not "did the LLM think it went well" but did the booking actually happen, did the search actually return the specific results, did we tell the user when it didn't. 4. In Continuous Integration (CI). GitHub Actions, run conditionally depending on what actually changed. 5. When production catches a failure or edge case from real users. That conversation becomes a new eval case so the same mistake can't creep back in. 6. When we add a new agent. Obviously! Agentic engineering works best with test-driven development. The holy grail is closing the loop properly: a failing eval raises a GitHub issue, an agent picks up the issue and proposes a fix, another agent makes the necessary change(s), the evals run again, and a human reviews the diff before anything ships. Same process we'd use for a colleague's PR. Which is sort of the whole point. I'm always happy to wake up in the morning & see that the robots have been self-improving! 🤓 What triggers evals for you? Do you have a continuous loop in place?