From AI Gateways and Evals to cost control and why prompts are code. Scaling AI Infra brings a whole new set of engineering challenges. Joined the AI Engineering podcast to dive into how we scale AI Infra at @mondaydotcom (plus a $50k lesson learned along the way 😅). open.spotify.com/episode/66g…

What happens when the most powerful LLM in the world disappears within 72 hours? (And why AI infra engineers shouldn't sleep too comfortably at night) This week, I planned to write a post about how we evaluated the integration of the new @claudeai Mythos Class models within @mondaydotcom’s AI infrastructure. Instead, recent events delivered an interesting lesson: the growing tension between AI, regulation, and infrastructure resilience. For anyone building AI-powered products, three infra lessons stand out: 1. Automatic fallback is no longer a "nice-to-have": An LLM Gateway architecture capable of dynamically and robustly routing traffic between different providers and models (e.g., from Fable to Opus or GPT) is no longer an architectural luxury. It is the exact line between happy customers and a Tier-1 incident caused by a regulatory decision on the other side of the globe. If one model vanishes, your system must pivot seamlessly in milliseconds. 2. The case for open models keeps getting stronger Proprietary, closed-source models offer state-of-the-art capabilities, but they come with significant geopolitical and vendor-dependency risks. Companies investing in hosting open-source models gain an important layer of strategic independence, even if that comes with tradeoffs in quality, operational complexity, and cost. 3. Safety is now part of your infrastructure stack To appease regulators, Anthropic introduced a system that scans the context window in real-time. The moment it detects an abuse attempt (like cyber weapon development), it "freezes" the powerful model (Fable) and seamlessly downgrades the user to a safer, weaker model (Opus). They also forced users to waive the standard Zero Data Retention (ZDR) policy to store data for 30 days. Yet, the government deemed it insufficient and shut it down anyway. Safety can no longer be delegated entirely to model providers. Input sanitization, prompt injection protection, PII scrubbing, policy enforcement, and context filtering increasingly need to live inside your own infrastructure. The more critical AI becomes to your business, the less you can rely on third parties as your only line of defense. Anyone who thinks building AI infrastructure is just about calling the fastest or cheapest API got a brutal lesson this week. Resilience, abstraction, and proactive risk management are the real tools in our engineering toolkit. Would your architecture survive losing its primary model tomorrow? @mondaycom_eng

BYOA - Bring Your Own Agent directly into your everyday workflows on @mondaydotcom. 🚀 Check out the video to see @claudeai Managed Agents in action.