SF Tech Week has more than 1,600 events, and LA Tech Week has around 800. Finding a handful that match your interests and schedule can take hours. We built Tech Week Picks in Lovable, partly to explore what we could build with the platform. The app asks five questions about your role, goals, and interests, then recommends a personalized lineup without scheduling conflicts. We chose TypeSafe's Jev because we needed fast, structured judgments. Jev classifies events by topic, audience, and format, then helps rerank the strongest matches. The scheduling and ranking rules stay in code. The next challenge was tuning the recommendations for quality. We installed Plumloom Autoeval directly in the Lovable development environment with one command: npm i -g @plumloom/cli From there, the Lovable coding agent could run evaluations against the actual app outputs, across multiple visitor profiles and trials, and examine relevance and factuality scores. Autoeval uncovered unsupported explanations, weak recommendations toward the bottom of lineups, and noise in the event data that was producing false matches. The coding agent used those findings to refine the recommendation logic, and we reran the evaluations to check the changes. This evaluation and tuning took a couple of hours. Manually testing and refining recommendations across those visitor profiles would have taken several days. The early response has been encouraging. For SF Tech Week, 79% of visitors who started the quiz completed it. What stood out from this experiment was how quickly we could move from building the app to evaluating and improving its recommendations, using Autoeval directly in the Lovable workflow. Try Autoeval (open source): github.com/AlchemyInCode/plu…

For higher-risk AI agents, production shouldn’t be where foreseeable failures are discovered. Offline evals move known risk earlier. Production reveals what the eval suite missed. And because both models and judges vary, one eval score is still only one observation.