Taylor

Lessons from fighting LLMs for the past few years: - The first step after a compelling AI demo should be creating evals - Don’t stop investing in evals until devs get near instant feedback, ideally a score - Try nailing the smallest possible version of your task first, even if you plan to launch something bigger - Do the task yourself so that you know whether you’re giving the LLM enough information, and have examples you can put in your prompt - Outsource labeling and ground truth creation as much as possible, but expect to spend irrational amounts of time on it yourself - Keep an experiment log detailing hypotheses and outcomes for everything you try - Look at your data, especially the formatted messages you send to the LLM - Use tiktokenizer dot vercel dot app to see where you’re wasting tokens - Don’t use a LLM for anything you can do deterministically - Forget one of these rules every project and painfully re-learn them