I can't tell you how elated I am that I could generate this thread from @headinthebox's talk. I've written about how I've been taking screenshots of YouTube videos and podcast players for a decade, and how I've used various LLMs to analyze those images to extract: - podcast name, episode name, current playtime I then overlay that info over the transcript, and I can easily retrieve the relevant exciting moments, and generate summaries... More here: linkedin.com/posts/realgenek… I've seen people like @tsarnick and @swyx generate fantastic video excerpts, but the SaaS tools I found required too much manual interaction — I'd have to input the time codes myself. But I was talking with Steve Yegge, and the idea dawned on me that I could write my own program to do this using ffmpeg. Steps: Get the videos, extract the designated time ranges, burn in the captions using the transcripts. Maybe with a coding assistant, I could do it in under 2 hours? We paired together, with Steve being my CHOP coach (Chat Oriented Programming). And to my utter amazement, using @SourcegraphCody, I got my first video excerpt generated 1h 45m into our session — of which the first hour was mostly getting oriented, getting data together, etc. We recorded the whole thing, and he'll publish it soon. But here are my reflections: - CHOP is a new skill, but it can definitely be learned — Steve would catch me often saying, "This isn't very CHOP. You're doing a lot of typing." This was code for: stop typing, and think instead about how to get the LLM to do more work. - Cody for IntelliJ is great — it uses Claude 3.5 Sonnet under the hood, and I used the following two modalities: - Chat window for multi-turn iteration: write this function, write some tests, no different tests, tests are failing (here's the error messages), fix the code, etc. - Inline chat: highlight the function, and write a prompt like: "make the ffmpeg captions more Tarantino-like". Haha. - LLMs are great at writing tests! This becomes critical to gain confidence that the code it generates actually works. (As you'll see in the video, having super fast feedback is critical: it was like night/day when I started using Hyperfiddle RCF, from @dustingetz because I could get feedback with milliseconds) - there were at least two times when I was a little surprised at how the LLM struggled to fix the code it wrote (e.g., merge overlapping time ranges) — I finally got things working by giving it hints (e.g., add an :end map entry, so you can do the computation more explicitly). - it's super handy to have ChatGPT or whatever open to the side — to do ad hoc research. - I feel like I got done in 2 hours what normally would have taken 2 days — before we started, I told Steve that this problem falls into the category of what I'd call: "not this month" Merely the idea of struggling with ffmpeg would make me not want to tackle this. But CHOPping (writing a prompt) to "write a function that takes an MP4 file and an SRT transcript, and overlays the captions at the bottom", and seeing it work within 5 minutes was absolutely amazing. - my big takeaway and puzzle: that function was super easy. But other problems were much more difficult — like merging overlapping time ranges to extract, finding SRT time ranges that overlapped causing duplicate captions. It's like the "how to draw an owl" joke. Sometimes the LLM can draw the whole freaking owl in one shot. Amazing! But so many other times, when you're dealing with lots of constraints, it takes a lot more effort to draw the owl — it's like you have to create 4-5 different "tween frames", to guide the LLM on what actions to take. I feel like there's an intuition I still need to gain about what LLMs can and can't do well. Just like in CS, we know that regular expressions cannot count — that requires a stack. Similarly, there's probably heuristics of what LLMs can and can't do, and knowing those limitations will make using CHOP much more effective. More to come later! And I'll post a link to the video of me pairing with Steve when I get it.