As a challenge to learn about reinforcement learning, I decided to train an agent to beat Pokémon Red. The goal is to defeat the Pokémon League without giving the agent cheats like infinite money or automatic HM use. So far, the agent can win against Misty, but there are still plenty of challenges ahead. I'll stream the latest attempts on Twitch and share the successes and failures here on X.

The capacity to verify that software you didn't write is correct is probably the most valuable skill right now.

All solo drills for skill acquisition in jujitsu, and all combat sports in fact, are a waste of time. Every movement you make should be a result of what your opponent is giving you. If you remove the opponent, there is no information for you to act upon, and your training environment is no longer representative of your performance environment. Simple ecological dynamics.

Took me a while to try it out, but the Codex desktop app is better than the CLI app. Especially if you're doing FE work.

Nothing is as annoying in BJJ as that guy who is always stopping the roll to "teach" you something. Don't be that guy.

An issue I've had with tools that block distractions is that it's hard to categorize what is a distraction. E.g a YouTube video can be a distraction or a useful tutorial. Turns out small AI models are great for this. Just explain why you need the app/website for the task and let the AI decide if you really need it or not.

It's easier than ever to build inaccessible apps.

I've been going to the gym for 15 yrs now, and while improving a PR feels nice it doesn't come even close to hitting a cool move in a sport. Still proud of this scramble and sub I hit a while ago in BJJ.

Bought two RTX PRO 6000s last November for around €7k each. The same retailer lists them at €15k each now. I should have bought four.

Spent some time with Jev trying to see how it played Pokémon Red. The TLDR is that playing a JRPG is not a good use for Jev, which, to be fair, is in line with their description that the model is suited for system 1 tasks, not system 2. That's not to say it's impossible to beat Pokémon Red with Jev. If you build a good harness around it, where instead of Game Boy buttons as decisions you give it general actions like "Navigate to Oak's house" or "Pick Bulbasaur" and also short-term milestones to progress the game like "Pick Oak's parcel at Viridian City Pokémart" it's probably doable. But at that point, the real engineering feat would be building a harness that makes general enough suggestions to allow Jev to be creative while still being possible to reach the goal of finishing the game. Otherwise, we would just be doing a scripted run with extra steps. In case you're curious, here is a video of Jev playing Pokémon Red till the point of leaving the player's house. It cost 2.9 cents to get here, but to be fair, I could probably have made some optimizations.

Got early access to Jev from TypeSafe AI. Looking forward to seeing how well it plays Pokémon Red 👀

Update on beating Pokémon Red by training an AI agent from scratch with RL: The architecture I went with had several policies, mostly one for each milestone of the game. E.g. Policy A to go from Oak's Lab to defeating Brock, Policy B to get to Misty, etc. This allowed me to tackle the challenge incrementally. Instead of trying to train a single policy for days, I can train multiple for hours and see the AI progress day by day. A weakness of this approach is a dependency of the later policies on the earlier ones. For example, the policy that beats Misty expects the previous policy to have chosen Charmander because that's part of the environment it was trained on. This becomes a problem when you realize you need to change a behavior in earlier sections to make the game easier to beat. This problem became apparent now that I need the AI to deal with Flash. To get Flash in Pokémon Red, you need: - To have caught 10 Pokémon - Have a Pokémon that can learn Flash. As of now, none of the policies have learnt to catch Pokémon, so I'm a bit stuck 🥲 I'm not 100% sure what I'm going to do yet to solve this, but I think decomposing the game into hard milestones to be achieved no matter what is not the way. I'm more inclined to believe that training the AI to achieve exploratory goals like catching as many new Pokémon as possible and exploring all the zones might yield better results.

The hard part so far of training RL agents to beat Pokémon Red is reliability. It's not difficult to get an agent to reach a goal once. What's hard is to get it there at least 50% of the time. The latest milestone is reaching Rock Cave. It's not reliable yet, but it's close.

The AI is finally able to beat Lt. Surge! This has been the hardest segment so far. Mostly due to: - Teaching and using Cut - Solving the gym puzzle I had to adjust the training curriculum so it could solve both challenges reliably. We also have a Charizard on the team now!

Today I learned a Reinforcement Learning approach called Reverse Curriculum Generation. The idea is that we start training the AI as close as possible to the solution. As it becomes reliable, we move the episode’s starting point backward until we reach the starting state we want the AI to perform from. I used this approach to teach the AI how to use Cut. Here's roughly the learning curriculum I used: 1. Use Cut when already facing a tree and having a Pokémon with Cut in the party. 2. Finish teaching Cut and use it. Cut has just been taught to a Pokémon, but dialogs remain open. 3. Choose a Pokémon to teach Cut and then use it. The AI starts with the HM already selected, has to teach it to a Pokémon, and then use Cut. 4. It has to cut the tree starting from a closed menu and no Pokémon knowing Cut. The trained AI is not as efficient as it could be, as it presses extra buttons, but it's reliable. Video of the AI below.

An issue I'm facing with using RL to beat Pokémon Red is dealing with Cut. The generic rewards aren't enough to get it to learn the menu sequence to both teach and use Cut. In Gen 1, you need to go to the menu to use an HM, which introduces complexity. My approach now is to detect when the player is facing a tree and introduce a policy that was trained just to teach and use Cut. However, the policy is not generalizing well yet. Massive failure below 🥲

Looks like the AI learnt Gen 1 Slash bug. Before it used to spam Ember, now it developed a liking for Slash since it will almost always crit.

Good progress today on beating Pokemon Red with an RL agent. After beating Misty the AI was able to get S.S Ticket from Bill and reach Vermilion city! Incentivizing exploration and penalizing the agent for taking too long has been working wonders.

Astra promised me a 200% improvement on my Pokemon Red RL training environment. It was closer to 6% 🫠

A behavior the AI tends to learn is to spam B. I guess it's because B helps it get unstuck from menus and dialogs. It's a useful behavior until it cancels Charmander's evolution 🥲 It broke my heart, so I added training stages just to teach the AI to let evolutions happen.

Going live on twitch.tv/canaibeatpokemon in about 10 minutes. The RL-trained agent can now go from the start to obtaining the S.S. Ticket with 60% reliability. The fastest time recorded during streaming was 18h 1m 32s. Let's see if it can do better on the stream 👀