“Sooooo…this is kinda garbage,” I told @medialyst founder @elvissun and graded his AI-generated media list with an F (27/100). Then I signed on as his company’s Founding Strategic Advisor. I've been hand-building media lists for 14 years. They each take days to research, vet, and refine. Now Medialyst can generate one in seconds. Even though the output has a failing Carly Score™ right now, we’re moving in the right direction. In July we were at 27.9/100. Elvis made changes and this week’s score is already up 5.8 points. I’ll be grading a fresh list output every Thursday and publish the number whether it's flattering or not. HOW WE CALCULATE THE CARLY SCORE™ We test: - Crossover between journalists I found and what the tool finds x .40 - Accuracy of the emails it returns x .30 - Whether I'd keep or cut the names on the list x .30 All weights add up to 1, no hidden multiplier. THE BASELINE July 20 run of a list for a job site’s Stress-Free Six-Figure Careers brief: - The tool found 7 of the 43 journalists on my hand-built list - But it only got 4 of their emails right - Of the 50 total names it returned, I'd keep 7 Score: 27.9/100 = GRADE F THE FIX - The tool now hunts for more of a journalist's published work before scoring or admits "not enough evidence" rather than inventing a score - Author index pages no longer count as articles someone wrote - Emails and LinkedIn profiles that match a completely different name are no longer treated as confirmed matches THE RESULT Same brief rerun on August 6: - Same crossover journalists score: 7/43 - Same correct emails score: 4/43 - Better kept journalists score: 7/21 Score: 33.7/100. GRADE: Still an F… but 5.8 points better because the tool spit out less trash. VALIDATION I sat with Elvis and walked through the most recent list line by line. For the first time the tool's scores matched my gut. It ranked reporters @Andy Medici at 95 (he actually covered the story), @Bryan Robinson at 95 (planning on covering in August), @Jennifer Mattson at 78 (who previously covered a similar report for us), and @Mary Cunningham at 55 (which is exactly where I’d put her for this story). MOVING FORWARD What's still broken: - The tool surfaced CPA Practice Advisor for a consumer story - Finding better emails for contacts - A site I couldn't even confirm was a real publication - Coursera, which is a program-review site These misses all get addressed before the next test. From now on, we’ll be posting a new Carly Score™ every Thursday. Link to the live scorecard if you wanna check it out below.