Hey, Claude: "I asked an earlier Claude to Remove the Squid from All Quiet on the Western Front. Now you can make movies and such. I need you to show how far you have come" & I pasted in the tweet below There is some clever stuff, models have a real sense of humor at this point

It is not inherently bad to publish research on the impact of AI that only refers to older models, but it requires a very careful discussion and has to be very clear to non-technical readers. If you show AI can do something, its fine. Generally, once AI has gained an ability, it does not regress in future generations. So papers that show that AI has crossed a threshold (such as "good as a human") or that seek to establish some sort of minimum impact or effect still hold up even if they used GPT-4. If the finding is that AI is bad or biased at something, you need to be much more careful. You cannot claim that because GPT-4 fails at something, that AI is bad at that task, because current or future models may do it. So what can you claim? These are just some examples: 1) You can just be explicit: "GPT-4 could not do X." This is a rare way to frame things because it is not really an interesting publishable paper in many cases. However, if you provide the information needed to reproduce your work, it can become a useful benchmark to measure progress. 2) You can measure trends: compare GPT-4 to GPT-5 to GPT-5.6 Sol or whatever (it is really important to include at least one reasoning model). You can then make better claims about abilities relating to this task. 3) You can make a strong, grounded argument that AI has a natural flaw or limitation that means it cannot do X, and then demonstrate evidence that this might be the case. 4) You can focus on a moderator or mediator: this sort of prompting or approach or social context or connection impacts the ability of GPT-4 to do X, and needs to be a concern for the future. 5) You can focus on humans: how do people react to AI? what are the failures and successes? what dangers or advantages or changes does it bring to us? Again, these aren't exhaustive, but generally negative capability claims have been much less durable than positive ones.