ALT What did I learn? Not a great deal about the psychology of programming itself. For the most part the field doesn't feel like a stumbling progression towards enlightenment, but just plain stumbling. Here are some common failure modes that frustrated me: Plain bad science, especially in the early years where a lot of the experiments are 'my pet project vs the world' and somehow the pet project always comes out looking good. My favorite example has a graph where the control group are clearly performing better, and the author explains this away, saying it's because the control group were cheating, and in the conclusion of the paper declares the treatment a success. And it was published! Failing to validate instruments. In particular, a lot of papers that involve coding qualitative data didn't bother to have two people code the data independently to check for agreement. 'We have to do some science, and this is science'. Many of the experiments are from the start clearly inc

ALT On the positive side: There are a ton of negative results. PPIG seems to be totally willing to publish failed experiments, which is awesome. Things get better over time. Somewhere around the mid-2010s I started finding papers that actually seemed like results. The series of papers on testing whether students form consistent models is a particular highlight - it starts with an interesting correlation, then some failed replications, then refinements of the test, then some successful replications, then more refinements, then combining it with interviews to test validity. It's not sexy, but it does seem to be actual progress towards a reliable measure of one specific leak in the education pipeline. I have a sort of vague idea that most of this work is just attacking things at too high a level. Programming is a big, diverse, complex skill and we don't really understand really basic subskills yet, or even know how to break it down into subskills. Without that it's impossible to kn