This longer post goes out to everyone interested in estimating causal effects. Suppose you’re evaluating whether a new after school tutoring program causes higher student test scores. Before comparing test scores between those who were tutored and those who were not, you first want to know: Were the students in the treatment and control groups similar before the program began? Did they have similar prior test scores, socioeconomic statuses, or levels of parental education? Answering this question is informative of whether an estimate of the tutoring program's effects is, indeed, causal. To assess this question, researchers use what they call "covariate balance tests." Covariate balance tests compare treated and untreated individuals' characteristics to each other to see whether the two groups look comparable. They usually do this with one characteristic at a time. They'll iteratively estimate regression models that predict baseline characteristics with treatment assignment. Sometimes, they'll test all of the baseline characteristics at the same time. One way to do this is to regress treatment assignment on all baseline characteristics. You then report the joint significance of all baseline covariates. This is often referred to as an "omnibus test." Testing for covariate balance like this is standard practice in causal inference across many fields. It appears in the social sciences--being prevalent in economics, political science, sociology, and psychology studies that seek to estimate causal effects. It also regularly shows up in causal studies in the hard sciences. But a recent paper by Bicalho, Bouyamourn, & Dunning in the journal Political Analysis argues that there’s an important blind spot in this approach. Simply put: not all baseline covariates are equally informative. Say in the tutoring example, we finding balance on students' eye color across the treatment and control groups. Bicalho, Bouyamourn, & Dunning argue that because eye color tells us almost nothing about future test scores, this test of covariate balance is not as valuable as, say, one on student test scores before the tutoring program began. The difference between eye color and baseline test scores is "prognosis": some pre-treatment variables are much better than others at predicting what would have happened without the treatment. Bicalho, Bouyamourn, & Dunning argue that researchers should first measure which covariates are actually predictive of the outcomes they are studying. Then, they should place more weight on those variables when evaluating covariate balance. The authors argue that researchers should measure prognosis by fitting a model of the outcome using only the control group and reporting: • an overall Prognosis R² (how much the covariates jointly explain the untreated outcome), and • the individual prognosis of each covariate (its standardized regression coefficient), showing which variables are actually predictive. Building on earlier work on prognostic scores, they argue that researchers should measure and report prognosis, collect more prognostic covariates when possible, and prioritize those covariates when interpreting or constructing balance tests. The broader lesson is simple but important: before asking whether treatment and control groups are balanced, ask whether they’re balanced on the variables that actually predict the outcome you are interested in. This is a thoughtful and important paper on how we evaluate causal inference studies!