Case study

Methodology	Counterfactual	Problem	Problem example
Pre-Post Evaluation	The subjects before they had enrolled in the program.	Wrong sampling comparison (Sample Assigning)? Unobservable factors affects the experiment	We do not know how children who enrolled in the program would have done if they HAD NOT enrolled in the program. The counterfactual implies that these children would have remained at the same reading level throughout the course of the year. It is quite plausible, however, that children would have improved in reading even without participating in the program due to other factors. In that same year, for example, families may have had a better harvest than usual, which could have improved children’s nutrition, which in turn could have improved their reading outcomes.
Simple Difference Post comparison of program participants and program non-participants correct	Children who did not enroll in the program, whose reading outcomes were measured after the implementation of the program.		The children who enrolled in the program may be significantly different from children who did not enroll. Children who enrolled in the program are likely to be low-performing compared to their peers—after all, the program intended to target these types of children. As a result, even if the program improved their reading outcomes, the children who enrolled still may not completely catch up to their higher-performing peers. Due to this bias, it may appear as though the program was ineffective.
Differences in Differences	The counterfactual is represented by children who did not enroll in the program, whose reading outcomes were measured both before and after the implementation of the program (in order to obtain their improvement in reading level over the course of the year).		Children who enrolled in the program are likely to be low-performing compared to their peers. As a result, starting from a lower initial reading level, these children are likely to improve more than unenrolled children because they have more room for improvement. This factor may bias our results upward.
Multivariate Analysis	The counterfactual is represented by children who did not enrolled in the program, controlling for (or holding constant) their age, sex, grade level, and parents’ education level???.		Despite controlling for many confounding variables, it is likely that some (potentially unmeasured or immeasurable) variables that are correlated with program enrollment have not been included (i.e. omitted variables bias). Examples include motivation or other unobserved factors. The children who enrolled in the program were not randomly assigned to the program, so these other factors were unaccounted for.  Due to this “selection effect”, the results could be biased.