Posted: 12:31PM April 12th, 2016 | Comments
One thing we are trying to get out of the impact evaluation process is “Are we really making a difference?” Why is this difficult? Well, human society is a complicated system, on which is almost impossible to implement rigorous scientific experimental methods to examine the effect of a treatment. For example, there does not exist a pair of fully identical communities that allows us to easily distinguish the effect of setting up a recycling system in one of them. Fortunately, we still have some alternative, though not perfect, approaches to measure the impacts.
Think about this: A job training program was implemented in Wisconsin, but the rate of unemployment did not drop. Is the program ineffective? What if the unemployment rate was originally climbing up? Or if all other states had an increasing unemployment rate during the same period? You probably got it! We need a “counterfactual” comparison to know whether or not our intervention or treatment has its impacts. Common methods include comparing across time (before v. after) and across groups (treatment v. non-treatment). Of course, a better way will be combining the two, where you can rule out the environmental or structural factors (e.g. a financial crisis that undermines the effect of the training program).
What else should be aware of? Well, there are plenty of them. One of them is selection bias. For example, if a green team performs well in terms of doing recycling project, can that be fully credited to the MPower program? Perhaps those who participated in the green team already did a great job in recycling at home. If that is the case, then the effect of implementing the same program to others may not be as outstanding as assumed from the outcome of the green team. Other threats to the validity of evaluation include multiple treatments, structural trends, and more. However, the core question to keep in mind is “Are we precisely measuring what we intend to measure?”