Experiments and Causal Inference
You have have heard the phrase “correlation does not imply causation”. This is usually used to say that just because two variables are associated with each other, that does not mean one causes each other. This website can generate spurious correlations for you. For example, the number of people who drowned by falling into a pool correlates with the number of films Nicolas Cage appeared in. Does this mean that Nicolas Cage causes people to drown? Probably not.
One problem which comes up a lot is that given enough data, you will find a lot of correlations. And many of those correlations will be meaningless. This can relate to a practice known as HARKing (hypothesizing after the results are known), where scientists get a large dataset, create findings based on that dataset, and then draw conclusions only after seeing what the data already says.
On the other hand, we often want to come away with causal conclusions from research. We want to know, for instance, if a particular approach to content moderation is effective, as was the case in Chandrasekharan et al. (2017)’s study of Reddit’s quarantine policy.
So if correlation does not imply causation, how do we establish causation? Generally, we use experiments. For example, in medicine, we might test the effectiveness of a new drug by running a randomized control trial (RCT). People are split into a treatment group, where they receive the drug, and a control group, where they do not receive the drug. Assuming that the group was randomized correctly, the we can attribute the difference in outcomes between the two groups to the drug.
Ideally we would be able to have experiments for everything, but this is not always practical. Causal inference gives us tools to answer why questions when we cannot use randomized control trials.
The fundamental problem of causal inference is that we cannot observe both a treated unit and its untreated counterfactual at the same time. The goal of causal inference approaches to provide a framework to answer causal questions regardless.