When presented with observational evidence to support a claim, we need to be wary. It is said that there are lies, damned lies, and statistics, and that you can devise a survey to prove anything that you want. If we are told “A study has shown that…” then we should think twice before we accept the conclusion that is drawn from it.

Misrepresenting the Data

The most basic mistake in interpreting evidence is simply misrepresenting the data. If the observational data do not fit the inference drawn, then there is a problem. There is the possibility of deliberate distortion, accidental misinterpretation, and selectivity.

Insufficient Data

A more common error is drawing a conclusion from insufficient data. Every study has a margin of error, and the smaller the study the greater this will be. Studies with a significant margin of error always leave doubt about any conclusions based on them, so it is important to consider the quantity of data in a study in assessing its validity.

Unrepresentative Data

A constant danger in empirical studies is unrepresentative data. A study that has a sufficient quanity of data may nevertheless be flawed due to insufficient quality of evidence. For a general conclusion to be drawn with any confidence from a limited data set, it must be reasonable to believe that the data set is representative.