Bent Flyvbjerg is a scholar in Denmark and an expert in cost-overruns and demand-shortfalls in public infrastructure projects. In one of his more recent publications (“Cost Overruns and Demand Shortfalls in Urban Rail and Other Infrastructure”), he demonstrates the dramatic demand shortfalls that most urban rail infrastructure is met with. According to his study, actual ridership was, on average, 50.8% lower than forecasted.
Rightly, Mr. Flyvbjerg excludes two statistical outliers as each “strongly diverge” from the figures. Whereas most other systems studied experienced dramatic demand shortfalls, these two outliers experienced demand that was 158 and 60% greater than forecasted.
Now I’m not asking Mr. Flyvbjerg to include those outliers. It is standard statistical practice to forget about them. I do, however, want to know what happened in those two instances; particularly given that each outlier was in a German city. Don’t you want to know what the Germans are doing right? Don’t you want to know the story behind these two abnormalities? Don’t you want to know what made these cities successful against the overwhelming evidence that suggested they should fail?
Don’t ignore statistical outliers just because they corrupt your models and poison your results. Maybe your model’s wrong. Forget about your model for a second and instead ask yourself why did that happen?
A statistical outlier, whether positive or negative, is something that is unique within your models, and uniqueness is deserving of inquiry. There is a story there. And in that story are answers to important questions.
Predictable results may make for easy answers, but (hopefully) your boss/client/supervisor isn’t looking for easy answers. And if he is, those answers are probably incorrect.
We need more people who don’t really care about easy answers and predictable results. Change occurs when people look for the unpredictable and then wrestle to find the story lurking somewhere underneath.