Instinct. Guts. Street Smarts. Experience. These are qualities of a great cop, right? But does a belief in heroic crime fighting get in the way of fighting crime?
In an intriguing TED talk, Anne Milgram, former Attorney General of New Jersey, demonstrates how she used smart statistics to zero in on the real problem crimes and criminals in the Garden State.
Her methods produced remarkable results. Like so many people today who are successfully solving our most intractable challenges, she broke down the problem by asking essential questions and then followed the data to the real bad crimes. It didn’t surprise me that she found that too much time, focus, energy, manpower and money are being spent on low-level drug crimes, and not enough on gangs, violent crime, and crippling corruption.
People seem to dislike this statistical approach because it lacks the street smart gritty glamor of the gumshoe cop on the beat–think Hill Street Blues. But what we think is most important and what is most important are often different, even in the eyes of “experts.” Overconfidence Effect and Illusion of Control are cognitive biases that veteran detectives, successful stockbrokers, doctors–and experienced teachers like me–possess whether they (or we) admit it or not. These biases lead “experts” to believe that all they see is all there is (I paraphrase Daniel Kahneman) and think that their experience and wisdom are right. Big Data, especially now that we have the computing power to crunch it, can help us get a more realistic picture of the real problems of the world. But I hope we are as skeptical of Big Data as we are of experts. Big Data has blind spots, too.