The Risk of Ranking

image from  here

Our society yearns for quantifiable, verifiable, non-biased ways to measure things.  And, of course, we want to measure so we can compare- so we can rank.  We do it with schools, hospitals, cars, neighborhoods, you name it- we want to measure something about it, compare it with other like things, and produce lists or rankings to help us make decisions.  Is this a good idea?

I've been thinking a lot about metrics and measurements lately.  Last year, I studied the topic through three books (reviewed here, here, and here) and a decent number of articles.  I learned the basics of how to measure.  It was a good experience, but I couldn't escape the 'danger' sense in my mind.

We can take numbers too far, and we see what happens when we do so.  An obvious example is schools.  Today we measure the quality of a school based (in part) on standardized test scores.  And guess what's happened: teachers are teaching students how to take standardized tests to the exclusion of much else.  Schools look good on paper but fail in their primary mission.  Why?  Because there's more to it than numbers.

The Necessity of Judgment

When we measure for the purpose of ranking, we're trying to use quantitative measures to make qualitative judgments.  We're trying to take ourselves out of the equation- to remove our interpretation or bias.  Seems like a good idea, but that's impossible: we cannot escape our judgment.

In this scientific age, measurements and numbers are upheld as supreme and unassailable.  "Numbers don't lie," we're told.  Science is king.  Here's the challenge: data always requires interpretation.  Facts, on their own, don't say much (this is described in more detail in The Soul of Science, reviewed here).  As an example, look at politicians.  Politicians throw out numbers and statistics all the time to back their positions . . . and it's easy for people on both sides of an issue to find data to back up their arguments.  But, of course, those on the other side of the argument will [often rightly] point out what they believe are flaws or caveats that render the data misleading and conclusions invalid.  In other words, people can look at the same data and, using their judgment, interpret it differently.  It's a fact of life: we cannot escape our judgment.

So, measurements have multiple interpretations.  And, even what we choose to measure is a bias; what we choose to ignore the same.  We can make many measurements, looking at something from different perspectives, to try to overcome some of this . . . but each measurement must be weighted appropriately with respect with the others, which (again) is a judgment call.  The bottom line: we cannot escape our judgment.

An Example

Here's a silly example for you: How much do I read in a given year?  I've been keeping track of this for a few years.  Here's the initial data:

2013: 81 books (23,314 pages)
2014: 84 books (20,734 pages)
2015: 85 books (20,509 pages)
2016: 33 books (7,998 pages)

This, on its own, looks like helpful information, but means almost nothing- especially when comparing how much I read vs. someone else.  Why?  Because this data says nothing about the size of the pages, words per page, presence of illustrations, complexity of the words, or readability of a work.  Am I reading Dickens or Dr. Suess?  In-depth historical treatises or graphic novels?  War and Peace or Harry Potter?  And how much did I remember, or how was my comprehension?  Some years, I read more than 20 graphic novels- hardly the same as chopping through one Tolstoy work.  My point: the data needs interpretation, and many questions are impossible to quantitatively capture.  Thus, it's hard- or impossible- to accurately rank me along other bibliophiles.  Even saying "John reads a lot" is based on the knowledge (or assumption) that most in our culture don't read that much (as articles like this report).

Conclusion

So what's the point?  Am I saying we shouldn't measure anything?  No.  There is a place for numbers; I'm just saying that measurements are not where the story ends.  They complement and aid, but do not replace, discernment.  They help provide insight but do NOT provide the whole story.  We cannot escape our own judgment in the matter, and we must therefore be open to hearing a variety of perspectives to increase the probability that we'll correctly interpret the data presented to us.  As Douglas Hubbard says, "measurements inform uncertain decisions," but they do not, and cannot, make the decisions obvious.  They can't remove 'us' from the equation.  Paraphrasing Martin Klubeck, metrics are nothing more than indicators.  They provide insight but not truth; they should inform but not drive decisions.  The right response to any measurement is investigation- look at why and how it was measured, and realize all the while that we cannot escape the human element.  We all need discernment in addition to measurement . . . numbers alone do not satisfy.