Measurement: You can't measure everything

A colleague of mine enjoys saying that “you can’t manage what you can’t measure.”  When I point out to him the things that we need to manage but can’t measure, he responds by saying that that is the challenge, to find measurements for those things, because, if we don’t, then we won’t be able to manage them. 

When my colleague looks at the gap between what we would like to be able to measure, and our current abilities, he sees this as a technological gap.  Give us more time, more effort, more research, more hard work, and we can close that gap, he thinks.

            In today’s America, this is the majority opinion.  In fact, it appears to be an overwhelming majority opinion.

So, I know that my opinion is heretical.  But what can I do?  The overwhelming majority is wrong on this issue.

Whose burden of proof?

First, let’s ask an intriguing opening question.  Whose responsibility is it to prove their side?  Right now, the majority thinks that their position is obvious.  All things are measurable, although we might not know how to measure everything yet.  If you don’t agree, prove it.

But mightn’t the “status quo ante” be that not everything is measurable, and it is up to those who think everything is measurable to prove their position?  Given that, in our common experience, there are many, many things that “resist” measurement, and given that we have no direct evidence of the correctness of the notion that all things can be measured, no experience of it whatsoever, why is the starting point the position for which there is no experiential evidence?

My view is that we should start from the assumption that measurement is far more difficult to do than we think, and that some things are inherently unmeasurable.  Now it is up to the measurement gurus to prove that there is only one category (things that are tough to measure) rather than two (things that are tough to measure, and things that can’t be measured).

And yet, so much public policy and programming today blithely assumes that all things are measurable, the evidence notwithstanding.

Finding one counter-example

If the statement that “everything is measurable” is true, then I should not be able to come up with any examples of things that you think are “not measurable.”  In other words, logically, finding one counter-example makes the statement false. 

So, let me ask you, which of your parents (or kids, or siblings, or spouses, or significant others, or friends) do you love more, and exactly how much more?

Or, think back to the best teacher you have ever had.  Think about the thing that he or she taught you that was the most important lesson you ever learned.  Can you measure the value of that lesson?  And exactly how much better was it than the second best lesson you ever learned?

If you have ever had serious medical care, was there someone who seemed to be incredibly caring, who gave you a sense that you were getting really good quality of care?  I don’t mean that the medical diagnosis or the procedures were accurate and well done.  I mean that you felt like the quality of the healing you received made you feel better, just for being the recipient of that quality of care?

And then, can you please put all four of these into one measurement, one “metric,” so that we can discuss the trade-offs that always have to be made in running an organization or in  public policy decision making?

Another colleague and I did a presentation a couple of years ago where we were questioning how well we can measure the effectiveness of graduate education in public health.  In order to try to explain our position, we asked a series of questions of our audience:

$          How good a parent are you?

$          What is the value of justice in our society?

$          What is a sufficient amount of freedom?

$          How much do you love your family?

$          How beautiful is a sunset?

$          How much creativity does it take to change important things?

Or, take two pieces of music, one that is exquisite, and another that is a ditty.  Exactly how much better is the artful piece over the mundane piece?

I’ll bet everyone reading this blog can think of something in their lives that is “beyond measure.”  Don’t you think?  And if so, then why do we persist in thinking that “everything is measurable?”

Yet, we do.

Yet, we do.

We want to know who the best teachers are.  So, we purport to measure how well their students perform on standardized tests, and then we attribute that to their teaching.  What is the evidence that this is a good measure?  In fact, the evidence suggests that it is not a good measure.  For instance, there are data that show that one year a teacher can have a high performance on this measure, and the next year, the same teacher, can be seen as a complete failure. 

Yet we persist in the illusion that our technology is adequate.  That it “can be adequate.”

Around the country, schools of higher education are coming up with measurable “competencies” to use to build their curricula.  Accrediting bodies make us do this.  We have to have measurable competencies for each degree program and then measurable competencies for each class.  Then we have to demonstrate that we have measured these competencies during the semester, to “prove” that we are teaching what we say we are teaching. 

One of the learning objectives for any class I teach is to make the students better “critical thinkers.”  For instance, in teaching how to manage not for profits, I think the difference between how well one of my former students manages and how well someone else manages will be, in large part, because one is better at judging the situations that they find themselves in, and is better at coming up with thoughtful, creative, consensus-building responses, based upon the literature and common experiences in the field.  So, can we measure critical thinking, placing that ability to do so on a yardstick?

 Measurement is important.  But it is not everything.  It probably isn’t even the most important thing.