Friday, February 13, 2015

Measurement: You can't measure everything



A colleague of mine enjoys saying that “you can’t manage what you can’t measure.”  When I point out to him the things that we need to manage but can’t measure, he responds by saying that that is the challenge, to find measurements for those things, because, if we don’t, then we won’t be able to manage them. 
When my colleague looks at the gap between what we would like to be able to measure, and our current abilities, he sees this as a technological gap.  Give us more time, more effort, more research, more hard work, and we can close that gap, he thinks.
            In today’s America, this is the majority opinion.  In fact, it appears to be an overwhelming majority opinion.
So, I know that my opinion is heretical.  But what can I do?  The overwhelming majority is wrong on this issue.
Whose burden of proof?
First, let’s ask an intriguing opening question.  Whose responsibility is it to prove their side?  Right now, the majority thinks that their position is obvious.  All things are measurable, although we might not know how to measure everything yet.  If you don’t agree, prove it.
But mightn’t the “status quo ante” be that not everything is measurable, and it is up to those who think everything is measurable to prove their position?  Given that, in our common experience, there are many, many things that “resist” measurement, and given that we have no direct evidence of the correctness of the notion that all things can be measured, no experience of it whatsoever, why is the starting point the position for which there is no experiential evidence?
My view is that we should start from the assumption that measurement is far more difficult to do than we think, and that some things are inherently unmeasurable.  Now it is up to the measurement gurus to prove that there is only one category (things that are tough to measure) rather than two (things that are tough to measure, and things that can’t be measured).
And yet, so much public policy and programming today blithely assumes that all things are measurable, the evidence notwithstanding.
Finding one counter-example
If the statement that “everything is measurable” is true, then I should not be able to come up with any examples of things that you think are “not measurable.”  In other words, logically, finding one counter-example makes the statement false. 
So, let me ask you, which of your parents (or kids, or siblings, or spouses, or significant others, or friends) do you love more, and exactly how much more?
Or, think back to the best teacher you have ever had.  Think about the thing that he or she taught you that was the most important lesson you ever learned.  Can you measure the value of that lesson?  And exactly how much better was it than the second best lesson you ever learned?
If you have ever had serious medical care, was there someone who seemed to be incredibly caring, who gave you a sense that you were getting really good quality of care?  I don’t mean that the medical diagnosis or the procedures were accurate and well done.  I mean that you felt like the quality of the healing you received made you feel better, just for being the recipient of that quality of care?
And then, can you please put all four of these into one measurement, one “metric,” so that we can discuss the trade-offs that always have to be made in running an organization or in  public policy decision making?
Another colleague and I did a presentation a couple of years ago where we were questioning how well we can measure the effectiveness of graduate education in public health.  In order to try to explain our position, we asked a series of questions of our audience:
$          How good a parent are you?
$          What is the value of justice in our society?
$          What is a sufficient amount of freedom?
$          How much do you love your family?
$          How beautiful is a sunset?
$          How much creativity does it take to change important things?
Or, take two pieces of music, one that is exquisite, and another that is a ditty.  Exactly how much better is the artful piece over the mundane piece?
I’ll bet everyone reading this blog can think of something in their lives that is “beyond measure.”  Don’t you think?  And if so, then why do we persist in thinking that “everything is measurable?”
Yet, we do.
Yet, we do.
We want to know who the best teachers are.  So, we purport to measure how well their students perform on standardized tests, and then we attribute that to their teaching.  What is the evidence that this is a good measure?  In fact, the evidence suggests that it is not a good measure.  For instance, there are data that show that one year a teacher can have a high performance on this measure, and the next year, the same teacher, can be seen as a complete failure. 
Yet we persist in the illusion that our technology is adequate.  That it “can be adequate.”
Around the country, schools of higher education are coming up with measurable “competencies” to use to build their curricula.  Accrediting bodies make us do this.  We have to have measurable competencies for each degree program and then measurable competencies for each class.  Then we have to demonstrate that we have measured these competencies during the semester, to “prove” that we are teaching what we say we are teaching. 
One of the learning objectives for any class I teach is to make the students better “critical thinkers.”  For instance, in teaching how to manage not for profits, I think the difference between how well one of my former students manages and how well someone else manages will be, in large part, because one is better at judging the situations that they find themselves in, and is better at coming up with thoughtful, creative, consensus-building responses, based upon the literature and common experiences in the field.  So, can we measure critical thinking, placing that ability to do so on a yardstick?

 Measurement is important.  But it is not everything.  It probably isn’t even the most important thing.

Muddleheadedness, when others fetishize simplemindedness

I think about too many things.  It would be a lot better to be a complete and utter expert on exactly one important thing.  Then I could offer a set of blogs on that thing, and everyone would know what to expect.  But that isn’t the way my mind works.

One early 20th century English philosopher said that the world can be divided into the simpleminded and the muddleheaded.  Everyone was one or the other.  A very few people can be both, but never simultaneously. 

The simpleminded can focus on “the instant,” a specific item or event or phenomenon.  They can know a very great deal about that instant, in detail and with precision.  But the effort to understand the complexity of that simple thing keeps them from being able to understand fully the context of that thing.  They can draw a brilliant portrait, but won’t have much idea of background.  Think Rembrandt.

The muddleheaded understand the milieu, the pattern and how everything within it inter-relates.  They can master complexity of context.  But in doing so, they have no remaining ability to understand in detail the specific components of that context.  They might paint beautiful landscapes, in which the people are not much more than smudges.  Think Turner.

Society needs both types of minds.  I want all the bridges I drive over to be designed by simpleminded engineers.  I want them to know, in exquisite detail, the strengths and the loads, and the qualities of the materials.  God forbid we ask a muddleheaded person to do that work.

But too often we ask the person who designs the bridge to tell us where to build it.  Not the same skill set.  For deciding where to build the bridge, you need to know about the context, the land uses, the life of the people in the area, their history and their future.  Siting bridges is muddleheaded work.

However, we live in a society that currently fetishizes simplemindedness.  Want to make kids learn better?  Give them tests.  What to improve the quality of healthcare?  Design a measurement tool.  Want to be a Good person?  Follow these rules.  Want to get a promotion?  You have to show that you are the best at the given task you have been asked to perform.  Want a grant?  You have to demonstrate your simpleminded devotion to the details of what you wish to study. 

Tenure?  Only to the simpleminded.  But, after a professor has done 25 years of increasingly detailed simpleminded work, and been acknowledged and lauded for it, we then ask him or her to answer muddleheaded questions in his or her field.  What is there about decades of simplemindedness that prepares one to respond to muddleheaded problems?  Not much.

Some few people can be both, but only at different times.  In a kind of Heisenbergian catch, the best of us can only understand one or the other at any one moment.

Public health, the field where I have done most of my work, is inherently muddleheaded.  We want to know about how things inter-relate in communities, how systems can be made to respond to problems, how power is used within the body politic, how groups of people cooperate in the Public Good.  Of course, we have to have simpleminded people, too.  We need people to study the chemistry, the microbiology, the statistics.  But there work must be entered into a muddleheaded system in order to make a difference.

Politics is a muddleheaded pursuit.  So is ethics. And religion.  And management.  And cooking. Sometimes, the practice of  law.  Many of the things to which I am drawn.  So, this blog that I am starting today will cover a lot of different topics.  Sometimes they will relate to each other; sometimes they will not. 

I believe some of my ideas are important, and I feel the need to share them.  Some of my ideas are merely interesting, and I hope you find them so.  Some of the things I will share are just simply fun.  Along the way, I hope you will find these concepts presented clearly and well.  I will be disappointed if we don’t find a few laughs, too.

Time to begin.

DS