metrics

Goodhart's Law and the Church

On a recent episode of Planet Money entitled The Laws Of The Office, they bring to light Goodhart’s law. The hosts define Goodhart’s law in this way, “if a company decides to measure something, workers will find a way to respond with good numbers.”

https://towardsdatascience.com/unintended-consequences-and-goodharts-law-68d60a94705c

https://towardsdatascience.com/unintended-consequences-and-goodharts-law-68d60a94705c

The Wikipedia entry quotes Marilyn Strathern who summarized the law as: "When a measure becomes a target, it ceases to be a good measure."

This got me thinking about the Church, specifically when the Church decides to measure things. Like other organizations, the Church measures a number of inputs in an attempt to get a full picture of the state of the Church. If Goodhart’s Law is true, once we choose a measure to measure our Churches, it is no longer a helpful measure because people will find ways to respond with good numbers.

In theology terms, this is called living under the Law. Living under the Law means that when we find ways to measure, humans, who are susceptible to Sin, will find ways to look good under the parameters of the Law. Knowing where we stand in relation to others is a key characteristic to Law living.

The Gospel smashes these hierarchies and comparisons. The Gospel proclaims that everyone is forgiven and make whole. This leveling of the playing field is met with great suspicion when we live under the Law (how can we know who is the best or at least who has “earned” the honor we give them?) Recall when Jesus’ parable of the workers who each received the same wage regardless of hours spent working? Or the idea that the first will be last and the last will be first? Jesus proclaimed a Gospel of freedom from the Law.

Even Goodhart’s Law.

Tyranny of Metrics

Jerry Muller's book, The Tyranny of Metrics, examines how fixating on creates a number of problems. The author states:, 

This book is not about the evils of measuring. It is about the unintended negative consequences of trying to substitute standardized measures of performance for personal judgment based on experience. The problem is not measurement, but excessive measurement and inappropriate measurement—not metrics, but metric fixation.

The book reads as a cautionary tale for the Church. The more the Church reads reports that "our numbers are in decline" the tighter we cling to metrics. As the Church faces a legitimacy and relevancy crisis in the culture, there is a temptation to fixate on metrics as a way to show legitimacy and relevancy. 

As the author says, metrics are not evil. It is difficult to diagnose a problem if there are not some measurements we can look at over time to make adjustments. It was once said to me that the numbers we look at in the church are similar to the numbers a doctor looks at when you go in for a check up. These "check up" numbers do not tell you everything about your health, but they are a starting point. If your breathing is consistently slow it could mean you are a super healthy marathoner but it could also mean you are close to death.

It is the fixation on metrics that is creeping into the Church that is a cause for concern. 

So what does metric fixation look like? Muller describes it in this way:

  • The belief that it is possible and desirable to replace judgment, acquired by personal experience and talent, with numerical indicators of comparative performance based upon standardized data (metrics)
  • The belief that making such metrics public (transparent) assures that institutions are actually carrying out their purposes (accountability)
  • The belief that the best way to motivate people within these organizations is by attaching rewards and penalties to their measured performance, rewards that are either monetary (pay-for-performance) or reputational (rankings)

Paradoxically using metrics as the means to gain a sense of clarity of a situation and using metrics as the primary measure of success creates misaligned incentives. For instance, the surgeon who wants to have more patients may talk about how many successful surgeries they have preformed. However, what she/he fails to communicate is that the they only take cases that have a high probably of success to begin with. Thus the "successful' surgeon may not who you want for your complicated surgery. This example might be called, "creaming" the numbers. That is the surgeon counts only the "cream of the crop." The patient and the surgeon have misaligned incentives. 

Muller points out a series of reoccurring flaws in using metrics:

  1. Measuring the things that are easy to measure (such as people in worship) 
  2. Measuring inputs (such as money) over outcomes (such as a transformed life)
  3. Creaming (counting only the best)
  4. Lowering standards (calling a gathering a worship, then add those numbers to weekly worship total)
  5. Omitting or distorting data (such as double counting a person who attends more than worship hour on Sunday)
  6. Cheating (such as when the preacher adds to the numbers because "it felt like there were more people there...")

The reality is metric fixation is killing clergy and creating cultures where churches are dominantly assessed through what is easily counted rather than through the Spirit. How we overcome metric fixation is a difficult but not impossible process. Metrics are only a small picture of reality, but because it is a number it weighted heavier. Metrics give the impression of concreteness and accessibility to situations that are ambiguous and complex. There is a desire to simplify the complexities of the world giving us a false sense of control and understanding. 

Fixating on metrics means that when a church provides their "check up" numbers, we forget that sometimes the heartbeat of the church looks good because there is a pacemaker modifying the heartbeat. Just because a church looks strong in the metrics might mean they are "juicing" in subtle (and even purely motivated) ways. Just because a church looks weak in the metrics does not mean that the church is dying or failing. 

And even if it is dying, the Church of Jesus Christ is not dead for too long because Sunday is coming. 

Teaching to the test - Minister to the metrics

When standardized testing in our schools came into being, I imagine that it was generally seen as a fine idea. Most teachers already used tests to assess learning and it is easy to see how making a standard test, while not perfect, could be valuable to assess student learning.

I imagine that the first generation of teachers and administrators used these standardized tests as they were intended to be used. That is as an assessment of student comprehension and learning. Again, testing is not perfect, but it is a tool among many tools that are helpful.

Generations have passed since the beginnings of standardized tests and now we have teachers and administrators no longer able to focus on student comprehension and learning, but are now (in many ways) forced to "teach to the test". 

Call it unintended consequences, but teaching to the test is rather common not just on standardized tests in grade school but also in things like the SAT, ACT, GMAT, GRE, and a slew of other standardized tests. Focusing on the test makes us able to do well on the test but not always do well in life.

The UMC recently has implemented a way to measure "fruitfulness" in the local church. These numbers, called "metrics", are just a handful of numbers that each local church is asked to plug into a dashboard each week. They include numbers such as, number of small groups, number of people in mission, offering collected, people in worship, etc. While we can debate the difficulty of capturing some of these numbers (such as how do you know how many people are in mission in a given week?) generally I do not think that keeping track of metrics is a bad thing.  Numbers are a helpful tool to assess and open conversation about the ministry of the local church.

But just as standardized tests may have some unintended consequences, so to do the metrics. I have concern that over time we will begin to minister to the metrics. 

For instance we can create specific programs or re-define existing ministries that will enhance the metric count. For instance, in some ways we could count choir practice as worship, and thus enhance the worship metric. Perhaps we count money that is collected for a fellowship meal as "money given to missions" because we also feed those who show up but are unable to pay. These are silly examples of course but the point is at some point you have to wonder if the church will begin to minister to the metrics?

I am not against these current metrics, but if we are going to count something and what we count will eventually shape the ministry we do, then could we count some additional things? For instant:

  • Number of people brought out of poverty
  • Number of homeless people you know by name
  • Number of people who fasted this week
  • Dollars given to mission as a result of a boycott
  • Number of failures
  • Number of new groups started
  • Number of groups that ended/concluded
  • Number of people your church is mentoring into ministry
  • Amount of time members were in silence

As a Church, we will minister to the metrics, and that is to be expected, the question really is what metrics will we minister to?

Heisenberg and church metrics

The UMC is getting on the bandwagon of big data. The church has always collected data such as number of people in worship and the number of people who transfer out of a congregation, but these numbers have always been kept on paper. Recently the UMC has shifted to keeping track of these and other numbers through computer programs. But it has only been the past couple of years that the UMC is beginning to make big data a conversation point.

Each week every United Methodist local church is to log into a system and upload a series of numbers. Some of these numbers are easy to track, such as worship attendance or dollars given to mission. Other numbers, such as mission outreach and small group participation are a little harder to report. In the beginning there were about 5 numbers, now there are 8.

Once that data is loaded up, we are then given visual representations of the data. (You can see SUMC's data here)

Great.

decline_420.jpg

While the debate rages on about how these numbers and data will be used, I wonder about Heisenberg's principle of uncertainty

This quantum physics concept says that you can either know the position or the speed of an object, but you cannot know both at the same time.

And so I wonder, while we are measuring the current position of the church, we are not able to see the direction the church is headed. 

It is easier to measure the current position of the church, I am much more interested in measuring the momentum the church is moving in and if/how that can be changed if needed. 

It is like in a basketball game. You can look at the scoreboard and see it is 85-95 and know the current position of the game. But you cannot know which team has the momentum. It only takes a three shots for the team down by 10 to be right back in the game, but the scoreboard cannot tell you who has the momentum of the game. 

The UMC has built a nice scoreboard, but frankly that is not the part of the game that gets me motivated to watch the game.