Metrics – perverse incentives?

Trivia time! What do following events have in common?

  • In the American Southwest in the 1850s there was a high reward for the scalps of members of violent and dangerous Indian tribes. This led scalp hunters to slaughter thousands of peaceful agricultural Indians and Mexican citizens, women and children alike, for their valuable scalps.
  • In Vietnam, under French colonial rule, there was a program paying people for each rat pelt handed in. It was originally intended to exterminate rats, but it led to the farming of rats instead.
  • In the 19th century,  palaeontologists traveling to China used to pay peasants for each fragment of dinosaur bone that they produced. The measure was an instant success! It took them a while to discover that peasants dug up the bones and then smashed them into multiple pieces to maximise their earnings.

All these are examples of perverse incentives:  measures that have unintended and undesirable effects which go against the interest of the incentive makers. They become counterproductive in the end.

I’m probably suffering from an acute case of testing analogitis again, because over the years I have seen these things happen in testing as well:

  • Managers evaluating testers by the amount of bugs found.
    This resulted in the submission of tons of trivial and low-priority bugs. People that used to thoroughly investigate bugs and put a lot of time in pinpointing started lowering their standards. 
  • Managers evaluating testers by the amount of test scripts executed.
    This resulted in testers only focusing on scripts, not allowing themselves go off-script and investigate. This often meant going against their intuition for suspicious “smells” in the software, and it certainly did not encourage the use of exploratory testing.
  • Managers evaluating testers by the amount of “rejected” bugs.
    The rationale behind this was: less rejections mean more valid bugs, better bug descriptions and better researched bugs. But the result of the metric was that testers were reluctant to enter complex, difficult or intermittent bugs out of fear of them being rejected. But these are the bugs we want the team to tackle, right? 
  • Managers evaluating testers by the quality of the software.
    First of all, what is quality? If we use Jerry Weinberg’s definition, “value to someone (who matters)”, it becomes clear that any manager’s assessment of quality is highly subjective. If the rewards for testers depend on the quality of the software, that is highly unfair. We are no gatekeepers of quality; we cannot assure quality, because we do not control all aspects of it. The only thing such an incentive achieves is a highly regulated cover-your-ass culture with formal hand-offs, and certainly not team collaboration, continuous improvement or better software. 

These are all examples of metrics used as incentives for testers, but in most cases they just ended up creating a blame culture where quantity and pathetic compliance is valued above quality and creativity.

Dear managers, I’d say: focus on collaboration and team achievements, set goals for the team. Make the whole team responsible for the quality and the product. Then see what happens.

Volcanic systems thinking

General systems thinking and the effects of a volcano outburst

I believe testing is about looking at things from as many perspectives as you can. Testing is also about relating things to one another, seeing things in a greater context.  In that sense you could say that testing is applied systems thinking. 

Years ago, Michael Bolton pointed me to the amazing book “Introduction to general systems thinking” by Jerry Weinberg. Actually, he pointed me to practically every publication by Jerry Weinberg – I’m still trying to prioritise my reading list. The book taught me that taking a holistic view of a system within its environment, may enable us to see patterns of behavior/actions and recognize interactions and interdependencies among its components. That way, we can better understand the system, maybe even predict how it will evolve over time.

The recent volcanic eruptions of the Eyjafjallajökull in Iceland provide a great example of (and also an exercise in) real-life general systems thinking and how several systems are possibly interconnected. I started wondering… What can the possible consequences of the recent volcanic eruption be, worldwide? There is a fair chance that other volcanoes nearby will erupt too. Scientists say history has proven that when the Eyjafjallajökull volcano erupts, the Katla volcano follows — the only question is how soon. And Katla, located under the massive Myrdalsjokull icecap, threatens disastrous flooding and explosive blasts when it blows.

If we can rely on history repeating itself, we’re onto something big. Let’s look at a true story. In June 1783, the eruption of the Icelandic Laki volcano was the start of a chain of unlikely events that affected everyone’s lives:

The immediate impact was catastrophic: around 25% of the Icelandic population died in the famine and fluorine poisoning after the eruptions ceased. Around 80% of sheep, 50% of cattle and 50% of horses died because of dental and skeletal fluorosis.

The rest of Europe soon followed:

  • A thick, poisonous smog cloud floated across the jet stream, resulting in many thousands of deaths throughout 1783 and the winter of 1784. Inhaling sulfur dioxide gas caused victims to choke. 
  • The thick fog caused boats to stay in port, unable to navigate.
  • Weather patterns started changing across western Europe:
    • The summer of 1783 was the hottest on record at that time. 
    • Severe thunderstorms with hailstones even killed cattle.
    • In 1784, a most severe winter caused 8,000 deaths in the UK.
    • During the melting that followed in spring, all of Europe reported severe flood damage.

And these were only the short term effects. The meteorological impact of the Laki eruption resonated on, contributing significantly to several years of extreme weather in Europe and the rest of the world:

  • In New Orleans, the Mississipi river froze, and ice started appearing in the gulf of Mexico.
  • African and Indian monsoon circulations weakened, leading to precipitation anomalies over the Sahel that resulted in low flow in the river Nile.
  • In France, a surplus harvest in 1785 caused poverty for rural workers.
  • For years afterwards, severe droughts followed. There were a series of bad winters and summers, including a violent hailstorm in 1788 that destroyed crops.
  • All this contributed significantly to the build up of poverty and famine that eventually triggered the French Revolution in 1789.

Wow. Time-out. Really? Could it be that the eruption of an Icelandic volcano lies at the basis of the French revolution, six years later? Since all these seemingly independent systems (meteorological, economic, agricultural and sociological) *are* connected, that’s perfectly plausible. Apparently, the indirect and long-term consequences of the eruption have greater impact than the initial effects of the event. Even art was affected. The most beautiful sunsets started appearing in late 18th-century paintings.

And what were the effects of the French revolution again? The abolition of Feudalism, for starters. The creation of a new order based on the famous ‘Declaration of the Rights of Man’. The main theme of the French Revolution, ‘Liberty, Equality and Fraternity’, later became one of the most famous political dogmas across the world. You could say that the Revolution paved the way for democracy. It brought about a lot of economic and social reforms, not only in France, but across Europe. Culture was also affected, at least in the short term, with the revolution permeating every creative endeavour. It changed the face of Europe: national identities joined forces, everywhere. In short: the revolution helped shape the future course of the world.

If we now think about the current eruptions of that rather friendly tongue-twister called Eyjafjallajökull, we notice that there are many more systems in play than there were in 1783. There’s commercial aviation and worldwide travel now; both of them fuel our economies and came to an abrupt stop. Airline stocks dropped rapidly. All this resulted in massive economic losses and temporary unemployment.  Several events are being cancelled because people cannot get there in time. People are trapped abroad or forced to stay at home. And these are only the short-term effects. I wonder what this year’s summer will be like. I already look forward to walking or skating across the channel to visit London. Or maybe throw a snowball or two at the gendarmes of Saint-Tropez.

And that’s just Eyjafjallajökull. Not Katla, not Laki. Just sayin’.