Testing analogitis (a phantom menace)

Blame this post on my wandering mind. I’m suffering from a severe case of analogitis. I’m starting to see testing analogies everywhere.

On  April 25, 2007, a 22-year-old female police officer was fatally shot in Heilbronn, Germany. The analysis of the DNA that was found at the crime scene revealed some astonishing information. The DNA belonged to a woman and started popping up in several seemingly unrelated cases, some of them dating as far as fifteen years back. Traces were found:

• on a cup after the killing of a 62-year-old woman in Idar-Oberstein, Germany
• on a kitchen drawer after the killing of a 61-year-old man in Freiburg, Germany
• on a syringe containing heroin near Gerolstein, Germany
• on the leftovers of a cookie in a trailer that was forcefully opened in Budenheim, Germany
• on a toy pistol after the robbery of Vietnamese gemstone traders in Arbois, France
• on a projectile after a fight between two brothers in Worms, Germany
• on a stone after a burglary in Saarbrücken, Germany
• after a burglary at an optometrist’s store in Gallneukirchen, Austria
• after 20 burglaries and thefts of cars and motorbikes Germany and Austria
• on a car used to transport the bodies of three Georgians killed in Heppenheim, Germany
• after a burglary in a disused public swimming pool in Niederstetten, Germany
• after four cases of home invasion in Quierschied, Tholey and Riol, Germany
• after an apartment break-in in Oberstenfeld-Gronau, Germany
• after the robbery of a woman in a club house in Saarhölzbach, Germany
• in the car of an auxiliary nurse who was found dead near Weinsberg, Germany

The so-called “phantom of Heilbronn” was born. An unknown woman was scattering her DNA all over the place, committing murders, breaking into houses, eating cookies, drinking beer, toting toy guns, she was even shooting up heroin. Profilers could tell that she was quite a busy lady, but she had no identity – the police had absolutely no clue. No Mentalist or CSI Heilbronn coming to the rescue – things were looking bleak.

But in March 2009, the case took a new turn. Investigators discovered the very same DNA sequence on the burned body of a male asylum-seeker in France – an anomaly, since the sequence was of a female. They eventually found out that the phantom serial killer did not actually exist and that the laboratory results were due to contamination of the cotton buds used for DNA probing. They discovered that the cotton swabs were already contaminated before shipping, and that they all came from the same factory. That factory, in turn, employed several Eastern European women who fit the type the DNA was assumed to match. The cotton swabs were not intended for analytical, but only medical use. They were sterile, but not certified for human DNA collection.

At this point you are probably wondering what all this has to do with testing. Bear with me a little.

Reading the Phantom of Heilbronn story reminded me of a phenomenon I briefly described in my 2007 paper “Software testing, profession of paradoxes?“. When we investigate something, the outcome of our investigations is sometimes influenced by the observation itself. The very act of testing influences its outcome. This axiom is also often referred to as the ‘observer effect’.

The idea is that since any form of observation is also an interaction, the act of testing itself can also affect that which is being tested. For example:

• When log files are used in testing to record progress or events, the application under test may slow down drastically
• The act of viewing log files while a piece of software is running can cause an I/O error, which may cause it to stop
• Observing the performance of a CPU by running both the observed and observing programs on the same machine will lead to inaccurate results because the observer program itself affects the CPU performance
• Observing (debugging) a running program by modifying its source code (e.g., adding extra output or generating log files) or by running it in a debugger may cause certain bugs to diminish or change their behavior, creating extra difficulty for the person trying to isolate the bug (also known as a ‘Heisenbug’)

Paradoxically, software testing is not always considered as the best way toward better quality. Just like in the Phantom of Heilbronn case, the advanced state of the technology might work against us, rather than with us. In 1990, Boris Beizer described the “complexity barrier principle” (1990): software complexity (and therefore that of bugs) grows to the limit of our ability to manage that complexity. 

Sometimes, testing is not the best option. Sometimes, testing and fixing problems does not necessarily improve the quality and reliability of the software. Oh sweet paradox, how can I embrace thee? Let me count the ways…