One month ago, my oldest daughter (6) started taking on rope skipping. The last time I had seen her practising, two weeks ago, she was still having trouble getting the rope neatly over (and under) herself, but yesterday she was able to complete several jumps in one go in a fluent movement. It was the first time I had seen her do that, so I was pretty impressed.
She was clearly in learning mode. I sat down to observe her more closely.
– “Wow, where did you learn all that?”
– “I’ve been watching older girls do that in school, daddy. Watch”.
She started jumping and counting out loud.
– “One, two, three, four, five, six, …”
She tripped on the rope.
– “Woohoo! Six!”
– “You go, girl!”
– “Again! One, two, three, four, five, nooooo…”
– “Five is good”.
– “No, daddy, five is not good. Again!”
She repeated the process a couple of times. She jumped seven (“Yes!”), four (“Nooo!”), five (“Pfff!”), six (“Yippie!”). I started noticing a pattern. It struck me that she alternated frustration with joy, and she let it depend on the number of jumps. Time for some questioning.
– “Why are you happy with anything above or equal to six, but unhappy with anything lower?”
– “It has to be at least six, daddy”.
– “Why six?”
She seemed really annoyed that I didn’t see her point. She thought I was pulling her leg.
– “Because I’m _six_ years old, daddy. Didn’t you know? What else could it be?”.
I was totally flabbergasted. She managed to impose some totally arbitrary pass/fail criteria on herself. Where did that come from? I thought that using pass/fail tests actually sabotages kids’ natural learning processes? But this appeared to come out of herself. No-one told her that she had to make at least six.
I wondered – maybe she just chose her age as a starting point, just to set some initial learning goals for herself? Was she planning on raising the bar later on when reaching six would have become too easy? Unfortunately I didn’t have the chance to follow-up on that – lunchtime!
Flash forward to work. All this reminded me of commonly defined pass/fail criteria such as
“90% of all tests must pass”
Really?
In “Are your lights on?”, Jerry Weinberg uses the well-known “Mary had a little lamb” nursery rhyme to show how a seemingly straightforward statement is prey to multiple interpretations, depending on which word you emphasize. An invaluable heuristic when looking at requirements. Why not try that on the familiar pass/fail criterium stated above?
– “90%”? What if the tests that would have revealed some serious errors happen to be in that 10% you so confidently dismissed? Why not 89 or 91?
– “All”? You know “all” possible tests that can be performed? Are they all documented? Some of them might still be residing in your head. What if in the meanwhile we performed some more important tests that revealed serious risks? Are these tests part of “All”?
– “Tests”? Do you only count scripted tests, or do you also take exploratory ones into account? What about important usability issues some users might have found? Or acceptance test checklists? Or automated checks?
– “Must”? What if not all 90% passes? Does this mean your solution is without value? The customer might value other things than you do. Is it up to you to decide how much value is in there?
– “Pass” ? What about behavior that is totally acceptable for your client, but that we find annoying? Pass or fail? What about tests that pass all steps, but that reveal important problems as a side-effect? Sometimes a test’s pass/fail decision is not binary.
My daughter went to school this morning and – for the first time – took her own jump rope with her. I wonder how many % of her rope jump cases will pass this time.
Test Side Story 
89% pass rate acceptable? Why that is just lunacy!
I like the lessons you draw on arbitrary standards and how fuzzy even strict numeric standards can be. As I read the first part of the story, though, my mind took it in a different direction–how pass/fail criteria can aid learning and motivation.
It seemed significant that “she alternated frustration with joy, and she let it depend on the number of jumps.” I suspect that the number “six” was the passing standard because it was achievable, but not all the time. A well-chosen criteria can goad us forward to strive harder. A goal that is either too easy or out of reach will not generally do that.
Setting our own goals, rather than having them externally imposed, makes it easier to set them in this “just right” zone. That’s not always the case, though. Sometimes we undermine ourselves by setting goals of perfection or expecting ourselves to perform at a really high level when we’re just learning. Setting our goals out of reach is a good way to fail in learning something.
Setting our own goals also allows us to make the reward and punishment appropriate. Notice that your daughter rewarded herself with a “yippee” and punished with a “pfff.” Anything more serious than that, and the reward and punishment would likely dominate to the detriment of the learning. Sometimes we do this to ourselves, also.
When trying to help someone else learn, it’s even more difficult to use pass/fail criteria at exactly the right level to enhance learning.
Exactly! Carefully chosen pass/fail criteria in learning are of a totally different order. She chose “six” as her initial passing standard (she chooses the number six a lot since she turned six a month ago – that’s why I suspect it was arbitrary). It would be interesting to see whether she will continue to set goals within reach like that and whether she will eventually raise that standard herself.
Thanks for taking the time to comment!
Good points Zeger!
I remember years ago we used to have a test phase with a 95% pass rate for exit criteria. Some people (especially PM’s) loved this – it made an aspect of their job easy. I was a test lead and whether we’d met it or not I’d always question the “finishing line” – i.e. “is this what you really want, and what do you think it is this is telling you?” Some got it, and some didn’t.
But I think these criteria still exist – sometimes they’re just hidden by someone’s perception (cognitive) bias – they like certain numbers (especially high pass rates and low numbers of unsolved faults).
I very rarely use or talk about test execution results using numbers anymore – the testing teams I work with realise this – and we get to talking about the product, issues and the silent evidence before any numbers are ever mentioned. Then you start talking about things that matter – quicker.