Unrest assured

“Eight! Five! Four! Three!”.

Four words that I am tired of hearing by now.

“Eight! Five! Four! Three!”.

My oldest (8yo) daughter finally saved up some money for that secret digital diary, and she was so excited she could hardly get to sleep. “Tomorrow I will go buy my secret diary! And it’s totally secret! You can only unlock it by whispering the secret password! It’s gonna be neat, daddy!”.

Next day I looked up what the fuss was all about, and found this description on the manufacturer’s website:

“Rest assured that your secrets will be kept fully safe in this digital diary.
With VTech’s revolutionary voice recognition technology, you can set an audio password that only you can enter to the diary. Even when others find out about your password, the voice recognition feature will only recognise the tone of your voice, and only let you in.”

Wow. That actually sounded pretty cool for a kid’s toy. I would love to get my hands on that one, explore it to test their pretty bold claims. When I came home from work the next day, my daughter beat me to it. She had locked herself up in her room with her new secret toy, only to come out a couple of hours later.

“This is so cool, daddy. Look!”

She pressed a button on the pink gizmo.

“Say your password” – Beep – “Eight! Five! Four! Three!”. “Password not recognized”. Umph.
“Say your password” – Beep – “Eight! Five! Four! Three!”. “Password not recognized”.

No signs of frustration on the 8 year-old face yet. “Hold on.”

“What’s happening here? It doesn’t work?”.

“Yes it does. But not always. Sometimes it just doesn’t recognize the password.”

“Your very secret password? The 8543 you keep on yelling at it?”

“Yes”

“Say your password” – Beep – “Eight! Five! Four! Three!”. “Welcome” <harp music>.

“See? Now it works.”

“Well, that’s strange, don’t you think? What did you do differently now compared to your previous attempts?”

“No idea, daddy”. She closes the thing again. Beep. “8!5!4!3!”. “Password not recognized”.

It puzzles me that she’s not really interested in the cause. The thing only works one out of four attemps, and she seems perfectly fine with it.

A couple of 8543-filled days later, as the oldest daughter is off to a birthday party, the youngest one (5 yo) decides to seize the opportunity. She had been quietly watching her sister the past days, not even bothering to try – probably knowing that the oldest would strongly object.

“Can I have a look at sis’ secret diary daddy?”

“Sure. But you probably won’t get too far, it is made especially to prevent others from…”

“Say your password” – Beep – “Eight Five Four Three”. “Welcome” <harp music>. Big triumphant smile.

On. Her. First. Try. And their voices don’t even sound alike!
Granted, they’re sisters, their voices may be similar, but still there is a significant difference. And I would think that this thing is made specifically to shield your secrets from curious sisters.

Is there a problem here? What happened? After this revelation I tried to open it, and it manages to keep me locked out, where I should be. My wife is not able to enter the realm of secrets either. But little sisters, on the other hand, seem to have a bigger accuracy rate than the actual voice owners.

So, two very apparent problems:

– The diary doesn’t open when it should
– The diary opens when it shouldn’t

It turned out that the reason why the oldest had a hard time getting passed the passphrase was her impatience. Impatience is her middle name – she just couldn’t wait to yell 8-5-4-3 until a while after the beep – most of the time she started the password phrase at the same time of the beep. When she respected the little pause, her success rate tripled.

The second problem is a bit trickier. I’m trying to imagine if and how this toy was tested. Security for kids is probably not very high on most stakeholders’ priority lists. They probably tested a couple of scenarios with kids of different ages and gender, maybe a deep-voiced parent here and there. Did they include voices from siblings? Siblings aren’t corner cases – they represent realistic real life scenarios, with a high probability of diary sneaking. Granted, the kids don’t really make much of it, but the oldest was not amused when she found out about the security breach. If you make these bold claims about your product, and your product’s whole advertising and unique selling proposition is centered on this very feature, can you please make sure that it – you know – “works”?

V-Tech – frustrating tester parents since 2013.

Advertisement

Delivering the message

We are testers. We communicate, collaborate and report. We are the headlights of a project. We light the way, break the news, deliver messages – and they’re not always pretty. How do we make sure our point is taken? It all boils down to adapting our style of delivery depending on the content and the audience. Context is indeed everything.

The delivery of a speech is mostly done through the nonverbal channel (whereas the content itself is verbal). This includes all speech elements other than the words themselves: eye contact, voice, articulation, gestures, facial expressions, body language – even appearance. Using all of these effectively requires timing and practice, lots of practice. An additional problem is that there is a wealth of delivery styles to choose from. How do we get the message across in the best possible way?

A really smart person once said: “Smart people learn from their mistakes, but REALLY smart people learn from other people’s mistakes.”

I think that’s true. In the video below, councillor Phil Davison – a candidate for the position of Republican treasurer for Stark County, Ohio –   delivers a passionate nomination speech in front of a receptive audience. His delivery style is – well… – pretty peculiar. 

Spoiler: he wasn’t  elected.

Failure is always an option – part 2 (wartime failures)

Wartime failures

In my search for information on failed software development projects, I was frequently reminded of the fact that it’s not always software projects that fail. In many cases, I even wondered why these projects were even started in the first place. Some of them seem to come straight from a Monty Python movie – downright absurd. Needless to say that their eventual cost far outweighed the benefits, if any.

I discovered* that wartime was a true breeding ground for many beautiful and poetic failures. Anything goes when there’s an enemy waiting to be crushed in the most creative ways possible:

  •  The Acoustic Kitty project:
    A CIA project in the 1960s attempting to use cats in spy missions. A battery and a microphone were implanted into a cat and an antenna into its tail. Due to problems with distraction, the cat’s sense of hunger had to be addressed in another operation. Surgical and training expenses are thought to have amounted to over $20 million. The cat’s first mission was eavesdropping on two men in a park. The cat was released nearby, but was hit and killed by a taxi almost immediately. Shortly thereafter the project was considered a failure and declared to be a total loss.
  • Operation Cornflakes:
    A World War II mission in 1944 and 1945 which involved tricking the German postal service Deutsche Reichspost into inadvertently delivering anti-Nazi propaganda to German citizens through mail. The operation involved special planes that were instructed to airdrop bags of false, but properly addressed mail in the vicinity of bombed mail trains. When recovering the mail during clean-up of the wreck, the postal service would hopefully confuse the false mail for the real thing and deliver it to the various addresses. The content was mainly anti-Nazi-propaganda. In addition, the postage stamps used were subtly designed to resemble the standard stamp with Adolf Hitler’s face, but a close examination would reveal that his face is made to look like an exposed skull or similarly unflattering imagery. The first mission of Operation Cornflakes took place in February 1945, when a mail train to Linz was bombed. Bags containing a total of about 3800 propaganda letters were then dropped at the site of the wreck, which were subsequently picked up and delivered to Germans by the postal service. Not too sure how many German families were converted by these letters.
  • The Bat Bomb project:
    Bat bombs were bomb-shaped casings with numerous compartments, each containing a Mexican bat with a small timed incendiary bomb attached. Dropped from a bomber at dawn, the casings would deploy a parachute in mid-flight and open to release the bats which would then roost in eaves and attics. The incendiaries would start fires in inaccessible places in the largely wood and paper construction of the Japanese cities that were the weapon’s intended target. Eventually, the program was cancelled it became clear that wouldn’ t be combat ready until mid-1945. By that time it was estimated that $2 million had been spent on the project. It is thought that development of the bat bomb was moving too slowly, and was overtaken in the race for a quick end to the war by the atomic bomb project.
  • Project Pigeon: .
    During World War II, Project Pigeon was B.F. Skinner‘s attempt to develop a pigeon-guided missile. The control system involved a lens at the front of the missile projecting an image of the target to a screen inside, while a pigeon trained to recognize the target pecked at it. As long as the pecks remained in the center of the screen, the missile would fly straight, but pecks off-center would cause the screen to tilt, which would then, via a connection to the missile’s flight controls, cause the missile to change course. Although skeptical of the idea, the National Defense Research Committee nevertheless contributed $25,000 to the research. Skinner’s plan to use pigeons in Pelican missiles was considered too eccentric and impractical; although he had some success with the training, he could not get his idea taken seriously. The program was canceled on October 8, 1944, because the military believed that “further prosecution of this project would seriously delay others which in the minds of the Division have more immediate promise of combat application.”

It’s probably no coincidence that the majority of these projects involved animals. In that case, failure is certainly an option – I heard that working with animals is highly unpredictable, hard to manage and time-consuming.

Strange, isn’t that what they say about software development too?

*source: wikipedia

Failure is always an option – part 1 (chaos)

About the Chaos report

One of the most popular reports people use to showcase failure of software development is the chaos report from The Standish Group. The Standish Group collects information on project failures in the software development industry in an attempt to assess the state of the industry. 

In 1994, they reported a shocking 16 percent project success rate, another 53 percent of the projects were challenged (not on time, over budget and with fewer functions than originally specified), and 31 percent failed outright.  Although the newer reports show better numbers, the overall results still paint a dire picture:

  1994 1996 1998 2000 2002 2004 2006 2009
Successful   16%   27%   26%   28%   34%   29%   35%   32%
Challenged   53%   33%   46%   49%   51%   53%   46%   44%
Failed   31%   40%   28%   23%  15%   18%   19%   24%

There aren’t a whole lot of other statistics out there on this topic, so obviously these numbers get big play. Guilty as charged, your honor. I have used them myself, in a presentation or two.

I won’t be doing that again.

I realized that I have some serious problems with these metrics. They measure a project’s success by solely looking at whether the projects were completed on time, on budget and with required features and functions. But what they do not take into account are things like quality, risk and customer satisfaction. Could it be that an extremely unstable, unusable and frustrating piece of sofware that was delivered on time and on budget qualifies as a success? I beg to differ.

The Standish Group’s methods are not fully disclosed, and the bits that are disclosed are apparently deeply flawed. Their figures are misleading, one-sided and meaningless – the results are completely unreliable. They present their figures as absolute facts, but I lack clear context. The most famous sceptics of the report are Jørgensen and Moløkken. They emphasize its unreliability and question the claim of a “software crisis”:

” Even the definition of challenged projects is not easy to interpret. It is defined as “The project is completed and operational but over budget, over the time estimated, and offers fewer features and functions than originally specified.” The problem here is the use of “and” instead of “or”, combined with the following definition of successful projects: “The project is completed on-time and on-budget, with all features and functions as initially specified.” Consider a project that is on-time, and onbudget, but not with all specified functionality. Is this project to be categorized as challenged or successful? Our guess is that it would be categorized as challenged, but this is not consistent with the provided definition of challenged projects. ” 

In the comments section of an interview with The Standish Group’s Jim Johnson, Jørgensen brought up his critique of the CHAOS report and asked Johnson two very fair questions. Johnson’s reply is pretty enlightening, to say the least. Here are a few excerpts:

…We are an advisory research firm much like a Gartner or Forrester. Neither they nor we can afford to give our opinions away for free. We have facilities, utilities, and personnel and we must, the same as you, be able to pay our bills. Just because someone asks a question, does not mean we will respond with an answer. In fact, we most likely will not

…Our current standard answer to a CHAOS inquiry is, first: please purchase our new book, ”My Life is Failure” in our online store. If that does not satisfy you, then you need to join CHAOS University. If you do not find your answer or answers there then you need to purchase our inquiry services. Then we will work to answer your questions…

…It is strange that Jørgensen has never applied or professed interest in joining us. Some answers can be found if you join us at CHAOS University 2007 or one of the many outreach events. So you can contribute to the CHAOS research by providing funding or sweat, but short of that you will and must be ignored by design…

Don’t get me wrong. I think there *are* lots of failing software development projects, but in other numbers and for other reasons than the ones Standish brings forth: deliveries that do not bring any value to its users, software that was poorly tested or poorly designed, resulting in failures in production.

The problem I have with the Chaos Report is that they claim to be some kind of “industry standard”, projecting a false image of the dire state of the software industry, based on poor metrics. And I certainly don’t believe in the “quality is dead” mantra that resonates from their reports. Sure, there’s plenty of chaos out there, but I like what Henry Miller said about that : “Chaos is the score upon which reality is written”.

I’m with Henry on this one.