Rapid Software Testing – skilled software testing unleashed

Up to 11

The software testing profession looks like a steadily maturing profession from the outside. After all, there are certifications schemes like ISTQB, CAT, IREB and QAMP (the one to rule them all), standards (ISO 29119) and companies reaching TMM (test maturity model) levels that – just like a Spinal Tap guitar amplifier – one day might even go up to 11. The number of employees that companies send off to get certified in a mere three days is soaring, and new certification programs are being created as we speak. Quick and easy. Multiple choice exams for the win!

The reality, however, is that the field of software testing is torn between different “schools” of testing. You could see these schools as determined and persistent patterns of belief, speech and behaviour. This means that different people – all calling themselves “test professionals” – have vastly different ideas of what testing is all about. Even something elementary as the definition of testing varies from “demonstration of fitness for purpose” to “questioning a product in order to evaluate it”, depending on who you talk with (for more info on the schools of software testing, I heartily recommend Brett Pettichord’s presentation on the subject).

And so it happens that different people think differently about “good” or “mature” software testing. I, for one, don’t believe in tester certification programs, at least not in the format they are in now and the way they are being used in the testing profession. The current business model is mainly designed to get as many people as possible certified within the shortest timeframe. Its prime focus is on certifiability, not on tester skill, and certainly not on the advancement of the craft. Advancement comes from sharing, rather than shielding.

Rapid Software Testing (RST)

So what are the options for a tester on a quest for knowledge and self-improvement? What is a budding tester to do?

I think there are valuable alternatives for people who are serious about becoming a world-class tester. One of these is Rapid Software Testing (RST), a 3-day hands-on course designed by James Bach and Michael Bolton.

Actually, calling this “a course” doesn’t do it justice. RST is at the same time a methodology, a mind-set and a skill set about how to do excellent software testing in a way that is very fast, inexpensive, credible and accountable. It is a highly experiential workshop with lessons that stick.

How is RST different?

During RST you spend much of the time actually testing, working on exercises, puzzles, thought experiments and scenarios—some computer-based, some not. The goal of the course is to teach you how to test anything expertly, under extreme time pressure and conditions of uncertainty, in a way that will stand up to scrutiny.

The philosophy presented in this class is not like traditional approaches to testing, which ignore the thinking part of testing and instead focus on narrow definitions for testing terms while advocating never-ending paperwork. Products have become too complex for that, time is too short, and testers are too expensive. Rapid testing uses a cyclic approach and heuristic methods to constantly re-optimize testing to fit the needs of your clients.

What’s in it for you?

  • The ability to test something rapidly and skilfully is critical. There is a growing need to test quickly, effectively, and with little information available. Testers are expected to provide quick feedback. These short feedback loops make for more efficient and higher quality development
  • Exploratory testing is at the heart of RST. It combines test design, test execution, test result interpretation, and learning into a seamless process that finds a lot of problems quickly. Experienced testers will find out how to articulate those intellectual processes of testing that they already practice intuitively, while new testers will find lots of hands-on testing exercises that help them gain critical experience
  • RST teaches you how to think critically and ask important questions. The art of questioning is key in testing, and a very important skill for any consultant
  • RST will provide you with tools to do excellent testing
  • RST will heighten your awareness

Bold claim bottom-line:

RST will make you a better tester.

RST comes to Belgium

Co-learning and Z-sharp are proud to announce that from 30 September – 2 October, Michael Bolton will visit Belgium to deliver the first ever RST course on Belgian soil, giving you the opportunity to experience this unique course in person. More info can be found here, or feel free to contact us for more info.

Brace yourself for an mind-opening experience that will energize and boost your mind.

All the way up to 11.

(for even more information and testimonials about RST, see Michael Bolton’s RST page)

 

Unrest assured

“Eight! Five! Four! Three!”.

Four words that I am tired of hearing by now.

“Eight! Five! Four! Three!”.

My oldest (8yo) daughter finally saved up some money for that secret digital diary, and she was so excited she could hardly get to sleep. “Tomorrow I will go buy my secret diary! And it’s totally secret! You can only unlock it by whispering the secret password! It’s gonna be neat, daddy!”.

Next day I looked up what the fuss was all about, and found this description on the manufacturer’s website:

“Rest assured that your secrets will be kept fully safe in this digital diary.
With VTech’s revolutionary voice recognition technology, you can set an audio password that only you can enter to the diary. Even when others find out about your password, the voice recognition feature will only recognise the tone of your voice, and only let you in.”

Wow. That actually sounded pretty cool for a kid’s toy. I would love to get my hands on that one, explore it to test their pretty bold claims. When I came home from work the next day, my daughter beat me to it. She had locked herself up in her room with her new secret toy, only to come out a couple of hours later.

“This is so cool, daddy. Look!”

She pressed a button on the pink gizmo.

“Say your password” – Beep – “Eight! Five! Four! Three!”. “Password not recognized”. Umph.
“Say your password” – Beep – “Eight! Five! Four! Three!”. “Password not recognized”.

No signs of frustration on the 8 year-old face yet. “Hold on.”

“What’s happening here? It doesn’t work?”.

“Yes it does. But not always. Sometimes it just doesn’t recognize the password.”

“Your very secret password? The 8543 you keep on yelling at it?”

“Yes”

“Say your password” – Beep – “Eight! Five! Four! Three!”. “Welcome” <harp music>.

“See? Now it works.”

“Well, that’s strange, don’t you think? What did you do differently now compared to your previous attempts?”

“No idea, daddy”. She closes the thing again. Beep. “8!5!4!3!”. “Password not recognized”.

It puzzles me that she’s not really interested in the cause. The thing only works one out of four attemps, and she seems perfectly fine with it.

A couple of 8543-filled days later, as the oldest daughter is off to a birthday party, the youngest one (5 yo) decides to seize the opportunity. She had been quietly watching her sister the past days, not even bothering to try – probably knowing that the oldest would strongly object.

“Can I have a look at sis’ secret diary daddy?”

“Sure. But you probably won’t get too far, it is made especially to prevent others from…”

“Say your password” – Beep – “Eight Five Four Three”. “Welcome” <harp music>. Big triumphant smile.

On. Her. First. Try. And their voices don’t even sound alike!
Granted, they’re sisters, their voices may be similar, but still there is a significant difference. And I would think that this thing is made specifically to shield your secrets from curious sisters.

Is there a problem here? What happened? After this revelation I tried to open it, and it manages to keep me locked out, where I should be. My wife is not able to enter the realm of secrets either. But little sisters, on the other hand, seem to have a bigger accuracy rate than the actual voice owners.

So, two very apparent problems:

– The diary doesn’t open when it should
– The diary opens when it shouldn’t

It turned out that the reason why the oldest had a hard time getting passed the passphrase was her impatience. Impatience is her middle name – she just couldn’t wait to yell 8-5-4-3 until a while after the beep – most of the time she started the password phrase at the same time of the beep. When she respected the little pause, her success rate tripled.

The second problem is a bit trickier. I’m trying to imagine if and how this toy was tested. Security for kids is probably not very high on most stakeholders’ priority lists. They probably tested a couple of scenarios with kids of different ages and gender, maybe a deep-voiced parent here and there. Did they include voices from siblings? Siblings aren’t corner cases – they represent realistic real life scenarios, with a high probability of diary sneaking. Granted, the kids don’t really make much of it, but the oldest was not amused when she found out about the security breach. If you make these bold claims about your product, and your product’s whole advertising and unique selling proposition is centered on this very feature, can you please make sure that it – you know – “works”?

V-Tech – frustrating tester parents since 2013.

The Eurostar 2012 diaries – part 3 (tuesday Nov 6)

Super Tuesday

Photo by Rik Marselis

Early tuesday, and my dreams were filled with empty auditoria and keynote speakers stuck in airports. A first reality check eases my mind a bit. The people I dreamt about are already checked in, and ready to rumble. As am I. I skip breakfast to be on site as early as possible, but halfway there I realize I left my phone in the hotel room. I hurry back and when I finally get to the premises, James and Julian beat me to it. While we help the tutorial speakers get all settled for the morning, the registration area is again being flashmobbed by testers. Delegates are now flowing in at a steady pace, but I notice remarkably few hiccups. Sure, there is the occasional delegate who is worried about his tutorial enrollment, but Siobhan seems to have a firm grip on payments and registrations. Siobhan handles all the adminstrative stuff throughout the year, a job that can never be underestimated. Rumor has it that she can even make Chuck Norris comply with Eurostar’s presentation materials deadline (which we couldn’t verify this year since his submission “I sit down in stand-up meetings” didn’t make the cut). The Eurostar team deals with the rush-hour queues swiftly, and before I can say “Morning coffee, anyone?”, the AM sessions are kicked off.

Photo by Randy Rice

Like yesterday, I wander around the now quiet and peaceful venue and do a temperature reading in the different tutorial rooms. Fiona Charles has everyone in her room up on their feet, milling around and fully engaged in her “Right-sizing Test Documentation”.  Paul Gerrard is testing the room capacity boundaries in his totally sold out “How to Create a Test Strategy”. Randy Rice – all the way from Oklahoma City, Oklahoma – is mighty popular with his “Free and Cheap Test Tools”. Michael D. Kelly – all the way from Indianapolis, Indiana – is spreading his wisdom about managing exploratory testing for a full house as well in his “Session Based Test Management”. When I check in with Alan Richardson’s and Simon Stewart’s Selenium Clinic, I see a screens full of code and two presenters on fire. All is good.

Again I feel sorry that I cannot sit in – imagine all the learning I’m mssing! – but as it turns out there is plenty of learning to be done elsewhere: rendez-vous at 11 AM in the main auditorium for a run-through of the conference opening. Finally, the auditorium is made Shrek-free and we can admire the beautiful – blueish – theatre. While the main idea of the run-through was a rehearsal of the opening remarks, we spend an hour test-driving four yellow tourist bikes through various aisles of the auditorium, checking where to brake and how to park – testing the risky bits first, so to speak. Daragh and Paul of the Eurostar team were able to secure these bikes last-minute as we are going to use them in the most vital part of the conference opening: the committee entrance. While James, Julian and Shmuel get in touch with their inner 13 year-old and do little uphill races on the blue carpet, ringing frantically, I try to make my slideset work on Alan Page’s nifty little Surface RT. Hungry? Not really. No stress, no sirree.

      Half past one. The official conference opening is upon us. The auditorium is packed, music is blasting out of the speakers and the four of us are in the – by now deserted – registration area, using our dubious trial bike skills to balance our yellow monsters in place. Lorraine gives us a go and we make our way to the front of the auditorium, all the time aware of the bizarreness of the situation: low visibility, a steep decline, funny brakes, loud music and a suit don’t make for a fluent biking experience. This concludes our very own “Men in Black”/”Boys are back in town” moment – and I’m glad we make it in one piece. We park the bikes and I climb onto the stage for the opening remarks. It is there that I have my first aha-moment. When asking how many people are attending the conference for the first time, I expect someting like 10-20% but see a *lot* of hands going up. I am a little thrown by that and I would like to see some official numbers to be sure what happened this year.

Ferran Adrià
Photo by Wired Magazine, 2012

I go ahead and explain the conference theme (Innovate/Renovate), telling the story of Ferran Adrià, the former chef from El Bulli, the best restaurant in the world for some time. Adrià started his career as a dishwater but managed to change the world of gastronomy by bringing elements of other disciplines into cooking: chemistry, psychology, physics. He expanded his cooking toolbox (his new toys were lyophilizers, liquid nitrogen, candy floss machines,…) and started investigating how the presentation of a dish influenced the perception of taste (did you know that strawberry mousse is perceived to be ten percent sweeter when served on a white plate compared to a black one?). My main message? In testing, it’s probably the easiest easy to say that innovation is not your job, rather something for those crazy boys and girls in R&D. But it’s not – it’s everyone’s job. Innovation is not just about products: it is also about business practices, processes, tools – it lies in everything we do. Everyone can be an innovator – testers too.

When the committee takes the stage for a personal address, expressing their wishes and hopes for the conference, I can actually stand back for a while and be amazed by the size of the whole endeavour. So many people here, in difficult economic times, all undergoing geographical and financial inconvenience to be here to learn and share experiences – this is great, and humbling at the same time. It is super tuesday alright.

Photo by Rik Marselis

Time now for Alan Page to step into the light, with his opening keynote “Test Innovation for Everyone” (a link to the presentation can be found here). It turns out to be a great talk, in which he points out that innovation is all about ideas, which makes test innovation mainly about test ideas. We innovate to solve problems – but are we solving the right problems? Try a lot, but keep checking whether you are doing the right thing. Alan’s talk is also book recommendation hour: “Where Good Ideas Come From” (Steven Johnson), “The Wisdom of Crowds” (James Surowiecki), “The Lean Startup” (Eric Ries), “Jimbo – Adventures in Paradise” (Gary Panter), “Brain Rules” (John Medina), “The Myths of Innovation” (Scott Berkun), “They All Laughed” (Ira Flatow), “Steal Like an Artist” (Austin Kleon), “The 5 Elements of effective thinking” (Edward Burger, Michael Starbird) and “The Innovator’s Dilemma” (Clayton Christensen). Some of these are already acquired as we speak.

AlanRichardsonKeynoteNext up are the first track sessions of the conference. Finally, time to watch the sessions we have been debating way back in march. I switch to track-hopping mode again and sample bits of as many tracks as possible. The afternoon flies by way too fast, and before I know, we are all inside the auditorium again for Alan Richardson‘s eclectic closing keynote, “Unconventional Influences”. I am very much looking forward to his talk – I chose this talk from his long list of possible subjects because I am all for bringing elements from other fields into our testing practice. And I am happy to see that Alan completely nails it with references to Dr. Seuss, H.P. Lovercraft’s “The Call of Cthulhu”, the “Fortean Times” magazine and ghostbusting. He tells a story about how he never allowed his testing to be limited by other people’s attitudes to testing or company mandates. He thinks that is an excuse that people embrace to stop them having to identify their beliefs about testing and challenge themselves to become better testers. Slides of his talk can be found here. He also wrote an unconventional paper to go with it, link here.

Tuesday evening is the traditional Expo drinks moment, and I take the opportunity to mingle and talk with as many people as possible. I drop by the Improve boot, where my very pregnant DEWT colleague Jeanne Hofmans hands me the first issue of “Quality Level Management – Managing quality in outsourcing”, a book she wrote together with Erwin Pasmans. Once I make it to the Test Lab, I see that Shmuel brought his infamous box with testing games and waste the rest of the expo drink time trying to solve puzzles together with great testers – and beer. How awesome is that? Totally sucked into puzzle solving, I fail to notice that the expo is closing, and together with Shmuel (still limping like a hunchback) I spend a good 30 minutes deciding whether we should ride our yellow bikes through the torrential rain or take a taxi. Taxi lines lines outside the RAI are rather discouraging, so we end up getting soaked on the bikes. Back at the Novotel, the Test Lab people set up a portable Test Lab in the bar, but even the most die-hard TestLabber gets hungry after a while. And so it happens that a group of twentysomething people find themselves in a great tapas restaurant for an evening of comido, bebidas y alegría. The athmosphere was so relaxed that I am tempted to stay up until the wee hours, but I am being a good chair and take the taxi back – mañana es otro día.

Here are Esther Gons‘ graphical recordings of this day:

Opening remarks

Test Innovation for Everyone – Alan Page

Curing our Binary Disease – Rikard Edgren

Value Inspired Testing – Neil Thompson

Unconventional Influences – Alan Richardson

… to be continued

The Eurostar 2012 diaries – part 2 (monday Nov 5)

So it begins… tutorial day

En route from the novotel to the RAI on this dark and rainy monday morning, I am pleasantly surprised to see “Eurostar Conference” signposted everywhere (next to Shrek the Musical, where it belongs) – who knows I might have ended up at a conference far, far away. It is only 7 AM, but the RAI is buzzing with activity. The registration desk is fully (wo)man(n)ed, and last minute checks are being done. Not too long before the hounds will be released, and the eurostar crew is very much “in the zone”.  I decide not to disturb them too much and go for a little orientation walk around the venue. The main auditorium remains closed this day (secret Shrek stuff, I assume) and the expo is still being built, but I am able to get a good idea of the venue layout. Loungy sitting areas, roomy but cosy session rooms – this place has the right vibe.

By the time I get back to the desk, the first tutorial speakers are already registering. Part of my job today is to show them to their rooms, make sure they get all settled and have everything they need. Dorothy Graham (“Managing Successful Test Automation”) and Janet Gregory (“Transitioning to Agile Testing”) are our first teachers on site, soon followed by Bob van de Burgt & Iris Pinkster O’Riordan (“Lessons Learned in Test Management”) and Rikard Edgren (“Exploratory Test Design”). I walk them up to their campsite for the day and when I get back down, the registration area looks like it is being flash-mobbed by multiple nationalities – minus the dancing. The desk is invaded with people waiting to get a name tag, conference bag and that very slick looking 20th-anniversary running shirt. They get a complimentary friendly word and welcome from the Eurostar crew, some advice and orientation, and off they are to learning heaven. By now I realize that I somehow missed Michael Bolton (“Critical thinking for testers”) entering the venue, but Siobhan reassures me that he /is/ in the house. I go upstairs again to say hi, and notice that Michael’s room is nearly full. Wait, is it 8:30 already? The last queues are cleared, and we have a lift off.

Standing by the registration desk, I notice keynote speakers Alan Page and Simon Stewart check in. As they are not teaching/speaking today, and they probably won’t be hanging around the venue the whole time, I make sure to remind them of the speaker’s drink taking place this evening in a bar behind the RAI. Apparently, I wasn’t the first:

The rest of the morning I spend wandering from room to room, sitting in for short periods of time, to catch the vibe and to see whether the delegates are enjoying themselves. This is a strange change of perspective. During the previous years, I was always in these tutorials myself, focused on learning. Now, I am checking the reactions of people and not really paying attention to what is being told. I am a lousy multi-tasker. Not that I don’t pick up stuff – I vividly recall Rikard walking around without shoes, talking about software potatoes. I catch Bob and Iris talking about a thin (cheese) slicing test management method (mmm… cheese), and Michael debriefing one of his many exercises and quoting Jerry Weinberg (*). I hear Dorothy highlighting and explaining stories from her award-winning book (“Experiences of Test Automation“), and Janet is scaling her normally much more intimate workshop to a way bigger audience – and she seems to handle that with style and grace.

The rest of the RAI is empty at this time of day, which makes for a strange contrast with the invasion of only an hour earlier. During the coffee breaks, the friendly chatter reappears, and I observe and talk. Finally, I bump into Shmuel, who is limping like someone with bad shoes who had a very long walk in Amsterdam on a rainy sunday. It adds to his overall funniness, although I think he already is funny enough as is. We discuss the subject of yellow tourist bikes to go back and forth between the Novotel and the RAI – bikes that will play a role in our conference opening as well.

Right before lunch, as I am welcoming delegates into the lunch restaurant, fellow committee member James Lyndsay enters the venue. He is sporting a big bag that mostly contains a stylish – and heavy – kilt. Guaranteed gala dinner goodness. During lunch, James is mentioning his upcoming gig with the London Bulgarian Choir – the friday after the conference. And I thought /I/ had a hectic schedule! When the tutorials start up again and the restaurant is emptying, Shmuel and I act as a (poor man’s excuse for a ) choir while James rehearses one of his deep-voiced solos, leaving the waiters and RAI staff wondering what they just witnessed. The RAI restaurant has good acoustics, actually. Rumor has it that James’ voice is still haunting the premises.

The afternoon flows smoothly, I feel, and around 4 pm I follow Lorraine’s recommendation to use the last couple of hours of the afternoon to relax a bit before things get really hectic tomorrow. The weather is crisp and clear, so I go for a walk around the area. Not too long though, since the speaker drinks kick off at 6.

At the drink, which turns out to take place in a cosy Austrian log cabin, I meet Julian Harty who just flew in from Nairobi with a short detour to the UK – to switch from summer to winter clothes. On thursday evening, he has to move on to go speak at the (equally fantastic) Oredev conference, which unfortunately takes place in the same week this year. Speaking of hectic schedules, I am convinced that Julian’s travel arrangements would make Kofi Annan look lazy. It is good to have our committee finally complete, and on site. Actually, this is the first time that the four of us together meet face to face, as Shmuel was skyping in while the rest of us were meeting in Galway in march.

As the committee is hosting the drinks, we do our best to make everyone feel welcome. Shmuel goes full reversed paparazzo and has his picture taken with everyone present – behavior that he will continue to exhibit throughout the whole conference, which makes me wonder: doesn’t that make his photo albums mighty Shmuel-centric? The athmosphere is really relaxed, and I am glad to see things turning out so nicely. I am now able to put faces to submissions, and voices to pictures. Michael D. Kelly – very thrilled to finally meet him, by the way – looked so young that I didn’t even recognize him at first. I don’t know why I had imagined him older, must be his reputation preceeding him.

My evening ended in a tasty Indonesian restaurant, where a large group of testers was diving into a rice table as we entered. I was invited to eat (heaps of) spicy leftovers from other people (thank you Rob Lambert and John Stevenson for feeding the hungry and the impatient – your good Samaritanism is highly appreciated). Again,  occasions like this work wonders in putting faces to twitter handles: be warned, @GeirGulbrandsen and @Kristoffer_Nord, you’re no longer safe from me.

After dinner : plenty of rest for the wicked. Tomorrow, there’s an official conference-opening to be done. 

… to be continued

——————————– 

(*) Which comes as no surprise, as Jerry Weinberg – Patron Saint of thinking testers – is very much worth quoting. And reading, even more so. If you haven’t read any of his books, I encourage you to do so.

The Eurostar 2012 diaries (the prequel)

What a year…

It has been a while since my last blog post, and being the programme chair for Europe’s biggest software testing conference probably had something to do with that. Now that the twentieth edition of Eurostar is over and the whole event is still very much in my system, I figured it is about time to revive Ye Olde TestSideStory blog.

The Eurostar office, Galway

The whole year leading up to this moment was one big trip into testing conference wonderland. I learned loads about conference-making (I’m pretending that this is a dictionary entry somewhere) in the small and the large. Selecting a committee, a theme, keynotes, tutorials, assembling a balanced programme out of 400+ submissions – these things in itself already were quite a challenge. This, combined with a steady flow of related side-activities proved to occupy the better part of my free time. Luckily, the Eurostar team in Galway (Ireland) made this into a very enjoyable and fluent experience. I had the privilige of visiting the Galway office a couple of times in the past year, and the team has a great energy that gets things going (and a love for Belgian chocolates and all things Guinness). Props to my employer CTG as well, for giving me the opportunity to spend time preparing the conference.

Working with my committee (Julian Harty James LyndsayShmuel Gershon) throughout the year was certainly a highlight. I have fond memories of our lengthy skype sessions, discussing about anything in the testing conference realm – we even managed to find some emerging behavior in skype chat in the process. In hindsight, I was particularly impressed with Julian’s pragmatism and fresh ideas, James’ note-taking fu in the face of a truckload of submissions, and Shmuel’s contagious enthusiasm.

The last weeks, pressure had been building gradually: seeing the early bird subscriptions take off, hearing about testlab preparations, tutorials filling up… Later on, a couple of speakers opted out and needed replacement – things were getting more real every week.

Rainy Amsterdam – Sunday November 4

After some uneventful aquaplaning all the way from Belgium, I met up with Israeli-Brazilian superstar (and programme committee member extraordinaire) Shmuel Gershon. Originally there was a visit planned to the RAI to get acquainted with the venue layout, but since Eurostar happened to coincide with Shrek The musical (Ogres in the main auditorium! Fionas mindmapping a test strategy!), this was no longer possible. We decided to dive headfirst into the city of Amsterdam, to explore. Some observations:

  • A couple of hours in Amsterdam can spawn more rain than six days in Ireland
  • Torrential rain will soak up even the sturdiest shoes
  • The Anne Frank house has bigger lines than the newly opened Amsterdam Apple Store
  • From now on, if the map and the territory disagree, I’m believing the territory
  • Serendipitous wandering can make you end up in one of the finer Indian Restaurants in Amsterdam
  • The finer Indian bread is very kosher – but expensive
  • Two men with identical bright blue Novotel umbrellas look funny (I guess people expected a Gene Kelly dance routine)

When arriving back at the Novotel, soaked to the bone, a bunch of testers had already gathered for an informal meetup in the bar. I was planning to change into dry clothes first, but got engaged in conversation and totally forgot about it. Sometimes you have to plan as you go along.

Conference pre-opening (photo by Huib Schoots)

While my shoes were drying slowly, I spent the rest of the evening chatting with new friends (Cyril Boucher, Jeanne Peng, Erkki Pöyhönen) and catching up with old ones (John Stevenson, Michael Bolton, Huib Schoots, Jean-Paul Varwijk, Rikard Edgren, Shmuel). John in particular was on fire that evening, quoting book titles like some kind of human reading tip generator. The two that I managed to note down are “The click moment” and “Everything is obvious“. The rest got lost in a pre-conference haze.

Later on I ran into the Eurostar crew as well. They had been on site since friday, unpacking stuff and basically building everything from scratch. They expanded their team for the conference, and it was nice meeting new faces there too. They all looked happy and confident, which was kind of reassuring to see: the logistic side is under control. Chatting with them also made me realize that things were about to be kicked off for real.

Are those nerves I feel? Anyway, time for bed – appointment at the RAI at 7 am.

… to be continued

Exploratory bug diagnosis

The prologue

At the Let’s Test conference last week, I attended a half-day tutorial on bug diagnosis by James Lyndsay, in which we tried to analyze the actions of testers when pinpointing bugs. We did all this by identifying our actions during some bug diagnosing exercises. My learnings kept lingering in the back of my mind throughout the conference (which was excellent, by the way). When I noticed on the way back home that something was wrong with the songs on my iPhone play-list, I decided to test my newly learned diagnosing-fu by describing my learnings in an exploratory essay while trying to find out what the problem really is.

At the time of writing, I don’t know what the cause of that problem is, yet. I will document my knowledge as it evolves (hopefully). So cover me, I’m going in…

The problem

On the way from Stockholm to Brussels, at a cruising altitude of 10.000 km, the hostess tells us it is safe to turn all electronic devices back on. I whip out my phone and start flipping through the albums that were uploaded via a newly created play-list. I select an album and hit play. But something’s amiss – that familiar album sounds less familiar this time around. It takes a while before I realize that the album didn’t start with its opening song. Did I hit shuffle unknowingly, by any chance? That has happened before… Nope, shuffle is off. I go to the details of the album and notice that the first song is not there.

First probing

Strange, that. I initially dismiss it as a one-off, but the next album I listen to, the same phenomenon occurs. All songs are there but the opening one. I check the other albums and it turns out that half of the albums uploaded to my play-list lack their first song. If a bug is something that bugs a user, this must be a capital B one. Having to listen to incomplete albums seriously bugs me; what’s even more: it takes away my desire to listen.

[I notice how I react quite emotionally to the strange behavior. Emotions are a powerful oracle. Although I have only limited knowledge of the problem now, I declare it officially a bug]

Defocusing & narrowing down

I feel frustrated because further bug investigation possibilities on my phone seem limited. I put it away and decide to defocus. A little in-flight snack and some reading manage to temporarily distract me, but an hour later the bug creeps up on me again. I follow my energy that leads me to start a static analysis. Although the symptoms are all on display here, I suspect the cause is not located in my phone but rather within my PC, iTunes or in the synching between iTunes and the phone.

[I just narrowed down the scope of the investigation with a first broad hypothesis]

From possible to plausible

I haven’t got iTunes at my disposal right now, but while I am at it, I refine my previous hypothesis into a couple of more specific ones that I will be able to confirm or refute when I get home:

  1. The original source mp3-folders contain incomplete albums
  2. The albums were uploaded wrongly in my iTunes library
  3. The albums were copied wrongly into the play-list
  4. The albums were synchronized wrongly from the play-list to the phone

[This list of 4 contains possible causes]

I can narrow these down further because hypotheses 1 and 2 are highly unlikely. The very same songs, from the very same source folders were recently used in other play-lists without a problem, and I haven’t noticed songs missing from albums in my music library.

[This makes hypotheses 3 and 4 the most plausible ones – better concentrate on these]

Checking hypotheses

Back home, I am reunited with the family, and with my iTunes library that resides on an external storage drive (which also happens to contain the original mp3-files). I quickly check option 1 and 2, because I am painfully aware of biases in my memory, and my thinking. [ Although I think these two options are less likely, you never know. I’m a tester, and we know things can be different, right? Well, in this case, not so much]. The suspicious albums in the source folders are complete, as they are in the music library.

[I notice that, rather than checking all albums, I tend to focus on one sample album to check my assumptions. Comparing the same sample throughout the hypothesis increases consistency and diminishes possible distortions. A possible risk is of course that this could turn out to be a not-so representative example]

This leaves me with 3 and 4, the plausible ones.

Were the albums copied wrongly in the suspected play-list? I know they have been correctly copied to other play-lists before, so I am curious to see if this can really be the case.

A-ha! Now we’re getting somewhere. Song number one, “Get Miles” got lost in the mists of the copy from library to play-list.

[I now come to realize that this was to be expected, since the synch process was designed to synch exactly what is in this play-list. Oh well, better be safe than sorry. This causes me to drop hypothesis 4, because the synch did exactly what is was supposed to do]

Reflecting & diving deeper

So, time-out for a second. What is happening? The contents of some albums were corrupted somewhere during the transfer to the play-list. First thing that strikes me: why only half of the albums? Why not all of them? They were all dragged to the play-list in the same session. Is there something I did differently for some albums? I recall that I started with importing individual albums into the library, but that I then resorted to a bulk import of the remaining albums. Maybe the “bulk-imported” albums are causing this? Then again, they are correctly loaded in the library, it is when they were transferred to the play-list that things went awry.

[While diving deeper within hypothesis 3, I develop a sub-hypothesis]

3.   The albums were copied wrongly into the play-list
3.1.   The problem with the play-list has something to do with bulk imports

I check hypothesis 3.1: I do a bulk import from several albums in one folder, and then transfer those to a newly created play-list. To no avail. I drag the suspicious album to a new play-list, but all 12 songs are there. I drag a couple of similar ones in there separately. Nothing wrong with them.

[This is not really working for me, and it starts to get boring. Let’s drag all of my available albums in, at the same time]

Bingo! Many albums there with the first songs missing. That was easy. Triumphantly, I clean the play-list and repeat the same action, to confirm.

[Repeating experiments can decrease uncertainty, but can also  free us from the illusion of control]

Nothing. All back to normal again. Huh? What did I do exactly, that first time? I launch several attempts to reproduce what I had first seen, including starting from a new play-list from scratch, all of them unsuccessful. It takes a while before I realize that I just copied the contents from the faulty play-list into the new one.

[So I start making mistakes. Back to square one. I abort hypothesis 3.1. and decide to catch some sleep]

New perspectives

Another day, a fresh perspective. What else is striking about this bug? It occurs to me that the solution might lie in the fact that every single one of those missing songs is the first song on the album. What does the missing “number one” tell me?

  • Order of play?
  • Something that started wrong and then went well?
  • Switching between albums?
  • Corrupting the first song and switching albums?
  • Switching from a bad to a good state?

[I am now focusing on the “why” of the first songs, whereas in the previous hypothesis I was focusing on the “why” of only half the albums]

Was there something I did that made the first songs go in a special state?  Suddenly, I remember… I keep forgetting that I moved the iTunes library to an external drive, it used to be on my laptop until a month ago. That means that iTunes does not recognize the songs in my library as long as the external drive is not connected to my laptop. That is no problem, as long as I don’t perform any actions on the songs, like dragging them or playing them. Otherwise the songs get a lovely exclamation mark in front of them. I disconnect the external drive and try to play the first song of the album in the library. The trusted exclamation mark appears:

I find myself investigating a new sub-hypothesis of hypothesis no. 3:

3.   The albums were copied wrongly into the play-list
3.1.   The problem with the play-list has something to do with bulk imports
3.2.   The problem has something to do with a disconnected library

Usually, the moment I notice I forgot to hook up the external drive, I quickly connect it and no harm is done. I wonder what would happen if I now connect the external drive again and drag the album to a new playlist in this state? [I have the feeling I’m nearly there. Could it really be…?]

I feel I am finally making some progress and refine 3.2 into 3.2.1.:

3.   The albums were copied wrongly into the play-list
3.1.   The problem with the play-list has something to do with bulk imports
3.2.   The problem has something to do with a disconnected library
3.2.1. Actions performed on songs while being disconnected from the library cause the songs to be skipped when copying albums to play-lists, even when the library is connected again at the time of copying

I do the experiment, and it confirms my hypothesis. I repeat the same procedure with an other album and this time, the same behavior occurs.

[I was kind of hoping and expecting that this would happen, which is normal behavior, but which can also be a danger during testing. We tend to focus more on things we really *want* to see]

The Cause & the Trigger

This last discovery leaves me with some mixed emotions. I feel happy to know what caused the missing songs, but I am also puzzled as to why so many songs have received the exclamation mark without me noticing. I can perfectly reproduce the problem, and I’m pretty sure it won’t happen to me again, since I will now be aware of the little exclamation marks while making play-lists. I have found the cause of the strange behavior that kept me busy for quite a while, but still… I am not sure how it got triggered in the first place.

I do have a trigger hypothesis (for now): flipping through albums using the cover flow view and hitting enter or trying to play them while the library is not there only marks the first songs with an exclamation mark. I recall that tried to listen to some albums, but not the complete amount that had missing songs. So there is still a decent amount of mystery involved.

Epilogue – is it a bug, really?

When I was first confronted with the problem, I proclaimed this a bug with capital B, because it annoyed me – the user – and it made me stop using the product. Has my opinion changed now that I have lived with the thing for a couple of days? I would argue that it has.

The behavior only seems to occur in very specific situations, and although the impact was quite big for me, it is unlikely that it will happen to me again. Is there a possibility that others will stumble upon this? Well, I stumbled upon it, so chances are that others will do too. And I certainly think there are other people like me that have their libraries on other media that are not by default connected to their computers. So yes, I think it IS a bug, although not as severe as I initially thought it was. This goes to show that we adapt severity and priority to our gradually evolving knowledge about the bug, and to the changing context (Something Rob Sabourin neatly pointed out as well in his brilliant Let’s Test keynote).

What I got to know about the problem so far leads me to believe that the product (iTunes) can be improved in a couple of ways (actually, there are plenty of other ways of improving it, but I digress). How about the following ones, for starters:

  • Doing a re-check of previously failed songs in case connectivity has been restored?
  • Removing obsolete exclamation marks when an external library is re-connected?
  • Adding a notification when trying to copy “songs not found” to playlists?
  • Making it more conspicuous to the user when the music library is not connected?

This concludes my adventure that started on the way back from Let’s Test. I wrote this post in several stages as I was trying to get a grip on that devious bug. It didn’t turn out to be the “clean” or “clear” bug I hoped it to be. Perhaps the iTunes product managers will even say it’s cosmetic or trivial. After all, they make the call. Oh well. I learned valuable stuff in the process. I learned that wording/noting your thoughts in the process helps you to see where your line of reasoning is heading, and what the (sometimes hidden) hypotheses are. It was all about the journey[1] of course, and not so much about the eventual outcome (which I felt was only a partial success).


[1] Although it was a personal journey, it was inspired by James Lyndsay, who encouraged me to share my thoughts on diagnosing bugs

Real Learning at a Virtual Conference

On September 13 last year, EuroSTAR went virtual for the very first time, without really knowing what they ventured themselves into. They called it a virtual conference, and it was exactly that: plenty of talks with Q&A after each presentation, discussions between attendees in the networking lounge, a test-tools virtual expo and a test-related resource centre. It turned out to be a huge success. People kept lounging in the lobby, engaging with others and sipping virtual cocktails. Actually, I made that last one up. But it would have been nice, wouldn’t it?

This year marks the 20th anniversary of the EuroSTAR conference, and preparations for the actual conference are in full swing. The programme was announced on May 3rd, and we do hope you like it enough to pay us a visit in beautiful Amsterdam later this year. But November is still half a year away, which is an eternity in these connected times. So why not give you something to warm up to, testing-wise?

A virtual conference, you say? Why, that’s a splendid idea!

The Eurostar team thought so too, and that’s why I recommend you save Wednesday May 16 in your calendar. That is the date of the second virtual conference, and it promises to be finger licking good. It marks a pivotal moment between past and future of the Eurostar conference.

• The conference will look back at the 2011 Manchester conference by featuring Bryan Bakker (Sioux Embedded Systems), who ranked among the highest scoring sessions last year. Bryan will talk about model driven development and its impact on testing, using results of a case study to illustrate his story.

• The virtual conference will also take a sneak peak at the future, by offering two of our Eurostar 2012 keynote speakers, Alan Page (Microsoft) and Alan Richardson (Compendium Developments), the opportunity to share their amazing ideas with the community (no, you didn’t have to be named Alan to secure a keynote spot, but it helped). Alan Page (co-author of “How we test software at Microsoft“) will discuss where (testing) ideas come from, and how anyone can use learning, creativity, pattern recognition and pragmatism to discover and apply new ideas anywhere – especially in software testing. He also has a blog – Tooth of the Weasel – that is very much worth checking out. Alan Richardson (the author of Selenium Simplified, of which a second edition was published recently) will share his experience of thinking visually in software testing – using models and diagrams to help his test planning and communication of testing. Alan is a also a hypnotizing tester – or was it a testing hypnotist? – who blogs and tweets as Evil Tester.

James Lyndsay (Workroom Productions), who concludes this all-star line-up, symbolizes the present of the conference: he is a valued programme committee member this year and never short on great ideas on testing. In the beginning of the year, he published an impressive blog series on managing exploratory testing, which he will try to condense/transform/shape into a talk called “There are Plenty of Ways to Manage Exploratory Testing”. Yes, I am curious about that as well.

But this is not all: apart from these four thought-provoking presentations, you’ll get the opportunity to get acquainted with the latest new tools and services in the expo, and get a chance to mingle and discuss with like-minded individuals.

Come get inspired, learn and share your knowledge with your testing peers. Get a taste from what’s to come in Amsterdam. And sip that virtual cocktail if you like.

Register for the Virtual Conference.

Keith Klain – Bridging the Gap: Leading change in a community of testers

I am at StarEast this week, and originally intended to give live-blogging a go, an art perfected by the ubiquitous Markus Gaertner. Alas, the wireless connection decided otherwise, so I am a bit on the late side with this. Here is a summary of the opening keynote of the StarEast conference, by Keith Klane: Bridging the Gap: Leading change in a community of testers.

In this presentation, Keith Klain described how Barclay’s Capital Global Test Center (GTC) managed to totally change the way they worked. The GTC is an independent testing service providing full lifecycle software testing support and specialist testing services across multiple businesses from 6 global locations.

Keith got triggered by something his boss had told him: to go reflect on a a couple of things, stating “In this business you’re either a bug or a windshield”. This made Keith wonder: what is motivating people? You cannot make people do anything unless they want to do it because it is in their best interest. That is real leadership.

On a leadership summit in Europe last year, Keith heard all sorts of depressing things: that testers don’t understand the business, that they are too slow and too expensive, that all testing should be automated. So clearly all these years of maturity models, metrics programmes, certifications, process improvements are not getting through, they’re not working.

Values

It is important to outline what your values are:

– Honesty, with ourselves and others. Be transparent about strengths and weaknesses.

– Integrity: earn the right to have an opinion. Provide clear and constant feedback.

– Accountability: Own it. Understand what “value” means in your business. Manage your own expectations.

The situation

When Keith arrived at the GTC, an improvement programme was already in place and ongoing, called “change programme phase 1”.

  • There were test maturity models and maturity improvement plans, based on industry standards. He decided to kill that.
  • There was a metrics programme, a test case efficiency model centered on test execution. It consisted of automation metrics based on test case coverage and program tracking sheets. They noticed that as soon as you want to put a number on something people are doing, people start focusing on the number, and not on the thing they’re supposed to do. He decided to kill the programme.
  • There was a career framework. The focus of test managempent was on operational control and team size. He killed that too.

Talent management

From that momentb on, the GTC started focusing on Talent Management. The keywords here: attract/develop/retain.

Attract

They raised the hiring (and existing employee) bar and transferred quality ownership to the team. They made sure that GTC was viewed as a “Top Project”, very hard to get in (in analogy with Google).

Develop

Keith invited James Bach to come and teach his Rapid Software Testing course and regularly provide consulting as well. On top of that, they improved business and testing skills. After a while, people started developing their own training courses, around ten in total. These were staff-led without any steering from management. They also started a new test management mentoring programme.

Retain

They flattened out the career framework, until it contained only four levels. They developed Induction Programmes, in which they told people what was expected from them. In addition, Keith told them what was to be expected from him.

Management principles

  • People start ignoring testing when it’s no longer relevant. It’s your own fault if people start ignoring you. Speak the language of your programs, learn what the values are and people will listen to you.
  • Being responsible sometimes means rocking the boat. Sometimes people will have to tell someone that something is not done well. That is no problem, as long as it’s done respectfully. “If you want a friend, buy a dog”.
  • No one has the market cornered on good ideas. Bill Gates once said that even the smartest people in the world only had one good idea in their whole life. New ideas are all around you, look around. Solve problems collectively.
  • Never stop asking why. It is a powerful and clarifying question that will get you where you want to. Question everything. Don’t mindlessly go about your job.
  • Invest 80% of your energy in your top 20%.
  • Leadership = simplification. The problem is that people cannot communicate simply about their solution. Tell something in a way that people do something about it. That is something that goes beyond an elevator pitch.
  • Don’t take it personally. You are not your ideas. These depend on a whole series of factors. If you take things too personally, you won’t be able to change your behavior
  • Think first, then do.

The Results

  • General investments in the GTC have gone up. This shows the importance that management started giving to the GTC.
  • Nonsense management overhead was removed, and all of a sudden loads of new talent started bubbling up. People got rewarded for their efforts on their test strategies, their approaches, their ideas. (“Runner up” CIO award).
  • The biggest improvement or cultural change was the GTC university. It was all there: class room training, brown bag sessions, mentoring, competitions, all done off the back, not steered by management.
  • Leading cultural change = a paradigm shift

Stop

Pretending that the value of your testing org is anyone other’s responsibility, and pretending that you’re going to” mature out” of that position

Start

Telling people what you expect from them. Support them by a training regime that helps them do that. Create an environment where people are not afraid to fail. Create a learning organization.

Continue

Driving out fear of failure by creating an environment that enables innovation and rewards collaboration through strategic objectives and constant feedback.

I really liked this keynote. It was an inspiring story that showed how you *can* make a difference when you start focusing on skill and building a learning organization.

An article on beauty, quality and relativity

Today, Stickyminds and Techwell published my article “On Beauty, Quality and Relativity”:

The article is based on a lightning talk I did at the Danish Alliance night at Eurostar 2010, and describes my thoughts after seeing the Joshua Bell video below (story in the article).

For those to whom the prospect of reading a 1200 word article sounds daunting (or for those of you who love executive summaries), here is a TL;DR:

  • Passers-by in subway don’t recognize sheer beauty because of incongruent context
  • This applies to testing as well: our ability to evaluate quality depends on feelings, expectations, words and distractions
  • To properly assess beauty/quality, viewing/testing conditions need to be optimal

<Service announcement: I will be presenting at StarEast on a slightly related topic: the links between the arts and testing. If you are there too, let’s chat!>

On being context-driven

This is the transcript/elaboration of a lightning talk I did at the Context-Driven theme night organised by TestNet in cooperation with our Dutch Exploratory Workshop on Testing (DEWT).

The hardest part

There comes a moment in the career of a context-driven tester where he is bound to have a sobering epiphany: for every situation where he knows the right approach, for every situation where he knows the perfect tools for the job, he comes to realize that there are numerous contexts where that approach isn’t the most appropriate one, where his ‘best practices’ are not usable. Or maybe they *are* usable, but they lead to suboptimal results.

Maybe I shouldn’t generalize. This is how it happened to me at least, and that was the moment when I became aware of the main principle of Context-Driven Testing: how you approach things is driven by the context of your project, not by your process. That is also the main difference with the common methodologies that try to replicate the same process over multiple contexts.

I admit that was a source of frustration for me. Going context-driven is certainly not taking the easy road – it would be much easier to implement the same processes everywhere I go. But none of that. Luckily, it happens to be the most exciting and challenging road I know.

Knowledge and awareness

What can we do to arm us against all these different changing contexts we find ourselves in? Gather knowledge, get experience, learn. Make your tester toolkit – with all your techniques and tips and tricks in there – as big as possible . That way you’ll be able to pick the right approach at the right moment.

Talk to fellow testers as much as possible. Grab every opportunity to network with them. Twitter is a blessing for these things – it has rocked my world and continues to do so. Conferences are hotspots for fascinating people, national and international alike. Events like this Testnet theme night are golden, they really are. Learn and read continuously. About testing, for sure, but also about other disciplines. I think there are many lessons to be learned from eg. psychology, sociology, philosophy and science. Some of the sharpest testers I know are physicists and philosophers!

Try out new ideas, new stuff, new approaches that you haven’t tried before. Apart from it being more fun, Marie Pasinski (staff neurologist at the Massachusetts General Hospital) recently wrote  that studies have shown that engaging in novel, stimulating activities promotes the growth of new neurons in the hippocampus (1). Putting this to work in your everyday life can be as simple as trying out a new recipe, taking a different route to work, reading up on the newest technology trends, or meeting new people.

Widen your awareness. Keep eyes and ears open, at all times. Absorb everything, as if you were a sponge. Are you familiar with the phenomenon where you happen upon some obscure piece of information – often an unfamiliar word or name – and soon afterwards encounter the same subject again, often repeatedly? It sure has happened to me, and it has a name as well: the Baader-Meinhof phenomenon. However strange it may seem, it is not that illogical. It just proves that our brains are fantastic pattern recognition engines. This is a characteristic that is highly useful for learning and I think we can use this to our advantage. The more we are aware of things surrounding us, the more we absorb knowledge, the higher the chance that it will keep lingering in our subconscious, and the likelier that a piece of knowledge will surface – and will stick in memory – when we need it.

The importance of knowledge in the context-driven community cannot be overestimated. That is the reason why there are so many initiatives for sharing that knowledge: free coaching by experts, peer workshops (DEWT, anyone?), tester meet-ups, weekend testing, Pair / Learn / Present, blogs, lots and lots of course materials available online…

The context-driven community is a very open community that focuses on sharing. If you are curious and want to know more, do get in touch. We’re more than willing to help where we can. 

(1) Marie Pasinski – Beautiful Brain Beautiful You, 2011