Rapid Software Testing – skilled software testing unleashed

Up to 11

The software testing profession looks like a steadily maturing profession from the outside. After all, there are certifications schemes like ISTQB, CAT, IREB and QAMP (the one to rule them all), standards (ISO 29119) and companies reaching TMM (test maturity model) levels that – just like a Spinal Tap guitar amplifier – one day might even go up to 11. The number of employees that companies send off to get certified in a mere three days is soaring, and new certification programs are being created as we speak. Quick and easy. Multiple choice exams for the win!

The reality, however, is that the field of software testing is torn between different “schools” of testing. You could see these schools as determined and persistent patterns of belief, speech and behaviour. This means that different people – all calling themselves “test professionals” – have vastly different ideas of what testing is all about. Even something elementary as the definition of testing varies from “demonstration of fitness for purpose” to “questioning a product in order to evaluate it”, depending on who you talk with (for more info on the schools of software testing, I heartily recommend Brett Pettichord’s presentation on the subject).

And so it happens that different people think differently about “good” or “mature” software testing. I, for one, don’t believe in tester certification programs, at least not in the format they are in now and the way they are being used in the testing profession. The current business model is mainly designed to get as many people as possible certified within the shortest timeframe. Its prime focus is on certifiability, not on tester skill, and certainly not on the advancement of the craft. Advancement comes from sharing, rather than shielding.

Rapid Software Testing (RST)

So what are the options for a tester on a quest for knowledge and self-improvement? What is a budding tester to do?

I think there are valuable alternatives for people who are serious about becoming a world-class tester. One of these is Rapid Software Testing (RST), a 3-day hands-on course designed by James Bach and Michael Bolton.

Actually, calling this “a course” doesn’t do it justice. RST is at the same time a methodology, a mind-set and a skill set about how to do excellent software testing in a way that is very fast, inexpensive, credible and accountable. It is a highly experiential workshop with lessons that stick.

How is RST different?

During RST you spend much of the time actually testing, working on exercises, puzzles, thought experiments and scenarios—some computer-based, some not. The goal of the course is to teach you how to test anything expertly, under extreme time pressure and conditions of uncertainty, in a way that will stand up to scrutiny.

The philosophy presented in this class is not like traditional approaches to testing, which ignore the thinking part of testing and instead focus on narrow definitions for testing terms while advocating never-ending paperwork. Products have become too complex for that, time is too short, and testers are too expensive. Rapid testing uses a cyclic approach and heuristic methods to constantly re-optimize testing to fit the needs of your clients.

What’s in it for you?

  • The ability to test something rapidly and skilfully is critical. There is a growing need to test quickly, effectively, and with little information available. Testers are expected to provide quick feedback. These short feedback loops make for more efficient and higher quality development
  • Exploratory testing is at the heart of RST. It combines test design, test execution, test result interpretation, and learning into a seamless process that finds a lot of problems quickly. Experienced testers will find out how to articulate those intellectual processes of testing that they already practice intuitively, while new testers will find lots of hands-on testing exercises that help them gain critical experience
  • RST teaches you how to think critically and ask important questions. The art of questioning is key in testing, and a very important skill for any consultant
  • RST will provide you with tools to do excellent testing
  • RST will heighten your awareness

Bold claim bottom-line:

RST will make you a better tester.

RST comes to Belgium

Co-learning and Z-sharp are proud to announce that from 30 September – 2 October, Michael Bolton will visit Belgium to deliver the first ever RST course on Belgian soil, giving you the opportunity to experience this unique course in person. More info can be found here, or feel free to contact us for more info.

Brace yourself for an mind-opening experience that will energize and boost your mind.

All the way up to 11.

(for even more information and testimonials about RST, see Michael Bolton’s RST page)

 

Advertisements

Home Swede Home – Øredev 2011

Last week I attended (and presented at) the Øredev conference in Malmö. Sigge Birgisson invited me to be part of a fully Context-driven test track, which I gratefully accepted. It turned out to be quite a memorable experience. Øredev was the first ever *developer* conference (be it with a testing twist) I attended, which gave the event a totally different vibe for me. Cosy, laid back and open-minded. Geeky too, in a good way:  they provided a cool conference app with a puzzle that could only be solved by obtaining other people’s codes. The side effect of that was that random people started addressing me with “Hi. Can I have your code?” moments before bolting off in their own space-time continuum. Speed dating for techies.

Another thing that really stood out were the graphic live-recordings by Heather and Nora from Imagethink. These talented ladies recorded every keynote live on stage, and made the beautifully looking artworks available as handouts later on. A brilliant idea.

As for the proceedings of the conference – here are some personal highlights:

Day 1

Day 1 had no real testing track, but there was enough fun to be had in other areas of the development spectrum. As the conference was centered around “the user” (Enter Userverse), it kicked off with “Only your mom wants to use your website”, an entertaining keynote by Alexis Ohanian, of Reddit and Hipmunk fame. Hey, the guy even spoke at TED about a whale called Mister Splashy Pants – top that! This time he told a compelling story about how the secret behind succesful websites is caring for your users. He told us that generally, the bar on websites is raised so low that it is really easy to stand out if you’re able to delight your user.

In “Collaboration by better understanding yourself”, Pat Kua stated that people have lots of in built reactions that hold us back from collaborating more effectively: power distance, physical distance, titles, even clothes. What could help us? Awareness, feedback, breaking the cycle, XP practices, courage. A good talk with good content, and some good book recommendations as well.

Johanna Rothman managed to keep me engaged for her whole talk about “Managing for collaboration”. She talked about how to manage the entire system for success, and how we should optimize and collaborate on the highest level, solving problems for the entire organization, not the project. I had the privilige of getting to know Johanna in her may 2011 PSL (Problem Solving Leadership) class, which she organizes together with Esther Derby and Jerry Weinberg. I knew she was a great storyteller, and she did not let us down: one gem was how she upset management by donating her entire bonus to her team and letting them decide who got what. 

Neal Ford closed off the day conference with “Abstraction distractions”, in which he dissected abstractions that have become so common that we started mistaking them for the real thing. An abstraction is a simplification of something much more complicated that is going on under the covers. As it turns out, a lot of computer programming consists of building abstractions. A file system, for instance, is a way to pretend that a hard drive isn’t really a bunch of spinning magnetic platters that can store bits at certain locations, but rather a hierarchical system of folders. And what’s that icon on a save button again? A floppy what? In addition, we shouldn’t name things that expose the underlying details. Users really don’t want save buttons, they just want their stuff to be saved. He also quoted Joel Spolsky’s Law of Leaky Abstractions: All non-trivial abstractions, to some degree, are leaky.

The day ended with drinks, dinner and some live jazz. I ended up talking testing (among other things) with Pradeep Soundarajan over dinner, when suddenly a late night evening session was announced: Copenhagen Suborbitals. At that moment, it reeked of a mediocre techno-act from the late nineties and I didn’t really feel like joining in. But Pradeep was curious enough and I decided to tag along. 

Flash forward one hour. Pradeep and I were literally blown away by a passionate tale of two Danes with a dream to build and launch their own manned rocket into space. Peter Madsen told a compelling and inspiring story about dreams, constraints, possibilities, enthusiasm, courage and rocket fuel.

Day 2

Day two was kicked off by Dan North who talked about “embracing uncertainty”. Fear – he said – leads to risk, risk leads to process, process leads to hate… and suffering and Gantt charts. Dan stressed that people would rather be wrong than uncertain, and that adding more process in times of uncertainty is wasteful and counter-productive. He also contrasted the original intentions of the agile manifesto in 2001, and what has become of that now. He stated that our ability to survive is directly related to handling the unexpected. We should embrace uncertainty, expect the unexpected and anticipate ignorance.

I decided to put up my basecamp in the “Test” room today, since this was context-driven testing day: six testing tracks covering a wide variety of topics. The only drawback was that the room looked like it was designed by an architect on acid: unfinished, an enigmatic door way up high in a wall, bare cables and sockets and a very short and high stage that forced you either to stand in front of the projection screen or to stay cemented in the same spot the whole time. Sound isolation was kind of peculiar too, although that only seemed to be  a problem when Americans were presenting nextdoors. But I’m nitpicking here: the whole Slagthuset venue was nice, and organization and technical team were super helpful, the whole day.

Pradeep Soundararajan‘s talk was titled “How I wish users knew how I help them through context driven testing”. Pradeep started by pointing out that he had the shortest abstract and the longest bio in the conference booklet. True. He seems to like long titles for his talks, too. In combination with his name, this probably makes him a nightmare to introduce at conferences. But in contrast with the title, his talk was short, crisp and funny. He was brave enough to do some live-demoing of his twitter-driven exploratory testing approach: looking for user feedback by searching in tweets with negative emoticons and profanities combined with the product or website name. I hadn’t read his blogpost before now, and it made me laugh out loud. I love the smell of profanities in the morning. Brilliant idea, that.

Next up was Shmuel Gershon, who shared an experience report of a 100% exploratory testing project, “Case Study on Team Leadership with Context-Driven Exploratory Tests”. He came well-prepared, all set to win our hearts with charisma, handouts and chocolats. He told us about how he took his team on a journey towards more context-driven testing and how he dealt with that as his role was also changing. He told us a story on test management, session based testing, recruiting even. He urged us to let people tell their stories, don’t start asking why, leaving them feeling that they have to justify themselves.

The ubiquitous Gojko Adzic (I suspect there are several clones making the rounds of conferences worldwide. Where /doesn’t/ he speak?) was his energetic self in his graveyard shift session called “Sleeping with the enemy”. Independent testing, he said, should be a thing from the past. Testers should engage with developers and business users, in order to create opportunities to accomplish things they cannot do otherwise. I like Gojko’s style, always direct and uncompromising, but always thoughtful. After Gojko’s presentation, a heated hallway discussion ensued in the so-called chalk-talk area. This embodies what conferences are all about: conferring. 

With “Diversity in team composition”, Henrik Andersson took the small stage trying to convince us that when assembling good teams, diversity rocks and uniformity, well, not so much. With some simple examples (“can I please ask everyone wearing black clothes to stand up. You are now a team”), he showed us that there’s much more to it than randomly throwing some people together.

Then it was Selena Delesie‘s turn to shine in the beamer lights. In “Focusing Testing on Business Needs”, she explained how to focus the testing effort on customer needs. She asked some pertinent questions; Are you valued in your team? How do you know?

The last presentation slot of the day in the testing track was for yours truly. In Artful Testing, I talked about how I think testing can benefit from the arts. From thoughtfully looking at it, to develop our thinking. From critical theory and the tools used by art critics, to become software critics. From artists, and how they look at the world – through artist personas). I also touched on the importance of context in evaluating art and software. I received some great reactions and feedback afterwards, and some good tips from Pradeep and Rikard as well.

After that, there was dinner, drinks and Øredev Open, where Pradeep was invited to present “The next generation software tester”. In theory. But you know how these things go. In theory, theory and practice are the same; in practice they are not: dinner took a bit longer than expected, drinks were abundant and so it happened that Pradeep took the stage for some Beer-Driven Exploratory Presenting. It was a great stand-up routine.

Ola Hylten joined in and Shmuel decided to whip out his box with tester games and puzzles. Time for some serious thinking, mixed with laughs. When the Øredev Open closed, we took ourselves and our silly games to the hotel bar where innocent passers-by quickened their pace.

Day 3

Day 3 in the test track started with “Agile testing: advanced topics” by Janet Gregory who highlighted five topics that had emerged since the release of “Agile Testing” by Lisa Crispin and herself. She mentioned feature acceptance (when you’re not able to deliver everything, focus on the features that matter), collaborative automation, large organizations, distributed teams, continuous learning.

Next up was my favorite Swedish philosopher (granted, I only know one), Rikard Edgren, who delivered did a spot-on and thought-provoking session called “Curing Our Binary Disease”. He stated that software testing is suffering from a binary disease: pass/fail addiction, coverage obsession, metrics tumor and sick test design techniques (sick as in “ill”, not “wicked” – my interpretation). Couldn’t agree more. He also mentioned his infamous “software potato”, which made for following legendary phrase: “A tester might not even know that he’s in the potato”. 

All this binary goodness got me thinking: Stay/Go? Focus/Defocus? Defocus it was. I chose to do a final round of the expo and do a quick Copenhagen visit to get some fresh air while it was still light out.

That concluded Øredev 2011. It was great to finally meet Selena, Sigge and Pradeep. And Robert Bergqvist as well. It was great catching up with others (Johanna, Shmuel, Henrik, Janet, David, Ola, Rikard,…). Next up: Eurostar in Manchester next week. A full-blooded tester conference that will rock as well. Let’s meet there.

Finding Porcini

A couple of weeks ago I found myself in southwestern France, a region which – at the time – was being struck by a unseen spell of global wetting. Summer had arrived three months early, people said. April and may had been exceptionally dry and warm; but in July, autumn was knocking on the door of our vacation home. Early autumn, I was told by our gentle host Philippe, goes hand in hand with a peculiar phenomenon: fungi frenzy/mushroom madness. All the locals get this strange misty-eyed look and head for the woods to hunt for the precious boletus edulis, more commonly known as  fungo porcino, porcine mushroom, cep and lovingly referred to as “the brown plague”, as it tends to halt the local economy.

When Philippe invited me to join him on an early morning quest for porcini, I gladly accepted. It sounded like a treasure hunt, as fresh porcini are sold for outrageous prices to local restaurants. And who doesn’t enjoy a good treasure hunt after breakfast? We left for the woods, armed only with wooden baskets, a knife and a sturdy 4×4.

Mushroom hunting, I found out quickly, is an art in its own right. Slowly and carefully, like an old sensei, Philippe unveiled his mushroom hunting mysteries. And as the mysteries disappeared, a neat set of categorized heuristics came trickling through:

The Mission

  • “Ready Zeger? So, we’re looking for cèpes, têtes de negres and chanterelles. Leave the others be. Some are poisonous, others just don’t taste nice”
  • “When we stop? When we’re finding more mushrooms than we can carry. Or when we’re not finding anything anymore.”

Classification

  • “Wait, you don’t know what to look for, do you? Come over here for a sec. This is a vintage cèpe. A fungho porcino. Like all other boletes, it has tubes extending downward from the underside of the cap, rather than gills. The pore surface of fruit body is whitish when young, but ages to a greenish-yellow when older.”
  • “Be careful there. Some mushrooms look very much like porcini. You should look under the hood. If it’s yellow, don’t touch ’em.”
  • “When you’re not really sure, scratch the bottom of the hood. When the scratch turns purple, don’t touch.”

Timing

  • “Why we’re heading out all of a sudden? Porcini tend to appear after summer peaks, depending on the weather. Usually they pop up about a week after a wet spell.”
  • “Oh no, that’s not true, Zeger. The fact that we picked loads of mushrooms here doesn’t mean I’m not coming back here tomorrow. These things grow fast. They often push overnight, you know.”

Location

  • “Location is everything. You should look at open spots in the woods, where the sun can actually reach the ground. Look, that should be a good place over there. Do you see the sunbeams peaking through the leafs? Let’s head over there.”
  • “When you find one, mind your step. Where there is one, there are many. You might crush some perfectly good ‘shrooms hidden under some leaves or grass.”
  • “Spotting porcini takes a trained and experienced eye. Here’s a pro-tip: look for where the leaves bunch up – perhaps they are being pushed up by a growing mushroom.”
  • “Don’t spend your time looking near ferns, man. Ferns grow on intensely acid soil, porcini don’t”
  • “Hey Zeger! You see this black beauty here? This is a tête de nègre, a particularly tasty and expensive kind of cep. Look for them near oak trees.”
  • “This here’s a chanterelle. If you spot two of them, follow the line that connects them; they always grow on a straight line.”

Picking technique

  • “Use a knife. Never ever pull mushrooms out of the ground. Cut them. If you damage their mycelium, they won’t grow again next year.”
  • “Remember: be gentle, cut them near the bottom of the stem in a straight line. Don’t break the hood.”
  • “Wait! Don’t cut the really small ones – they’ll be worth much more later on.”

I was soaking with sweat after a couple of hours of intense scouting. In a short timespan, Philippe managed to transform me into a die-hard mushroom hunter. A novice still, but I felt I was learning quickly. Philippe’s heuristics (not best practices, mind you) helped me discern the good from the bad, finding porcini hidden under leaves and cantharelles in a neat straight line. I even developed my own heuristics as I went along: I started looking alongside paths through the woods – plenty of chances for the sun to peep through the deck of leaves, and easier to spot since the vegetation is less dense.

Testing porcini

As I was wandering through the woods with eagle eyes and at a snail’s pace, it all felt strangely familiar. When Philippe said “Where there is one, there are many”, it struck my tester chord. Here I am, a tester, looking for mushrooms, which doesn’t seem to be all that different than looking for bugs. No wonder I liked it so much. I also realized that when I’m looking for bugs, I use these kind of heuristics all the time, but all too often I’m not very aware of them. Which is a pity, because used consciously, these heuristics (“a fallible method for solving a problem”) can be a really powerful tool to boost your exploratory testing efforts.

  • Start with a mission – make sure you – and your team – know what to look for, since our conception limits our perception. Michael Bolton often quotes Marshall McLuhan on this: “I wouldn’t have seen it if I hadn’t believed it”
  • Make sure you’ve got your classification right. If you’re only interested in a specific kind of bug, maybe you shouldn’t waste time reporting others. You could consider parking them somewhere, or keeping the reporting rather lightweight by MIPping (mention in passing) them. But try to stick to your main focus for the session. And if you find a nice-looking bug, is it really? Scratch it, it might turn purple
  • Timing – as in mushroom picking – is also a factor to be considered in bug hunting. Are there typical times at which the application is less stable? When is an ideal testing time, really? Again, this is largely dependent on context
  • Location, location, location. Personally, I use many heuristics to guide me where to test. Which areas are more vulnerable? When you find one, there tend to be many others, indeed. As leaves bunching up *might* indicate a pushing mushroom, seemingly insignificant facts might be a tell-tale sign for bugs nearby: the code that developers write after a wild night of partying might not be all that good, for example. Or they can just have a bad day. I was once told by an old native American medicine man that developers are human too
  • As some mushrooms are picked and not cut, our bag-o-techniques should enable us to deal with any situation. As Lee Copeland points out in A Practitioner’s Guide to Software Test Design: a tester should carry his techniques with him at all times, just like a handyman’s toolbox follows him around everywhere he goes. Apply a specific technique, use an particular approach when the situation calls for it.

For the record: I’m not a mushroom master, yet. I lack practice, experience and domain knowledge to attain mastery. I’m not a testing master either, as I’m in constant learning mode. For every good practice I know, in context, I am aware that there’s always another context that I need to get myself familiar with. That prospect may seem humbling and daunting to many, but I wouldn’t want it any other way. That’s Context-Driven Testing for ya.

(For more info, see The Seven Basic Principles of the Context-Driven School as a starting place. There’s a lot more where that came from).

Exploring Rapid Reporter

Eploring Rapid Reporter

This is my attempt at an exploratory essay.

I will be documenting my first exploration of a note-taking tool (Shmuel Gershon‘s free Rapid Reporter) to assist exploratory testing sessions.  That’s right, I’ll be exploring an explorer’s tool – sounds like this has the makings of a pretty meta experience.

I first heard about the tool when Shmuel gave the Rebel Alliance at Star East a sneak peek of what he was working on at the time. The Rebel Alliance was a very cool “conference within a conference” organised by Matt Heusser, with plenty of lightning talks and great discussions. You can see a video of his short talk here.

Up till now, I have been using David Gilbert’s TestExplorer, Jonathan Kohl’s Session Tester and James Bach’s Scan Tool to document my exploratory testing sessions (with miscellaneous results), so when he finally offered it to the public during the Blogstar competition (in which Shmuel was a well-deserved runner-up by the way), I knew I just had to give it a try. 

I started out by reading the user guide on the site, skimmed through the known issues list on the page and read through the short faq section. During this little theory intake, I deliberately didn’t start the tool (although I had to suppress the urge). I wanted to start using it for a first time in a proper testing session. 

Start

I fired the RapidReporter.exe, was asked to enter a name and a charter, all pretty straightforward.

*Edit: Actually, I entered a session name when prompted for a name. Later, when looking in the .csv-file, I realised that it was supposed to be my own name, the name of the reporter. It confused me, so perhaps the naming was ambiguous? Would “Reporter” be a better option? (Made a NOTE about it).  

I was ready to test a piece of newly coded software. The cool thing was that RapidExplorer stayed on top, tucked away in a place that didn’t bother visibility: no alt+Tab switching, no interrupting my flow. Note-taking went smoothly. Switching the note type by the up/down arrows was intuitive and quick.

After a short while I noticed a blue progress bar ehm… progressing and it surprised me. That’s a nice feature, it gives you a visual clue of the session’s progress, but without the actual distraction and pressure of a detailed timer. The blue bar also implied a default duration, since I hadn’t indicated any preferred session length before starting. I wondered why the default duration wasn’t asked along with the name and charter, and figured that was probably because most people use the same timebox for their sessions anyway (60-90-120 minutes, whatever). So what is the duration of the current session? It isn’t indicated on the main bar. Is there a menu of some kind? The most intuitive thing would be a right-click, I’d say. Bingo! Three menu items there: Time until end, Open working folder and About. But where do I see the duration of this very session? The only time-related entry is the first one, but this only breaks down in a menu to select either ’60-90-120 min until end’ or to ‘stop the session’. Selected menu items in comparable products often have a selection checkmark or bullet next to them (NOTE).

I opened some menu items in the application under test and noticed an inconstistency, so taking a screenshot was in order. I didn’t remember reading about hotkeys, but I tried some anyway (CTRL+S, ALT+S, S – which just entered an ‘S’ in Rapid Reporter, of course). I was afraid I would lose the expanded menu if I clicked outside the application. I eventually ended up clicking the S button, but – as I expected – the screenshot didn’t show the expanded menu items. Apparently, there is no keyboard shorcut yet, which makes it impossible to capture these kinds of phenomena (NOTE). For all the other screenshots I ended up taking, clicking the S did the trick just fine. The captures were saved as time-stamped jpegs in the startup folder. A huge improvement, since I was used to take the screenshot, paste it in MSPaint, name it & save it. By the time I got to the application, I often forgot what I was actually doing in the first place. Darned short-time memory.

I got interrupted by a phone call and wanted to pause the timer. Mmm. No pause option to be seen. I tried hitting the spacebar to pause – don’t know where that came from. Media players or games, maybe? There *was* a the stop option in the menu, so I tried that. But this actually stopped the timer and reset the progress bar to zero. Is there an option to pause and “restart” where I paused it? I can think of some occasions in which that would be useful (NOTE).

I was still pondering the timer reset, when I noticed that the progress bar started progressing again. Is the “stopped” state really a stopped state? Tried it again, but now stopping really stopped it. Maybe I started a session without knowing that first time? Maybe something intermittent? (INVESTIGATE LATER)

When time was up, this was clearly indicated by the flashing stopwatch. It was still possible to add keep working with it even when time is up, which seems like a logical and practical thing to do. I can think of many situations in which you want to continue for a bit, clean up some loose ends.

I started a new session, worked with that for a while. I was really getting used to this new kind of notetaking. I wanted to extend the length of the session, and wondered what would happen if I changed it to 120 minutes. I expected the progress bar to redraw itself (keeping in mind the time spent and time still to go), since my current session was 60 minutes. But it didn’t. Did the duration really change? Possibly, but no visual clue to confirm that (NOTE).

I was curious about what was actually being recorded during the session and opened the csv-file in the startup folder. All my notes were there, time-stamped and delimited with commas. All nice and powerful.

Stop

I stopped the session while it was still ongoing (by using the cross in the upper right corner) and got the message that an error occured when writing  the note into a file. Yes, the csv-file was still open and although I didn’t enter a note since opening the file, the application attempts to write something into the file when closing. It would be nice if the excel closed automatically instead of throwing an error when nothing changed (NOTE).

I checked the screenshots taken during the session (by clicking “S”) and the Rapid Reporter bar is in the screenshots. At first I thought this could be annoying for a tester, but then I figured that it would always be in a non-disturbing place anyway. It can also convey useful information: how far you were in the session, what the note was at the time, etc. 

I decided to try taking a screenshot the regular way (using Printscreen), and it turns out that the Rapid Reporter bar is *not* in the picture. Strange. Seems like an inconsistency, but I might not have all information here. I’ll talk to Shmuel about it (NOTE).   

Conclusion

I decided to stop my first exploration of Rapid Reporter here. It was a pretty shallow tour of some basic functionalities – I didn’t even look into the reporting possibilities yet, I’ll certainly do that next time. This little write-up just documents my first two hours of exploring a new tool. I wanted to learn about it first, not necessarily find bugs.

To round it up: I must say I’m impressed. It’s easy to use, and non-intrusive indeed. When using other Session Based Test Management tools, I often ended up spending more time in the note-taking tools than in the application under test, which is a bit of a drag. Rapid Reporter puts all focus on the software being explored. It supports exploration, while allowing testers to stay in their flow. This is important for me anyway: everytime someone interrupts my flow, God kills a kitten. Ok, not really, but you catch my drift.

Of course, I stumbled upon some issues, but they didn’t annoy me, they didn’t make me stop using it. I didn’t expect perfection – after all, the readme file explicitly mentions that it’s still buggy. I would have gotten *really*suspicious if it would have claimed it was bug-free. So its current state is perfectly okay for me. After all, it is brand new. It’s work-in-progress. The best thing Shmuel could do, was to unleash it upon an international cohort of testers. And he did. It turned out to be a great gift to the testing community.

Will definitely use again. A+++

Collateral features

About collateral features – things that were not expected, but do provide value in the end

Last year, James Lyndsay introduced me to his “Nr1 diagram of testing” (© Workroom Productions): a deceivingly simple model that tries to capture the essence of testing.

The circle on the left represents our expectations – all the things we expect. This is the area where scripted tests or checklists come into play to tell us about the value that is present. The right hand circle represents the actual deliverable – what the software really does. This never completely matches what was originally expected, so this is where we use exploratory testing to find out about risk. 

The diagram divides the world into four distinct regions:

  • The overlap. These are the things we both expected and got. 
  • The region outside the circles. That is all we didn’t want, and didn’t receive in the end. A pretty infinite set, that is.
  • The left-hand arc. This is what we expected to get, but didn’t receive. This means that the deliverable turned out less valuable than we had hoped.
  • The right-hand arc. The software system under test does all kinds of things things we didn’t expect. There’s clearly some risk involved, here, and it’s up to us to discover just how much risk there is.

Simplicity is of course the main strength of such a model. It can certainly help to identify or classify things in our quest to quickly grasp the essence of things.

These four regions got me thinking. I’d like to expand on that. What about things that we expected and received, but do not prove value, for instance? Unneeded features – not too unrealistic. Or – even better – what about things that were not expected, but are there and actually do provide value in the end? Immediately the term “Collateral features” came to mind: no matter how hard we try to create designs for certain uses, people will always utilize them in their own way. These unintended uses can be strange sometimes, but some them are downright brilliant.

Take a look at Alec Brownstein. While most people use Google AdWords to promote their business (after all, that’s what it was designed for), Alec used it to get a job. He was trying to land a job as a copywriter with some of the top ad agencies in New York City. He assumed that the creative directors at these top agencies would “vanity google” their own name. So he bought AdWords of the names and put a special message in to each of them. The rest is history. He now works for Y&R, after a total investment of $6 (story here).

Collateral features also emerged in the microblogging world. Because Twitter provided no easy way to group tweets or add extra data, the twitter community came up with their own way: hashtags. A hashtag is similar to other web tags- it helps add tweets to a category. Hashtags weren’t an official feature, but they sure made their way into the daily twitter work lexicon of billions of people. 

In an article in Forbes magazine called You Can’t Predict Who Will Change The World, Nassim Nicholas Taleb pointed out that things are all too often discovered by accident—but we don’t see that when we look at history in our rear-view mirrors. The technologies that run the world today (like the Internet, the computer and the laser) are not used in the way intended by those who invented them.

There will always be unforeseen usage of your software. Some prove risky, others do contain value. Some of these collateral features even replace the intended use to become main features and make their way in the ‘expected’ circle.  Ultimately, your customers make the final call. They decide how to use your product or service. Not you, not your marketeers.

– “But no user would ever do that!”

– “Fair enough. Wanna bet?”

Through the looking-glass, and what testers found there – EWT05

Today it was weekend testing time again, and things got pretty crowded with a pleiad of international testers: facilitators Anna Baik and Markus Gärtner, !ndra from Hyderabad, Shiva Shankari, Krishnaveni K, Nagashree Manjunath, Jeroen RosinkAjay BalamurugadasJassi and myself.

The target today was Virtual Magnifying Glass, an open source screen magnifier for Windows, Linux, FreeBSD and Mac OS X.

The mission was basically: “Test this!”. Although a mission like that is way too vague to start from, it *is* a mission that we all encounter once in a while. It is our job to ask questions to make sure we understand what is needed.  So I wondered… do they just need bugs? Or an advice? Or information about how the application works? Ajay also went in questioning mode: “Test : create test ideas, hunt for bugs, compare with a different product, learn the product, so many things? What exactly is required?”. So with a little help of Ajay, the mission was rephrased to “Find quality related valuable information about the product”.

So far so good. I wanted to try pair testing over skype and Ajay was willing to team up with me. We set up a call and were soon discussing in person which approach to take. Although we encountered connection problems later on, it was a good experience that bears repeating.

Lessons learned:

  • It’s easy to feel frustrated because time is too short. After a round of introductions, some explanation and a discussion about the mission, there’s less than 30 minutes left to explore the software. Even for seasoned Rapid Testers this would be an uncomfortably short timebox. Avoid frustration by setting realistic goals for yourself.
  • When pairing, you’re not starting testing right away. There’s some discussion first, some setting up needs to be done too. Pairing is probably more beneficial when there is more time to test, so there’s plenty of time to talk things through. Now I felt a bit rushed because the clock was ticking.
  • Read-me files are a good place to start when exploring an unknown piece of software. In this case, they provided me with a good model of the software, a good basis to start exploring from.
  • Logging bugs takes time. Time not being spent testing. An issue also addressed in the “Why is testing taking so long (part1)(part2)” blogposts by Michael Bolton. 

Some nice, thought-worthy quotes were dropped during the debriefing too:

  • “Clearing traps is a skill. Recognizing traps is a bigger skill”
    Ajay Balamuragadas (c) 2010
  • “Minds are shaped when guided under pressure in a certain direction trying to maintain vision and control” 
    Jeroen Rosink (c) 2010
  • “Watch out where the huskies go, and don’t you eat that yellow snow” 
    Frank Zappa (c) 1974

Well… I guess all that snow is finally getting to me. Frank never made it to this European Weekend Testing session. He wasn’t much of a tester either. But he sure knew how to play that guitar. And I think he would have been a great explorer, having fun all the way.

The ongoing weekend testing adventure – EWT03

A write-up of European Weekend Testing session 3 – EWT03

I had to skip the previous weekend testing session involving Bing maps (which was a pity – I absolutely adore map applications), but today I was able to participate in EWT03. It was again an interesting experience. Not too many participants this time, which made for a more cozy atmosphere. Today’s testers were Marlena Compton, Anna Baik, Markus Gärtner and myself.

The Mission: You work in a small-medium company, and your manager has been asked to evaluate switching the company over to using Google calendar. He needs a quick assessment from you before his conference call this afternoon. Use the FCC CUTS VIDS touring heuristic to guide you.

We started firing off a bunch of questions at the boss, but he was in a hurry and kind of unresponsive – something about an important golf game he had to attend. He just wanted a quick assessment of  the tool, he apparently wasn’t into answering today. So much for questioning – we were on our own pretty soon. I wonder if there’s any way to keep an imaginary boss from leaving a virtual meeting. Still not sure about that.

We decided to divide the coverage since we started late and there was less than half an hour of testing time left. We ended up cutting the FCC CUTS VIDS into – surprise! – FCC, CUTS and VIDS (yes, sometimes you just go for the obvious). I settled for the VIDS:

  • Variability tour: look for things you can change in the application – and then you try to change them.
  • Interopeability tour: what does this application interact with?
  • Data tour: identify the major data elements of the application.
  • Structure tour : find everything you can about what comprises the physical product (code, interfaces, hardware, files, etc…)

In the meanwhile the boss had magically reappeared to give us a quarter of an hour extra – he probably forgot his bag of golf clubs. But I quickly realised that time would still be way too short to look at all aspects and concentrated on variability and interoperability – and on a list of existing bugs as well, something I remembered from the first session. In my quest for things that can be changed in the application, I settled for the system date, not really a part of the application but still something that can change while you’re using it. Some interesting bugs ensued, including a very severe one in Skype group chat – not really part of the mission. I thought I had lost connection but it turned out that the order of the chat messages was totally messed up. It took a while before I realised what had happened and lost a good deal of time because of that. Later on, I switched to interoperability but I also lost my internet connection for for a while, which was annoying but in retrospect also pointed me to a synchronisation bug.

The debriefing, moderated by Anna, revealed some good points from the other participants. Marlena covered the FCC (feature, complexity, claims) part and did a feature tour first, which proved very helpful. She explored all functionalities to get more familiar with them before she went off to investigate other areas. She wondered how the rest of us could start testing without doing such a feature tour/exploration first. But in fact, I think we all started with a bit of general exploration. I know I did. It feels kind of natural, probing around the surface for a while before before diving into the abyss of detail. Concerning the splitting up of the heuristic among several testers, this turned out to be a bit distracting, confusing and counter-productive. There is quite some overlap between the different areas and sticking to only some of them might mean that you’re missing out on otherwise obvious problems. This led us to the use of sapience in testing, being able to picking the heuristics that are useful for the given features.

We didn’t get to brief the boss in the end, so you could say that our mission didn’t really succeed. But hey, some say that failure breeds success. Soon, things will be looking so bright we’ll have to wear shades.