Tag: Exploratory testing

Rapid Software Testing – skilled software testing unleashed

Up to 11

The software testing profession looks like a steadily maturing profession from the outside. After all, there are certifications schemes like ISTQB, CAT, IREB and QAMP (the one to rule them all), standards (ISO 29119) and companies reaching TMM (test maturity model) levels that – just like a Spinal Tap guitar amplifier – one day might even go up to 11. The number of employees that companies send off to get certified in a mere three days is soaring, and new certification programs are being created as we speak. Quick and easy. Multiple choice exams for the win!

The reality, however, is that the field of software testing is torn between different “schools” of testing. You could see these schools as determined and persistent patterns of belief, speech and behaviour. This means that different people – all calling themselves “test professionals” – have vastly different ideas of what testing is all about. Even something elementary as the definition of testing varies from “demonstration of fitness for purpose” to “questioning a product in order to evaluate it”, depending on who you talk with (for more info on the schools of software testing, I heartily recommend Brett Pettichord’s presentation on the subject).

And so it happens that different people think differently about “good” or “mature” software testing. I, for one, don’t believe in tester certification programs, at least not in the format they are in now and the way they are being used in the testing profession. The current business model is mainly designed to get as many people as possible certified within the shortest timeframe. Its prime focus is on certifiability, not on tester skill, and certainly not on the advancement of the craft. Advancement comes from sharing, rather than shielding.

Rapid Software Testing (RST)

So what are the options for a tester on a quest for knowledge and self-improvement? What is a budding tester to do?

I think there are valuable alternatives for people who are serious about becoming a world-class tester. One of these is Rapid Software Testing (RST), a 3-day hands-on course designed by James Bach and Michael Bolton.

Actually, calling this “a course” doesn’t do it justice. RST is at the same time a methodology, a mind-set and a skill set about how to do excellent software testing in a way that is very fast, inexpensive, credible and accountable. It is a highly experiential workshop with lessons that stick.

How is RST different?

During RST you spend much of the time actually testing, working on exercises, puzzles, thought experiments and scenarios—some computer-based, some not. The goal of the course is to teach you how to test anything expertly, under extreme time pressure and conditions of uncertainty, in a way that will stand up to scrutiny.

The philosophy presented in this class is not like traditional approaches to testing, which ignore the thinking part of testing and instead focus on narrow definitions for testing terms while advocating never-ending paperwork. Products have become too complex for that, time is too short, and testers are too expensive. Rapid testing uses a cyclic approach and heuristic methods to constantly re-optimize testing to fit the needs of your clients.

What’s in it for you?

  • The ability to test something rapidly and skilfully is critical. There is a growing need to test quickly, effectively, and with little information available. Testers are expected to provide quick feedback. These short feedback loops make for more efficient and higher quality development
  • Exploratory testing is at the heart of RST. It combines test design, test execution, test result interpretation, and learning into a seamless process that finds a lot of problems quickly. Experienced testers will find out how to articulate those intellectual processes of testing that they already practice intuitively, while new testers will find lots of hands-on testing exercises that help them gain critical experience
  • RST teaches you how to think critically and ask important questions. The art of questioning is key in testing, and a very important skill for any consultant
  • RST will provide you with tools to do excellent testing
  • RST will heighten your awareness

Bold claim bottom-line:

RST will make you a better tester.

RST comes to Belgium

Co-learning and Z-sharp are proud to announce that from 30 September – 2 October, Michael Bolton will visit Belgium to deliver the first ever RST course on Belgian soil, giving you the opportunity to experience this unique course in person. More info can be found here, or feel free to contact us for more info.

Brace yourself for an mind-opening experience that will energize and boost your mind.

All the way up to 11.

(for even more information and testimonials about RST, see Michael Bolton’s RST page)

 

Finding Porcini

A couple of weeks ago I found myself in southwestern France, a region which – at the time – was being struck by a unseen spell of global wetting. Summer had arrived three months early, people said. April and may had been exceptionally dry and warm; but in July, autumn was knocking on the door of our vacation home. Early autumn, I was told by our gentle host Philippe, goes hand in hand with a peculiar phenomenon: fungi frenzy/mushroom madness. All the locals get this strange misty-eyed look and head for the woods to hunt for the precious boletus edulis, more commonly known as  fungo porcino, porcine mushroom, cep and lovingly referred to as “the brown plague”, as it tends to halt the local economy.

When Philippe invited me to join him on an early morning quest for porcini, I gladly accepted. It sounded like a treasure hunt, as fresh porcini are sold for outrageous prices to local restaurants. And who doesn’t enjoy a good treasure hunt after breakfast? We left for the woods, armed only with wooden baskets, a knife and a sturdy 4×4.

Mushroom hunting, I found out quickly, is an art in its own right. Slowly and carefully, like an old sensei, Philippe unveiled his mushroom hunting mysteries. And as the mysteries disappeared, a neat set of categorized heuristics came trickling through:

The Mission

  • “Ready Zeger? So, we’re looking for cèpes, têtes de negres and chanterelles. Leave the others be. Some are poisonous, others just don’t taste nice”
  • “When we stop? When we’re finding more mushrooms than we can carry. Or when we’re not finding anything anymore.”

Classification

  • “Wait, you don’t know what to look for, do you? Come over here for a sec. This is a vintage cèpe. A fungho porcino. Like all other boletes, it has tubes extending downward from the underside of the cap, rather than gills. The pore surface of fruit body is whitish when young, but ages to a greenish-yellow when older.”
  • “Be careful there. Some mushrooms look very much like porcini. You should look under the hood. If it’s yellow, don’t touch ’em.”
  • “When you’re not really sure, scratch the bottom of the hood. When the scratch turns purple, don’t touch.”

Timing

  • “Why we’re heading out all of a sudden? Porcini tend to appear after summer peaks, depending on the weather. Usually they pop up about a week after a wet spell.”
  • “Oh no, that’s not true, Zeger. The fact that we picked loads of mushrooms here doesn’t mean I’m not coming back here tomorrow. These things grow fast. They often push overnight, you know.”

Location

  • “Location is everything. You should look at open spots in the woods, where the sun can actually reach the ground. Look, that should be a good place over there. Do you see the sunbeams peaking through the leafs? Let’s head over there.”
  • “When you find one, mind your step. Where there is one, there are many. You might crush some perfectly good ‘shrooms hidden under some leaves or grass.”
  • “Spotting porcini takes a trained and experienced eye. Here’s a pro-tip: look for where the leaves bunch up – perhaps they are being pushed up by a growing mushroom.”
  • “Don’t spend your time looking near ferns, man. Ferns grow on intensely acid soil, porcini don’t”
  • “Hey Zeger! You see this black beauty here? This is a tête de nègre, a particularly tasty and expensive kind of cep. Look for them near oak trees.”
  • “This here’s a chanterelle. If you spot two of them, follow the line that connects them; they always grow on a straight line.”

Picking technique

  • “Use a knife. Never ever pull mushrooms out of the ground. Cut them. If you damage their mycelium, they won’t grow again next year.”
  • “Remember: be gentle, cut them near the bottom of the stem in a straight line. Don’t break the hood.”
  • “Wait! Don’t cut the really small ones – they’ll be worth much more later on.”

I was soaking with sweat after a couple of hours of intense scouting. In a short timespan, Philippe managed to transform me into a die-hard mushroom hunter. A novice still, but I felt I was learning quickly. Philippe’s heuristics (not best practices, mind you) helped me discern the good from the bad, finding porcini hidden under leaves and cantharelles in a neat straight line. I even developed my own heuristics as I went along: I started looking alongside paths through the woods – plenty of chances for the sun to peep through the deck of leaves, and easier to spot since the vegetation is less dense.

Testing porcini

As I was wandering through the woods with eagle eyes and at a snail’s pace, it all felt strangely familiar. When Philippe said “Where there is one, there are many”, it struck my tester chord. Here I am, a tester, looking for mushrooms, which doesn’t seem to be all that different than looking for bugs. No wonder I liked it so much. I also realized that when I’m looking for bugs, I use these kind of heuristics all the time, but all too often I’m not very aware of them. Which is a pity, because used consciously, these heuristics (“a fallible method for solving a problem”) can be a really powerful tool to boost your exploratory testing efforts.

  • Start with a mission – make sure you – and your team – know what to look for, since our conception limits our perception. Michael Bolton often quotes Marshall McLuhan on this: “I wouldn’t have seen it if I hadn’t believed it”
  • Make sure you’ve got your classification right. If you’re only interested in a specific kind of bug, maybe you shouldn’t waste time reporting others. You could consider parking them somewhere, or keeping the reporting rather lightweight by MIPping (mention in passing) them. But try to stick to your main focus for the session. And if you find a nice-looking bug, is it really? Scratch it, it might turn purple
  • Timing – as in mushroom picking – is also a factor to be considered in bug hunting. Are there typical times at which the application is less stable? When is an ideal testing time, really? Again, this is largely dependent on context
  • Location, location, location. Personally, I use many heuristics to guide me where to test. Which areas are more vulnerable? When you find one, there tend to be many others, indeed. As leaves bunching up *might* indicate a pushing mushroom, seemingly insignificant facts might be a tell-tale sign for bugs nearby: the code that developers write after a wild night of partying might not be all that good, for example. Or they can just have a bad day. I was once told by an old native American medicine man that developers are human too
  • As some mushrooms are picked and not cut, our bag-o-techniques should enable us to deal with any situation. As Lee Copeland points out in A Practitioner’s Guide to Software Test Design: a tester should carry his techniques with him at all times, just like a handyman’s toolbox follows him around everywhere he goes. Apply a specific technique, use an particular approach when the situation calls for it.

For the record: I’m not a mushroom master, yet. I lack practice, experience and domain knowledge to attain mastery. I’m not a testing master either, as I’m in constant learning mode. For every good practice I know, in context, I am aware that there’s always another context that I need to get myself familiar with. That prospect may seem humbling and daunting to many, but I wouldn’t want it any other way. That’s Context-Driven Testing for ya.

(For more info, see The Seven Basic Principles of the Context-Driven School as a starting place. There’s a lot more where that came from).

Exploring Rapid Reporter

Eploring Rapid Reporter

This is my attempt at an exploratory essay.

I will be documenting my first exploration of a note-taking tool (Shmuel Gershon‘s free Rapid Reporter) to assist exploratory testing sessions.  That’s right, I’ll be exploring an explorer’s tool – sounds like this has the makings of a pretty meta experience.

I first heard about the tool when Shmuel gave the Rebel Alliance at Star East a sneak peek of what he was working on at the time. The Rebel Alliance was a very cool “conference within a conference” organised by Matt Heusser, with plenty of lightning talks and great discussions. You can see a video of his short talk here.

Up till now, I have been using David Gilbert’s TestExplorer, Jonathan Kohl’s Session Tester and James Bach’s Scan Tool to document my exploratory testing sessions (with miscellaneous results), so when he finally offered it to the public during the Blogstar competition (in which Shmuel was a well-deserved runner-up by the way), I knew I just had to give it a try. 

I started out by reading the user guide on the site, skimmed through the known issues list on the page and read through the short faq section. During this little theory intake, I deliberately didn’t start the tool (although I had to suppress the urge). I wanted to start using it for a first time in a proper testing session. 

Start

I fired the RapidReporter.exe, was asked to enter a name and a charter, all pretty straightforward.

*Edit: Actually, I entered a session name when prompted for a name. Later, when looking in the .csv-file, I realised that it was supposed to be my own name, the name of the reporter. It confused me, so perhaps the naming was ambiguous? Would “Reporter” be a better option? (Made a NOTE about it).  

I was ready to test a piece of newly coded software. The cool thing was that RapidExplorer stayed on top, tucked away in a place that didn’t bother visibility: no alt+Tab switching, no interrupting my flow. Note-taking went smoothly. Switching the note type by the up/down arrows was intuitive and quick.

After a short while I noticed a blue progress bar ehm… progressing and it surprised me. That’s a nice feature, it gives you a visual clue of the session’s progress, but without the actual distraction and pressure of a detailed timer. The blue bar also implied a default duration, since I hadn’t indicated any preferred session length before starting. I wondered why the default duration wasn’t asked along with the name and charter, and figured that was probably because most people use the same timebox for their sessions anyway (60-90-120 minutes, whatever). So what is the duration of the current session? It isn’t indicated on the main bar. Is there a menu of some kind? The most intuitive thing would be a right-click, I’d say. Bingo! Three menu items there: Time until end, Open working folder and About. But where do I see the duration of this very session? The only time-related entry is the first one, but this only breaks down in a menu to select either ’60-90-120 min until end’ or to ‘stop the session’. Selected menu items in comparable products often have a selection checkmark or bullet next to them (NOTE).

I opened some menu items in the application under test and noticed an inconstistency, so taking a screenshot was in order. I didn’t remember reading about hotkeys, but I tried some anyway (CTRL+S, ALT+S, S – which just entered an ‘S’ in Rapid Reporter, of course). I was afraid I would lose the expanded menu if I clicked outside the application. I eventually ended up clicking the S button, but – as I expected – the screenshot didn’t show the expanded menu items. Apparently, there is no keyboard shorcut yet, which makes it impossible to capture these kinds of phenomena (NOTE). For all the other screenshots I ended up taking, clicking the S did the trick just fine. The captures were saved as time-stamped jpegs in the startup folder. A huge improvement, since I was used to take the screenshot, paste it in MSPaint, name it & save it. By the time I got to the application, I often forgot what I was actually doing in the first place. Darned short-time memory.

I got interrupted by a phone call and wanted to pause the timer. Mmm. No pause option to be seen. I tried hitting the spacebar to pause – don’t know where that came from. Media players or games, maybe? There *was* a the stop option in the menu, so I tried that. But this actually stopped the timer and reset the progress bar to zero. Is there an option to pause and “restart” where I paused it? I can think of some occasions in which that would be useful (NOTE).

I was still pondering the timer reset, when I noticed that the progress bar started progressing again. Is the “stopped” state really a stopped state? Tried it again, but now stopping really stopped it. Maybe I started a session without knowing that first time? Maybe something intermittent? (INVESTIGATE LATER)

When time was up, this was clearly indicated by the flashing stopwatch. It was still possible to add keep working with it even when time is up, which seems like a logical and practical thing to do. I can think of many situations in which you want to continue for a bit, clean up some loose ends.

I started a new session, worked with that for a while. I was really getting used to this new kind of notetaking. I wanted to extend the length of the session, and wondered what would happen if I changed it to 120 minutes. I expected the progress bar to redraw itself (keeping in mind the time spent and time still to go), since my current session was 60 minutes. But it didn’t. Did the duration really change? Possibly, but no visual clue to confirm that (NOTE).

I was curious about what was actually being recorded during the session and opened the csv-file in the startup folder. All my notes were there, time-stamped and delimited with commas. All nice and powerful.

Stop

I stopped the session while it was still ongoing (by using the cross in the upper right corner) and got the message that an error occured when writing  the note into a file. Yes, the csv-file was still open and although I didn’t enter a note since opening the file, the application attempts to write something into the file when closing. It would be nice if the excel closed automatically instead of throwing an error when nothing changed (NOTE).

I checked the screenshots taken during the session (by clicking “S”) and the Rapid Reporter bar is in the screenshots. At first I thought this could be annoying for a tester, but then I figured that it would always be in a non-disturbing place anyway. It can also convey useful information: how far you were in the session, what the note was at the time, etc. 

I decided to try taking a screenshot the regular way (using Printscreen), and it turns out that the Rapid Reporter bar is *not* in the picture. Strange. Seems like an inconsistency, but I might not have all information here. I’ll talk to Shmuel about it (NOTE).   

Conclusion

I decided to stop my first exploration of Rapid Reporter here. It was a pretty shallow tour of some basic functionalities – I didn’t even look into the reporting possibilities yet, I’ll certainly do that next time. This little write-up just documents my first two hours of exploring a new tool. I wanted to learn about it first, not necessarily find bugs.

To round it up: I must say I’m impressed. It’s easy to use, and non-intrusive indeed. When using other Session Based Test Management tools, I often ended up spending more time in the note-taking tools than in the application under test, which is a bit of a drag. Rapid Reporter puts all focus on the software being explored. It supports exploration, while allowing testers to stay in their flow. This is important for me anyway: everytime someone interrupts my flow, God kills a kitten. Ok, not really, but you catch my drift.

Of course, I stumbled upon some issues, but they didn’t annoy me, they didn’t make me stop using it. I didn’t expect perfection – after all, the readme file explicitly mentions that it’s still buggy. I would have gotten *really*suspicious if it would have claimed it was bug-free. So its current state is perfectly okay for me. After all, it is brand new. It’s work-in-progress. The best thing Shmuel could do, was to unleash it upon an international cohort of testers. And he did. It turned out to be a great gift to the testing community.

Will definitely use again. A+++

The ongoing weekend testing adventure – EWT03

A write-up of European Weekend Testing session 3 – EWT03

I had to skip the previous weekend testing session involving Bing maps (which was a pity – I absolutely adore map applications), but today I was able to participate in EWT03. It was again an interesting experience. Not too many participants this time, which made for a more cozy atmosphere. Today’s testers were Marlena Compton, Anna Baik, Markus Gärtner and myself.

The Mission: You work in a small-medium company, and your manager has been asked to evaluate switching the company over to using Google calendar. He needs a quick assessment from you before his conference call this afternoon. Use the FCC CUTS VIDS touring heuristic to guide you.

We started firing off a bunch of questions at the boss, but he was in a hurry and kind of unresponsive – something about an important golf game he had to attend. He just wanted a quick assessment of  the tool, he apparently wasn’t into answering today. So much for questioning – we were on our own pretty soon. I wonder if there’s any way to keep an imaginary boss from leaving a virtual meeting. Still not sure about that.

We decided to divide the coverage since we started late and there was less than half an hour of testing time left. We ended up cutting the FCC CUTS VIDS into – surprise! – FCC, CUTS and VIDS (yes, sometimes you just go for the obvious). I settled for the VIDS:

  • Variability tour: look for things you can change in the application – and then you try to change them.
  • Interopeability tour: what does this application interact with?
  • Data tour: identify the major data elements of the application.
  • Structure tour : find everything you can about what comprises the physical product (code, interfaces, hardware, files, etc…)

In the meanwhile the boss had magically reappeared to give us a quarter of an hour extra – he probably forgot his bag of golf clubs. But I quickly realised that time would still be way too short to look at all aspects and concentrated on variability and interoperability – and on a list of existing bugs as well, something I remembered from the first session. In my quest for things that can be changed in the application, I settled for the system date, not really a part of the application but still something that can change while you’re using it. Some interesting bugs ensued, including a very severe one in Skype group chat – not really part of the mission. I thought I had lost connection but it turned out that the order of the chat messages was totally messed up. It took a while before I realised what had happened and lost a good deal of time because of that. Later on, I switched to interoperability but I also lost my internet connection for for a while, which was annoying but in retrospect also pointed me to a synchronisation bug.

The debriefing, moderated by Anna, revealed some good points from the other participants. Marlena covered the FCC (feature, complexity, claims) part and did a feature tour first, which proved very helpful. She explored all functionalities to get more familiar with them before she went off to investigate other areas. She wondered how the rest of us could start testing without doing such a feature tour/exploration first. But in fact, I think we all started with a bit of general exploration. I know I did. It feels kind of natural, probing around the surface for a while before before diving into the abyss of detail. Concerning the splitting up of the heuristic among several testers, this turned out to be a bit distracting, confusing and counter-productive. There is quite some overlap between the different areas and sticking to only some of them might mean that you’re missing out on otherwise obvious problems. This led us to the use of sapience in testing, being able to picking the heuristics that are useful for the given features.

We didn’t get to brief the boss in the end, so you could say that our mission didn’t really succeed. But hey, some say that failure breeds success. Soon, things will be looking so bright we’ll have to wear shades.

Weekend Testing hits Europe – EWT01

The first European Weekend Testing session (January 16, 2010)

Last saturday, weekend testing hit European ground. The Weekend testing phenomenon has been the talk of testingtown for a couple of months now. They originally started out as the Bangalore Weekend Testers and are gradually taking their brilliant mix of cybergathering / testing / learning / mentoring / communication / pairing to a higher level. I  became intrigued when I first heard about them through a column by Fiona Charles and some namedropping at Eurostar shortly after. I liked the idea of personal and professional development, while serving the open-source community as well. My first thoughts were “what a shame we don’t have this over here” and “wouldn’t European testers like this too?”.

They do, apparently.

Markus Gärtner and Anna Baik – whom I met online in the Software Testing Club, set up the first European chapter and planned the first session with the help of Ajay Balamurugadas, a weekend testing veteran from Bangalore. On saturday january 16, some great  testers – and me –  gathered online for a unique experience. Among the participants: Markus, Anna and Ajay of course, Phil Kirkham, Jeroen De Cock, Anne-Marie Charrett, Thomas Ponnet, Maik Nogens and Jassi.

After some initial babylonic confusion in a skype group call, we decided to settle for a group chat. There was a round of introductions first, during which everyone stated his name, experience and what he/she wanted to take away from this session. After that, Ajay (aka the great facilitator) revealed the mission and the target to us, whilst stating enigmatically “that there were traps”. Right then I knew that this wasn’t going to be just another round of testing. The target turned out to be a piece of open-source image-editing software by the name of Splashup. From that moment on, there was a lot of questioning going on – one of the most important testing skills. People explored, asked questions, gave feedback. Some started off on the wrong foot, but quickly recovered and learned from that. We made a lot of assumptions as well, and not always good ones. In the beginning I was a bit distracted by all the chat messages flying around while testing, but once I developed that second pair of eyes, it worked out just fine. The first hour flew by and everyone sent in their bug reports on time (yes, there *was* a deadline). Peer pressure, in a good way.

Next up was an interesting group discussion. Everyone shared their impressions, approaches, things they learned and the traps they fell into. Ajay did a great job in moderating this, and made sure all testers were involved. Here are a couple of things I took away from this first session:  

  • When confronted with testing within a really short timespan, it is good to quickly decide on your personal mission and focus, and stick to that. Try to stay focused on the areas you chose and don’t go wandering off, unless of course you accidentally open pandora’s box and bugs are thrown at you. I sticked to the file opening and saving functions – and there’s plenty to report about that. 
  • I think the way we start testing largely depends on our domain knowledge as well. When confronted with a completely new product, you probably start exploring in an attempt to primordially learn about the application first. Knowing the domain makes you less of an explorer – and more of a bug hunter, which may be dangerous – make sure you don’t block out the learning. I had worked with Splashup before, which gave me a (misplaced?) sense of ‘expertise’, so I dove in and went on an immediate bughunt.
  • Always question the mission you get. Question your assumptions about the mission. The main goal might not always be a mere bughunt.
  • There are some great exploratory testing tools out there. I used Session Tester, which allowed me to quickly generate an html-report at the end. It is great for notetaking too. I really like the ‘Prime Me’-function in there – great to get new testing ideas when you’re stuck. I didn’t have to use it this time – ideas galore.
  • When working with image editing programs, I immediately tend to go in that ‘comparable product’-mode and start comparing it with Photoshop. While that may be ok to have some idea of the expected behaviour of filters and effects, it might not be ideal for an open source application. It makes you expect too much. Free tools often cater for other needs – their users have different expectations.
  • Afterwards we all agreed that it was strange that we didn’t pair up while testing, or that we didn’t do a short ‘who will do what’-briefing before we started.  Divide and conquer, we didn’t. I think that was mainly due to the fact that it was all pretty new to us. This will be easier once we get to know eachother better. Another lesson learned in software testing.

As some scientist already pointed out (and actually proved too), time is relative. The planned two hours quickly became three. I have the impression that everyone was thrilled about this fresh kind of collaboration. Interacting with passionate fellow testers, talking to them, learning from them – it all made for a great experience. Sure, it is probably not always easy to attend the sessions, you actually have to plan for it – make sure the kids are entertained elsewhere. But the reward is significant.

People might argue “weekends, isn’t that supposed to be quality time away from work?” – I admit that this thought crossed my mind too. I now realise that this *is* quality time, and far away from work as well. Quality learning time. And fun fun fun – did I mention it’s lots of fun, too?

Happy two thousand and System.NullreferenceException!

A comment on the Y2.01K bugs detected at Symantec and the DSVG bank group.

First of all, happy 2010 everyone!

And a happy new year for the German banks as well, althought it must have been a not-so-happy-one for the DSVG bank group. Yesterday, they issued a statement  that some 20 million debit cards issued by the banks belonging to the group were affected by a “millennium bug”-like problem. Apparently the problem stemmed from a chip on the cards which, due to a programming fault, wouldn’t correctly process the number 2010. The group said cash machines were adjusted hours after the problem emerged to ensure that customers can withdraw money, but there may still be problems using some debit-card terminals. Those should be fixed by Monday, it said.

Monday? Like in Monday, January 11th? Like in “almost a week from now”? In full winter sales period, with the number of payment transactions generally hitting historical highs (who’s paying in cash, nowadays?), that seems kind of disastrous to me.  

The same problem hit Symantec as well. The Symantec Endpoint Protection Manager (SEPM) server handled all dates greater than December 31, 2009 11:59pm as “out of date”.

Back in 1999, computer experts widely believed that hardware and software systems would fail as the clocks rolled over to the year 2000 because computers and other devices, which used only two digits to represent the year, would mistake the year 2000 for the year 1900. In the end, however, the so-called “millennium bug” caused few problems, because a lot of companies had anticipated the turn of the millennium with all possible resources. The computer software business was booming at the time. There were lots of new jobs. Granted, most of them were repetitive and rote programming and testing jobs, but jobs nontheless. Freshly graduated students were sent off to bootcamps to become ruthless programmers. I still dream about riding dinosaurs while yelling Cobol commands at them – but I digress. There also was a high demand for testers. A wide range of different people were transformed into software testers. Finally, companies were acknowledging the need for testing. Great! But apparently, not all testing (and programming, for that matter) was up to standard.

Now I don’t know the exact processing happening at Symantec or in the DSVG bank terminals, but this indeed looks like a sibling of that good old millennium bug. Maybe this was the result of a cheap and dirty Y2K bug fix where programmers put in a simple if <10 = 20xx otherwise the date is 19xx. In that case, this unwanted emergent behaviour doesn’t seem too hard to detect. Or does it? My 2 cent’s worth: this is what you get when performing scripted testing based on poorly thought-out boundary value analysis (aka domain testing) without exploring the software to know more about the risks. Or maybe they did explore, but they were only focused on that one known boundary called 2000. In both cases, testers failed to acknowledge other boundaries in the software that maybe simple talks with the developers would have detected anyhow. Michael Bolton talked about this in his Eurostar tutorial this year: the actual boundaries in a system may not be the ones we are told about – that’s why we need to explore. He also wrote this interesting article (pdf) on domain testing in Better Software magazine.

Eurostar 2009 – a week to remember

A write-up of the Eurostar 2009-conference in Stockholm

I absolutely *love* Stockholm in wintertime. Pepparkakor, glögg, gravad lax… and Eurostar too. People keep telling me that I would probably love it even more in summertime, but I’ll always associate those dark days with Eurostar. I presented my first Eurostar track there in 2007 – nothing but good memories – and I was selected this year as well. The Eurostar line-up is always pretty impressive, so it can be both intimidating and exciting to be a part of that. It’s just a matter of keeping the intimidation level below the excitement level, I guess. As a boyscout, good old Baden Powell always told me to “be prepared”. Now sometimes I wouldn’t recognize a life lesson if it punched me in the face, but here’s one that I did remember. So I found myself writing a paper and assembling a presentation during those hot holiday nights in Southwestern France. You just gotta love those early deadlines!

November 29

After an uneventful flight from Brussels to Arlanda, set foot on Swedish soil. Met up with fellow Belgian Mieke Gevers, a member of this year’s program committee and in charge of the track chairs as well. I helped her carry some excess bagagge that turned out to contain presents for the trackchairs – you can’t go wrong with Belgian chocolates and “jenever“. We took the Arlanda express (easy and quick) to Stockholm C and a cab to the Rica Talk hotel.

November 30 – Tutorial day

On monday I attended a full-day tutorial by Michael Bolton called “Exploratory Testing Masterclass” (slides available here). Two years ago I attended his tutorial on Rapid Software Testing, which I found very valuable. Michael Bolton is an engaging speaker and teacher who invites you to think, rather than just sit and absorb theoretical matter. There were lots of exercises, including one on factoring (identifying dimensions of interest in a product). We were asked to identify all dimensions of a wineglass that may be relevant to testing it, using the “San Francisco Depot” – heuristic (Structure, Functions, Data, Platform, Operations, Time) – not new to me but always worth repeating. A lot of mnemonic wizardry to be found here. What about that handy mnemonic for oracles – HICCUPPS/F (History, Image, Comparable product, Claims, User expectation, Product, Purpose, Statutes, Familiar problems) – never again say that you don’t know why something should be considered a bug. Care to take a ride on that test reporting heuristic called MCOASTER? Well I’ll see your CRUSSPIC STMPL, and raise it with a FCC CUTS VIDS (Mike Kelly’s application touring heuristic). Mnemomania!

Of course, there were plenty of other impressions that kept lingering for a while.

  • A quote by Jerry Weinberg: “A tester is someone who knows things can be different” – true.
  • “If it ain’t exploratory, it’s avoidatory” – made me laugh. 
  • “A good tester doesn’t just ask “Pass or Fail?”. A good tester asks “Is there a problem here?”.
  • CHECks are CHange detECtors, testing is exploring.
  • A complete debunking of some boundary value analysis truisms: it is generally accepted that the behaviour at boundaries is more likely to show erratic behaviour, but how do we know these boundaries? The actual boundaries in a system may not be the ones we are told about. That’s why we must explore.
  • Testing is “storytelling” – I liked that take on testing:

“You must tell a story about the product, about how it failed, and how it might fail – in ways that matter to your various clients. But you must also also tell a story about testing, how you configured, operated and observed it – about what you haven’t tested, yet… or won’t test, at all – and about why what you did was good enough.”

The end of the session was foreseen at 5 PM. The discussions kept going on until 5.45 PM. I think that says it all. Later that evening, an international amalgam of testers set out to explore the possibilities of finding food in Gamla Stan. Eventually we found an Indian restaurant using that good old I.NEWTON heuristic (Indian, Nearby, Edible, Welcoming, Tasty, Open, Not-too-expensive). The end of a great day. Had some nice conversations with Rikard Edgren, Tone Molyneux, Ray Arell and  John Watkins (my trackchair) as well.

December 1

The second day started with a tutorial as well, be it a half-day one: Managing Exploratory Testing by Jonathan Kohl. Of course there were a lot of similarities with the first tutorial, but this was more of a hands-on session, where we could put Michael Bolton’s concepts from the day before into practise. There was some theory about coverage models – SF Depot anyone? We ended up describing a whole bunch of characteristics of a table that we had never associated with an ordinary table before. Practical and fun. Certainly an eye-opener.

At that point I was still trying to get a hold of the person I was supposed to trackchair on wednesday. Originally I would be trackchairing my colleague Wim De Mey’s track about regression testing in a migration project, but Wim had to cancel his presentation at the very last moment because of unfortunate familial circumstances. A replacement was found in the person of Mika Katara, from Finland – but no sign of him, yet. Oh well, time for a quick lunch, a tour of the expo and the actual kick-off of the conference.  Dorothy Graham opened the 17th Eurostar conference in style. She introduced the program committee (Tone and Mieke made sure Isabel Evans was also represented by carrying an air-filled balloon with a face drawn on it – I’m not sure if Isabel would be too happy with the analogy 🙂 ) and set the scene for the first keynote speaker.

Lee Copeland started this very first talk of the conference about nine of the most important innovations in software testing: the context-driven school, test-first development, really good books, open source tools, session-based test management, testing workshops, freedom of the press, virtualization and “testing in the cloud”. Strange that he sees the context-driven school as an innovation – as far as I know it was founded in 1999; the first book that explicitly named it was already published in 2001. I agree with the freedom of the press thing. Testing blogs are appearing everywhere (guilty, your honour), twitter is on the rise. Lee is apparently not a fan of twitter. Neither was I – I always thought of it as encouraging the spreading of triviality, but I’m actually starting to come back from that. I noticed that a lot of people within the testing community are using it to share their ideas, give advice or call for help. And it gives a great deal of extra coverage to an event like this (see twitter.com/esconfs), so maybe I’ll give it a try. Later. 

The rest of the afternoon consisted of a series of  short 20-minute tracks, which is mostly just enough to launch some provoking ideas, but not really ideal for a lot of content. Johan Jonasson talked about how he managed to save a project with the introduction of a structured exploratory testing approach. This track would have benefited from a 45 minute timeslot – there was no time to go into detail, which I found a pity. Next up was Julian Harty, who explained the concept of “trinity testing”: short session of around 90 minutes per feature, where the feature owner, the developer and the test engineer work interactively through the software to share knowledge and ideas. Pretty interesting, since I also found out later that “the trinity test” was also the name of the very first nuclear test ever conducted, marking the very start of the nuclear age. Julian is probably aware of this – I didn’t hear him mentioning it, though.
Geoff Thompson then talked about reporting – “If only we could make them listen!”. Well actually, it’s more the communicator’s job to make sure he gets heard. It was a great talk – he was able to slip in the Challenger disaster and the Heathrow terminal 5 debacle as examples of how important messages were apparently not deemed important enough, with horrendous results. Knowing your recipients is key, and knowing what information they want as well. Noteworthy: a lot of people are color-blind. If you absolutely want to make sure that everyone understands your reports, shouldn’t you avoid the reds and greens?
Besides being a sapient testing evangelist, Michael Bolton is also a human quote machine. He did this cross between stand-up routine and  political televangelism called “Burning Issues of the Day” (available here). A lot of wisecracks and eye-openers, the funniest moment at Eurostar for me. He was even able to win a bet by slipping in a quote about agilists and sex:

“The agilistas did not discover pairing or test-first programming. They’re like teenagers who’ve just discovered sex. It IS great, but calm down”.

The last speaker of the day was the same as the first one. Jonathan Kohl talked about how our urge to be “Agile” can distract us from our mission to deliver software that our customers value, while supporting our team. Agile can distract from skill development too. The term “Agile” has become big business, and lost a great deal of it’s significance. So let’s stop worrying about whether what we do is “agile” or not, and go back to calling it “software development”. As far as I’m concerned, he hit the nail on head. I wouldn’t have minded him talking about this a little longer.

The day ended with drinks in the expo and my attempt at playing a memory game at one of the stands. I kept failing epically. While I was trying to get asleep I found the ideal excuse: my head was already full of things to remember – no room for these trivial button sequences.

December 2

Right before the first keynote of the day I finally met Mika, whom I was supposed to be trackchairing in the afternoon. He was invited as a backup speaker on friday to speak on wednesday, was able to make it, but had to leave immediately after his talk. A true case of hit-and-run guerilla presenting at Eurostar! Naomi Karten then delivered an interesting keynote about “changing how you manage and communicate change”. Her talk was built around the Satir change model. There’s an initial status quo, then a foreign change-inducing element causing a ‘POW’, then chaos, after that an adjustment and in the end a new status quo. When people are confronted with change, they are experiencing a loss of control, and they often react to that in an emotional way. Important: listen, be empathic, regularly communicate the status of the change, even when there is nothing to report. She also used a quote that I well certainly use myself when feeling cornered:  

“Hofstadter’s Law: It always take longer than you expect, even when you take into account Hofstadter’s Law”

By then it was time to pay a visit to the Test Lab that was set up by James Lyndsay and Bart Knaack. It was a Eurostar first, and I am actually wondering now why it took so long to have some actual “testing” going on at a testing conference. The software they were running was Open EMR, an open source patient management and appointment book system. What made it even more interesting for me is that I have been testing and working with a similar (not open source, though) system for a long time, so I more or less know what to expect (or what actual users of the software would expect). I paired up with Rikard for a while and found a whole bunch of issues by merely touring the application – we noted them for later reference. It is always nice to pair with fellow testers to see what they focus on, and what their reasoning is. The state of the software under test was something else. It showed some pretty alarming behaviour, and it was far from intuitive or user-friendly.

By then it was time for Eurostar veteran Erik Boelen, speaking at Eurostar for the fifth time already. I’ve known Erik for some time now, and his talks are always entertaining and relaxing in a way. “The power of risk” was his view on how to use a risk-based test strategy that “makes people talk”, like Läkerol. His main message was (apart from the implicit one that testing can be fun *and* will rule the world) that they defined all the risks and used them as entry paths for exploratory testing. For the highest and medium risks they documented their test cases, and for low risks they just reported the results.

After lunch I introduced Mika Katanen (from the university of Tampere in Finland) and his talk about Automatic GUI test generation for smartphone applications. I am totally new to model-based testing and I was impressed with the brief demo he showed. His track went well, and there were a lot of people approaching him for a chat at the end. I do hope that he was able to catch his plane on time. Parallel with this track, Shrini Kulkarni held his talk about software metrics which I was unable to attend. People said it was good – I hope I will be able to see him speak some place else in the future.

Remembering the memory game disaster from the day before, I decided to unfocus for a while – my mind was getting stuck again. I teamed up with some CTG colleagues plus a wildcard named Tom and enrolled ourselves for the quiz that was supposed to take place in the evening. We aptly named ourselves “The Handsome Oracles”, but it wasn’t meant to be. The quiz was canceled later on, so we weren’t able to put the money where our mouth was. We also worked out some testing limericks for the limerick competition – we didn’t win. I thought they were good, but that’s probably just another example of parents not recognizing the ugliness of their own babies. There’s a good joke and an interesting analogy about that here. Gitte Ottosen ended the day with a talk about combining agile and maturity models which was chosen best presentation last year in The Hague. I had the impression she was a little nervous – which is completely understandable. I was telling to myself that delivering a keynote for a full auditorium like that sure looked like a daunting task – until I suddenly realised that I would be standing in that same room tomorrow. My unfocused mind started wandering off.

While the temperatures were taking a dive, the Handsome Oracles went into town for dinner. I returned a bit earlier than the rest to rehearse my talk and to get a good night’s sleep while the (by then just plain) Oracles went barhopping. Haha! Life’s good, but not fair at all. 

December 3

The last day of the conference, and people started looking weary. Ray Arell gave us a good wake-up call with his keynote on moving to an agile environment, based on his experiences at Intel. Ray’s a great speaker (and a fun guy too – I might add). He described his hits and misses; the ‘misses’ are often the most interesting parts of experience reports. Lot’s of good advice and some nice puns (Wagile, FRagile, Scrumfalls).

I stayed in the agile track in the big auditorium where John Watkins presented some material from his book on agile testing, aptly named “Agile Testing”. John had gathered case study material from twenty agile projects and proposed agile methods for small, medium, large, off-site, and even off-shore projects. Intriguing, but upon hearing the idea of “agile best practises”, my context-driven genes started to play up.

John was also my great trackchair and introduced me as “Filmstar, Rockstar, Tester!” At least, that was his own juicy summary after I mentioned to him that I had worked as a movie distributor before and had also played in a rock band. Granted, I also admitted playing a zombie once – a serious case of method acting. Anyway, his introduction loosened the audience a bit and I was able to present my track “A lucky shot at agile?” without any problems. I wanted to tell a testing story and I think it went well. I felt at ease (those wireless microphones are really great) and there were many questions afterwards. During the rest of the day people I didn’t know came up to me to congratulate me with the presentation, which was nice. I took a long lunch and had a walk around the expo. I went back to the Test Lab to report the bugs that we found earlier. I didn’t succeed in entering them all, which made me feel kind of guilty – I wished that I would have spent more time there. But I had a hard time choosing. It’s a pity that test labbing also meant skipping tracks as well.

The last regular talk of the conference was held by Rikard Edgren, who is also a Eurostar regular. I had seen his presentation on testing creativity (“Where testing creativity grows”) in 2007 and I liked it a lot, since it is also a subject that is dear to me. There’s far too many people that think that testing is not a creative or challenging activity. This time he talked about  “More and better test ideas“. He promoted the use of oneliners as test ideas – a brief statement of something that should be tested. These test ideas can then be used as a basis for test cases, or as a guideline for other types of testing, or even discarded when there irrelevant or when there is simply not enough time. I think Rikard’s subjects will always be a bit polarizing due to their innovative nature – you either like them or you don’t. I am a believer and it was a good way for me to finish the conference.

I missed the first part of the Test Lab result presentation since they changed the timing and I totally forgot about that. But I got the most important statistics. Over two and a half days, more than 50 bugs were found. My first reaction was: “Only 56? Man, there’s hundreds of them hiding in there”, but then I realised that people had been testing in the lab only for short periods, in between tracks, just as I did. I wonder what would have happened if hundreds of testers had a go at it, all at the same time. Bugfest!

After a short panel discussion with John Fodeh (next year’s programme chair), Geoff thompson, Tobias Fors and Nathalie Van Delft it was time for the award ceremony. Naomi Karten received the Best Tutorial Award and the European Testing Excellence Award went to Anne Mette Hass. In the meanwhile I was dozing off in my not-so-comfy chair – these 4 days of conferencing were finally getting to me. A friendly woman on the stage was mentioning someting about a longlist of papers, and a shortlist, and a final selection of three, containing two Dutch and one Belgian paper. Now wait a minute… how many Belgians sent in a paper? 1…2… before I could make the math, my name was announced as winner of the ENEA Best Paper Award. Two talks at Eurostar, two papers, two awards… what are the odds of that? I was absolutely flabbergasted. That’s actually three in a row for my company CTG, since Bert Jagers won the award last year in The Hague. The pressure is on for next year :-).

I spent the rest of the evening in the hotel bar, where all the testers with an early flight on friday morning were flocking. We ended the day singing an eclectic mix of Irish traditionals, Dylan, early Springsteen and – of course! – Abba, accompanied by a non-certified tester, who plays a mean mandolin. I love Stockholm in wintertime. It was a good Eurostar. Yes sirree.

The winner takes it all (a 3-day workshop with Cem Kaner)

A review of the Cem Kaner workshop during the Cem Kaner Week, held in the last week of October 2009 in Nieuwegein, The Netherlands.

Last August I received a mail from a company called Immune-IT that promoted their upcoming “Cem Kaner week“. The concept: during the last week of october Cem Kaner would visit The Netherlands to lead a three day workshop and give a freely accessible presentation as well. Considering the cost of the workshop I settled for the free presentation, which I figured was also quite unique. Nieuwegein in The Netherlands is only a 160 km drive from my home (ok, you’d have to get past Antwerp, Breda and the Utrecht traffic jams, but still: worth the hassle). Later I heard that there was also a possibility to get a free workshop place by entering a contest on their website. Long story short: I participated, won the contest, checked the mail again (still said I won it) and jumped from joy. On October 26, I had a date with testing history.

I have always looked up to Cem Kaner, admired him for being one of the pioneers in the software testing field. He is a professor of computer sciences at Florida Institute of Technology and the (co-)author of truly great books like “Testing Computer Software” (alledgedly the best selling software testing book of all time) and “Lessons learned in Software Testing” (which inspired and influenced me big time). He founded the “context-driven school of software testing” (I’m on their side) and has all these excellent free courses available online for Black Box Software Testing. All this was rushing through my head while cruising through the lowlands. Later on, traffic came to a standstill, and this gave me even more time to get my imagination running… What if I don’t like him? What if he turns out to be a belligerent ghoul (gratuitous “The Smiths“-reference)? Well, none of all that. Cem is a nice person, almost humble in his ways. I like him. A friendly, human encyclopedia of black box software testing.

The size of the workshop was pretty small – only 12 people enrolled, which made it super-interactive. Immune IT had asked Cem to talk specifically about testing in difficult economic times. He did talk about that, briefly, but luckily he covered a whole range of other things as well. Some highlights:

  • Exploratory checklists and a truly hilarious piece of scientific proof that following scripts is a best practise (yes, you heard it right, in some cases there *are* best practises) for brain-damaged rats. In 1977, Cem incidentally stumbled upon proof that normal lab rats use ‘checklists’ to drive their behaviour, while brain-damaged ones resort to following the same script over and over, no matter the context. More on this can be found here.
  • Scripting and learning. Cem debunked some common myths surrounding scripted tests.
    #1. “Test scripts are ideal as ‘training wheels’ for new testers. After several months of following a wide range of scripts, the new tester will have learned by example a lot about the application domain, the program and how to test it”. So they say. But have you ever noticed what happens when you drive to a new place with a GPS that gives you a script like “Turn left at the next lights etc…?”. If you try to go there again, do you actually remember the route? You learn faster when exploring and discovering, rather than by ‘being told’.
    #2. “Test scripts specify all initial entry conditions”. But you cannot capture everything, there are so many variables that it becomes impossible (system state, program state, system config, system resources, other processes) to describe this upfront.
    #3.  “Test scripts specify the expected results”. But – same as above – scripts cannot specify all possible outcomes. There are simply too many variables involved. Our tests cannot address all possibilities.
    #4.  “Test scripts involve a comparison that machines or humans should make”. But what about confirmation bias? Or inattentional blindness? We ignore things based on their meaning, before we ever become aware of them.
  • An extensive part about exploratory testing from the godfather himself, it doesn’t get any better than that. He explained the renewed definition of exploratory testing (which I endorse but find a bit too long – I liked the old one better).  *taking a big breath* here goes:

Exploratory Software Testing:
– is a style of software testing
– that emphasizes the personal freedom and responsibility
– of the individual tester
– to continually optimize the value of her work
– by treating
     * test-related learning,
     * test design,
     * test execution, and
     * test result interpretation
– as mutually supportive activities
– that run in parallel throughout the project

  • An intriguing part on testing an investment modeling tool called VectorVest.
     
  • “Exploratory Test Automation”. Slides available here. I specifically liked his experiences at a phone manufacturing company called Telenova.  They were able to track down a stack overflow bug in one of their telephone sets that had been undetected even though testing achieved 100% statement and branch coverage in the relevant parts of the code. In the field, this would require a long sequence of calls to a continuously-active phone. They succeeded in isolating it through “Exploratory Test Automation”: they created a simulator that generated long chains of random events, emulating input to the system’s 100 phones. That eventually exposed the bug (and several others as well).
  • Cem is obviously not a great fan of automated tests, but he makes an exception for “Exploratory Test Automation” , “High-volume Test Automation” and “Extended Random Regression Testing”. Traditional test techniques tie us to a small number of tests. Extended random regression and long simulations exposes bugs the traditional techniques probably won’t find.

These three days were a real treat. I was able to talk and discuss with Cem Kaner personally – and learn from him, which was priceless.  I also met some other great people – passionate about testing and always prepared for an healthy argument. At the end Cem signed the copies of “Lessons learned in software testing” that we all got for free, which means I now got two of them. One to use and one to cherish.