Month: February 2010

Through the looking-glass, and what testers found there – EWT05

Today it was weekend testing time again, and things got pretty crowded with a pleiad of international testers: facilitators Anna Baik and Markus Gärtner, !ndra from Hyderabad, Shiva Shankari, Krishnaveni K, Nagashree Manjunath, Jeroen Rosink, Ajay Balamurugadas, Jassi and myself.

The target today was Virtual Magnifying Glass, an open source screen magnifier for Windows, Linux, FreeBSD and Mac OS X.

The mission was basically: “Test this!”. Although a mission like that is way too vague to start from, it *is* a mission that we all encounter once in a while. It is our job to ask questions to make sure we understand what is needed.  So I wondered… do they just need bugs? Or an advice? Or information about how the application works? Ajay also went in questioning mode: “Test : create test ideas, hunt for bugs, compare with a different product, learn the product, so many things? What exactly is required?”. So with a little help of Ajay, the mission was rephrased to “Find quality related valuable information about the product”.

So far so good. I wanted to try pair testing over skype and Ajay was willing to team up with me. We set up a call and were soon discussing in person which approach to take. Although we encountered connection problems later on, it was a good experience that bears repeating.

Lessons learned:

  • It’s easy to feel frustrated because time is too short. After a round of introductions, some explanation and a discussion about the mission, there’s less than 30 minutes left to explore the software. Even for seasoned Rapid Testers this would be an uncomfortably short timebox. Avoid frustration by setting realistic goals for yourself.
  • When pairing, you’re not starting testing right away. There’s some discussion first, some setting up needs to be done too. Pairing is probably more beneficial when there is more time to test, so there’s plenty of time to talk things through. Now I felt a bit rushed because the clock was ticking.
  • Read-me files are a good place to start when exploring an unknown piece of software. In this case, they provided me with a good model of the software, a good basis to start exploring from.
  • Logging bugs takes time. Time not being spent testing. An issue also addressed in the “Why is testing taking so long (part1)(part2)” blogposts by Michael Bolton. 

Some nice, thought-worthy quotes were dropped during the debriefing too:

  • “Clearing traps is a skill. Recognizing traps is a bigger skill”
    Ajay Balamuragadas (c) 2010
  • “Minds are shaped when guided under pressure in a certain direction trying to maintain vision and control” 
    Jeroen Rosink (c) 2010
  • “Watch out where the huskies go, and don’t you eat that yellow snow” 
    Frank Zappa (c) 1974

Well… I guess all that snow is finally getting to me. Frank never made it to this European Weekend Testing session. He wasn’t much of a tester either. But he sure knew how to play that guitar. And I think he would have been a great explorer, having fun all the way.

Advertisement

Behold, the “better Tester”

Reaction to a discussion in the “Software Testing & Quality Assurance” LinkedIn-group

There are some strange and intriguing questions being posed in the Software Testing & Quality Assurance LinkedIn group lately. “Who are Testing Criminals?“, “Who should take the blame for defects in production software ? ” or “Test Plan ! Do we really follow it ! ” (an awful lot of exclamation marks for a question, if you ask me) to name but a few. I read them through, bit my tongue, controlled my breathing and managed to ignore them in the end. 

But two days ago, another gem was posted:

Who is better Tester?

@Who finds highest defects..?
@Who can say in 100% confidence that this product\application is BUG\DEFECTS free..?
@Who creates highest testing scenarios,Testcases,etc..?
@Who has as many as software testing certifications like ISTQB or something like that…?

Mmm… I thought about Pradeep Soundararajan’s blog post about the infamous Black Viper Testing Technique as a provoking response to another discussion in that same group, and decided to react.

Who is the better tester? Well… None of them, actually.

  • “Highest number of defects”
    What does that tell you, out of context? Maybe the person that found the highest number of defects has tested more hours than the rest. Maybe he tested a buggy part of the product, while others were concentrating on a more “mature” part. Maybe he only found “low priority” bugs, while others – with less bugs on their list – found more severe showstoppers. Which tester would you prefer?
  • “100% Bug free”
    I wouldn’t hire a tester that dares to make claims such as “this software is 100% bug free”. It isn’t. It can’t be. If you continue testing, more bugs will be found. This is the catch-22 of testing. We stop testing when we’re finished, but we’re never finished. So we stop testing because of other reasons.
  • “Highest number of test cases/scenarios”
    Again, without context, this is a worthless statement. Maybe he works faster, but more sloppy. Maybe the other testers were investigating bugs they found while scripting. Time spent hunting/pinpointing bugs is valuable, and if testers are asked to engage in that, they’re also not “writing tests”. Maybe other people are designing tests in a slower pace because they tend to talk to stakeholders about the requirements first, to be sure that they understand what is really going on. The person with the highest test case count may be designing tests badly, or testing things the wrong way. Or maybe he’s just testing the wrong things. The productivity of a tester can/may never be measured in number of test cases, but this seems to happen all too often.
  • “Number of certifications”.
    What do certifications tell you? That a tester was able to sit through a short course and was able to score +28 out of 40 on a multiple choice exam? Does certification actually tell you anything about the real skills the tester possesses? Is he able to think critically, to question the product in order to evaluate it?

I would say the better tester is the one that adds the most value with his testing, one that is able to explore the software and recognizes problems. A good tester doesn’t just ask ‘Pass or Fail?’. A good tester asks ‘Is there a problem here?’ (props to Michael Bolton for that one).

There. I said it. I actually felt relieved after writing that down. Who is better Self-therapist?

C is for Conferring

On conferences and how to keep it cheap

The blogosphere is pregnant with conference posts these days. And good ones too I might add.
  • In his blogpost conferences on the cheap, Matt Heusser gives some helpful tips on keeping the cost of attending testing conferences down in these difficult times.
  • Matt also created a conference wiki where everyone can add any kind of testing event or conference that is taking place. A great source of information, that is still expanding as we speak. If you know of any noteworthy events that are not yet listed, feel free to add them. 
  • Very recently, the Software Testing Club spawned an interesting testing conference discussion whether or not people are paying for conferences out of their own pocket. 

I think being at conferences is great fun. You meet peers. You talk and share ideas. And get ideas, as well. Occasionally, you attend a track that makes you go “what was that all about?!”, but in general I get tremendously inspired. Not only from the track sessions – most of the time the hallway discussions provide some decent food for thought as well. Of course these benefits aren’t always very measurable, but conferences are doing their very best to provide attendees with ways to convince their managers and prove the added value. Eurostar even provides a justification kit and a Conference Evaluation & ROI Worksheet for reporting on the ROI back home.

But there’s always other ways: 

  • You can get in for free as a speaker, but in that case there’s some effort involved. Writing abstracts, papers, making a presentation. Can be quite a hassle – tit for tat. And you have to get selected of course.
  • Some conferences give away conference tickets in lotteries. Slim chance, but you never know.
  • Step out of the dark side and show your face to the community
    – be a video star. Are you ready for your close-up, <insert name here>?