An oracle is a principle or a mechanism that helps us in recognizing if the software under test is working according to some criteria. A heuristic is a fallible way of solving a problem; it can also help us discover and explore. James Bach and Michael Bolton have provided us with a lot of helpful heuristics already. One of them is a consistency heuristic mnemonically called HICCUPPS. Michael wrote an article about his use of this particular heuristic here (pdf). The idea is that a product should be consistent with:- History (consistent with it’s past behavior)
- Image (consistent with the image the organization wants to project)
- Comparable Products (consistent with a similar product)
- Claims (product behaves the way some document or person says it should – e.g. in advertisements)
- User’s expectations (consistent with what user wants or expects from the product)
- Product (behavior should be consistent throughout the whole product)
- Purpose (consistent with its apparent purpose)
- Statutes (behaving in compliance with legal or regulatory requirements)
Although this is all about hardware, I was reminded of the ‘Claims’-part of this heuristic when I recently saw a video of a BBC reporter at the Consumer Electronics Show (CES) who broke the alledgedly unbreakable SONIM phone on his first attempt. Examples don’t come any clearer than this. Don’t go about telling that your product is unbreakable unless you’re 100% sure it is. Then again, how can you ever be 100% sure? When you put up a fish tank for everyone to use, expect the unexpected. That’s an open invitation for creative people to think ‘out of the tank’. I guess they’ll have to revise their marketing tagline. How about ‘Unbreakable by anything but edges of fish tanks’?.
* edited typo: ‘Statutes’ instead of ‘Statuses’
Thank you for the kind words, and for the nice summary of the list.
Just a minor typo that might introuduce a little confusion: where you’ve type “Statuses”, that’s “Statutes”—that is, laws—but there’s also some incompleteness that we’ve introduced: the S is also for “Standards”—relevant, explicit published standards (like RFCs or the Microsoft Windows User Interface Guidelines) or implicit standards, like the arrangement of the clutch, brake, and accelerator pedals on a car. (The latter leaks into the Comparable Products heuristic; that’s okay. These categories are designed to be overlapping and mutually reinforcing, to reduce the possibility that you’ll miss an important problem.)
There’s also one more: we suspect a problem when we see something that’s consistent with patterns of familiar problems. That is, people often look for and recognize the kinds of problems that they’ve seen before. In that sense, in contrast with the other heuristics, it’s an inconsistency heuristic; we want to see inconsistency with those familiar patterns.
Love the video! Well done!
All the best,
—Michael B.
Michael, thanks for the clarification. I edited the Statutes typo. I didn’t include the one on ‘Familiar problems’ on purpose, because it is not – as you pointed out – a consistency heuristic as all the others are. Although it is a very useful one for judging bugs, I feel that it should probably be in an other kind of category – maybe ‘inconsistency heuristics’?
Thanks for the feedback, highly appreciated!
— Zeger
Although it is a very useful one for judging bugs, I feel that it should probably be in an other kind of category – maybe ‘inconsistency heuristics’?
We do that when we teach. We mention it at the same time as the others, and note the inconsistency!
It’s important enough that we don’t want to leave it out.
—Michael B.