Software Quality Assurance (also known as QA) is often misunderstood.
It’s not just about checking requirements.
It’s also not just about finding bugs.
QA is all about understanding whether a system truly works — for the people who will use it.
Quality is “fitness for use”
If you’re new to the concepts of Software Quality Assurance or new to my site, you can learn about Joseph M. Juran’s (1904-2008) definition of “quality” on my Quality as Fitness for Use page. That will be important for understanding my basic Philosophy of Quality.
Who is a tester?
“It’s important for [QA testers] to have the skills to be able to explore and experiment, to anticipate problems that are deeply buried or rare or subtle or emergent. Those people need to be able to model the whole system, and to investigate it with the specific goal of looking for trouble.” (Michael Bolton, software testing guru, quote extracted from his blog in 2019)
Bolton has spent a lot of time in the trenches and thinks very deeply about software testing. I agree that we expect testers to think deeply about their system (both in its entirety and in the most minute details), to explore it thoroughly, and to realize that bugs exist — and that their job is to find and help eliminate them.
Becoming a great tester
I build on Michael’s thoughts by reminding testers that it’s their job to care deeply about two things:
1. Whether the system works correctly
We make sure that we test each requirement, each field, and each interaction and compare what we find to what the designers or developers expected us to find. At its core: Is the system working as designed?
2. The voice of the end user
We don’t test software for ourselves. We do it for someone else: The person who might sit in front of this system every day or the person using the app on their phone throughout their busy day. Would they think that it’s working well? Will they want to come back to it because it’s a good app or even fun to use?
A little secret between us
Testers aren’t always professionals — or at least, we aren’t always doing it to earn a paycheck. I have had a number of experiences where I have encountered a bug in an app that I use regularly. When I do, I’ll file a support request about what I’ve found. I always write the request like I would a bug ticket for work. (See my guide at How to Write a Good Bug Ticket)
The support team often reaches out — especially if they need clarification or additional information to replicate the issue. They almost always check in with me when the bug has been fixed and a new version released. Sometimes they even invite me to do some alpha and/or beta testing for them.
Occasionally it’s the developer themselves who reaches out. Some of my favorite apps are maintained by a solo developer as a passion project or side gig. They always very much appreciate that someone cares enough to help them eliminate bugs that they themselves hadn’t found yet.
The little secret? Anyone who is willing to reach out to support and provide clear information on how to replicate an issue reliably — because they care about the systems that they use — is a tester, whether that’s their job title or not.
What does a tester do?
Now that we’ve established the fact that anyone can be a tester — and some of us choose to do it as a profession — let’s discuss the work that testers do.
Actively work with the system to find bugs
Testers aren’t just “lucky” or “magic” (even though I’ve frequently been called both by others on development teams). Even I can’t randomly fiddle around on our computer, and the bugs come find me, ready to be reported. It’s quite the opposite: We sit with the “System Under Test” (SUT) and try to determine whether it’s working correctly or not.
Sometimes we’ll follow a script
A very basic example is for a login screen:
- Go to www.example.com
- In the “username” field enter
[email protected] - In the “password” field enter
P@ssword1234 - Click the
Loginbutton - Expect that the user will be logged in and land on the “Welcome” page
This kind of test verifies that the system behaves correctly under expected conditions. It also leads us to consider testing what happens in less-than-perfect conditions.
The script above can be expanded to include “negative tests,” like using an invalid username or password and making sure that the system provides a clear error message. Scripts can also go on for many steps to cover complex system processes or user journeys. Such scripts are great for ensuring that the system is behaving correctly according to the requirements and specifications (also known as fitness).
Sometimes we’ll write scripts for “robots”
Many testers write code that can be used by a testing system (often known as a “framework”) to handle mundane testing tasks like logging in or long, tedious user journeys. The framework can follow the (coded) script to repeat the test often without becoming bored — or missing an important step. The “robots” also execute the script way faster than any human could perform the same test steps.
Once the “robot” finishes the test script, it will report its findings (pass/fail and information about what failed and how). Testers will review the report from the framework to determine where issues might have occurred — or to affirm that this particular part of the system is working as designed.
Often we’ll use the system like a user would — without any script
Scripts are great for testing against rules and requirements. But that’s only part of the story.
I once knew a tester who followed all the steps of a script and marked the test as Passed — even though the screen was literally flashing a red ERROR message at the top of the screen throughout most of the process. Yes, they could execute the steps as described, but there was something seriously wrong that they either ignored or failed to see because it wasn’t mentioned in a step of the script.
We need to sit with the System Under Test like the users will:
- hands on the keyboard, mouse, or device
- watching the whole screen, not just the single field or fields mentioned in a script step
- moving around and between multiple screens — perhaps even “out of order” from the expected ways of moving through the system
- thinking about whether the system “makes sense” and “looks good” and “works well” (expectations that may never be written down concretely in a requirement)
When we test this way, we’re being user advocates — speaking for the users when they aren’t even present in the conversation. This is where testing moves beyond verification and becomes exploration: We’re testing for fitness for use.
My page on Exploratory Testing provides even more information on how to test like a user.
Testers tell the truth — even when that’s difficult
Let’s turn to Michael Bolton again for his thoughts:
“Testing is the part of [software development] that is focused on learning about the actual status of the product; dispelling illusions about it; seeking and finding trouble so that customers don’t. That’s a socially difficult role, because many people have a natural aversion to thinking about problems and errors and bugs.” (Bolton, quote extracted 2019 from his blog)
It’s not always easy being a tester.
Sure, we get to try new features and new systems before anyone else does. But new systems often come with all sorts of unexpected behaviors and weird bugs. It’s our job to seek them out, find them, report them, and work with the developers to make sure that they are fixed.
And then there are the moments when the project is on a tight deadline, and the Big Boss wants to get the app into users’ hands right away — and we find a pretty bad bug? I’ve been that tester. I’ve had to say, “No, the product isn’t ready, and Yes, we must fix this bug before we release the product.”
That takes guts. No, I wasn’t the most popular person in the room at that moment. But we got through it, fixed the bug, released a bit later, and never had users complain because we had let that one “get out the door.”
When we’re good at our job as testers, we often tell the truth, exposing the fact that the system isn’t as perfect as the team hopes that it is.
That’s because our ultimate goal is software that works well for the end users.
A parting thought
Edsger Dijkstra (1930-2002), a Dutch computer scientist put it best:
“Testing can prove the presence of bugs, not their absence.”
No matter how much we test, no matter how good we are at our jobs, end users will still find bugs. Or one will hide in a part of the system that was developed ages ago but hasn’t been touched, then suddenly rear its head at the weirdest time. And that’s the way it goes.
But once we’ve discovered it, good testers will ensure that it’s reported on a bug ticket and will soon be squashed!
Software Quality Assurance isn’t about eliminating every bug. It’s about making sure the right things work, for the right people, in the right ways.
Copyright ©2019 Jay R. Newlin and Hopeworks.