We Need To Talk About Testing

Author: Dan North Link: https://dannorth.net/2021/07/26/we-need-to-talk-about-testing/

The purpose of testing

Whenever we make changes to software — adding a new feature, fixing bugs, refactoring to improve design — we incur some amount of risk. There are all number of bad things that could happen as a result of some seemingly innocent change, things like:

• Functional correctness: It doesn’t produce the results we expect.
• Reliability: It mostly shows correct answers but sometimes it doesn’t.
• Usability: Sure it works but it is inconsistent and frustrating to use.
• Accessibility: Like usability, but exclusionary and probably illegal.
• Predictability: It has random spikes in resources such as memory, I/O, or CPU usage, or occasionally hangs for a noticeable amount of time.
• Security: It works as designed but it exposes security vulnerabilities.
• Compliance: It works but it doesn’t handle personal information correctly, say.
• Observability: It mostly works, but when it doesn’t it is hard to identify why.

For each type of thing, there is a person or role who cares about that thing who we call a stakeholder.

Thus, we arrive at a reason for why we test:

💡 The purpose of testing is to increase confidence for stakeholders through evidence.

There are three helpful qualities that can be associated with a testing mindset:

1. Empathy - the ability to get inside a stakeholder’s head, see things from their perspective, and understand their concerns.
2. Scepticism - the ability to doubt the work you are doing, even as you are doing it. This involves putting aside any ego or confirmation bias. This scepticism aligns with the scientific method, which seeks to falsify an hypothesis, not “prove” it.
3. Ingenuity - the ability and determination to do whatever it takes to provide that assurance to the stakeholder — or to discover that their scepticism was justified in the first place. Testing is non-linear and often emergent and exploratory. Test automation is but one tool in a good tester’s toolbox, and they will use all of those tools.

From that definition, it follows that:

💡 You are testing if and only if you are increasing confidence for stakeholders through evidence.

Test thinking includes making design choices that rule out entire classes of defects, or replacing custom-made components with a set of well-documented, battle-tested components whose behavior is well understood.

Test thinking means applying these ideas, and having these thoughts and conversations early, with the right people. It means taking the time to think about all the stakeholders of a new piece of work, and what kinds of things they may be worried about.

On the flip side, if you are doing work that doesn’t demonstrably increase confidence in at least one of those stakeholders with tangible evidence, you are not testing, whatever else you may be doing. While this work may hold some other kind of value, from a testing perspective, at best you are covering an already well-worn path, at worst you are engaging in test theatre — activities that give the illusion of testing, while generating no useful information.

In the context of software development, the most common form of surfacing this evidence, is through either manually interacting with the system, or by designing, writing and running automated tests.

Test-Driven Development (TDD)

The goal of TDD is to produce small executable code examples that help drive the developer towards a good design. These code examples, while providing useful feedback to the developer during development, are not necessarily good tests.

The confusion around TDD and the purpose of testing is what led to the creation of Behaviour-Driven Development.

… the purpose and essence of testing, and the role of its practitioners, has been diluted by the invasive species of “automated tests”. Even the infamous automation testing pyramid is usually almost entirely about functionality rather than any of the cross-cutting safety concerns like security or compliance, or about operability or observability characteristics.

❓ Primary point: tests created for the purpose of automation overwhelmingly target functionality, and not any of the other, equally relevant concerns.

To summarise: TDD, BDD, ATDD, and related methods categorically do not replace testing, whatever their names may suggest. They are primarily design and development techniques.

❓ A subsequent point is made around the way "unit tests" are written:

These code examples were never designed as tests, but as guides. For instance, as a programmer I may write a “test” with a single value to check a calculation, and maybe one more to triangulate for a more complex case. Then I declare victory. With a testing mindset I will think about edge cases, small or large values, huge negative values, values close to plus/minus zero or other key transitions, missing or invalid values, calling things out of sequence, and various other devious inspections.

❓ Most unit tests focus exclusively on the happy path. Even the ones that don't are often limited in the scope in which they try to "break" the system.

❓ A point is made about the confidence of the developer being a poor indicator of quality, "If I didn't think I was right, I would have written different code". The confidence stems from a lot of cognitive biases working against us, confirmation bias, fundamental attribution error, Dunning-Kruger.

We are starting from a position of assumed success rather than assumed failure.

On manual testers retraining as automation engineers:

While programming skills can be useful for a tester, I don’t recognise the roles of automated tester or manual tester. Automation is one of many useful tools in a good tester’s tool belt.

On outsourcing testing:

Recognising risk, understanding impact, getting inside stakeholders’ heads — which often involves building relationships with key individuals or groups — and squirrelling around surfacing evidence, these are not activities or capabilities that it is prudent to outsource. If you have an “outsourced testing function”, even if it is to another team in the same organisation, the chances are they are engaged in testing theatre, and not doing anything to reduce risk or increase confidence.

Is test coverage a useful metric?

One thing test coverage can reveal is code that has no automated tests at all.

There is a commonly held idea that we should strive towards having high code coverage, some people even say 100% coverage should be the target. Based on all the above, we know that tests in this category are often the source of a false sense of security. A lack of coverage isn’t necessarily always a concern, for example a UI that developers and testers see multiple times a day is unlikely to have a glaring layout defect, even if there are no automated tests to prove it.

What does it mean to “shift testing left”?

❓ There's an idea here that to shift testing left doesn't just mean starting testing activities earlier, but it also means *doing different things*. Things like applying a testing mindset to architecture and design, as well as considering all of those stakeholders from earlier. In turn, this means that we're shifting left on security, accessibility.

When and where should we be testing?

As early as possible, and as often as is practical; wherever in the development cycle we can start to surface evidence to give assurance.

A common fallacy is that a feature needs to be “finished” before we can consider testing it, however this is easily debunked. We can assess what data is being accessed, where from, in what way, for how long, and for what purposes. We can assess lightweight UI sketches for consistency or accessibility.

From this we can have an opinion about security, privacy, compliance. How much data comes back each time? What are some worst-case scenarios for data volumes or values, or screen update times? This can give us insights into reliability, robustness, and resilience.

A programmer can’t be thinking like all the stakeholders all the time or they would never get any work done! But this test thinking should be happening continually, with testers embedded in development teams and elsewhere along the value chain.

How much is enough testing?

❓ Depending on the type of change, and the context in which it happens, the "amount" of testing required to achieve a suitable level of confidence will vary. If a change is likely to be very high impact, or the consequences of getting it wrong, catastrophic; then a lot of testing will be required in order to achieve that level of confidence.

❓ When we think about "enough" testing, or the practical implications of testing some projected change, it can sometimes lead to constructive discussions about alternatives with different implications. There will generally be trade-offs involved with these alternatives, it is rare that there is one objectively best solution.