Integrated Tests Are A Scam

Author: J.B. Rainsberger Watched/Read: November 18, 2022 Link: https://www.youtube.com/watch?v=fhFa4tkFUFw

TLDR;

Don’t use system tests or integrated tests to verify correctness of code. Separate domain behaviour from technology integration, and you will need fewer “integrated tests”.

System test/integrated test/end-to-end test - have I wired things up correctly, am I using the correct implementations?

Microtest/unit test/developer test - is this particular component behaving correctly, considering only itself and its direct collaborators?

…programmers writing tests to show the basic correctness of their code. By basic correctness I refer to the myth of perfect technology: if I ran the system on perfect technology, would it (eventually) compute the right answer every time? I would call such a system entirely basically correct.

While integration tests offer value in other contexts, too many programmers use them to show basic correctness, and when they do that they waste a tremendous amount of time and effort. - J. B. Rainsberger



What is the scam?



Have you found yourself in this situation?

1. 100% of unit/developer tests pass, but you still find a bug ⇒ common response to this is to say there are “holes” in my tests, they are not thorough enough ⇒ I need to write integrated tests because the problems are generally in the interaction between modules.
2. Integrated tests don’t constrain our design. Tests with larger scope don’t give as good feedback about the quality of our design.
3. Integrated tests are lenient towards bad/tangled designs, so we tend to write more bad/tangled code.
4. Bad/tangled code is harder to write unit/developer tests for, so we write less of them ⇒ goto 1


To clarify: integrated tests have their place, and that place will become clear later in the post.

So if integrated tests aren’t the solution to our problem…



Collaboration tests check the layer above, contract tests check the layer below.


Collaboration tests will…

• Run my code, isolated from modules it depends on
  • Dependencies are behind interfaces
  • Interfaces are implemented by test doubles
• Focus on the behaviour of the module in question, but also its interactions with its direct neighbours

If these tests are annoying to write: if they are long, excessively complex, have too much test setup, too many mocks ⇒ this is gives us feedback about our module and its interaction with those direct neighbours - maybe the design of the code is bad?

• If the test is complicated, this means the design is complicated, and maybe the interactions are complicated.


Contract tests will…

• help us answer the question “do I implement this interface correctly?”
  • What does it mean to implement that interface?
  • When I say I implement an interface, can you trust that?

There are multiple parts to a contract:

• Syntax - the shape of the interface
  • Methods
  • Input types
  • Output types
• Rules of behaviour
  • If I have three different possible return values, under which circumstances can I expect to see each?
    • What do those different values mean?
  • Do I return null or throw an exception?
    • What kind of exception?

Integrated tests by comparison, are an indirect, and often incomplete way of checking an object or module implements its interface correctly.

What does this look like?

What is the contract?

Aspects of the shape and behaviour of an interface that you rely on - that is the contract.


Collaboration tests

Consider a point of integration between my service - the client - and another service providing me with data - the supplier. Collaboration tests are concerned with how my service reacts when given certain responses from the supplier.

Rather than trying to test against a real supplier, test doubles are a way for me to express the contract directly; using the fewest words, or the least code possible.

Some people reach for lightweight implementations of interfaces, e.g. in memory databases, over using test doubles. Eventually these become complex enough that you want to write tests for the lightweight implementation - writing tests for your tests. Mistake.

Example: Repository pattern

I have a method that finds all customers… I could:

• Create an in-memory representation of my repository
• Not add any customers to the repository

OR I could:

• Pretend, with a test double implementing the repository interface, that the method returned no customers

Stub queries - expect actions

We end up with two broad categories of collaboration test.

1. Tests validating my module’s behaviour having received a particular response from a collaborator - I stub interfaces of these collaborators to return canned results - you can implement an interface to return a hard coded result
2. Tests asserting that some function was called in the next layer - side effects
   1. We don’t want to assert on the result of those side effects, only that they were triggered

Consider: What is the key interaction for this test? That is the thing we should be asserting on.

Aim for one assertion per test

If you find yourself wanting to write multiple assertions in all of your tests, this is more feedback on the design of the system - are your interactions too complex?

---

Contract tests


For each class of collaboration test, there should be a matching contract test showing how that particular case can arise e.g.:

If I have a collaboration test looking at what happens when a collaborator returns null, then I need to have a contract test that shows the conditions in which that collaborator can return null.

If I have a collaboration test without a matching contract test, what value is that collaboration test providing? If I can’t make the collaborator return null, why is null part of the interface?

If there are certain outcomes that are impossible, we should remove those outcomes from the contract of the interface.


So, circling back to the scam, we can now fix the “holes” in our unit testing with contract tests.

---

Testing the system

To verify correctness of the system, we end up with 4 categories of tests.

1. At the individual module level, e.g. for behaviour happening internally, operating exclusively on objects in memory - unit tests. We don’t need anything more fancy.
2. For the interactions between services within my organisations ecosystem, including services not necessarily owned by me or my team - collaboration and contract tests. Visualised as a tree, my system grows down, with collaboration and contract tests at each layer.


1. For integrations with the outside world, e.g. APIs I consume that are entirely outside my organisation - integrated tests.

   These tests are not verifying behaviour of my system and how it integrates with the outside service, they are characterisation tests or learning tests. We are validating our assumptions about how this service that we depend on but have zero control over works.


1. For verifying configuration in the system, e.g. are things wired up correctly? Am I using the correct implementations? - some integrated/end-to-end tests.

---

Bonus

We may find ourselves in a situation where we have multiple modules talking with the same thing in the outside world. We can deduplicate, and separate that interaction with the outside world behind an interface, and now we have something we can write collaboration and contract tests against.

E.g.: I have two modules interaction with the same database or the same external API. Rather than both modules having their own integration tests against the external dependency, we can de-duplicate and decouple by introducing a layer in between. Now only the new module needs integration tests, and the original two modules can have collaboration and contract tests.