Author: Dave Farley Link: https://www.amazon.com.au/Modern-Software-Engineering-Really-Better/dp/0137314914

Preface

Software engineering is more than the code that we write and the tools that we use. Software engineering is not production engineering in any form; that is not our problem. If when I say engineering it makes you think bureaucracy, please read this book and think again.

Software engineering is the application of an empirical, scientific approach to finding efficient, economic solutions to practical problems in software.

we need to become experts at learning to succeed

we need to become experts at managing [that] complexity

Part 1: What is Software Engineering?

Chapter 1: Introduction

Software engineering is the application of an empirical, scientific approach to finding efficient, economic solutions to practical problems in software.

…to become experts at learning, we need the following: Iteration Feedback Incrementalism Experimentation Empiricism

To become experts at managing complexity, we need the following: Modularity Cohesion Separation of Concerns Abstraction Loose Coupling.

practical tools to drive an effective strategy for any software development:

• Testability
• Deployability
• Speed
• Controlling the variables
• Continuous delivery

If our “software engineering” practices don’t allow us to build better software faster, then they aren’t really engineering, and we should change them!

In other disciplines, engineering simply means the “stuff that works.” It is the process and practice that you apply to increase your chances of doing a good job.

Chapter 2: What is Engineering?

Waterfall processes are production lines for software. They are the tools of mass production. They are not the tools of discovery, learning, and experimentation that are, or at least should be, at the heart of our profession.

Unless we are foolish in our software development choices, for us, production consists of triggering the build!

It is automatic, push-button, immensely scalable and so cheap that it is best considered free. We can still make mistakes and get it wrong, but these are problems that are understood and well addressed by tools and technology.

“Production” is not our problem. This makes our discipline unusual. It also makes it subject to easy misunderstanding and misapplied thinking and practices, because this ease of production is so unusual.

The assumption was not that you could plan and get it right the first time, rather that you treated all ideas, solutions, and designs with skepticism until you ran out of ideas about how things could go wrong.

The other engineering principle that is embodied in Hamilton’s early work is the idea of “failing safely.” The assumption is that we can never code for every scenario, so how do we code in ways that allow our systems to cope with the unexpected and still make progress?

Engineering is the application of an empirical, scientific approach to finding efficient, economic solutions to practical problems.

…the system should first be made to run, even though it does nothing useful except call the proper set of dummy subprograms. Then, bit-by-bit it is fleshed out, with the subprograms in turn being developed into actions or calls to empty stubs in the level below

Managing Complexity

Later, as tooling, production techniques, and engineering understanding and discipline increased, these mass-produced weapons exceeded the quality, as well as the productivity, of even the greatest master craftsmen, and at a price that anyone could afford.

A simplistic view may interpret this as a “need to standardize,” or a need to adopt “mass production for software,” but this is, once again, confusing the fundamental nature our problem. This is not about production—it is about design.

If we design a gun that is modular and componentized in the way that the arms manufacturers of the American Civil War did, then we can design parts of that gun more independently. Viewing this from a design perspective rather than from a production engineering or manufacturing perspective, we have improved our management of the complexity of building guns.

Before this step, the gunsmith master-craftsmen would need to think of the whole gun if they wanted to change some aspect of its design. By componentizing the design, the Civil War manufacturers could explore changes incrementally to improve the quality of their products step-by-step. Edsger Dijkstra said:

The art of programming is the art of organizing complexity.

Repeatability and Accuracy of Measurement

Let us for a moment imagine a complex software system. After a few weeks of operation, let’s say the system fails. The system is restarted, and two weeks later it fails again in much the same way; there is a pattern. How would a craft-focused team cope with this compared to an engineering-focused team?

The crafty team will probably decide that what they need is to test the software more thoroughly. Because they are thinking in craft terms, what they want is to clearly observe the failure.

This isn’t stupid; it makes sense in this context, but how to do it? The commonest solution that I have seen to this kind of problem is to create something called a soak test. The soak test will run for a bit longer than the normal time between failure, let’s say three weeks for our example. Sometimes people will try to speed up time so that the soak will simulate the problem period in a shorter time, but usually not.

The test runs, the system fails the test after two weeks, and the bug is, eventually, identified and fixed.

Is there any alternative to this strategy? Well, yes!

Soak tests detect resource leaks of one form or another. There are two ways to detect leaks; you can wait for the leak to become obvious, or you can increase the precision of your measurement so you catch the leak early before it becomes catastrophic.

I had a leak in my kitchen recently. It was in a pipe, buried in concrete. We detected the leak once it had soaked the concrete sufficiently for water to start to puddle on the surface. This is the “obvious” detection strategy.

We got a professional in to help us fix the leak. He brought a tool, an engineered solution. It was a highly sensitive microphone that “listened” for the sound of the leak underground.

Using this tool, he could detect the faint hiss of leaking water buried in concrete with sufficient, super-human precision to allow him to identify the location within a few inches and dig a small trench to get at the defective piece of pipe.

So back to our example: the engineering-focused team will use accurate measurement rather than waiting for something bad to happen. They will measure the performance of their software to detect leaks before they become a problem.

This approach has multiple benefits; it means that catastrophic failure, in production, is largely avoided, but it also means that they can get an indication of a problem and valuable feedback on the health of their system much, much sooner. Instead of running a soak test for weeks, the engineering-focused team can detect leaks during regular testing of the system and get a result in a matter of minutes. David Parnas said:

Software engineering is often treated as a branch of computer science. This is akin to regarding chemical engineering as a branch of chemistry. We need both chemists and chemical engineers, but they are different.

Engineering, Creativity, and Craft

The problem with craft-based solutions to problems is that they are not scalable in the way that engineering-based solutions are.

Craft can produce good things, but only within certain bounds.

Engineering discipline in virtually all human endeavors increases quality, reduces costs, and generally provides more robust, resilient, and flexible solutions.

It is a big mistake to associate ideas like skill, creativity, and innovation only with craft. Engineers in general, but certainly design engineers, exhibit all of these qualities in abundance all of the time. These attributes are central to the process of design engineering.

So taking an engineering approach to solving problems does not, in any way, reduce the importance of skill, creativity, and innovation. If anything, it amplifies the need for these attributes.

Why What We Do Is Not Software Engineering

In 2019, Elon Musk’s company SpaceX made a big decision; it was working on creating spacecraft that will one day allow humans to live and work on Mars and explore other parts of the solar system. In 2019, it switched from building its Starships out of carbon fiber to building them from stainless steel instead. Carbon fiber was a pretty radical idea; they had done a lot of work, including building prototype fuel tanks from the material. Stainless steel was also a radical choice; most rockets are built from aluminum because of its lightness and strength.

The SpaceX choice of stainless steel over carbon fiber was based on three things: the cost per kilogram was dramatically lower for steel; the high-temperature performance, to cope with re-entry temperatures, was better than aluminum; the low-temperature, cryogenic performance was dramatically better than both of the alternatives.

Carbon fiber and aluminum are significantly weaker than steel at very low and high temperatures.

When was the last time you heard anyone make a justification for a decision associated with software creation that sounded even vaguely like that?

This is what engineering decisions look like. They are based on rational criteria, strength at a certain temperature, or economic impact. It is still experimental, it is still iterative, it is still empirical.

You make a decision based on the evidence before you and your theory of what that will mean, and then you test your ideas to see if they work. It is not some perfectly predictable process.

Trade-Offs

All engineering is a game of optimization and trade-offs. We are trying to attempt to solve some problem, and, inevitably, we will be faced with choices. In building their rockets, one of the biggest trade-offs for SpaceX is between strength and weight. This is a common problem for flying machines, and actually for most vehicles.

Understanding the trade-offs that we face is a vital, fundamental aspect of engineering decision-making.

If we make our system more secure, it will be more difficult to use; if we make it more distributed, we will spend more time integrating the information that it gathers. If we add more people to speed up development, we will increase the communication overhead, coupling, and complexity, all of which will slow us down.

The Illusion of Progress

The tools matter only to the degree to which they “move the dial” on some more fundamental things.

Time For a Rethink?

When we get these things [the ideas in the book] right, and many teams do, we see greater productivity, less stress and burnout in team members, higher quality in design, and more resilience in the systems that we create.

The systems that we build please their users more. We see dramatically fewer bugs in production, and teams that employ these ideas find it significantly easier to change almost any aspect of the systems that they work on as their learning evolves. The bottom-line result of this is usually greater commercial success for the organizations that practice in this way. These attributes are the hallmarks of engineering.

Summary

Certainly in some circles we have come to see engineering as an unnecessary, onerous, and burdensome thing that gets in the way of “real software development.” Real engineering in other disciplines is none of these things. Engineers in other disciplines make progress more quickly, not less. They create work of higher quality, not lower.

When we begin to adopt a practical, rational, lightweight, scientific approach to software development, we see similar benefits. Software engineering will be specific to software, but it will also help us to build better software faster, not get in the way of us doing that.

Chapter 3: Fundamentals of an Engineering Approach

An Industry of Change?

When Fred Brooks wrote that there were no order-of-magnitude gains, he missed something. There may not be any 10x gains, but there are certainly 10x losses.

I have seen organizations that were hamstrung by their approach to software development, sometimes by technology, more often by process. I once consulted in a large organization that hadn’t released any software into production for more than five years.

The Importance of Measurement

Most metrics applied to software development are either irrelevant (velocity) or sometimes positively harmful (lines of code or test coverage).

It also goes on to dispel a commonly held belief that “you can have either speed or quality but not both.” This is simply not true. Speed and quality are clearly correlated in the data from this research. The route to speed is high-quality software, the route to high-quality software is speed of feedback, and the route to both is great engineering.

Applying Stability and Throughput

Clearly the addition of extra review and sign-offs are going to adversely impact on throughput, and such changes will inevitably slow down the process. However, do they increase stability?

For this particular example the data is in. Perhaps surprisingly, change approval boards don’t improve stability. However, the slowing down of the process does impact stability adversely.

We found that external approvals were negatively correlated with lead-time, deployment frequency, and restore-time, and had no correlation with change fail rate. In short, approval by an external body (such as a manager or CAB) simply doesn’t work to increase the stability of production systems, measured by time to restore service and change fail rate. However, it certainly slows things down. It is, in fact, worse than having no change approval process at all.5

It is not obvious that CABs are a bad idea. They sound sensible, and in reality that is how many, probably most, organizations try to manage quality. The trouble is that it doesn’t work.

Without effective measurement, we can’t tell that it doesn’t work; we can only make guesses.

Experts at Learning

There is considerable consensus among people who many of us consider to be thought leaders in our industry on this topic. Despite being well known, these ideas are not currently universally or even widely practiced as the foundations of how we approach much of software development. There are five linked behaviors in this category:

• Working iteratively
• Employing fast, high-quality feedback
• Working incrementally
• Being experimental
• Being empirical

Learning is at the heart of everything that we do. These practices are the foundations of any effective approach to software development, but they also rule out some less effective approaches.

Waterfall development approaches don’t exhibit these properties, for example. Nevertheless, these behaviors are all correlated with high performance in software development teams and have been the hallmarks of successful teams for decades.

Experts at Managing Complexity

As a software developer, I see the world through the lens of software development. As a result, my perception of the failures in software development and the culture that surrounds it can largely be thought of in terms of two information science ideas: concurrency and coupling.

These are difficult in general, not just in software design. So, these ideas leak out from the design of our systems and affect the ways in which the organizations in which we work operate.

You can explain this with ideas like Conway’s law,6 but Conway’s law is more like an emergent property of these deeper truths.

There are five ideas in this category, too. These ideas are closely related to one another and linked to the ideas involved in becoming experts at learning. Nevertheless, these five ideas are worth thinking about if we are to manage complexity in a structured way for any information system:

• Modularity
• Cohesion
• Separation of concerns
• Information hiding/abstraction
• Coupling

Part 2: Optimize For Learning

Chapter 4: Working Iteratively

Waterfall-style thinking starts from the assumption that “if we only think/work hard enough, we can get things right at the beginning.”

Agile thinking inverts this. It starts from the assumption that we will inevitably get things wrong. “We won’t understand what the users want,” “we won’t get the design right straight away,” “we won’t know if we have caught all the bugs in the code that we wrote,” and so on and so on. Because they start off assuming that they will make mistakes, agile teams work in a way that, quite intentionally, mitigates the cost of mistakes.

Practical Advantages of Working Iteratively

One of the common ideas from both Scrum and Extreme Programming (XP) was that we should work on small units of work to completion. The agile thought process was, “Progress in software development is hard to measure, but we can measure finished features, so let’s work on smaller features so that we can see when they are finished.”

This reduction in batch size was a big step forward. However, it gets complicated when you want to know how long it will take to “finish.” This iterative approach to development is different from more traditional ways of thinking. For example, in continuous delivery we work so that every small change, multiple times per day, is releasable. It should be finished to the degree that we can safely and reliably release our software into production at any point. So what does “finished” really mean in that context?

Each change is finished because it is releasable, so the only sensible measure of “finished” is that it delivers some value to its users. That is a very subjective thing. How do we predict how many changes are needed to represent “value” to our users? What most organizations do is to guess at a collection of features that, in combination, represent “value,” but if I can release at any point in the life of my software, this is a somewhat blurry concept.

One of the more subtle advantages of working iteratively is that we have a choice. We could iterate on the products that we create and steer them, based on good feedback from our customers and users, toward higher-value outcomes. This is one of the more valuable aspects of this way of working that is often missed by more traditional organizations that attempt to adopt it.

Agile planning depended, to a significant degree, on decomposing work into small enough pieces that we could complete our features within a single sprint, or iteration. Initially this was promoted as a way of measuring progress, but it had the much more profound impact of delivering definitive feedback on the quality and appropriateness of our work on a regular basis. This change increases the rate at which we can learn. Does this design work? Do our users like this feature? Is the system fast enough? Have I eliminated all of the bugs? Is my code nice to work in? and so on.

Iteration as a Defensive Design Strategy

Dan [North] described the difference between waterfall and agile thinking as, effectively, a problem in economics. Waterfall thinking is promulgated on the assumption that change gets more expensive as time goes on.

This worldview is problematic. It means that if this model is correct, the only sensible solution is to make the most important decisions early in the life of a project. The difficulty with this is that early in the life of a project, we know the least that we will ever know about it. So we are making crucial decisions in the life of a project based on ill-informed guesses, however hard we work at this point to inform them.

Dan North’s alternate view was this: given that the classic Cost of Change model clearly doesn’t help us, what would? How much nicer would it be if we could flatten the Cost of Change curve? (See Figure 4.2.)

What if we could change our minds, discover new ideas, discover errors, and fix them, all at roughly the same cost whenever that happened? What if the Cost of Change curve was flat?

It would give us the freedom to discover new things and benefit from our discoveries. It would allow us to adopt an approach that would allow us to continuously improve our understanding, our code, and our user’s experience of our products.


So, what would it take to achieve a flat Cost of Change curve?

We can’t afford to spend lots of time in analysis and design without creating anything, because that means more time not learning what really works. So we need to compress things. We need to work iteratively. We need to do just enough analysis, design, coding, testing, and releasing to get our ideas out into the hands of our customers and users so that we can see what really works. We need to reflect on that and then, given that learning, adapt what we do next to take advantage of it.

This is one of the ideas at the heart of continuous delivery


The Lure of the Plan

Industry data says that for the best software companies in the world, two-thirds of their ideas produce zero or negative value.3 We are terrible at guessing what our users want. Even when we ask our users, they don’t know what they want either. The most effective approach is to iterate. It is accepting that some, maybe even many, of our ideas will be wrong and work in a way that allows us to try them out as quickly, cheaply, and efficiently as possible.

Once we have an idea that we would like to execute on, we need to find a way to decide when to stop. How do we call a halt on a bad idea?

We need to find a way to try our ideas with minimum cost, so that if it is bad, we can find that out quickly and at relatively low cost. A 2012 survey of software projects carried out by the McKinsey Group in association with Oxford University found that 17% of large projects (budgets over $15M) went so badly that they threatened the existence of the company that undertook them. How can we identify these bad ideas? If we work in small steps, get real reaction to the progress or otherwise, and constantly validate and review our ideas, we can see soonest, with lowest investment, when things start to work differently to our hopes and plans. If we work iteratively in small steps, the cost of any single step going wrong is inevitably lower; therefore, the level of this risk is reduced.

An iterative approach is very different. We can begin when we know almost nothing and yet still make useful progress. We can start with some simple, understandable facet of the system. Use this to explore how our team should work on it, try out our first thoughts on the architecture of our system, try out some technologies that we think might be promising, and so on. None of these things is necessarily fixed. We have still made progress even if we found that the tech was a bad idea and our first concept of the architecture was wrong. We now know better than we did before. This is an inherently open-ended, infinite process. As long as we have some kind of “fitness function,” a way of telling if we are heading toward our goal or away from it, we can continue in this vein forever, refining, enhancing, and improving our understanding, our ideas, our skills, and our products. We can even decide to change our “fitness function” along the way if we decide that there are better goals to aim for.

The lure of the plan is a false one. This is not a more diligent, more controlled, more professional approach. Rather, it is more limited and more based on hunch and guesswork and can, realistically, work only for small, simple, well-understood, well-defined systems.

An iterative approach to planning and execution allows us to always have the most up-to-date picture of the situation that we are really in, rather than some predictive, theoretical, always-inaccurate version of that situation. It allows us to learn, react, and adapt as changes happen along the way. Working iteratively is the only effective strategy for a changing situation.

Practicalities of Working Iteratively

So, what can we do to work this way? The first thing is to work in smaller batches. We need to reduce the scope of each change and make change in smaller steps; in general, the smaller the better. This allows us to try out our techniques, ideas, and technology more frequently.

Working in small batches also means that we limit the time-horizon over which our assumptions need to hold. The universe has a smaller window of time within which it can intrude on our work, so things are less likely to change in damaging ways. Finally, if we make small steps, even if a small step is invalidated by changing circumstance or just misunderstanding on our part, there is less work lost. So, small steps really matter.

The obvious incarnation of this idea in agile teams is the idea of iterations or sprints. Agile disciplines promote the idea of working to completed, production-ready code, within a small, fixed period of time.

At a completely different scale, you can think of the practices of continuous integration (CI) and test-driven development (TDD) as being inherently iterative processes.

In CI we are going to commit our changes frequently, multiple times per day. This means that each change needs to be atomic, even if the feature that it contributes to is not yet complete. This changes how we approach our work but gives us more opportunities to learn and to understand if our code still works alongside everyone else’s.

TDD is often described by the practices that contribute to it: Red, Green, Refactor.

• Red: Write a test, run it, and see it fail.
• Green: Write just enough code to make the test pass, run it, and see it pass.
• Refactor: Modify the code and the test to make it clear, expressive, elegant, and more general. Run the test after every tiny change and see it pass.

This is a deeply fine-grained, iterative approach. It encourages a substantially more iterative approach to the fundamental technicalities of writing code.

Chapter 5: Feedback

Feedback in Integration

When I commit my code, it will trigger my continuous integration system and evaluate my change in the context of everyone else’s. I get a new level of feedback at this point. I gain deeper understanding. In this case, I can now learn if something in my code has “leaked out” and caused some other part of the system to fail.

The definition for CI states:

(CI) is the practice of merging all developers’ working copies to a shared mainline several times a day.

Most CI experts will relax “several times a day” to “at least once per day” as an acceptable, though not desirable, compromise.

So, by definition, CI is about exposing changes in small increments to evaluation at least once per day.

Branching, of any kind, also by definition, is about isolating change:

Branches allow contributors to isolate changes.

In basic, definitional terms, CI and FB then are not really compatible with each other. One aims to expose change as early as possible; the other works to defer that exposure.

Instead of working on a feature until it is “finished,” or “ready for production” continuous integration and its big brother continuous delivery demand of us to make changes in small steps and have something ready for use after every small step. This changes how we think about the design of our system in some important ways.

This approach means that the process to design our code is more like one of guided evolution, with each small step giving us feedback, but not necessarily yet adding up to a whole feature. This is a very challenging change of perspective for many people, but it is a liberating step when embraced and is one that has a positive impact on the quality of our designs.

Feedback in Architecture

Continuous delivery is a high-performance, feedback-driven approach to development. One of its cornerstones is the idea that we should produce software that is always ready for release into production. This is a high standard and demands a very high frequency and quality of feedback.

Achieving this requires organizations to change many different aspects of their development approach. Two aspects that come to the fore may be considered architectural qualities of the systems that we build. We need to take the testability and deployability of our systems seriously.

I advise the companies that I work with to aim for creating “releasable software” at least once per hour. This means that we must be able to run probably tens, maybe hundreds of thousands of tests every hour.

If any single test takes longer than an hour to run or if your software takes longer than an hour to deploy, it won’t be possible to run your tests this quickly, however much money you spend on hardware.

So the testability and deployability of our system add constraints to our ability to gather feedback. We can choose to design our systems to be more easily testable and more easily deployable, allowing us to gather feedback more efficiently and over shorter time periods.

We’d prefer tests that take seconds or milliseconds to run and deployment to complete in a handful of minutes or, even better, a few seconds.

There are two effective routes: either you can work to build monolithic systems and optimize them for deployability and testability, or you can modularize them into separate, individually “deployable units.” This second approach is one of the driving ideas behind the popularity of microservices.

Prefer Early Feedback

In general, it is an effective practice to try to get definitive feedback as early as possible. When I am coding, I can use my development tools to highlight errors in my code as I type. This is the fastest, cheapest feedback loop, and one of the most valuable. I can take advantage of this by using techniques like type systems to give me fast definitive feedback on the quality of my work.

I can run the test (or tests) in the area of the code that I am working on in my development environment and get feedback very quickly—usually in less than a few seconds.

My automated unit tests, created as the output of my TDD approach, give me my second level of feedback as I work and regularly run them in my local development environment.

My full suite of unit and other commit tests will be run once I have committed my code. This gives me a more thorough, but more costly in terms of time, validation that my code works along with other people’s code.

Acceptance tests, performance tests, security tests, and anything else that we consider important to understanding the validity of our changes give us further confidence in the quality and applicability of our work, but at the cost of taking longer to return results.

So working to prefer to identify defects, first in compile-ability (identified in our development environment) and then in unit tests and, only after those validations have succeeded, in other forms of higher-level tests, means that we can fail soonest and get the highest quality, most effective feedback.

Continuous delivery and DevOps practitioners sometimes refer to this process of preferring early failures as shift-left, though I prefer the less obscure “Fail fast!”

Feedback in Product Design

How do we know that the ideas that we have, the products that we create, are good ones?

The real answer is that we don’t know until we get feedback from the consumers of our ideas (our users or customers).

Applying the principles of employing and optimizing for fast, high-quality feedback enables organizations to learn faster; to discover what ideas work, or don’t, for their customers; and to adapt their products to better meet customer needs.

Feedback in Organization and Culture

…if your team has an idea to improve its approach to something, take a leaf from the scientist’s book and be clear about where you think you are now (current state) and where you would prefer to be (target state). Describe a step that you think will take you in the correct direction. Decide how you will decide whether you are closer to, or further away from, your target state. Make the step and check to see if you are closer to, or further from, the target and repeat until you are at the target.7

Chapter 6: Incrementalism

Importance of Modularity

Divide the problem into pieces aimed at solving a single part of a problem. This approach has many advantages. Each component of the system is simpler, more focused on the task at hand. Each component is easier to test, is faster to deploy, and sometimes may even be deployed independently of the others. Once you reach that point, and not before, you are really in the realm of microservices.

Taking a modular approach forces you to consider the boundaries between the modules of the system and take them seriously. These boundaries are important; they represent one of the key points of coupling in the system, and focusing on the protocols of information exchange between them can make a significant difference to how easy it is to isolate work and increase flexibility.

Organizational Incrementalism

A modular approach frees teams to work more independently. They can each make small incremental steps forward without needing to coordinate, or at least with minimal coordination, between teams. This freedom allows organizations that embrace it fully to move forward and innovate at unprecedented pace.

When we are discussing software development, we are nowhere close to being able to eliminate human creativity from this endeavor. So to enable human creativity, we need to leave room in the process and policies that structure our work for creative freedom. One of the defining characteristics of high-performing teams in software development is their ability to make progress and to change their minds, without asking for permission from any person or group outside of their small team.

Let’s pick this apart a little. Let us start with “small teams.” Although we now have more data to back up the assertion,4 it has long been known that small teams outperform large ones. In his book The Mythical Man Month, Fred Brooks wrote:

The conclusion is simple: If a 200-man project has 25 managers who are the most competent and experienced programmers, fire the 175 troops and put the managers back to programming.

These days, most agile practitioners would consider a team of 25 to be a large team. Current thinking is that the optimum team size is eight or fewer people.

Small teams are important for a variety of reasons, but their ability to make progress in small, incremental steps is an important one. To carry out organizational change, the most effective strategy is to create many small, independent teams and allow them the freedom to make their own changes. This progress can, and should still, be structured. It should be constrained to some degree to allow separate, independent teams to head in a roughly similar direction, targeted at fulfilling a larger-scale organizational vision, but still this is a fundamentally more distributed approach to organizational structure than has been traditional for most big firms.

The key transformation then that most organizations need to make is toward greater autonomy for people and teams to deliver high-quality, creative work. Distributed, incremental change is the key.

Modular organizations are more flexible, more scalable, and more efficient than more traditional organizational structures for software development.

Tools of Incrementalism

The most profound tools to enable incrementalism are feedback and experimentation, but we also need to focus on modularity and separation of concerns.

If my code is a big spaghetti ball-of-mud and I make a change in one place, I may inadvertently affect another part of the code. There are three important techniques that will allow me to make such a change more safely.

I can architect my system to limit the scope of the change. By designing systems that are modular and have a good separation of concerns, I can limit the impact of my changes beyond the area of the code that is my immediate focus.

I can adopt practices and techniques that allow me to change the code with lower risk. Chief among these safer practices is refactoring. That is the ability to make changes in small, simple, controlled steps that allow me to improve or at least modify my code safely.

Finally, there is testing. Testing, and specifically automated testing, gives us protection to move forward incrementally with significantly more confidence.

A test-driven approach to automated testing demands that we create mini executable specifications for the changes that we make to our systems. Each of these little specifications describes the necessary conditions to begin the test, executes the behavior under test, and then evaluates the results.

To manage the amount of work necessary to achieve all of this, we are crazy if we don’t try to make our lives easier by keeping the tests as simple as we can and by designing our system as testable code.

Since testable code is modular with a good separation of concerns, automated testing creates a positive feedback loop that enhances our ability to design better systems, limit the blast radius of mistakes, and make changes more safely. Ultimately, the combination of these three techniques provides a massive step forward in our ability to make changes incrementally.

Limiting the Impact of Change

To make the pieces of our system more independent, we can use the powerful technique of the Ports & Adapters pattern.

At any interface point between two components of the system that we want to decouple, a port, we define a separate piece of code to translate inputs and outputs, the adapter. This allows us more freedom to change the code behind the adapters without forcing change on other components that interact with it through this port.

This code is the core of our logic, so being able to change this without coordinating with other teams or people is a big win. As a result, we can safely make incremental progress in this part of the code and then deal with the significantly trickier and costly changes in the agreed-upon protocols of information exchange between components. These changes should, ideally, happen a lot less often, so teams will break one another’s code significantly less often, too.

We should always treat these integration points, these ports, with a little more care than other parts of our systems because they cause more pain when things need to change here. The Ports & Adapters approach gives us a strategy to embody that “more care” in our code.

The other important, and often overlooked, tool in managing the impact of change is speed of feedback. If I write some code that breaks your code, then how much that matters is very different depending on when we find out that I broke it.

If we only discover that I broke something months later, then the implications may be serious. If our code is already in production when we find the problem, the implications could be very serious.

If, on the other hand, we find out within a few minutes of my making the change, then it is no big deal. I can resolve the problem that I created, maybe before you even notice. This is the problem that continuous integration and continuous delivery solve.

This means that we can use either, or both, of these strategies to limit the impact of change. We can design our systems to enhance our ability to make changes, without forcing the need to change on others, and we can optimize our working practices to make changes in small, incremental steps. Committing those small changes to some shared evaluation system and then optimizing that evaluation system give us feedback quickly enough to allow us to react to it and to manage any problems that our changes may cause.

Incremental Design

First, we need to accept that change, missteps, and the impact of the unexpected, as our knowledge deepens, are all simply inevitable, whether you acknowledge them or not. It is simply the reality of all complex creation of any kind, and in the context of software development specifically, it is the nature of the beast.

Accepting that we don’t know, doubting what we do know, and working to learn fast is a step from dogma toward engineering.

The techniques of managing complexity are important for several reasons, but in this context of software development, as an act of discovery it is a vital one because they allow us to limit the “blast radius” when our “step forward” turns out to be a misstep. You can think of this as defensive design or defensive coding, but a better way to think of it is as incremental design.

We can choose to write code in ways that are merely a sequence of steps organized, or rather not organized, as a big ball of mud, poorly compartmentalized. Alternatively, we can write code in ways that effectively acknowledge and manage its complexity as it evolves.

If we do the former, then the more tightly coupled, less modular, less cohesive the code, the more difficult it is to change. That is why the properties that allow us to manage the complexity in our code that I keep repeating are important. If we adopt these ideas at every level of granularity pervasively in our work, then we close fewer doors on change, and we leave more options open to make change—even unexpected change—in the future. This is different from over-engineering and writing code that copes with every eventuality. This is code that is organized to make change easier, not code that does everything that you can think of right now.

Chapter 7: Empiricism

Grounded in Reality

…our production systems will always surprise us, and they should! Ideally they will not surprise us too often in very bad ways, but any software system is really only the best guess, so far, of its developers. When we publish our software into production, this is, or should be, an opportunity to learn.

Features are built because teams believe they are useful, yet in many domains most ideas fail to improve key metrics. Only one third of the ideas tested at Microsoft improved the metric(s) they were designed to improve.

Empiricism, making decisions based on evidence and observations of reality, is vital to making sensible progress. Without that analysis and reflection, organizations will continue to proceed on the basis of only guesswork and will continue to invest in ideas that lose them money or reputation.

Avoiding Self-Deception

The first principle is that you must not fool yourself – and you are the easiest person to fool.

Science is not what most people think it is. It is not about large hadron colliders or modern medicine or even physics. Science is a problem-solving technique. We create a model of the problem before us, and we check to see if everything that we currently know fits the model. We then try to think of ways in which we can prove the model is wrong

Inventing a Reality to Suit Our Argument

The language researchers had started with a theory—that parallelism was the answer—but they had gotten so caught up in an implementation that they never thought to test their starting premise, which was that this would result in a faster outcome. It resulted in a slower outcome and more complex code.

Guided by Reality

The best way to start is to assume that what you know, and what you think, is probably wrong and then figure out how you could find out how it is wrong.

It has long been understood that parallelism costs when you need to “join results back together.” Amdahl’s law shows that there is a harsh limit to the number of concurrent operations that make sense, unless they are wholly independent of one another.

The academics assumed that “more parallelism is good,” but that is an idea that is based on some kind of imaginary, theoretical machine, where the costs of concurrency were low; such machines don’t exist.

Chapter 8: Being Experimental

Most software development is consciously carried out as an exercise in craft where someone guesses what users may like. They guess about a design and/or technology that could achieve their product goals. Developers then guess about whether the code that they write does what they mean it to, and they guess about whether there are any bugs in it. Many organizations guess about whether their software is useful or made more money than it cost to build it.

What Does “Being Experimental” Mean?

We must move away from making decisions based on what the most important, charismatic, or famous people say, even if it is Richard Feynman, and instead make decisions and choices based on evidence.

Four characteristics define “being experimental” as an approach:

• Feedback: We need to take feedback seriously, and we need to understand how we will collect results that will provide us with a clear signal and deliver them efficiently back to the point at which we are thinking. We need to close the loop.
• Hypothesis: We need to have an idea in mind that we are aiming to evaluate. We are not wandering around willy-nilly, randomly collecting data. That is not good enough.
• Measurement: We need a clear idea of how we will evaluate the predictions that we are testing in our hypothesis. What does “success” or “failure” mean in this context?
• Control the variables: We need to eliminate as many variables as we can so that we can understand the signals that our experiment is sending to us.

Hypothesis

To be scientific, once we have a guess, in the form of a hypothesis, we start making some predictions, and then we can try to find ways to check those predictions.

We need to be able to test our hypotheses. Our tests can take a variety of forms. We can observe reality (production), or we can carry out some more controlled experiment, perhaps in the form of an automated test of some kind.

Measurement

It is too easy to fool ourselves by trying to “fit the facts to the data.” We can achieve some level of protection from such mistakes by thinking carefully, as part of the design of our experiment, what measurements we think will make sense. We need to make a prediction, based on our hypothesis, and then figure out how we can measure the results of our prediction.

I can think of lots of examples of measuring the wrong things. At one of my clients, they decided that they could improve the quality of their code by increasing the level of test coverage. So, they began a project to institute the measurement, collected the data, and adopted a policy to encourage improved test coverage. They set a target of “80 percent test coverage.” Then they used that measurement to incentivize their development teams, bonuses were tied to hitting targets in test coverage.

Guess what? They achieved their goal!

Some time later, they analyzed the tests that they had and found that more than 25 percent of their tests had no assertions in them at all. So they had paid people on development teams, via their bonuses, to write tests that tested nothing at all.

In this case, a much better measure would have been stability. What this organization really wanted was not more tests but better quality code, so measuring that more directly worked better.

Controlling the Variables

My take on continuous delivery as a generalized approach to software development is that it allows us to proceed with much more surety. It eliminates, to a large extent, the variables around the quality of our work so that we can concentrate on whether our product ideas are good. We can get a much clearer picture of “are we building the right things” because we have taken control of “are we building the things right.”

By controlling many of the technical variables in software development, continuous delivery allows us to make progress with significantly more confidence than before. This allows software development teams to take real advantage of the techniques of optimizing for learning that are at the heart of this book.

Putting the Experimental Results of Testing into Context

There are several studies, academic and informal, on the impact of TDD on defect reduction. Most studies agree that defect reduction is in a range from 40 percent to well over 250 percent. Source: https://bit.ly/2LFixzS, https://bit.ly/2LDh3q3, https://bit.ly/3cLT5F0

If we think of our software existing in a tiny universe that we create, however big or complex the software, then we can control that universe precisely and evaluate our software’s role in it. If we work to be able to “control the variables” to the extent that we can reliably and repeatably re-create that universe—infrastructure as code as a part of a continuous delivery deployment pipeline, for example—then we have a good starting point for our experiments.

The full set of all the tests that we have written, including the collection of experiments asserting our understanding of the behavior of our system in that controlled universe, is our body of knowledge of the system.

We can give the definition of the “universe” and the “body of knowledge” to anyone, and they can confirm that they are, as a whole, internally consistent—the tests all pass.

If we want to “create new knowledge” in the system, we can create a new experiment, a test, that defines the new knowledge that we expect to observe, and then we can add that knowledge in the form of working code that meets the needs of the experiment. If the new ideas are not consistent with previous ideas, meaning the “body of knowledge” in our mini, controlled universe, then experiments will fail, and we will know that this idea is wrong, or at least inconsistent with the recorded statement of the knowledge in the system.

Part 3: Optimize For Managing Complexity

Chapter 9: Modularity

Modularity is of vital importance in managing the complexity of the systems that we create. Modern software systems are vast, complicated, and often genuinely complex things. Most modern systems exceed the capacity of any human being to hold all of the details in their heads.

To cope with this complexity, we must divide the systems that we build into smaller, more understandable pieces—pieces that we can focus on without worrying too much about what is going on elsewhere in the system.

These days when I begin a software project, I will establish a check in the continuous delivery deployment pipeline, in the “commit stage,” that does exactly this kind of test and rejects any commit that contains a method longer that 20 or 30 lines of code. I also reject method signatures with more than five or six parameters. These are arbitrary values, based on my experience and preferences with the teams that I have worked on. My point is not to recommend these specific values; rather, it is that “guiderails” like these are important to keep us honest in our design. Whatever the time pressure, writing bad code is never a time-saver!

Hallmarks of Modularity

More practically, though, what we are looking for is something that divides our code into little compartments. Each compartment can be reused multiple times, perhaps in a variety of contexts.

The code in a module is short enough to be readily understood as a stand-alone thing, outside the context of other parts of the system, even if it needs other parts of the system to be in place to do useful work.

Undervaluing the Importance of Good Design

There are a few reasons why many software developers don’t pay attention to ideas like these. As an industry, we have undervalued the importance of software design. We obsess over languages and frameworks. We have arguments over IDEs versus text editors or object-oriented programming versus functional programming. Yet none of these things comes close to being as important, as foundational, as ideas like modularity or separation of concerns to the quality of our output.

This, though, is to me what software development is really about. How can we create code and systems that will grow and evolve over time but that are appropriately compartmentalized to limit damage if we make a mistake? How do we create systems that are appropriately abstracted so that we can treat the boundaries between our modules as opportunities to enhance our systems rather than liabilities that prevent us from changing them?

Any programming language is only a tool. I have been privileged to work with a few world-class programmers. These people will write good code in a programming language that they have never used before. They will write nice code in HTML and CSS or Unix shell scripts or YAML. One of my friends even writes readable Perl!

The Importance of Testability

If our tests are difficult to write, it means that our design is poor. We get a signal, immediately. We get feedback on the quality of our design as we attempt to refine it for the next increment in behavior. These lessons are delivered automatically to us if we follow the Red, Green, Refactor discipline of TDD. When our tests are hard to write, our design is worse than it should be. If our tests are easy to write our code, the stuff that we are testing inevitably exhibits the properties that we value as hallmarks of high quality in code.

What driving our designs from tests does is encourage us to create testable code and systems and so, given the limits of our experience and talent, enhances the outcome.

Designing for Testability Improves Modularity

If I want to test the effectiveness of the airfoil of a wing on an airplane, I can build the airplane and go flying. This is a terrible idea that even the Wright Brothers, who built the first powered, controlled airplane, realized wouldn’t work.

If you take this rather naive approach, then you have to do all of the work first before you learn anything. When you try to learn this way, how will you measure the effectiveness of this airfoil versus another? Build another airplane?

Even then, how do you compare the results? Maybe the wind was gustier when you flew the first prototype versus the second. Maybe your pilot had a bigger breakfast on the first flight than the second. Perhaps the air pressure or the temperature varied, so the wings delivered different amounts of lift because of that. Maybe the fuel batch was different between the two, so the engine was producing different power levels. How can you manage all these variables?

If you take this whole-system, waterfall approach to solving this problem, the complexity of the system is now expanded to encompass the entire environment in which the airfoil operates.

The way to scientifically measure an airfoil is to take control of these variables and standardize them across your experiments. How can we reduce the complexity so that the signals that we get back from our experiment are clear? Well, we could put the two airplanes into a more controlled environment, maybe something like a big wind tunnel. That would allow more precise control of the airflow over the wings and the wind. Maybe we could do this in a temperature and pressure-controlled environment. Only with this sort of control can we expect to get to more repeatable results.

If we are going to start down this road, we don’t really need an engine or flight controls or the rest of the airplane for that matter. Why not just make two models of the wings with the airfoils that we would like to test and try those in our temperature- and pressure-controlled wind tunnel?

That would certainly be a more accurate experiment than just going flying, but this still requires us to build the whole wing twice. Why not make a small model of each airfoil? Make each model as precisely as possible, using exactly the same materials and techniques, and compare the two. If we are going to go that far, we could do this on a smaller scale, and we’d need a simpler wind tunnel.

These small pieces of airplane are modules. They are parts that certainly add to the behavior of the whole plane, but they are focused on a specific part of the problem. It is true that such experiments will give you only a partially true picture. The aerodynamics of airplanes are more complex than only the wings, but modularity means that we can measure things that we couldn’t measure without it, so the part, the module, is certainly more testable than the whole.

In the real world, this is how you conduct experiments to determine how the shape of wings, and other things, affect lift.

Modularity gives us greater control and greater precision in the things that we can measure. Let’s move this example into the software world. Imagine that you are working on System B, which is downstream from System A and upstream from System C (see Figure 9.1).


This is typical of working on big systems in complex organizations. This presents a problem: how do we test our work? Many, maybe even most, organizations faced with this problem jump to the assumption that it is essential to test everything together to be sure that the system is safe to use.

There are many problems with this approach. First, if we measure only at this scale, we face the “test the whole airplane” problem. The whole system is so complex that we lack precision, reproducibility, control, and clear visibility of what any results that we do collect really mean.

We can’t evaluate our part of the system with any degree of precision, because the upstream and downstream parts, System A and System C, get in our way. There are many types of tests that are simply impossible as a result of this decision. What happens to System B if System A sends it malformed messages?

That case is impossible to measure, while the real System A is in place sending well-formed messages. How should System B respond when the communications channel to System C is broken?

Again, we can’t test that scenario while a real System C, with working comms, is in place, getting in the way of us faking a comms error.

The results that we do collect don’t tell us much. If a test fails, is that because there is a problem with our system, or is it one of the others? Maybe the failure means that we have the wrong versions of the upstream or downstream systems. If everything works, is that because we are ready to release? Or is it because the cases we are trying to evaluate are so simplistic, due to this mega-system not being testable that they aren’t really finding the bugs that are really there?

If you have a suite of even automated tests that you run to evaluate your software to determine if it is ready to release and those tests don’t produce the same result every time, what do those results really mean?

The real root cause of a lack of determinism in computer systems is concurrency. This can take various forms. The clock ticking away incrementing the system time is one form of concurrency; the OS re-organizing your disk when it thinks it has some spare time is another. In the absence of concurrency, though, digital systems are deterministic. For the same sequence of bytes and instructions, we get the same result every time.

One useful driver of modularity is to isolate the concurrency so that each module is deterministic and reliably testable. Architect systems so that entry into a module is sequenced and its outcomes are more predictable. Systems written this way are very nice to work on.

This may seem a fairly esoteric point, but in a system in which every behavior that its users observe is deterministic, in the way that I have described, it would be eminently predictable and testable with no unexpected side effects, at least to the limits of our testing.

Most systems are not built like this, but if we take an engineering-led approach to their design, they can be.

If, instead, we could apply our calipers to measure only our component (see Figure 9.3), we could measure with much greater accuracy and precision and with much more reliability. Stretching my analogy to breaking point, we could measure other dimensions of the problem, too.


So, what would it take to measure with this increased precision and specificity? Well, we would like our measuring points to be stable so that we get the same result from our measurement every time, all other things being equal. We would like our evaluations to be deterministic.

We would also like to not have to re-create our measurement points from scratch each time that the system changes.

To be clear, what I am describing here is a stable, modular interface to the part of the system that we would like to test. We compose larger systems of smaller modules, modules with clearly defined interfaces, for inputs and outputs. This architectural approach allows us to measure the system at those interfaces.

When I was involved in creating a financial exchange, we treated the whole enterprise system as a single system, but we established clear, well-defined integration points for every external interaction and faked those external systems. Now we had control; now we could inject new account registrations and collect data that, in real operation, would have been sent to banks or clearinghouses and so on.

This allowed us, for some of our testing, to treat the whole system as a black box, inject data to get the system into an appropriate state for a test, and collect its outputs to evaluate the system’s response. We treated every point at which our system interacted with a third-party system and every integration point as a point of measurement where we could plug in to our test infrastructure. This was possible only because our entire enterprise system was designed with testability in mind from day one.

Our system was also extremely modular and loosely coupled. So, as well as evaluating the system as a whole, we could do more detailed testing for single, service-level components, too. Inevitably all of the behavior within a service was also developed using fine-grained TDD techniques for pretty much every line of code. We could also test tiny pieces of the behavior of the system in isolation from everything else. As I said, modularity and testability are fractal.

Services and Modularity

From a purely practical perspective, we can think of a service as code that delivers some “service” to other code and hides the detail of how it delivers that “service.” This is just the idea of “information hiding” and is extremely important if we want to manage the complexity of our systems as they grow (see Chapter 12). Identifying “seams” in the design of our systems where the rest of the system doesn’t need to know, and shouldn’t care about, the detail of what is happening on the other side of those “seams” is a very good idea. This is really the essence of design.

The seams or boundaries should be treated with more care. They should be translation and validation points for information. The entry point to a service should be a little defensive barrier that limits the worst abuses of consumers of that service. What I am describing here is a Ports & Adapters kind of model at the level of an individual service. This approach should be just as true for a service that communicates via standard method or function calls as one that uses HTML, XML, or any other form of messaging.

Deployability and Modularity

One of the core ideas in my earlier book is the idea of the deployment pipeline, a mechanism that takes commits in at one end and produces a “releasable outcome” at the other. This is a key idea. A deployment pipeline is not simply a little workflow of build or test steps; it is a mechanized route from commit to production.

This interpretation has some implications. This means that everything that constitutes “releasability” is within the scope of your deployment pipeline. If the pipeline says everything is good, there should be no more work to do to make you comfortable to release—nothing…no more integration checks, sign-offs, or staging tests. If the pipeline says it is “good,” then it is “good to go!”

This, in turn, has some implications for the sensible scope of a deployment pipeline. If its output is “releasable,” it must also be “independently deployable.” The scope of an effective deployment pipeline is always an “independently deployable unit of software.”

Now this has an impact on modularity. If the output of the deployment pipeline is deployable, it means that the pipeline constitutes a definitive evaluation of our software. It’s definitive at least to the degree that we care, and consider safe and sensible, to establish its readiness for release.

There are only two strategies that make sense if we are to take that idea to its logical conclusion. We can build, test, and deploy everything that constitutes our system together, or we can build, test, and deploy parts of that system separately. There is no halfway solution. If we don’t trust the output of our deployment pipeline sufficiently and feel it necessary to test the results it generates with the output of other deployment pipelines, then that presents problems; the messages that our deployment pipeline is sending to us are now unclear, and since we are trying to be engineers, that isn’t good enough!

Deployability can up the stakes on modularity. As we have seen, deployability defines the effective scope of a deployment pipeline. Our choices of what really works, if we value high-quality work based on fast, efficient feedback, are really quite limited.

We can make the choice to build, test, and deploy everything that constitutes our system together and eliminate dependency-management problems altogether (everything lives in a single repository), but then we must take on the responsibility to create fast enough feedback to allow developers to do a good job, which may take a big investment in engineering to get the feedback that drives any high-quality process quickly enough.

Alternatively, we can work so that each module is, essentially, independent of every other module. We can build, test, and deploy each of these things separately, without the need to test them together.

This means that the scope of our builds, tests, and deployment is small. Each of them is simpler, so it is easier to achieve fast, high-quality results.

However, this comes at the sometimes very significant cost of a more complex, more distributed architecture in our systems. We are forced, now, to take modularity very seriously indeed.

Modularity at Different Scales

If, as I argue, the importance of modularity is a tool to help us manage complexity, then we need to take that to the point of readable code. Each class, method, or function should be simple and readable and, where appropriate, composed of smaller, independently understandable submodules.

Modularity in Human Systems

In this kind of world, the constant refrain has been “how do we scale?” Sometimes, rarely, that is about the software, but mostly when people in big organizations ask that question, what they really mean is “how can we add more people so that we can produce software faster?”

The real answer is that for any given computer system there are very serious limits to that. As Fred Brooks famously said:

You can’t make a baby in a month with 9 women.

If we need small teams to efficiently create good, high-quality work, then we need to find ways to seriously limit the coupling between those small teams. This is at least as much an organizational strategy problem as it is a technical one. We need modular organizations as well as modular software.

So if we want our organizations to be able to scale up, the secret is to build teams and systems that need to coordinate to the minimum possible degree, we need to decouple them. Working hard to maintain this organizational modularity is important and one of the real hallmarks of genuinely high-performing, scalable organizations.

In a study Quantitative Software Management (QSM) also found that the larger teams produced 5x more defects in their code. See https://bit.ly/3lI93oe.

Chapter 10: Cohesion

High-Performance Software

Let’s revisit our trivial example again. I have heard programmers make the argument that the code in Listing 10.1 is going to be faster than the code in Listing 10.2 because of the “overhead” of the method calls that Listing 10.2 adds. I am afraid that for most modern languages this is nonsense. Most modern compilers will look at the code in Listing 10.2 and inline the methods. Most modern optimizing compilers will do more than that. Modern compilers do a fantastic job of optimizing code to run efficiently on modern hardware. They excel when the code is simple and predictable, so the more complex your code is, the less help you will gain from your compiler’s optimizer. Most optimizers in compilers simply give up trying once the cyclomatic complexity4 of a block of code exceeds some threshold.

Link to Coupling

If we want to retain our freedom to explore and to sometimes make mistakes, we need to worry about the costs of coupling.

Coupling: Given two lines of code, A and B, they are coupled when B must change behavior only because A changed.

Cohesion: They are cohesive when a change to A allows B to change so that both add new value.

Driving High Cohesion with TDD

Yet again using automated tests, and specifically TDD, to drive our design gives us a lot of benefits. Striving to achieve a testable design and nicely abstracted, behaviorally focused tests for our system will apply a pressure on our design to make our code cohesive.

We create a test case before we write the code that describes the behavior that we aim to observe in the system. This allows us to focus on the design of the external API/Interface to our code, whatever that might be. Now we work to write an implementation that will fulfill the small, executable specification that we have created. If we write too much code, more than is needed to meet the specification, we are cheating our development process and reducing the cohesion of the implementation. If we write too little, then the behavioral intent won’t be met. The discipline of TDD encourages us to hit the sweet spot for cohesion.

Costs of Poor Cohesion

My point here, though, is that there is sweet spot for cohesion. If you jumble too many concepts together, you lose cohesion at a fairly detailed level. In example 1, you could argue that all the work is done inside a single method, but this is only naively cohesive.

In reality, the concepts associated with adding an item to a shopping cart, the business of the function, are mixed in with other duties that obscure the picture. Even in this simple example, it is less clear what this code is doing until we dig in. We have to know a lot more stuff to properly understand this code.

The other alternative, add_to_cart3, while more flexible as a design, still lacks clarity. At this extreme it is easy for responsibilities to be so diffuse, so widely dispersed, that it is impossible to understand the picture without reading and understanding a lot of code. This could be a good thing, but my point is that there is a cost in clarity to coupling this loose, as well as some benefits.

Cohesion in Human Systems

The findings from the “State of DevOps” report say that one of the leading predictors of high performance, measured in terms of throughput and stability, is the ability of teams to make their own decisions without the need to ask permission of anyone outside the team. Another way to think of that is that the information and skills of the team are cohesive, in that the team has all that it needs within its bounds to make decisions and to make progress.

Chapter 11: Separation of Concerns

Separation of concerns is defined as “a design principle for separating a computer program into distinct sections such that each section addresses a separate concern.”

Separating Essential and Accidental Complexity

The essential complexity of a system is the complexity that is inherent in solving the problem that you are trying to solve, how to calculate the value of a bank account, how to total the items in a shopping cart, or even how to calculate the trajectory of a spaceship, for example. Addressing this complexity is the real value that our system offers.

The accidental complexity is everything else—the problems that we are forced to solve as a side effect of doing something useful with computers. These are things like persistence of data, displaying things on a screen, clustering, some aspects of security…in fact anything that is not directly related to solving the problem at hand.

An effective approach to improving our designs through separation of concerns is to focus very clearly on separating the concerns of the accidental and essential complexities of our systems.

I want the logic of my system that cares about how to drive a car to be separate from the logic that knows how to display information on a screen, the logic that knows how to evaluate a trade to be separate from how that trade is stored or communicated.

Testability

If we work to ensure that our code is easy to test, then we must separate the concerns or our tests will lack focus. Our tests will also be more complex, and it will be difficult to make them repeatable and reliable. Striving to control the variables so that we can test encourages us to create systems that demonstrate the properties of high quality in software that we value: modularity, cohesion, separation of concerns, information hiding, and loose coupling.

Ports and Adapters

void doSomething(Thing thing) {
	String processedThing = process(thing);
	s3client.putObject("myBucket," "keyForMyThing," processedThing);
}

One take on cohesion is that within a particular scope, the level of abstraction should remain consistent. So what if we improved the consistency here? Listing 11.7 is a big improvement in this respect, even if all that we have done is rename a class and a method.

void doSomething(Thing thing) {
    String processedThing = process(thing);
    store.storeThings("myBucket," "keyForMyThing," processedThing);
}

You can think of the new abstraction as a port, or a vector through which information flows. Whether or not you decide to make the port polymorphic is entirely up to you and the circumstances in your code, but even where you don’t, this code is better. It is better because you have improved the separation of concerns, improved the cohesion by maintaining a more consistent level of abstraction, and improved both its readability and maintainability.

The concrete implementation of this port is an adapter that acts as a translation service, translating ideas from, in this example, the context of “things” to the context of “AWS S3 Storage.”

When to Adopt Ports and Adapters

When people discuss the Ports & Adapters approach, they are usually discussing it in the context of a translation layer at the boundaries between services (or modules).

This is good advice. In his book Domain Driven Design,2 Eric Evans recommends:

Always translate information that crosses between Bounded Contexts.

In designing a system from services, I, and others, advise that we should aim to align our services with a bounded context. This minimizes coupling and improves the modularity and cohesion of our services.

Combined, these two pieces of advice suggest a simple guideline of “Always translate information that flows between services,” or to put it another way, “Always communicate between services using Ports & Adapters.”

We can protect our code two ways, and we can use an adapter that translates things into our worldview as they arrive at the edges of our system, allowing us to validate our inputs to the degree to which we care about them. Or we can wrap up stuff that we don’t trust and ignore it so that we can protect our systems from dubious external changes.

What is an API?

An application programming interface (API) is all of the information that is exposed to consumers of a service, or library, that exposes that API.

Imagine, for a moment, a function that takes a binary stream of data as an argument. What is the API?

Is it only the signature of the function? Well, maybe, if the function treats the binary stream as a black-box and never looks inside the stream, then yes, the signature of the function defines its coupling with its callers.

However, if the function interacts with the contents of the binary stream in any way, that is part of its contract. The level of interaction defines the degree to which it is coupled, with the information in the stream.

If the first eight bytes in the stream are used to encode its length, and that is all that the function knows or cares about the stream, then the function signature, plus the meaning of the first eight bytes and how the length is encoded within them, are “the API.”

The more that the function knows of the content of the stream of bytes, the more coupled to it it is, and the greater the surface area of the API. I see many teams that ignore the fact that the data structures in its inputs that their code understands, and processes, are part of that code’s public API.

Our adapters need to deal with the whole API. If that means translating, or at least validating, the content of a binary stream of inputs, then so be it. The alternative is that our code may break when someone sends us the wrong stream of bytes. This is a variable that we can control.

As a default stance, or a guideline, I recommend that you always add Ports & Adapters where the code that you talk to is in a different scope of evaluation, such as a different repo or a different deployment pipeline. Taking a more defensive stance in these situations will make your code more testable, yet again, and more robust in the face of change.

Using TDD to Drive Separation of Concerns

Specifically in the context of separation of concerns, our tests become more difficult to write the more that concerns are conflated within the scope of a test. If we organize our development around testing and drive our development through testing, then we are confronted much earlier in the process by the costs and benefits of our design decisions.

Chapter 12: Information Hiding and Abstraction

Abstraction or Information Hiding

I conflate these ideas because I don’t think that the difference between the two is enough to really concern us. What I am talking about here is drawing lines, or seams, in our code so that when we look at those lines from the “outside,” we don’t care about what is behind them. As a consumer of a function, class, library, or module, I should not need, or care, to know anything about how it works, only how I use it.

It should be obvious that if our aim is to manage complexity so that we can build more complex systems than we can comfortably hold inside our heads, then we need to hide information.

Organizational and Cultural Problems

The causes are complex and diverse. One of the most common complaints that I hear from software developers and software development teams is “my manager won’t let me XXX,” where “XXX” is either “refactor,” “test,” “design better,” or even “fix that bug.”

The first thing to say is why do we, as software developers, need to ask for permission to do a good job? We are the experts in software development, so we are best placed to understand what works and what doesn’t.

If you hire me to write code for you, it is my duty to you to do the best job that I can. That means I need to optimize my work so that I can reliably, repeatably, and sustainably deliver code over a long period of time. My code needs to solve the problem that I am faced with, and it needs to fulfill the needs of my users and ambitions of my employers.

If you hired me as a chef, you would never say, “you have permission to sharpen your knives” or “it is your responsibility to clean your work area,” because as a professional chef, you, and I, would assume that those things are a fundamental part of being a professional. As a chef, that would be part of my duty of care.

As software professionals, it is our duty to understand what it takes to develop software. We need to own the responsibility for the quality of the code that we work on. It is our duty of care to do a good job. This is not altruistic; it is practical and pragmatic. It is in the interest of our employers, our users, and ourselves.

I have certainly seen organizations that, either intentionally or unintentionally, applied pressure on developers to speed up. Often, though, it is developers and development teams that are complicit in deciding what “speeding up” entails.

It is usually the developers that rule out quality, not the managers or organization. Managers and organizations want “better software faster,” not “worse software faster.” In reality, even that is not the trade-off. As we have already seen, the real trade-off, over long periods of time, is between “better software faster” and “worse software slower.” “Better” goes hand in hand with “faster.” This is important for all of us to recognize and to believe. The most efficient software development teams are not fast because they discard quality but because they embrace it.

Technical Problems and Problems of Design

Specifically, in the context of avoiding and correcting big balls of mud, though, there is a mindset that is important to adopt. This is the mindset that it is a good thing, a sensible thing, to change existing code.

Many organizations are either afraid to change their code or have some kind of reverence for it that belies the reality. I would argue the reverse: if you can’t, or won’t, change the code, then the code is effectively dead. To quote Fred Brooks again:

As soon as one freezes a design, it becomes obsolete.

Fear of Over-Engineering

The real solutions to the problem of being afraid to change our code are abstraction and testing. If we abstract our code, we are, by definition, hiding the complexity in one part of the system from another. That means that we can more safely change code in one part of the system, with a much higher level of confidence that our change, even if wrong, will not adversely affect other parts. To be even more sure of this, we also need the testing, but as usual the value of testing is not that simple.

Improving Abstraction Through Testing

You don’t write specification after you have completed the work; you need them before you start. So we will write our specifications (tests) before we write the code. Since we don’t have the code, our focus is more clearly fixed on making our life easier. Our aim, at this point, is to make it as simple as possible to express the specification (test) as clearly and simply as we can.

Inevitably then, we are, or at least should be, expressing our desires for the behavior that we want, from our code from the perspective of a consumer of it, as clearly and simply as we can. We should not be thinking about the implementation detail that will be required to fulfill that mini-specification at this point.

If we follow this approach, then, by definition, we are abstracting our design. We are defining an interface to our code that makes it easy to express our ideas so that we can write our test case nicely. That means that our code is also easy to use. Writing the specification (test) is an act of design. We are designing how we expect programmers to interact with our code, separate from how the code itself works. All this before we have gotten to the implementation detail of the code. This approach, based on abstraction, helps us separate what the code needs to do from how it does it. At this point, we say little or nothing about how we will implement the behavior; that comes later.

Leaky Abstractions

Leaky abstractions are defined as “an abstraction that leaks details that it is supposed to abstract away.”

An authorization service that reports functional failures as HTML errors and a business logic module that returns NullPointerExceptions are both breaking business-level abstractions with technical failures. Both of these are a kind of break in the continuity of the illusion that the abstraction is intended to convey.

One take on this is that abstraction, all abstraction, is fundamentally about modeling. Our aim is to create a model of our problem that helps us reason about it and helps us to do work. I like this quote from George Box:

All models are wrong, some models are useful.

This is always the situation that we are in. However good our models, they are representations of truth, not the truth itself. Models can be enormously useful even when fundamentally untrue.

Our aim is not to achieve perfection but to achieve useful models that we can use as tools to solve problems.

Picking Appropriate Abstractions

Abstraction, and the modeling that is at its heart, is a fundamental of design. The more targeted the abstractions are to the problem that you are trying to solve, the better the design. Note, I didn’t say “the more accurate the abstraction.” As Harry’s Tube map so clearly demonstrates, the abstraction doesn’t need to be accurate to be enormously useful.

The subtlety here, though, and the enormous value that TDD delivers, is that if I have written my abstract specification, focusing on what the code should do and not how it achieves that outcome, then what my test is expressing is my abstraction. So if the test is fragile in the face of change, then my abstraction is fragile in the face of change. So I need to think harder about better abstractions. I know of no other way of getting this kind of feedback.

Isolate Third-Party Systems and Code

As soon as we allow third-party code into our code, we are coupled to it. In general, my preference and advice is to always insulate your code from third-party code with your own abstractions.

Some caveats before we proceed with this idea. Obviously, your programming language and its common supporting libraries are “third-party code,” too. I am not suggesting that you write your own wrapper for Strings or Lists, so as usual my advice is a guideline rather than a hard-and-fast rule. However, I advise that you think carefully about what you allow “inside” your code. My default position is that I will allow language concepts and libraries that are standard, but not any third-party libraries that don’t come with my language.

Chapter 13: Managing Coupling

Coupling is defined as “the degree of interdependence between software modules; a measure of how closely connected two routines or modules are; the strength of the relationships between modules.”

…we should aim to prefer looser coupling over tighter coupling, but also to understand the trade-offs that we make when we make that choice.

…you don’t get better software faster by throwing people at the problem. There is a fairly serious limit on the size of a software development team, before adding more people slows it down (refer to Chapter 6).

If your team and my team are developmentally coupled, we could maybe work to coordinate our releases…

…The overhead of keeping everyone in step rapidly spirals out of control.

There are ways in which we can minimize this overhead and make this coordination as efficient as possible. The best way to do this is through continuous integration.

We will keep all our code in a shared space, a repository, and each time any of us changes anything, we will check that everything is still working. This is important for any group of people working together; even small groups of people benefit from the clarity that continuous integration brings.

This approach also scales significantly better than nearly everyone expects. For example, Google and Facebook do this for nearly all of their code. The downside of scaling up in this way is that you have to invest heavily in the engineering around repositories, builds, CI, and automated testing to get feedback on changes quickly enough to steer development activities. Most organizations are unable or unwilling to invest enough in the changes necessary to make this work.

Microservices are as follows:

• Small
• Focused on one task
• Aligned with a bounded context
• Autonomous
• Independently deployable
• Loosely coupled

I am sure that you can see that this definition closely aligns with the way that I describe good software design.

The trickiest idea here is that the services are “independently deployable.” Independently deployable components of software have been around for a long time in lots of different contexts, but now they are part of the definition of an architectural style and a central part.

Our service will need to be cohesive so that it is not too dependent on other services or other code. It needs to be very loosely coupled with respect to other services so that it, or they, can change without either one breaking the other. If not, we won’t be able to deploy our service without testing it with those other services before we release, so it isn’t independently deployable.

Loose Coupling Isn’t the Only Kind That Matters

Michael Nygard4 has an excellent model to describe coupling. He divides it into a series of categories:

Prefer Loose Coupling

The trick is to draw the seams of abstraction so that high-performance parts of the system fall on one side of that line or another so that they are cohesive, accepting that the transition from one service, or one module, to another will incur additional costs.

These interfaces between services prefer looser coupling to the extent that each service hides details from another. These interfaces are more significant points in the design of your system and should be treated with more care and allowed to come at a little higher cost in terms of runtime overhead as well as lines of code. This is an acceptable trade-off and a valuable step toward more modular, more flexible systems.

DRY Is Too Simplistic

DRY is excellent advice within the context of a single function, service, or module. It is good advice; beyond that, I would extend DRY to the scope of a version control repository or a deployment pipeline. It comes at a cost, though. Sometimes this is a very significant cost when applied between services or modules, particularly if they are developed independently.

The problem is that the cost of having one canonical representation of any given idea across a whole system increases coupling, and the cost of coupling can exceed the cost of duplication.

The advantage of DRY is that when something changes, we need to change it in only one place; the disadvantage is that every place that uses that code is coupled in some way.

So if you are creating a microservice-based system, with each service being independently deployable, and each service having its own deployment pipeline, you should not apply DRY between microservices. Don’t share code between microservices.

Async as a Tool for Loose Coupling

The previous chapter discussed the leakiness of abstractions. One of those leaky abstractions is the idea of synchronous computing across process boundaries.

As soon as we establish such a boundary, whatever its nature, any idea of synchrony is an illusion, and that illusion comes at a cost.

The illusion, the leaky abstraction, of synchrony can exist, but only to the point where one of these failures happens—and they will happen. Figure 13.1 shows the places where a distributed conversation can go wrong.


1. There may be a bug in A.
2. A may fail to establish a connection to the network.
3. The message may be lost in transmission.
4. B may fail to establish a connection to the network.
5. There may be a bug in B.
6. The connection to the network may fail before B can send a response.
7. The response may be lost in transmission.
8. A may lose the connection before it has the response.
9. There may be a bug in A’s handling of the response.

Apart from 1 and 9, each of the points of failure listed is a leak in the abstraction of synchronous communications. Each adds to the complexity of dealing with errors. Nearly all of these errors could leave A and B out of step with one another, further compounding the complexity. Only some of these failures are detectable by the sender, A.

This is not really the place to go into too much detail of specific approaches to design, but I am a believer in treating process boundaries as asynchronous and communicating between distributed services and modules via only asynchronous events. For complex distributed systems, this approach significantly reduces the impact of abstraction leaks and reduces the coupling to the underlying accidental complexity that sits beneath our systems.

Imagine for a moment the impact of a reliable, asynchronous messaging system on the list of failure points in Figure 13.4. All of the same failures can occur, but if Service A only sends asynchronous messages, and some time later receives only a new async message, then now Service A doesn’t need to worry about any of them after step 2. If a meteorite has hit the data center that contains Service B, then we can rebuild the data center, redeploy a copy of Service B, and resubmit the message that Service A sent originally. Although rather late, all the processing continues in precisely the same way as though the whole conversation had taken only a few microseconds.

Loose Coupling in Human Systems

In my professional life, I see many large organizations hamstrung by organizational coupling. They find it almost impossible to release any change into production, because over the years they have ignored the costs of coupling, and now making the smallest change involves tens, or hundreds, of people to coordinate their work.

If you want consistency across a large, complex piece of software, you should adopt the coordinated approach. In this you store everything together, build everything together, test everything together, and deploy everything together.

This gives you the clearest, most accurate picture but comes at the cost of your needing to be able to do all of these things quickly and efficiently. I generally recommend that you strive to achieve this kind of feedback multiple times per day. This can mean a significant investment in time, effort, and technology to get feedback quickly enough.

The distributed approach is currently more in favor; it is a microservices approach. In microservices organizations, decision-making is intentionally distributed. Microservice teams work independently of one another, each service is independently deployable, and there is no direct coordination cost between teams. There is, though, an indirect cost, and that cost comes in terms of design.

To reduce organizational coupling, it is important to avoid the need to test services together later in the process. If services are independently deployable, that means they are tested independently too, since how can we judge deployability without testing? If we test two services together and find out that version 4 of one works with version 6 of another, are we really then going to release version 4 and version 17 without testing them? So they aren’t independent.

Both of these approaches—the only two that make any real sense—are all about different strategies to manage the coupling between teams. You manage coupling by speeding up the frequency with which you check for mistakes when coupling is high, or you don’t check at all, at least prior to release, when coupling is low.

Part 4: Tools To Support Engineering In Software

Chapter 14: The Tools of an Engineering Discipline

Measurement Points

If we want our code to be testable, we need to be able to control the variables. We want to be able to inject precisely the information that we need and only that information. To get our software into a state where we can test it, we invoke some behavior, and then we need the results to be visible and measurable.

Problems with Achieving Testability

The strongest argument against TDD that I sometimes hear is that it compromises the quality of design and limits our ability to change code, because the tests are coupled to the code. I have simply never seen this in a codebase created with “test-first TDD.” It is common—I’d say inevitable—as a result of “test-after unit testing,” though. So my suspicion is that when people say “TDD doesn’t work,” what they really mean is that they haven’t really tried TDD, and while I am sure that this is probably not true in all cases, I am equally certain that it is true in the majority and so a good approximation for truth.

How to Improve Testability

The testability of our system is fractal. We can observe it and use it as a tool, at both the level of whole enterprise systems and at the narrow focus of a few lines of code, but it is one of the most powerful tools in our tool chest.

Deployability

So what does “releasable” mean? Inevitably that is somewhat contextual.

We’d certainly need to know that the code did what the developers thought it did, and then it would be good to know that it did what the users needed it to do. After that, we’d like to know if the software was fast enough, secure enough, resilient enough, and maybe compliant with any applicable regulations.

Actually, when describing deployment pipelines, I make a distinction between releasable and deployable. It is a subtle point, but from a development perspective, I want to separate the idea of being “ready to deploy a change into production” from “releasing a feature to users.”

In continuous delivery we want the freedom to create new features over a series of deployments. So at this point I am going to switch from talking about releasability, which implies some feature completeness and utility to users, to deployability, which means that the software is safe to release into production, even if some features are not yet ready for use and are hidden in some way.

This idea of deployability is an extremely useful tool at the system and architectural level. If the deployment pipeline says that the system is deployable, it is ready to be deployed into production.

Lots of people misunderstand this about continuous delivery, but this is what a deployment pipeline is for. If the deployment pipeline says that the change is good, there is no more testing to be done, no more sign-offs, and no further integration testing with other parts of the system before we deploy the change into production. We don’t have to deploy into production, but if the change was approved by the pipeline, it is ready, if we choose to.

The scope of our evaluation should always be an independently deployable unit of software. If we can’t confidently release our change into production without further work, then our unit of evaluation, the scope of our deployment pipeline, is incorrect.

Speed

When I consult with teams to help them adopt continuous delivery, I advise them to focus on working to reduce the time it takes to gain feedback.

I usually offer some guidelines: I tell them to work to optimize their development process so that they can achieve a releasable outcome, a production-quality deployable unit of software, multiple times per day, with a strong preference for shorter times. As a target, I generally recommend aiming to have something that you could deploy into production in less than one hour from the commit of any change.

Controlling the Variables

If we want to be able to quickly, reliably, and repeatably test and deploy our systems, we need to limit variance, and we need to control the variables. We want the same results every time that we deploy our software, so we need to automate the deployment and manage the configuration of the systems that we deploy as far as we are able to do so.

Where we can’t exert control, then we have to treat those margins of the system that touch on the uncontrolled world with great care. If we are deploying software to an environment outside of our control, we want to depend on it to the least degree that we can. Abstraction, separation of concerns, and loose coupling are key ideas to limit our exposure to anything outside of our direct control.

We want the tests that we create to give precisely the same results every time that we run them for the same version of the software under test. If test results vary, then we should work to exert greater control to better isolate the test from outside influences or to improve the determinism in our code. Modularity and cohesion, separation of concerns, abstraction, and coupling are yet again key ideas in allowing us to exert this control.

Reliably testable code is not multithreaded within the scope of a test, except for some very particular kinds of test.

Concurrent code is difficult to test because it is not deterministic. So if we design our code to be testable, we will think carefully about concurrency and work to move it to controlled, well-understood edges of our system.

Chapter 15: The Modern Software Engineer

One of my favorite models comes from Jan Bosch; he describes it as “BAPO versus OBAP.”1 Figure 15.1 and Figure 15.2 help to explain his idea.


Most firms follow an OBAP model (see Figure 15.1). They first fix the organization, departments, teams, responsibilities, and so on. Then they decide on a business strategy and how to generate revenue and profit or other business outcomes, based on the constraints of those organizational decisions. Next they decide on a suitable architecture to base their systems on, and finally on a process that can deliver that system architecture.

This is kind of crazy. The business vision and goals are constrained by organizational structure.

A more sensible model is to treat the structure of our organizations as a tool: BAPO.

We identify business vision and goals, decide how we could achieve that technically (architecture), figure out how we could build something like that (process), and then pick an organizational structure that will support the necessary activities.

1. Jan Bosch describes these ideas in his blog post “Structure Eats Strategy” at https://bit.ly/33GBrR1 and in his book Speed, Data and Ecosystems. See https://amzn.to/3x5Ef6T.