Accelerate The Science of Lean Software and DevOps: Building and Scaling High Performing Technology Organizations

Author: Gene Kim, Jez Humble, Nicole Forsgren Publishing/Release Date: March 1, 2018 Watched/Read: April 14, 2022 Link: https://www.amazon.com.au/Accelerate-Software-Performing-Technology-Organizations/dp/1942788339


Chapter 1 - Accelerate

To summarize, in 2017 we found that, when compared to low performers, the high performers have:

46 times more frequent code deployments 440 times faster lead time from commit to deploy 170 times faster mean time to recover from downtime 5 times lower change failure rate (1/5 as likely for a change to fail)

Chapter 2 - Measuring Performance

There have been many attempts to measure the performance of software teams. Most of these measurements focus on productivity. In general, they suffer from two drawbacks. First, they focus on outputs rather than outcomes. Second, they focus on individual or local measures rather than team or global ones.

A successful measure of performance should have two key characteristics. First, it should focus on a global outcome to ensure teams aren’t pitted against each other. Second, our measure should focus on outcomes not output: it shouldn’t reward people for putting in large amounts of busywork that doesn’t actually help achieve organizational goals.

In the design part of the lead time, it’s often unclear when to start the clock, and often there is high variability. For this reason, Reinertsen calls this part of the lead time the “fuzzy front end”

4 Key Metrics

Shorter product delivery lead times are better since they enable faster feedback on what we are building and allow us to course correct more rapidly. Short lead times are also important when there is a defect or outage and we need to deliver a fix rapidly and with high confidence.

The second metric to consider is batch size…batch size is hard to measure and communicate across contexts as there is no visible inventory. Therefore, we settled on deployment frequency as a proxy for batch size since it is easy to measure and typically has low variability.

Delivery lead times and deployment frequency are both measures of software delivery performance tempo. However, we wanted to investigate whether teams who improved their performance were doing so at the expense of the stability of the systems they were working on.

Traditionally, reliability is measured as time between failures. However, in modem software products and services, which are rapidly changing complex systems, failure is inevitable, so the key question becomes: How quickly can service be restored? We asked respondents how long it generally takes to restore service for the primary application or service they work on when a service incident (e.g., unplanned outage, service impairment) occurs, offering the same options as for lead time (above).

Finally, a key metric when making changes to systems is what percentage of changes to production (including, for example, software releases and infrastructure configuration changes) fail.

Tradeoffs (or not)

…there is no tradeoff between improving performance and achieving higher levels of stability and quality. Rather, high performers do better at all of these measures.

This is precisely what the Agile and Lean movements predict, but much dogma in our industry still rests on the false assumption that moving faster means trading off against other performance goals, rather than enabling and reinforcing them.

bliki: TradableQualityHypothesis

bliki: DesignStaminaHypothesis

Measuring organisational culture

“in pathological and bureaucratic organizational cultures, measurement is used as a form of control, and people hide information that challenges existing rules, strategies, and power structures. As Deming said, ’whenever there is fear, you get the wrong numbers’”

Chapter 3 - Measuring and Changing Culture

Westrum Model of Organisational Culture

• Pathological (power-oriented) organizations are characterized by large amounts of fear and threat. People often hoard information or withhold it for political reasons, or distort it to make themselves look better.
• Bureaucratic (rule-oriented) organizations protect departments. Those in the department want to maintain their “turf,” insist on their own rules, and generally do things by the book—their book.
• Generative (performance-oriented) organizations focus on the mission. How do we accomplish our goal? Everything is subordinated to good performance, to doing what we are supposed to do.

Westrum’s further insight was that the organizational culture predicts the way information flows through an organization. Westrum provides three characteristics of good information:

1. It provides answers to the questions that the receiver needs answered.
2. It is timely.
3. It is presented in such a way that it can be effectively used by the receiver.

Good information flow is critical to the safe and effective operation of high-tempo and high-consequence environments, including technology organizations. Westrum describes the characteristics of organizations that fall into his three types below.


Westrum’s theory posits that organizations with better information flow function more effectively.

…First, a good culture requires trust and cooperation between people across the organization, so it reflects the level of collaboration and trust inside the organization.

Second, better organizational culture can indicate higher quality decision-making. In a team with this type of culture, not only is better information available for making decisions, but those decisions are more easily reversed if they turn out to be wrong because the team is more likely to be open and transparent rather than closed and hierarchical.

Finally, teams with these cultural norms are likely to do a better job with their people, since problems are more rapidly discovered and addressed.

We hypothesized that culture would predict both software delivery performance and organizational performance. We also predicted that it would lead to higher levels of job satisfaction.

For modern organizations that hope to thrive in the face of increasingly rapid technological and economic change, both resilience and the ability to innovate through responding to this change are essential. Our research into the application of Westrum’s theory to technology shows that these two characteristics are connected. Initially developed to predict safety outcomes, our research shows it also predicts both software delivery and organizational performance. This makes sense, because safety outcomes are performance outcomes in a healthcare setting. By extending this to technology, we expected this type of organizational culture to positively impact software delivery and organizational performance. This mirrors research performed by Google into how to create high-performing teams.

Analysis showed that only lead time, release frequency, and time to restore together form a valid and reliable construct. Thus, in the rest of book, when we talk about software delivery performance it is defined using only the combination of those three metrics.

What is a latent construct?

A latent construct is a way of measuring something that can’t be measured directly. We can ask for the temperature of a room or the response time of a website—these things we can measure directly.

A good example of something that can’t be measured directly is organizational culture. We can’t take a team’s or an organization’s organizational culture “temperature”—we need to measure culture by measuring its component parts (called manifest variables), and we measure these component parts through survey questions. That is, when you describe organizational culture of a team to someone, you probably include a handful of characteristics. Those characteristics are the component parts of organizational culture. We would measure each (as manifest variables), and together they would represent a team’s organizational culture (the latent construct). And using survey questions to capture this data is appropriate, since culture is the lived experiences of those working on a team.

Google wanted to discover if there were any common factors among its best-performing teams… “who is on a team matters less than how the team members interact, structure their work, and view their contributions” (Google 2015). In other words, it all comes down to team dynamics.

How organizations deal with failures or accidents is particularly instructive. Pathological organizations look for a “throat to choke”: Investigations aim to find the person or persons “responsible” for the problem, and then punish or blame them. But in complex adaptive systems, accidents are almost never the fault of a single person who saw clearly what was going to happen and then ran toward it or failed to act to prevent it.

Thus, accident investigations that stop at “human error” are not just bad but dangerous. Human error should, instead, be the start of the investigation. Our goal should be to discover how we could improve information flow so that people have better or more timely information, or to find better tools to help prevent catastrophic failures following apparently mundane operations.

Chapter 4 - Technical Practices

What is continuous delivery?

Continuous delivery is a set of capabilities that enable us to get changes of all kinds—features, configuration changes, bug fixes, experiments—into production or into the hands of users safely, quickly, and sustainably. There are five key principles at the heart of continuous delivery:

• Build quality in
• Work in small batches
• Computers perform repetitive tasks; people solve problems
• Relentlessly pursue continuous improvement
• Everyone is responsible

In order to implement continuous delivery, we must create the following foundations:

• Comprehensive configuration management
• Continuous integration (CI)
  • High- performing teams keep branches short-lived, integrating with trunk/master frequently.
  • If the build is broken, the priority for the team is to fix it immediately.
• Continuous testing
  • Automated unit and acceptance tests should be run against every commit to version control to give developers fast feedback on their changes
  • Developers should be able to run all automated tests on their workstations in order to triage and fix defects.
  • Testers should be performing exploratory testing continuously against the latest builds to come out of CI.
  • Work is not “done” until relevant tests are written and passing.

Implementing continuous delivery means creating multiple feedback loops to ensure that high-quality software gets delivered to users more frequently and more reliably. When implemented correctly, the process of releasing new versions to users should be a routine activity that can be performed on demand at any time.

The impact of continuous delivery on teams

The research indicated that teams practicing continuous delivery experienced less deployment pain, and less burnout.

When teams practice CD, deployment to production is not an enormous, big-bang event—it’s something that can be done during normal business hours as a part of regular daily work.

If you want to improve your culture, implementing CD practices will help. By giving developers the tools to detect problems when they occur, the time and resources to invest in their development, and the authority to fix problems straight away, we create an environment where developers accept responsibility for global outcomes such as quality and stability.

Teams that did well at continuous delivery achieved the following outcomes:

• Strong identification with the organization you work for
• Higher levels of software delivery performance (lead time, deploy frequency, time to restore service)
• Lower change fail rates
• A generative, performance-oriented culture

Improvements in CD brought payoffs in the way that work felt. This means that investments in technology are also investments in people, and these investments will make our technology process more sustainable.

The impact of continuous delivery on quality

Unplanned work and rework are useful proxies for quality because they represent a failure to build quality into our products.

High performers reported spending 49% of their time on new work and 21% on unplanned work or rework. In contrast, low performers spend 38% of their time on new work and 27% on unplanned work or rework.

Continuous Delivery: Key capabilities

• Version control

  • Uncontroversial
  • Keep things like configuration and build scripts in version control as well as source code

• Test automation

  • Having automated tests that are reliable

    • Having a suite of reliable tests will give confidence when they pass, as well as being a genuine indicator of problems when they fail.
    • Conversely, unreliable tests are sometimes worse than no tests, as they may be producing false positives or negatives leading to either issues going unseen, or time wasted on non-issues.
      • Separate unreliable tests into a separate quarantine suite that is run independently.
      • Or just delete them.

  • Developers primarily create and maintain acceptance tests, and they can easily reproduce and fix them on their development workstations.

    • Having automated tests primarily created and maintained either by QA or an outsourced party is not correlated with IT performance.
    • When developers are the ones building and maintaining tests, two things happen:

      • The code becomes more testable, and a testable solution is more likely to remain maintainable for longer.

      • They will actually care about the tests, and invest time and effort in maintaining them.

    None of this means that we should be getting rid of testers. Testers serve an essential role in the software delivery lifecycle, performing manual testing such as exploratory, usability, and acceptance testing, and helping to create and evolve suites of automated tests by working alongside developers.

    Every commit should trigger a build of the software and running a set of fast, automated tests. Developers should get feedback from a more comprehensive suite of acceptance and performance tests every day. Furthermore, current builds should be available to testers for exploratory testing.

• Test data management

  • Successful teams had adequate test data to run their fully automated test suites and could acquire test data for running automated tests on demand.
  • Trunk-based development

    • Teams that did well had fewer than three active branches at any time, their branches had very short lifetimes (less than a day) before being merged into trunk and never had “code freeze” or stabilization periods.

    • The research showed that team size, org size and industry did not factor into this.

    Teams using branches that live a short amount of time (integration times less than a day) combined with short merging and integration periods (less than a day) do better in terms of software delivery performance than teams using longer-lived branches. Anecdotally, and based on our own experience, we hypothesize that this is because having multiple long-lived branches discourages both refactoring and intrateam communication.

• Information security

  High-performing teams were more likely to incorporate information security into the delivery process. Their infosec personnel provided feedback at every step of the software delivery lifecycle, from design through demos to helping with test automation. However, they did so in a way that did not slow down the development process, integrating security concerns into the daily work of teams. In fact, integrating these security practices contributed to software delivery performance.

Chapter 5 - Architecture

They present a list of different types of software that may be present in some form in an org:

• Greenfield: new systems that have not yet been released
• Systems of engagement (used directly by end users)
• Systems of record (used to store business-critical information where data consistency and integrity is critical)
• Custom software developed by another company
• Custom software developed in-house
• Packaged, commercial off-the-shelf software
• Embedded software that runs on a manufactured hardware device
• Software with a user-installed component (including mobile apps)
• Non-mainframe software that runs on servers operated by another company
• Non-mainframe software that runs on our own servers
• Mainframe software

Deployability and testability

The type of system being built is not important in terms of achieving high performance, but there are two architectural characteristics that are. Agreement with the following statements is an indicator of high performance:

• We can do most of our testing without requiring an integrated environment.
• We can and do deploy or release our application independently of other applications/services it depends on.

To achieve these characteristics, systems must be designed intentionally to be loosely coupled — that is, they can be changed and validated independently of one another.

The biggest contributor to continuous delivery in the 2017 analysis—larger even than test and deployment automation—is whether teams can:

• Make large-scale changes to the design of their system without the permission of somebody outside the team
• Make large-scale changes to the design of their system without depending on other teams to make changes in their systems or creating significant work for other teams
• Complete their work without communicating and coordinating with people outside their team
• Deploy and release their product or service on demand, regardless of other services it depends upon
• Do most of their testing on demand, without requiring an integrated test environment
• Perform deployments during normal business hours with negligible downtime

In teams which scored highly on architectural capabilities, little communication is required between delivery teams to get their work done, and the architecture of the system is designed to enable teams to test, deploy, and change their systems without dependencies on other teams.

“organizations which design systems … are constrained to produce designs which are copies of the communication structures of these organizations” - Melvin Conway

Mentions “inverse conway manouver” — that is, reshaping team and org structure in order to drive the desired architecture.

Architectural approaches that enable this strategy include the use of bounded contexts and APIs as a way to decouple large domains into smaller, more loosely coupled units, and the use of test doubles and virtualization as a way to test services or components in isolation.

A loosely coupled architecture enables scaling

A loosely coupled, well defined architecture is firstly a key in unlocking better delivery performance, but it is also an enabler for greater org scaling, and by extension, better org productivity.

Org productivity can be calculated based on the number of deploys per day per developer. A commonly held belief is that while adding developers to a team might increase overall productivity, productivity of individual developers suffers due to increased overhead in communication and integration.

The research indicates that even this is not true. Mid-level performers deploy at a mostly constant frequency, and low performers see a net decrease in deployment frequency. Meanwhile, high performers enjoy continually increasing deployment frequency.


By focusing on the factors that predict high delivery performance—a goal-oriented generative culture, a modular architecture, engineering practices that enable continuous delivery, and effective leadership—we can scale deployments per developer per day linearly or better with the number of developers. This allows our business to move faster as we add more people, not slow down, as is more typically the case.

Allow teams to choose their own tools

Standardisation of developer tools is common in many organisations. It is usually done for one or more of the following reasons:

• Reducing the complexity of the environment
• Ensuring the necessary skills are in place to manage the technology throughout its lifecycle
• Increasing purchasing power with vendors
• Ensuring all technologies are correctly licensed

This lack of flexibility around tool choice has some downsides: teams are unable to pick the technologies that best suit their needs. It also prevents experimentation with new approaches to their problems.

When teams can decide which tools they use, it contributes to software delivery performance and, in turn, to organizational performance.

That said, there is a place for standardization, particularly around the architecture and configuration of infrastructure. The benefits of a standardized operational platform are discussed at length by Humble (2017). Another example is Steve Yegge’s description of Amazon’s move to an SOA, in which he notes, “Debugging problems with someone else’s code gets a LOT harder, and is basically impossible unless there is a universal standard way to run every service in a debuggable sandbox” (Yegge 2011).

Another finding in our research is that teams that build security into their work also do better at continuous delivery. A key element of this is ensuring that information security teams make preapproved, easy-to-consume libraries, packages, toolchains, and processes available for developers and IT operations to use in their work.

Architects should focus on engineers and outcomes, not tools or technologies

What tools or technologies you use is irrelevant if the people who must use them hate using them, or if they don’t achieve the outcomes and enable the behaviors we care about. What is important is enabling teams to make changes to their products or services without depending on other teams or systems. Architects should collaborate closely with their users—the engineers who build and operate the systems through which the organization achieves its mission—to help them achieve better outcomes and provide them the tools and technologies that will enable these outcomes.

Chapter 6 - Integrating Infosec into the Delivery Lifecycle

Shifting Left On Security

We found that when teams “shift left” on information security— that is, when they build it into the software delivery process instead of making it a separate phase that happens downstream of the development process—this positively impacts their ability to practice continuous delivery. This, in turn, positively impacts delivery performance.

What does “shifting left” entail? First, security reviews are conducted for all major features, and this review process is performed in such a way that it doesn’t slow down the development process.

…infosec experts should contribute to the process of designing applications, attend and provide feedback on demonstrations of the software, and ensure that security features are tested as part of the automated test suite. Finally, we want to make it easy for developers to do the right thing when it comes to infosec. This can be achieved by ensuring that there are easy-to-consume, preapproved libraries, packages, toolchains, and processes available for developers and IT operations.

We found that high performers were spending 50% less time remediating security issues than low performers. In other words, by building security into their daily work, as opposed to retrofitting security concerns at the end, they spent significantly less time addressing security issues.

Rugged Software

Chapter 7 - Management Practices for Software

Lean Management Practices

Limit Work In Process (WIP). Limiting WIP on its own is not a predictor of high performance, it is a tool used to expose obstacles to work flowing efficiently through the process. The goal should then be to improve the process to remove these obstacles.

Visual feedback of metrics from production should also be used. Things like response times, failure/error rates, throughput should all be presented to the team in a way that enables understanding at a glance what the current health of the system is.

Implement a Lightweight Change Management Process

…approval by an external body (such as a manager or CAB) simply doesn’t work to increase the stability of production systems, measured by the time to restore service and change fail rate. However, it certainly slows things down. It is, in fact, worse than having no change approval process at all.

Our recommendation based on these results is to use a lightweight change approval process based on peer review, such as pair programming or intrateam code review, combined with a deployment pipeline to detect and reject bad changes. This process can be used for all kinds of changes, including code, infrastructure, and database changes.

Segregation of Duties

In regulated industries, segregation of duties is often required either explicitly in the wording of the regulation (for instance, in the case of PCI DSS) or by auditors. However, implementing this control does not require the use of a CAB or separate operations team. There are two mechanisms which can be effectively used to satisfy both the letter and the spirit of this control.

First, when any kind of change is committed, somebody who wasn’t involved in authoring the change should review it either before or immediately following commit to version control. This can be somebody on the same team. This person should approve the change by recording their approval in a system of record such as GitHub (by approving the pull request) or a deployment pipeline tool (by approving a manual stage immediately following commit).

Second, changes should only be applied to production using a fully automated process that forms part of a deployment pipeline. That is, no changes should be able to be made to production unless they have been committed to version control, validated by the standard build and test process, and then deployed through an automated process triggered through a deployment pipeline.

This idea (approval by external bodies) is a form of risk management theater: we check boxes so that when something goes wrong, we can say that at least we followed the process. At best, this process only introduces time delays and handoffs.

Chapter 8 - Product Development

Lean Product Management Approaches

Working in small batches

The key to working in small batches is to have work decomposed into features that allow for rapid development, instead of complex features developed on branches and released infrequently.

…Working in small batches enables short lead times and faster feedback loops.

…the capability to work and deliver in small batches is especially important because it allows you to gather user feedback quickly using techniques such as A/B testing. It’s worth noting that an experimental approach to product development is highly correlated with the technical practices that contribute to continuous delivery.

Gathering customer feedback includes multiple practices: regularly collecting customer satisfaction metrics, actively seeking customer insights on the quality of products and features, and using this feedback to inform the design of products and features. The extent to which teams actually have the authority to respond to this feedback also turns out to be important.

Team experimentation

…if a development team isn’t allowed, without authorization from some outside body, to change requirements or specifications in response to what they discover, their ability to innovate is sharply inhibited.

…the ability of teams to try out new ideas and create and update specifications during the development process, without requiring the approval of people outside the team, is an important factor in predicting organizational performance as measured in terms of profitability, productivity, and market share.

To be effective, experimentation should be combined with the other capabilities we measure here: working in small batches, making the flow of work through the delivery process visible to everyone, and incorporating customer feedback into the design of products. This ensures that your teams are making well-reasoned, informed choices about the design, development, and delivery of work, and changing it based on feedback. This also ensures that the informed decisions they make are communicated throughout the organization. That increases the probability that the ideas and features they build will deliver delight to customers and add value to the organization.

Lean product management and software delivery performance are interconnected. The research shows that lean product management positively influences software delivery performance, stimulates a generative culture, and decreases burnout. Flipping the model, the research also showed that software delivery performance is a driver of lean product management practices. Improving software delivery capabilities is an enabler for working in smaller batches and performing user research that influences product direction. This is a reciprocal model, or colloquially, a virtuous cycle.

Chapter 9 - Making Work Sustainable

Deployment pain

(The research indicated a) high correlation between deployment pain and key outcomes: the more painful code deployments are, the poorer the IT performance, organizational performance, and organizational culture.

Fundamentally, most deployment problems are caused by a complex, brittle deployment process. This is typically the result of three factors. First, software is often not written with deployability in mind. A common symptom here is when complex, orchestrated deployments are required because the software expects its environment and dependencies to be set up in a very particular way and does not tolerate any kind of deviation from these expectations, giving little useful information to administrators on what is wrong and why it is failing to operate correctly. (These characteristics also represent poor design for distributed systems.)

Second, the probability of a failed deployment rises substantially when manual changes must be made to production environments as part of the deployment process.

Finally, complex deployments often require multiple handoffs between teams, particularly in siloed organizations where database administrators, network administrators, systems administrators, infosec, testing/QA, and developers all work in separate teams.

In order to reduce deployment pain, we should:

• Build systems that are designed to be deployed easily into multiple environments, can detect and tolerate failures in their environments, and can have various components of the system updated independently
• Ensure that the state of production systems can be reproduced (with the exception of production data) in an automated fashion from information in version control
• Build intelligence into the application and the platform so that the deployment process can be as simple as possible

Burnout

Managers who want to avert employee burnout should concentrate their attention and efforts on:

• Fostering a respectful, supportive work environment that emphasizes learning from failures rather than blaming
• Communicating a strong sense of purpose
• Investing in employee development
• Asking employees what is preventing them from achieving their objectives and then fixing those things
• Giving employees time, space, and resources to experiment and learn

Last but not least, employees must be given the authority to make decisions that affect their work and their jobs, particularly in areas where they are responsible for the outcomes.

Common problems that can lead to burnout

Six organizational risk factors:

1. Work overload: job demands exceed human limits.
2. Lack of control: inability to influence decisions that affect your job.
3. Insufficient rewards: insufficient financial, institutional, or social rewards.
4. Breakdown of community: unsupportive workplace environment.
5. Absence of fairness: lack of fairness in decision-making processes.
6. Value conflicts: mismatch in organizational values and the individual’s values.

How to reduce or fight burnout

Organisational factors highly correlated with burnout:

1. Organizational culture. Strong feelings of burnout are found in organizations with a pathological, power-oriented culture. Managers are ultimately responsible for fostering a supportive and respectful work environment, and they can do so by creating a blame-free environment, striving to learn from failures, and communicating a shared sense of purpose. Managers should also watch for other contributing factors and remember that human error is never the root cause of failure in systems.
2. Deployment pain. Complex, painful deployments that must be performed outside of business hours contribute to high stress and feelings of lack of control. With the right practices in place, deployments don’t have to be painful events. Managers and leaders should ask their teams how painful their deployments are and fix the things that hurt the most.
3. Effectiveness of leaders. Responsibilities of a team leader include limiting work in process and eliminating roadblocks for the team so they can get their work done. It’s not surprising that respondents with effective team leaders reported lower levels of burnout.
4. Organizational investments in DevOps. Organizations that invest in developing the skills and capabilities of their teams get better outcomes. Investing in training and providing people with the necessary support and resources (including time) to acquire new skills are critical to the successful adoption of DevOps.
5. Organizational performance. Our data shows that Lean management and continuous delivery practices help improve software delivery performance, which in turn improves organizational performance. At the heart of Lean management is giving employees the necessary time and resources to improve their own work. This means creating a work environment that supports experimentation, failure, and learning, and allows employees to make decisions that affect their jobs. This also means creating space for employees to do new, creative, value-add work during the work week—and not just expecting them to devote extra time after hours. A good example of this is Google’s 20% time policy, where the company allows employees 20% of their week to work on new projects.

Chapter 10 - Employee Satisfaction, Identity, and Engagement

…employees in high-performing organizations were 2.2 times more likely to recommend their organization as a great place to work, and other studies have also shown that this is correlated with better business outcomes

We found that the employee Net Promoter Score (eNPS) was significantly correlated with the following constructs:

• The extent to which the organization collects customer feedback and uses it to inform the design of products and features
• The ability of teams to visualize and understand the flow of products or features through development all the way to the customer
• The extent to which employees identify with their organization’s values and goals, and the effort they are willing to put in to make the organization successful

As we demonstrated in Chapter 8, when employees see the connection between the work they do and its positive impact on customers, they identify more strongly with the company’s purpose, which leads to better software delivery and organizational performance.

Changing organisational culture and identity

When leaders invest in their people and enable them to do their best work, employees identify more strongly with the organization and are willing to go the extra mile to help it be successful. In return, organizations get higher levels of performance and productivity, which lead to better outcomes for the business.

…investments in continuous delivery and Lean management practices, which contribute to a stronger sense of identity, may very well help reduce burnout. Once again, this creates a virtuous circle of value creation in the business where investments in technology and process that make the work better for our people are essential for delivering value for our customers and the business.

…the best thing you can do for your products, your company, and your people is institute a culture of experimentation and learning, and invest in the technical and management capabilities that enable it.

How does DevOps contribute to job satisfaction

Looking at the measures that correlate strongly with job satisfaction, we see some commonalities. Practices like proactive monitoring and test and deployment automation all automate menial tasks and require people to make decisions based on a feedback loop. Instead of managing tasks, people get to make decisions, employing their skills, experience, and judgment.

Chapter 11 - Leaders and Managers

Being a leader doesn’t mean you have people reporting to you on an organizational chart—leadership is about inspiring and motivating those around you.

What is transformational leadership?

…transformational leadership is essential for:

• Establishing and supporting generative and high-trust cultural norms
• Creating technologies that enable developer productivity, reducing code deployment lead times and supporting more reliable infrastructures
• Supporting team experimentation and innovation, and creating and implementing better products faster
• Working across organizational silos to achieve strategic alignment

To capture transformational leadership, we used a model that includes five dimensions (Rafferty and Griffin 2004). According to this model, the five characteristics of a transformational leader are:

• Vision. Has a clear understanding of where the organization is going and where it should be in five years.
• Inspirational communication.Communicates in a way that inspires and motivates, even in an uncertain or changing environment.
• Intellectual stimulation. Challenges followers to think about problems in new ways.
• Supportive leadership. Demonstrates care and consideration of followers’ personal needs and feelings.
• Personal recognition. Praises and acknowledges achievement of goals and improvements in work quality; personally compliments others when they do outstanding work.

High-performing teams reported having leaders with the strongest behaviors across all dimensions: vision, inspirational communication, intellectual stimulation, supportive leadership, and personal recognition. In contrast, low-performing teams reported the lowest levels of these leadership characteristics.

…though we often hear stories of DevOps and technology transformation success coming from the grassroots, it is far easier to achieve success when you have leadership support.

…leadership helps build great teams, great technology, and great organizations—but indirectly, leadership enables teams to rearchitect their systems and implement the necessary continuous delivery and Lean management practices.

The role of managers

Managers can do a lot to improve their team’s performance by creating a work environment where employees feel safe, investing in developing the capabilities of their people, and removing obstacles to work.

When it comes to culture, managers can improve matters by enabling specific DevOps practices in their teams and by visibly investing in DevOps and in their employees’ professional development.

…managers should make performance metrics visible and take pains to align these with organizational goals, and should delegate more authority to their employees.

What could investment in DevOps initiatives and my teams look like? There are a number of ways technology leaders can invest in their teams:

• Ensure that existing resources are made available and accessible to everyone in the organization. Create space and opportunities for learning and improving.
• Establish a dedicated training budget and make sure people know about it. Also, give your staff the latitude to choose training that interests them. This training budget may include dedicated time during the day to make use of resources that already exist in the organization.
• Encourage staff to attend technical conferences at least once a year and summarize what they learned for the entire team.
• Set up internal hack days, where cross-functional teams can get together to work on a project.
• Encourage teams to organize internal “yak days,” where teams get together to work on technical debt. These are great events because technical debt is so rarely prioritized.
• Hold regular internal DevOps mini-conferences. We’ve seen organizations achieve success using the classic DevOpsDays format, which combines pre-prepared talks with “open spaces” where participants self-organize to propose and facilitate their own sessions.
• Give staff dedicated time, such as 20% time or several days after a release, to experiment with new tools and technologies. Allocate budget and infrastructure for special projects.

Tips to improve culture and support your teams

As the real value of a leader or manager is manifest in how they amplify the work of their teams, perhaps the most valuable work they can do is growing and supporting a strong organizational culture among those they serve: their teams.

Enable cross-functional collaboration by:

• Building trust with your counterparts on other teams.
• Encouraging practitioners to move between departments.
• Actively seeking, encouraging, and rewarding work that facilitates collaboration.

Use Disaster Recovery Testing exercises to build relationships

Many large technology companies run disaster recovery testing exercises, or “game days,” in which outages are simulated or actually created according to a pre-prepared plan, and teams must work together to maintain or restore service levels.

Kripa Krishnan, Director of Cloud Operations at Google, runs a team that plans and executes these exercises. She reports, “For DiRT-style events to be successful, an organization first needs to accept system and process failures as a means of learning“.

Help create a climate of learning by:

• Creating a training budget and advocating for it internally.
• Ensuring that your team has the resources to engage in informal learning and the space to explore ideas.
• Making it safe to fail.
• Creating opportunities and spaces to share information.
• Encourage sharing and innovation by having demo days and forums.

Make effective use of tools:

• Make sure your team can choose their tools.

teams that are empowered to choose their own tools drive software delivery performance

• Make monitoring a priority.

Proactive monitoring was strongly related to performance and job satisfaction in our survey, and it is a key part of a strong technical foundation.

While many DevOps success stories highlight the fantastic grassroots efforts of the technical teams involved, our experience and our research shows that technology transformations benefit from truly engaged and transformational leaders who can support and amplify the work of their teams.