At SPAN we work with many organizations that employ Agile at enterprise scale. Despite the many differences between these organizations, their problems are surprisingly similar. Effective software development at scale is challenging to establish and even more challenging to maintain.

The Symptoms

Engineers within large agile organizations are frustrated. Making decisions at a strategic, or even tactical, level is hampered by poor situational awareness. Individuals and teams find it difficult to identify the most important issues in their area of concern, let alone the solution in general. Comprehending the interactions between issues is even harder, and making projections about future states based on their best current assessment is a black art which only the most experienced, and long-serving, employees can manage effectively.

When an organization has more than a few self-organizing agile teams collaborating on a single offering, it becomes difficult to produce a coherent release. Trade shows, earnings calls, and competitors offerings demand a coordinated response, but, with each team focused on different objectives and continuously “distracted” by urgent patches, and emergency releases, predictable release scheduling becomes a near impossible task.

Onboarding new employees takes months when internal systems are an obscure alphabet soup of acronyms with no high-level map of the overall solution. When vital operational information is mixed with outdated and misleading content in unstructured wikis, new employees struggle to get traction. Favoring working code over documentation is excellent for a single, fast-moving, team working on a new solution, but when the overall system is spread across many poorly documented code repo’s that have evolved over many years, finding the information you need becomes an archaeological expedition.

I am frequently asked “Is everyone this messed up?” to which I answer “No, only the most successful companies are this messed up! It takes time and effort to get like this!” Few organizations have a solution to this problem. Indeed, what many perceive as a problem may be the unavoidable price of enterprise agility, steady growth, and staff turnover. Rapid adaptation to emerging needs arises from a steady churn. Evolution proceeds through a series of wasteful experiments. However, there may be a way to reduce the pain. Many of these inefficiencies have the same root cause.

The Root Cause

The Agile Enterprise has evolved to exploit the advantages of agile development. Large organizations have sub-divided into agile teams that each independently build and operate their portion of the solution. This approach is susceptible to problems that build slowly over time, gradually becoming chronic. Technical debt accumulates, crucial knowledge evapourates, and teams struggle to adapt their solutions to a continually coevolving environment. Developers must continually re-educate themselves and their users. All these problems take time to emerge they are a function of enterprise longevity, organizational growth, and staff turnover.

The Agile Manifesto famously stated:

We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value:

• Individuals and interactions over processes and tools
• Working software over comprehensive documentation
• Customer collaboration over contract negotiation
• Responding to change over following a plan

That is, while there is value in the items on the right, we value the items on the left more.

With the experience of enterprise-scale agile, we can see that the Agile Manifesto is a small-scale solution that does not solve long-term problems. Specifically, the Agile Manifest does not solve the problem of staff turnover. Software Engineers typically stay in their jobs for 2 to 3 years before moving on. Most software organizations experience something like a +15% turnover per year. Agile projects lasting 6 to 9 months can often avoid turnover issues. Engineers consumed by a project tend not to look around for new opportunities. Together, these factors mean the agile manifesto works well for single teams on single projects, but in an organization with many teams and projects, the manifesto is itself the cause of long-term problems.

Valuing “Individuals and interactions over processes and tools” is fine if the individuals stick around. When individuals move on it is often only the processes and tools that remain.

Valuing “Working software over comprehensive documentation” is great if detailed knowledge of the software is already in the heads of the developers. However, when developers move on, the knowledge goes with them, all that remains is the code. Software inevitably breaks, leaving broken code with no documentation, which is the worst possible combination. Reverse engineering a complicated piece of broken code to fix a defect, with little understanding of what the code was originally intended to do, is fraught with peril.

Valuing “Customer collaboration over contract negotiation” works as long as the customers and the software developers are around to keep each other honest. When either party moves on it becomes hard to honor half-remembered agreements that were never written down. While negotiating contracts is seldom fruitful, some effort to document what we were trying to produce can avoid many subsequent misinterpretations.

“Responding to change over following a plan” is an act of faith that turns to hubris as the challenge scales. For a single skilled team, it can be justified. However, if ten-plus agile teams are trying to coordinate a coherent product release in time for next year’s trade show, trusting to responsiveness will inevitably lead to teams thrashing as they all try to adapt to each other’s changes. To be successful a project with so many moving parts requires a strategy for resolving conflicting demands. Agreed goals and interim objectives, clear ownership of responsibilities, internal commitments to deliverables and dates, and a robust escalation procedure to resolve the inevitable problems is one way to manage such conflict. Internal marketplaces where resources can be traded is another.

All this is not to say the Agile Manifesto is wrong. However, it is insufficient for large organizations with many teams. It is missing mechanisms to harvest, improve and transfer knowledge between employees as they join and ultimately leave the organization. Institutional memory is crucial to the success of enterprises that aim to survive for the medium to long-term.

Example 1: 5% growth per quarter

Starting from 100 employees and growing at a modest 5% per quarter the company will reach about 325 employees in 6 years, by that time:

• 30% have less than 9 months experience
• 21% have more than 9 but less than 18 months experience
• 25% have more than 18 but less than 36 months experience
• Less than 25% have more than 36 months experience

Example 2: 10% growth per quarter

Starting from 100 employees and growing aggressively at a rate of 10% per quarter the company will reach about 1000 employees in 6 years, by that time:

• 39% have less than 9 months experience
• 24% have more than 9 but less than 18 months experience
• 23% have more than 18 but less than 36 months experience
• Less than 14% have more than 36 month experience

The more aggressively you grow the worse the knowledge management problem becomes. Without an effective plan to harvest and redistribute knowledge, aggressive growth companies can easily forget how they became successful in the first place.

Organizations spend a lot of time and resources developing knowledge and capability. While some of it gets translated into procedures and policies, most of it resides in the heads, hands, and hearts of individual managers and functional experts. Over time, much of this institutional knowledge moves away as people take on new jobs, relocate, or retire. Knowledge also degrades when a new senior executive or CEO introduces a different agenda that doesn’t build on earlier knowledge, or contradicts what was done previously. And knowledge disappears even more rapidly when a firm reorganizes or merges with another and there is a subsequent reshuffling of the cast of characters.

How to Preserve Institutional Knowledge. by Ron Ashkenas

The Solution

Preserving institutional knowledge is essential if organizations want to retain agility and velocity as they scale. Knowledge lives in the heads of people; it is generated from information that is itself created by processing discrete, unorganized objective facts or observations.

In Enterprises, knowledge becomes embedded in static representations like documents, and wikis. Knowledge also becomes embedded in dynamic systems, processes, and working practices. Managing the transfer of knowledge from peoples heads to these static and dynamic representations and back again is essential for knowledge to persist as people move through the organization and the organization grows.

Institutional knowledge management requires an explicit strategy that identifies a few key areas of focus. Organizations should not attempt to retain or control all knowledge; institutional learning is as much about retiring outdated information, as it is about capturing and retaining new information. Knowledge turnover is essential for organizational learning. By identifying a few key focus areas, organizations can ensure they retain knowledge in vital areas while allowing new knowledge to replace outdated thinking.

Enforcing knowledge management top down has a poor track record. Instead, it must be encouraged to emerge bottom up, as the people who directly benefit either share or consume valuable information. Organizations must incentivize these individual transactions if they wish to ensure the emergent benefits.

To retain knowledge, information must be acquired, revised, classified, utilized, and repeatedly evaluated by the people who use it. Providing recognition and rewards to staff who undertake these activities is essential. If knowledge management is not recognized as valuable, it will not happen. The problem with the Agile Manifesto is that it’s only solution for knowledge transfer is either word of mouth or code. In reality, there are other, often more effective mechanisms for capturing and transferring knowledge. Large enterprises must use all these methods if they are to learn and endure.