These appendices are not essential readings to understand and use Amplio. In fact, are intended to be stand-alone articles that address issues that many Agile coaches face in their work.

Waterfall Is Never the Right Approach

I often hear that there are cases where Waterfall is best. I don’t agree.

Let me first point out, however, that the choice is not between waterfall and Agile. There are other approaches. There are also multiple aspects to consider:

do we know the steps we need to take to accomplish our task?
are all the people and capabilities we need to accomplish the task readily at hand?
what is the chance of making an error?

I don’t believe adopting waterfall as an approach is ever a good choice. Waterfall comes with the following mindset:

we don’t need feedback between the steps involved. In knowledge work this is often between analysis, design, code, and validation
we can hand work off with little or no cost
big batches are ok since they enable us to be more efficient
specialized skills working only on their specialty is good
we can understand the work to be done before we do it
written requirements can specify what we need

Going through each of these:

1) we don’t need feedback between the steps involved. In knowledge work, evidence shows we do so that we can correct our misunderstandings. In manufacturing, or set process type work, the feedback is useful to detect when an error has taken place. In both cases not getting feedback creates waste. We understand requirements better when we do analysis. We understand analysis while in design… Interestingly enough. Royce’s original paper said waterfall wouldn’t work. People liked the simplistic model he presented instead of the one with feedback cycles built-in.

2) we can hand work off with little to no cost. When we hand work off, we lose a lot of information. We either have to re-learn that information or, more likely; we go with misunderstandings.

3) big batches are ok since they enable us to be more efficient. This has been proven inaccurate in many industries and is the heart of Lean. Eli Goldratt said it best – “Often reducing batch size is all it takes to bring a system back into control.” Waterfall does little to have us work on small batches – a vital thing to bring value to market quickly.

4) specialized skills working only on their specialty is good. Synergy is good. Cross-functional teams, when possible, are good.

5) we can understand the work before we do it. I hear a common refrain that “our customers know what they want after we show them what they don’t want.” This is because requirements elicitation is a discovery process.

6) written requirements can specify what we need. I’ll leave this to people’s experiences, and one of my favorite Churchill quotes – “This report, by its very length, defends itself against the risk of being read.”

In other words, the waterfall mindset is just not accurate. I would suggest where waterfall worked, people went beyond this mindset and did something else. For those who say, “Well, no one does pure waterfall, ” I suggest looking at what they did outside the waterfall that worked.

What Can We Do?

In knowledge work, we first need to realize that we should not be trying to do Agile. We should be trying to be as agile as possible – increasing our agility as an organization. This means:

focusing on delivering business value quickly by selecting the most important things of value (this typically means smaller chunks to be delivered)
having proper team structures to get the job done
removing delays in workflow by avoiding working on too many things (work on the most important ones without exceeding your capacity)
shortening feedback cycles is always a good thing to get clarity on where you are and to detect errors quickly

There are more, of course. But I’ve picked these because these are things we can always do to some extent.

In manufacturing or processes where we understand what is to be done, Lean thinking helps by detecting errors quickly and continuously learning how to do things better.

One other note. We should remember that waterfall is to Agile as mass manufacturing is to lean manufacturing. In manufacturing, we know all of the steps to do but working in small batches with feedback is still better than big batches with testing at the end.

Why Lean-Agile Should Be More Predictable Than Waterfall

Many executives have been led to believe that Agile is inherently less predictable than a waterfall approach. However, when wrapped in Lean thinking, Agile can be more predictable because it enables working directly on the true causes of unpredictability in software development. Waterfall’s large projects and stage-gate approach cause delays in feedback, workflow, testing, and integration. These delays inherently create a significant amount of rework (redoing requirements, reworking code that missed requirements, finding bugs, thrashing during integration). This work, of course, is never planned for and therefore results in inherently bad estimation, not to mention that eliminating this extra work improves productivity.

In addition, Lean-Agile methods have us focus on smaller, higher-density pieces of work. This alone helps bring our systems under control, as observed by Eli Goldratt – “Often reducing batch size is all it takes to bring a system back into control.”

The key to having predictable results is eliminating delays, creating unplanned work. This requires smaller batches of work that are appropriately sequenced in importance, properly formed teams (or groups of teams), and solid technical practices. Agile suggests additional methods such as Acceptance Test-Driven Development, automated testing, and continuous integration. These directly improve understanding of requirements (through quick feedback), code quality, and risk reduction.

Although waterfall and Agile approaches have different mindsets, some of Agile’s methods can be incorporated into a waterfall process. For example, making both projects and planning cycles shorter. Improving methods will not make an organization Agile, but it may help set the stage for a true transformation.

I’ve heard it said that “we can’t go Agile because we need predictability.” This is an understandable sentiment because many Agilists have dismissed predictability as impossible. However, there are two camps in the Agile community – one that is team-centric and another that wraps Agile with Lean thinking (we call this Lean-Agile). A team-centric approach will often not achieve predictability when applied at scale because the team is only one part of the value-add cycle. But a Lean approach is holistic, and although it may not be possible to predict precisely what must be built, it is possible to predict how much can be built. With proper selection of the work to be done,

However, more than simple iterations are required to achieve this. Lean-Agile should not be just a mindset shift but must incorporate solid product management and technical practices. Let’s consider why projects (regardless of approach) often do not achieve the predictions they make. The most discussed are:

what the customer wants is not understood
the time to build something is unclear
the code is fragile and hard to change

There’s an even bigger one lurking here (which I’ll discuss a little later) but for now, let’s look at these.

What the customer wants is not understood. Customers often don’t know what they want until they see what they don’t want. This implies they should be shown “what they don’t want” as soon as possible. This is why it’s important to build small segments of work quickly. This also shortens the time between all stages of the work (requirements, design, code, test, integration). This combination of quick feedback with coding and testing being done together enables quick course corrections as both the customer and the development group learns what is truly of value.

Acceptance Test-Driven Development (having customers, developers and testers specify requirements in the form of test specifications) achieves better clarity on what is needed and avoids many misunderstandings altogether. Getting the customer to provide requirements in this precise manner changes the nature of the requirements. Vagueness is no longer allowed, and customers’ understanding of what they are looking for improves.

The time to build something is unclear. The answer to this is again to go small. Estimating the time it will take to complete a large chunk of work is typically less accurate than estimating the time for smaller things. Some people have ridiculed estimation because they are often wildly inaccurate, and managers sometimes abuse the teams for that. But bad estimates are more of a symptom of other issues – unclear requirements, poor code quality, and even fear of management. But avoiding estimation abandons looking at the actual underlying reasons.

The code is fragile and hard to change. Poor code quality will slow things down, but this does not have to be the path taken. When code is fragile and is difficult to change, it will not be possible to accurately predict how long things will take, even for small changes. But that’s because of the poorly written code, which is both difficult and risky to change.

Waterfall’s approach addresses this issue by doing all coding first and then testing afterward. There is often a perceived need to do this because of the need to batch test. But this doesn’t get at the issue (poor code quality). Accommodating poor code quality will not improve it but tends to get the code in a slow death spiral that will eventually need to be re-written.

Poor code can be improved with automated acceptance tests. The time test automation saves is only part of its value. It also enables refactoring of the code in safety because defects introduced will be detected quickly. Developers fix these bugs much faster because they are detected while still fresh in their minds. This is a considerable time-saving. Most developers think they spend a lot of time fixing bugs. But on reflection, they see that most of their time is spent finding the bug. This is not mere semantics. Bugs detected immediately are fixed much faster than bugs detected weeks later (even when the code hasn’t changed).

The Common Themes

The first common theme here is that smaller batches of work allow for greater efficiency by providing quicker feedback, better estimation, and shortening the delays between requirements, code, and testing. This efficiency is due to the fourth issue I earlier mentioned I was going to discuss. This is because delays in feedback, workflow (e.g., waiting for people), and detecting errors will cause additional work.

It is common to focus on how to work faster or better. But it is more effective to discover how to avoid creating the need for rework. Although much rework is due to a misunderstanding, having quick feedback significantly lowers the amount of rework required. Consider how much time is spent on redoing requirements, building features of less value than initially thought, fixing bugs, manual testing, thrashing during integration, …. It should be apparent a significant amount of work done in projects was created by the aforementioned delays. The focus must shift from doing things faster to avoiding delays between the work steps and achieving fast feedback.

Why Waterfall Will Inherently Provide Bad Estimates

Consider the waterfall planning process. Larger chunks of work are planned to be done in separate steps – requirements, design, code, functional test, integration, and system test. This exacerbates the aforementioned issues by increasing delays in feedback between each step and overall feedback achievable at completion. This creates a significant of work due to these delays. However, how much of this time is planned for at the start of the project?

None. Zero. Nada.

Estimates in waterfall don’t consider the work that derives from delays – work that often takes as much or more as the planned work. Planning sessions typically don’t include the question, “how much time should we set aside for missed or wrong requirements, pitfalls in our design, or all of the extra time it’ll take to find the bugs because testing was delayed?” In other words, one reason estimates are bad is that they do not include a significant amount of the work that needs to be done.

So Why Should Lean-Agile Be More Predictable?

Lean-Agile takes a different approach with requirements, design and coding, and testing. Working in small batches and continuously course-correcting reduces time spent correcting misunderstood requirements. However, technical methods are still needed. Test-first and automated testing also results in better code. Considering writing tests before writing code is an important lesson from design patterns (although not stated explicitly in the iconic Gang of Four Patterns book, both Design Patterns Explained and Essential Skills for the Agile Developer describe this in-depth). And finally, by using smaller batches of work, the time between code and test decreases, thereby lowering the finding time even more.

Another difference exists in the approach to architecture. In waterfall, developers are supposed to design for the stated requirements. However, they have learned that requirements will change and tend to over-design to accommodate this. On the other hand, Lean-Agile suggests designing to accommodate change. This is known as emergent design.

An Extra Benefit of Lean-Agile

Building things in small batches results in learning faster about what is needed and enables pivoting as required. Feedback from the customer uncovers the actual value of each feature or story. Furthermore, customers learn that they don’t need to ask for everything upfront. Instead, they can adjust their requirements as they see what’s being built and better understand their needs.

Waterfall, oddly enough, encourages customers to ask for everything upfront because little justification is required other than “I need it” when initial requirements are stated. Anything specified later, however, must go through a change review board. Customers quickly learn to ask for whatever they might need up-front. Lean-Agile teaches customers they only need to specify what they need soonest and that they can adjust as they learn more. While a general understanding of the entire system is required to enable an estimate for building, only the general size of the work is needed. The details will come later as more is learned. Accurate estimates on the rate of value delivery enable this.

Management’s Role in Lean-Agile

Management’s role is to create a great work environment for teams to work to achieve the organization’s strategic vision autonomously. They should leave teams to self-organize but need to help create the context within which the teams can work. This is done by getting input from the teams and other parts of the organization (e.g., product management, business intelligence, ops). Managers do not manage people but rather improve the eco-system within which people work.

Lessons from Lean-Agile Can Improve Waterfall

Although Lean-Agile is as much a mindset shift as it is a change in how work is being done, it is possible to take some of its lessons and apply them to a waterfall approach and get an immediate benefit. The key is to better work on smaller chunks of value. In other words, two 3-month waterfall projects will almost certainly deliver more value than one 6-month project. The reasons are that delays are removed by having a smaller development cycle, and pivoting on requirements is possible after delivering the first chunk of value. A further benefit is that half the entire project’s value is delivered in half the time. To learn more about this, see The Business Case for Agility.

Summation

It is true that many companies that are not hitting their targets using Waterfall methods may not hit them initially when they switch to Lean-Agile methods. But Lean-Agile provides a vehicle for improving the underlying causes of lack of predictability. Repeated failings in waterfall persist because management often thinks it would work if “people would just follow the plan.” But the plan doesn’t include things that are certain to happen (extra work due to delays). This makes “following the plan” inherently impossible. Lean-Agile provides the opportunity to get to the root cause of delays that are creating additional work that isn’t being planned for. By working on the root causes of the problem instead of the symptoms, Lean-Agile enables a path for true improvement.

Making Scrum Lean (ASG)

The Scrum Guide speciously claims Scrum is based on empiricism and Lean thinking. I first stated that Scrum could be thought of as a partial implementation of Lean in 2007. The creators of Scrum flatly denied this at the time. While Scrum does use a few semi-Lean practices, it is not based on Lean thinking. This is important to understand since thinking it is based on Lean thinking obscures what Lean is. Lean can be used to both improve Scrum and go beyond it, but only if you look to learn.

“It is impossible for a man to learn what he thinks he already knows.” Epictetus

These chapters are intended to provide an overview of how Amplio Development can be used to improve the workflow of those doing Scrum. In a nutshell, it will convert you from Scrum to Lean. You won’t be doing Scrum anymore but you will be more effective with less effort. And if your manager requires you to do Scrum, just call it “Scrum And.”

Removing the Immutability of Scrum (ASG)

Scrum provides you with a simple framework, that if you live within it, will work. It provides a quick start but requires the team to figure out what is needed to actually get the work done. It is based on two assumptions. One, that you’re in a situation where Scrum works. Oddly enough, it doesn’t tell you what these situations are. And two, that people will figure out the specific way to get things done within the framework. It is immutable because it does not provide a way to tell if a suggested change to Scrum will be an improvement or not. Hence, it demands the immutability of its key practices.

The main challenge here is mirrored in Edgard Schein’s maxim – “we don’t talk and think about what we see, we see what we are able to think and talk about.” By providing a guide that gives us a framework to live within, people talk about the framework. But the framework provides no real guidance on how to figure things out – so people don’t see solutions.

Amplio is based on a different mindset. It has identified first principles and a method for using them (the Factors for effective value streams) that can be used to accurately predict if a workflow change will be an improvement.

See A Fundamental Question That Separates Many in the Agile Community for more on this belief earlier in this book.

Without such a guide, people often make poor decisions on what might improve their work methods. This often happens after trying to remove an impediment to no avail. Abandoning certain Scrum practices makes sense at times – nothing works everywhere. But the objectives of those practices must still be met. Unfortunately, as Edgar Schein observes, “We don’t think and talk about what we see; we see what we are able to think and talk about.” Alternatives with greater efficacy and requiring lower energy to implement are not seen and therefore not taken.

Ironically, when teams are struggling with a Scrum practice and see no alternative, they often abandon the practice. Scrum proponents often chastise them for not following Scrum and call them ScrumButs. Instead of blaming people for not being motivated enough or for not committing to putting in more energy to follow Scrum, I suggest we find an alternative approach that lowers the energy required to get past an impediment.

Amplio is based on the belief that no practice is the best one everywhere. Instead of focusing on the practices of Scrum, we can shift our focus to the objectives of its practices. Then, we can ask ourselves, “are there other ways of meeting this objective?” This parallels how patterns are defined. Patterns are “solutions to recurring problems in a context.” By thinking of the Scrum practices as a solution to a particular problem when we have a cross-functional team using timeboxing to solve a complex problem, we can ask ourselves, “what would a solution to the same problem but in our context be?”

If we use a solution to this problem in our context, it will be fit for purpose, easier to implement, and more effective. The Scrum framework can be thought of as a framework of patterns, not practices. This is the way Amplio Development is designed. See Amplio Is a Pattern Language.

Using double-loop learning can help us identify when we should change our practices. Use the factors for effective value streams to determine if a new practice would be better (See How to Improve or Change a Practice for more).

Lean-Theory can provide the guidance needed to transition to Scrum practices and find alternative practices when applicable. Ironically, you won’t be doing Scrum anymore. I say ironically because if Scrum were genuinely based on Lean thinking, using it would not take you away from Scrum.

Re-thinking ScrumBut and ScrumAnd – Introducing ScrumSub (ASG)

Abstract

The Scrum Guide tells us that its roles, events, artifacts, and rules are immutable. Scrum is based on empirical process control’s “inspect and adapt” and complex adaptive systems. It does not incorporate Lean thinking in explicitly, as recently claimed. This lack of theory makes it difficult to ensure a change to any Scrum practice will be an improvement. Teams sometimes get frustrated with their impediments and abandon a practice they think is causing it.

Scrum.org defines “ScrumBut” as meaning “that Scrum has exposed a dysfunction that is contributing to the problem but is too hard to fix. A ScrumBut retains the problem while modifying Scrum to make it invisible so that the dysfunction is no longer a thorn in the side of the team.” For example, a team may say “we do Scrum, but we don’t do sprints.”

Ken Schwaber has since stated he prefers to talk about “ScrumAnd” where something is added to Scrum. But there is a third possibility – substituting a Scrum practice which removes the impediment while changing a Scrum practice.

We define a “ScrumSub” as “we are substituting a practice based on established theory that removes an impediment better than a prescribed Scrum practice. While the practice changes, the intent of the practice being substituted for is still being met.”

We are suggesting that Scrum practices should not be immutable but rather changed only when it’s an improvement for the team to do so. This recognizes that while arbitrarily changing practices is likely bad, changing practices to meet the context of the team, when the practices are grounded in solid theory, can be an improvement.

Why Having Options And Understanding Some Theory Is So Important

Consider being on a team that is having trouble finishing the stories by the end of a sprint. People aren’t sure what to do. The Scrum model expects people to figure this out but Scrum does not provide the required theory of Flow and Lean to do so. There are several options now. Let’s consider each from different perspectives of the team.

Just stick at it and follow Scrum

Scrum would suggest just to keep trying to get things done by the end of the sprint. It presumes that when an impediment is uncovered, the team needs to figure out how to solve it. But what if they don’t know how to do this on their own? For example, in this case, understanding Kanban would be a ready solution, or at least give insights on how to solve it. But many teams are mandated to learn Scrum and are not also given Kanban training.

Consider a ScrumMaster who only knows Scrum. They’ll likely propose sticking with Scrum because they don’t have a ready alternative. This is even more likely if they’ve been brought in to do Scrum since that’s their area of expertise. It also forces them to move out of their comfort zone and may make management question why there are there if Scrum is not being used. This, by the way, is why it’s best to have Agile coaches know both Scrum and Kanban (Amplio provides the mindset of both).

Any suggestions by the team regarding changing a practice, whether good or bad, is likely to be met with “but then we won’t be doing Scrum.” The challenge with this is, of course, the team’s focus in now on doing Scrum and not solving their own problems.

Falling into ScrumBut

At some point, if the team doesn’t learn how to solve this problem, they will eventually come up with something like “let’s not have sprints, let’s do Kanban.” Of course, most of the time, this is really just not having sprints (Kanban is not Scrum without sprints). They are now in uncharted territory without a map for improvement. Things will likely get worse and many people will say “you can’t blame Scrum because they are not doing it.” I would suggest this is an inherent problem with any approach that has preset practices.

Pick from a set of Options

One of the key concepts of Amplio, and other approaches based on first principles, is that there is no one size fits all practices. Instead, they should be based on the context a team is in. Amplio provides options to choose from, enabling a team to accomplish what’s needed in a suitable way. Picking a more appropriate option than one of the immutable aspects of Scrum may have the team not be doing Scrum anymore but getting the job done nonetheless.

Use a Theory to Figure out a Solution

Sometimes a good practice is a good theory. In other words, if it’s unclear what to do, then understanding some basics of flow, Lean, and the Theory of Constraints may be all you need. Look at your problem, see if there’s another way to solve it, try it and see if you got a good result.

If people just abandon practices without adopting a new one to fulfill the intent of the practice, ScrumBut as a bad thing is likely to ensure. Substituting practices requires understanding the intention of the practice being substituted and a set of alternatives to choose from.

See Know how to Select A More Appropriate Practice to learn how to substitute a better practice for the one you are using.

Introducing “Scrum Sub”

Redefining ScrumBut is too complicated now. The connotation that not doing a Scrum practice is bad is too prevalent. When the situation is “We do Scrum, but we substitute this practice for a Scrum practice” because it works better for our situation.

Not doing Scrum is not always bad. Sometimes there’s something more suitable for your team.

Blogs on Scrum and Lean

Unfortunately, many people in the Agile community think they understand Lean by having read a book or two. It’s worth considering where you get your information from. There are three levels of understanding lean:

Lean is based on the Toyota Production System (TPS) and is about manufacturing.
Lean can be understood by looking at the TPS and directly translating it into knowledge work.
One must look at the universal principles underlying the TPS and use them for your situation.

Many people talk about Lean from the first two perspectives. While there is some value there, that is not genuinely Lean thinking. One of the best books to start with is Lean Thinking: Banish Waste and Create Wealth in Your Corporation by Womack and Jones.

I have been studying Deming and Lean since the late 80s. It is not a simple topic. My understanding continues to evolve. I have collected several blogs I have written over the last three years on Lean and Scrum. The purpose for collecting these is threefold:

It’s important not to accept the misstatement that Scrum is based on Lean thinking because that will obscure the value of Lean
Learn Lean thinking to improve Scrum
Learn Lean thinking to go beyond Scrum

Blogs on the topic – most recent ones on top: