Many executives have been led to believe that Agile is inherently less predictable than a waterfall approach.  However, when wrapped in Lean-Thinking, Agile can be more predictable because it enables working directly on the true causes of unpredictability in software development. Waterfall’s large projects and stage-gate approach cause delays in feedback, workflow, testing, and integration. These delays inherently create a significant amount of rework (redoing requirements, reworking code that missed requirements, finding bugs, thrashing during integration).  This work, of course, is never planned for and therefore results in inherently bad estimation, not to mention that eliminating this extra work improves productivity.

 In addition, Lean-Agile methods have us focus on smaller, higher-density pieces of work. This alone helps bring our systems under control, as observed by Eli Goldratt – “Often reducing batch size is all it takes to bring a system back into control.”

The key to having predictable results is eliminating delays, creating unplanned work.  This requires smaller batches of work that are appropriately sequenced in importance, properly formed teams (or groups of teams), and solid technical practices.  Agile suggests additional methods such as Acceptance Test-Driven Development, automated testing, and continuous integration.  These directly improve understanding of requirements (through quick feedback), code quality, and risk reduction.

Although waterfall and Agile approaches have different mindsets, some of Agile’s methods can be incorporated into a waterfall process. For example, making both projects and planning cycles shorter.  Improving methods will not make an organization Agile, but it may help set the stage for a true transformation.

I’ve heard it said that “we can’t go Agile because we need predictability.”  This is an understandable sentiment because many Agilists have dismissed predictability as impossible.  However, there are two camps in the Agile community – one that is team-centric and another that wraps Agile with Lean-Thinking (we call this Lean-Agile). A team-centric approach will often not achieve predictability when applied at scale because the team is only one part of the value add cycle.  But a Lean approach is holistic, and although it may not be possible to predict precisely what must be built, it is possible to predict how much can be built. With proper selection of the work to be done,

However, more than simple iterations are required to achieve this.  Lean-Agile should not be just a mindset shift but must incorporate solid product management and technical practices.  Let’s consider why projects (regardless of approach) often do not achieve the predictions they make.  The most discussed are:

  • what the customer wants is not understood
  • the time to build something is unclear
  • the code is fragile and hard to change

There’s an even bigger one lurking here (which I’ll discuss a little later) but for now, let’s look at these.

What the customer wants is not understood. Customers often don’t know what they want until they see what they don’t want.  This implies they should be shown “what they don’t want” as soon as possible.  This is why it’s important to build small segments of work quickly. This also shortens the time between all stages of the work (requirements, design, code, test, integration). This combination of quick feedback with coding and testing being done together enables quick course corrections as both the customer and the development group learns what is truly of value.

Acceptance Test-Driven Development (having customers, developers and testers specify requirements in the form of test specifications) achieves better clarity on what is needed and avoids many misunderstandings altogether.  Getting the customer to provide requirements in this precise manner changes the nature of the requirements.  Vagueness is no longer allowed, and customers’ understanding of what they are looking for improves.

The time to build something is unclear.  The answer to this is again to go small.  Estimating the time it will take to complete a large chunk of work is typically less accurate than estimating the time for smaller things.  Some people have ridiculed estimation because they are often wildly inaccurate, and managers sometimes abuse the teams for that. But bad estimates are more of a symptom of other issues – unclear requirements, poor code quality, and even fear of management.  But avoiding estimation abandons looking at the actual underlying reasons.

The code is fragile and hard to change.   Poor code quality will slow things down, but this does not have to be the path taken.  When code is fragile and is difficult to change, it will not be possible to accurately predict how long things will take, even for small changes.  But that’s because of the poorly written code, which is both difficult and risky to change.

Waterfall’s approach addresses this issue by doing all coding first and then testing afterward.  There is often a perceived need to do this because of the need to batch test.  But this doesn’t get at the issue (poor code quality).  Accommodating poor code quality will not improve it but tends to get the code in a slow death spiral that will eventually need to be re-written.

Poor code can be improved with automated acceptance tests.  The time test automation saves is only part of its value.  It also enables refactoring of the code in safety because defects introduced will be detected quickly.  Developers fix these bugs much faster because they are detected while still fresh in their minds.  This is a considerable time-saving.  Most developers think they spend a lot of time fixing bugs.  But on reflection, they see that most of their time is spent finding the bug.  This is not mere semantics.  Bugs detected immediately are fixed much faster than bugs detected weeks later (even when the code hasn’t changed).

The Common Themes

The first common theme here is that smaller batches of work allow for greater efficiency by providing quicker feedback, better estimation, and shortening the delays between requirements, code, and testing.  This efficiency is due to the fourth issue I earlier mentioned I was going to discuss. This is because delays in feedback, workflow (e.g., waiting for people), and detecting errors will cause additional work.

It is common to focus on how to work faster or better.  But it is more effective to discover how to avoid creating the need for rework.  Although much rework is due to a misunderstanding, having quick feedback significantly lowers the amount of rework required.  Consider how much time is spent on redoing requirements, building features of less value than initially thought, fixing bugs, manual testing, thrashing during integration, ….  It should be apparent a significant amount of work done in projects was created by the aforementioned delays. The focus must shift from doing things faster to avoiding delays between the work steps and achieving fast feedback.

Why Waterfall Will Inherently Provide Bad Estimates

Consider the waterfall planning process.    Larger chunks of work are planned to be done in separate steps – requirements, design, code, functional test, integration, and system test.  This exacerbates the aforementioned issues by increasing delays in feedback between each step and overall feedback achievable at completion. This creates a significant of work due to these delays.  However, how much of this time is planned for at the start of the project?

None.  Zero.  Nada. 

Estimates in waterfall don’t consider the work that derives from delays – work that often takes as much or more as the planned work. Planning sessions typically don’t include the question, “how much time should we set aside for missed or wrong requirements, pitfalls in our design, or all of the extra time it’ll take to find the bugs because testing was delayed?” In other words, one reason estimates are bad is that they do not include a significant amount of the work that needs to be done.

So Why Should Lean-Agile Be More Predictable?

Lean-Agile takes a different approach with requirements, design and coding, and testing.  Working in small batches and continuously course-correcting reduces time spent correcting misunderstood requirements.  However, technical methods are still needed. Test-first and automated testing also results in better code.  Considering writing tests before writing code is an important lesson from design patterns (although not stated explicitly in the iconic Gang of Four Patterns book, both Design Patterns Explained and Essential Skills for the Agile Developer describe this in-depth). And finally, by using smaller batches of work, the time between code and test decreases, thereby lowering the finding time even more.

Another difference exists in the approach to architecture.  In waterfall, developers are supposed to design for the stated requirements.  However, they have learned that requirements will change and tend to over-design to accommodate this. On the other hand, Lean-Agile suggests designing to accommodate change.  This is known as emergent design.

An Extra Benefit of Lean-Agile

Building things in small batches results in learning faster about what is needed and enables pivoting as required.  Feedback from the customer uncovers the actual value of each feature or story.  Furthermore, customers learn that they don’t need to ask for everything upfront.  Instead, they can adjust their requirements as they see what’s being built and better understand their needs.

Waterfall, oddly enough, encourages customers to ask for everything upfront because little justification is required other than “I need it” when initial requirements are stated. Anything specified later, however, must go through a change review board.  Customers quickly learn to ask for whatever they might need up-front.  Lean-Agile teaches customers they only need to specify what they need soonest and that they can adjust as they learn more.  While a general understanding of the entire system is required to enable an estimate for building, only the general size of the work is needed.  The details will come later as more is learned.  Accurate estimates on the rate of value delivery enable this.

Management’s Role in Lean-Agile

Management’s role is to create a great work environment for teams to work to achieve the organization’s strategic vision autonomously.  They should leave teams to self-organize but need to help create the context within which the teams can work.   This is done by getting input from the teams and other parts of the organization (e.g., product management, business intelligence, ops).  Managers do not manage people but rather improve the eco-system within which people work.

Lessons from Lean-Agile Can Improve Waterfall

Although Lean-Agile is as much a mindset shift as it is a change in how work is being done, it is possible to take some of its lessons and apply them to a waterfall approach and get an immediate benefit.  The key is to better work on smaller chunks of value. In other words, two 3-month waterfall projects will almost certainly deliver more value than one 6-month project.  The reasons are that delays are removed by having a smaller development cycle, and pivoting on requirements is possible after delivering the first chunk of value.  A further benefit is that half the entire project’s value is delivered in half the time. To learn more about this, see The Business Case for Agility.

Summation

It is true that many companies that are not hitting their targets using Waterfall methods may not hit them initially when they switch to Lean-Agile methods.  But Lean-Agile provides a vehicle for improving the underlying causes of lack of predictability.  Repeated failings in waterfall persist because management often thinks it would work if “people would just follow the plan.”  But the plan doesn’t include things that are certain to happen (extra work due to delays).  This makes “following the plan” inherently impossible.  Lean-Agile provides the opportunity to get to the root cause of delays that are creating additional work that isn’t being planned for.  By working on the root causes of the problem instead of the symptoms, Lean-Agile enables a path for true improvement.

 

Go to Amplio Development: The Path to Effective Lean-Agile Teams 

Go to Being a Professional Coach

Latest Article

Why Leaders Should Focus on Value Streams

Read on Cutter's site >>

Latest Presentation

Intro to BLAST and How To Solve Challenges in the Workflow

Watch Video >>

Upcoming Event

The Amplio Community of Practice (Free)

Learn More >>

New Workshop

The Amplio Development Masterclass

More Info >>

Latest Presentation

Intro to BLAST and How To Solve Challenges in the Workflow