Lean Is Not Agile

Lean Software Development ostensibly tries to apply many of the principles of Lean Manufacturing to software development. I first remember reading about Lean Manufacturing, and the Japanese Quality Assurance movement, more than 20 years ago - before I had even heard of Scrum and well before the Agile Manifesto. In fact when I first read about XP (in '98 or '99) I remember thinking that a lot of ideas came from the quality assurance movement and even Lean.

However, in the last few years I have seen a lot of people saying or assuming that Lean Software Development is a "variety of Agile", like XP, Scrum, etc are. For example, I recently saw an advertisement for a course at a reasonably well respected university on SCRUM vs LEAN: Agile Methodologies Analysed and Compared.

Let's be clear on this, Lean is not a variety of Agile.

“ Lean is not
a variety
of Agile ”

Brief History

Lean manufacturing grew up as part of the Japanese Quality Assurance Movement around the middle of the last century. This movement was a major contributing factor in Japan's rise in a few decades to be the world's manufacturing powerhouse. The main focus of Lean Manufacturing is eliminating waste in its many forms.

The success of Japan's QA movement meant the ideas spread to other countries and many different industries. The ideas of QA and Lean Manufacturing found their way into SQA (software quality assurance) around the early 1990's. A great deal of them also found their way into Agile (perhaps via XP) in the late 1990's, but Agile is more than just a rehashing of the ideas of SQA and Lean.

A fundamental principle of QA/Lean is the concept of "Right First Time", which is trying to do things properly by eliminating mistakes. Agile outright rejected this idea, not in all things, but in the most crucial aspect of the software development process - how the software is specified and designed.

Agile is based on evolving the design based on responsive feedback. The assumption is that waste is unavoidable and that an emphasis on eliminating waste is counter-productive.

Lean Software Development

About 10 years ago Mary and Tom Poppendieck wrote a book called Lean Software Development: An Agile Toolkit, which apparently started a Lean movement "within Agile". I think I first heard of this book a few years ago but I have only just started reading it. I haven't had time to read all of it yet, but what I have read I agree with absolutely. It explains many of the principles and practices of Agile really well.

The only problem I have with the book is that it tries to apply "Lean Manufacturing" to justify Agile practices. The implication is that the core idea behind Agile is the lean philosophy of eliminating waste. This I find entirely fallacious.

The critical flaw in the premise of the book is demonstrated by this direct quote:

  “Just as manufacturing processes are continuously improved by production workers, so must a software system be continuously improved by developers.”

  - see Tool 19: Refactoring on page 137

The problem here is that it is confusing two different things: the manufacturing process and the product being manufactured.

A core principle of Lean Manufacturing and QA in general is to attempt to continually improve the process by which a product is created. But the actual product is not normally modified only the way in which it is made. For example, a watch manufacturer does not sell a watch (version 1.0) with only an hour hand, with the promise that later upgrades would add features such as a minute hand, increased accuracy, etc.

Applying this principle from Lean Manufacturing to software development means continually improving the way the software is developed. That is, Lean does not say to continuously refine the software (the product), only the process.

Rework is a QA term meaning work that has to be fixed or repeated because it was not done correctly the first time.

Hence the primary application of Lean ideas to software would be to improve that part of the process which far and away causes the most mistakes and rework - poorly understood, poorly communicated and continually changing requirements. Lean Manufacturing says to eliminate mistakes in the development process. In other words we must look at how to remove mistakes in the requirements.

By applying Lean Manufacturing principles to software development we must try harder to get the requirements right the first time. This is intended to avoid the enormous waste caused by repeatedly changing the design and the code. But this idea of BDUF is now generally regarded as futile.

BDUF?         

BDUF stands for Big Design Up Front and is the name given by Agilists for the way that software is normally developed using waterfall methodology. 

Decades of experience and studies have shown that, no matter how much effort is put into the initial design, there will be a disparity between what users need and what the software actually does.

I discussed this recently in Agile Design.

Agile Software Development

Agile takes the opposite approach to that of Lean Manufacturing. The assumption is that you can never get the requirements right the first time. There will always be mistakes.

The Agile approach is to start with the very basics of what you know, creating a minimal working initial version and then evolve the design based on feedback.

Software Development vs Manufacturing

Lean tries to take ideas from a manufacturing process and apply it to the software development process. However, these processes are very different - eg:

• Manufacturing is about making many identical copies of a product. Every software product is different.
• Manufacturing involves a repetitive process. Although there is some repetition in software development many things are unique to a project.
• Eliminating even a small waste in manufacturing can make a large cost saving. The cost of eliminating waste in the software development process often outweighs the saving.

It should not be assumed that principles beneficially applied to manufacturing are similarly beneficial to the software development process.

Software Development vs Engineering

OK, software development is not like manufacturing, but isn't it like the design as opposed to the manufacture of products?

On the surface, this argument seems much more reasonable than the one above. A software developer is just another type of engineer; just as a civil engineer designs bridges or an aeronautical engineer designs planes, so a software engineer designs programs.

This is getting a bit off track but it seems relevant to clarify this here. Creating software is not like creating physical objects. There are important reasons when designing physical objects to get a whole lot of things right up front. For example, you don't want to be almost finished as skyscraper when you realise the foundations are inadequate. Software does not suffer (as much) from these sort of limitations and so can be easily evolved as required (subject to use of Agile practices such as Unit Tests). I discussed this at length in the Software is Different section of Agile Design.

The thing to remember: software developers are not engineers. The idea of "software engineering" has bugged me for many years and I will probably write about it in this blog soon.

Similarities

I should be clear that the QA (and Lean) share a lot in common with Agile. I touched on this in SQA and Agile Methodologies. Some examples are:

• emphasis on JIT and deciding as late as possible
• customer focus
• automation of repetitive tasks
• retrospectives and continuous improvement
• importance of visibility of the process
• quality is everyone's responsibility
• empowering workers to make decisions
• emphasis on training and learning

Despite that I do not believe some of the QA ideas typified by slogans such a "Right First Time" and "Quality Is Free" are always applicable. I might discuss this further in the future.

Conclusion

The first principle of Lean Manufacturing is to eliminate waste. This has been shown to work well for manufacturing. For a repetitive process designed to mass produce identical products, eliminating even the smallest waste can produce large savings.

However, this principle of Lean Manufacturing cannot be applied to the software development process. Every software project is different. The main type of "waste" that traditionally (and still commonly) occurs in software development is due to rework (of the design and the code) due to the requirements. To eliminate this waste the Lean philosophy says to improve the process - in other words try harder to get the requirements right from the start.

Agile takes the opposite tack. We start with the minimum requirements needed to get something working and evolve the design based on responsive feedback. In effect, Agile methods (especially Unit Tests) provide a means to deal with the "waste" in a far simpler and less costly manner than trying to eliminate it. This waste in unavoidable and trying to eliminate it is counter-productive.

This is not to say that Lean does not have anything to offer the software industry. In fact most of the principles of Lean are also shared by Agile. Even the philosophy of eliminating waste can be usefully applied to many areas of software development. But at their heart Lean and Agile are diametrically opposed.

Vendor Mentality

Last week I talked about production line mentality. Now it's time to consider vendor mentality.

How are they related? In essence:

• production line mentality - I just do what the boss/job spec. says
• vendor mentality - I just provide what the customer asks for

Note that I did not invent the term vendor mentality (though I did make up the term production line mentality).

It is apparently a well-known effect to which I was first introduced by a colleague - thankyou Rixt.

Vendor Mentality in Developers

In the software industry many developers and contractors suffer from vendor mentality. It is based on this simplistic approach: the client says what they want, the "software vendor" creates the software and the client pays for it. This works fine for (simple) manufactured goods. Creating software is usually much more involved than simply exchanging money for a product.

Here are some reasons:

• the customer rarely knows exactly what they want
• when the customer knows what they want, it may not be what they need
• a customer will often specify what they see as the solution without really defining the problem or their objective
• different customer representatives will have different understandings of requirements
• what the customer wants will change as they understand their real objectives
• what the customer wants will change due to external factors
• what the customer wants will evolve as they try the software
• what the customer wants can often change for no apparent reason

Vendor Mentality in Clients

Clients and customers can also suffer from vendor mentality - "just give me what I want". However, in my experience clients are far more amenable to feedback, negotiation and looking at the problem from a different perspective than developers believe. I have already discussed this problem on the customer side (see Customer Management) - which details how and why you need to educate your client if they have a tendency towards vendor mentality.

The Developer/Client Relationship

To overcome this problem both sides need to work together to establish a partnership rather than consider it a simple buyer/seller relationship.

• developers need to ask questions rather than just take requests at face value
• developers need to understand the business of the customer
• developers often need to consider the customer's customer (see The Customer's Customer in Customer Management)
• developers should not be afraid to say no to the customer
• developers should identify ways to be of value to the customer
• the customer needs a better understanding of the development process
• the customer needs to actively guide the developers (via product owner)
• the customer needs to be open about their problems and objectives
• both parties should be honest and open with each other
• both work together to come up with the most satisfactory solution

When Things Go Wrong...

Mistakes happen. A great indicator of the health of the relationship between a software supplier and their client is what happens when things go awry. In an unhealthy relationship a supplier will try to cover up their mistakes, or (if that is not possible) excuses will fly or attempts made to shift the blame elsewhere. On the other hand, mistakes by the client are often not pointed out and the price is grudgingly born by the supplier.

This situation is commonly attributed to a difficult customer. Indeed, the customer's representatives may make unreasonable demands, but this is usually due a lack of understanding on their part. First, they may not appreciate the difficulties of the development process. More often the problem arises due to a customer who does not really know what they want. In this case you need to clarify, for everybody's sake, exactly what they are trying to achieve.

However, rather than the customer it is more often the developers, and particularly managers, that suffers from vendor mentality. They will insist on staff bending over backwards to accommodate the client, and do anything to prevent the company (particularly themselves) from looking bad. In such a blaming culture mistakes are punished which results in a large amount of effort spent in trying to deflect blame and in trying to avoid getting caught up in other people's problems.

Better than adopting a vendor mentality (and trying to cover up mistakes or deflect blame) is to be open and honest with your customers. They (even large companies) are almost always less demanding and overbearing than their suppliers believe, especially when they fully understand the situation. It may be up to you, as the developer, to help them to fully understand the situation as already discussed. But if a customer cannot accept what you tell them and insist then you may be better off without them.

A major problem is that many developers do not have the skills and personality to effectively interact with customers. They may be afraid to question the client and point out mistakes in their thinking. This is exacerbated in a culture of fear which does not encourage risk taking.

Sometimes it takes courage to do what is right but in the long-term the customer will thank you. Here are some examples of what you as a developer can do:

• never agree to unrealistic deadlines
• never agree to take on too many projects at the same time
• never take a request at face value - ask more questions
• if what the customer is asking for makes no sense then try to ascertain their objectives (see below)
• simply say no if it is not in the interests of the customer (or the customer's customer)
• don't be afraid to tell the client of their mistakes
• admit your own mistakes

5 Whys

OK, so how do you handle a client that asks for a software modification that makes no sense? Often a client will state exactly how they think the software should work, making many assumptions. Developers need to look beyond the request and ascertain the client's objectives. A particularly good technique I have found to arrive at the root cause of a problem is the 5 Whys technique. (Thankyou Inbar for introducing me to this technique!)

The 5 Whys technique simply says to ask "Why" to the initial request then to the subsequent answers until you can get no further. You can typically ask "Why?" up to five times (possibly more or less) before you get to the root cause of the problem.

Consider this example (where C stands for client or customer and D stands for developer):

C: We want to ask and record why the operator has chosen the recovery operation.
D: Why?
C: Because operators are selecting the operation when it is inappropriate.
D: Why?
C: Because they do not understand its purpose.
D: Why?
C: Because they have not been trained on its use.
D: Why?
C: We don't have time to give all operators training before they use the software.
D: Why?
C: Because training is expensive and we only have courses twice a year.
D: I understand. An alternative would be to move the "recovery operation" to the supervisor menu so only experienced users can select it. This assumes that all supervisors have been trained and understand its purpose.
C: Yes, all supervisors have undergone training. That sounds like a great idea!

Despite its usefulness there are things to watch out for. If you use the technique on a different client representative or group (or even the same representative at a different

The Wrong Approach :)   

C: I would like you to launch off a surface and into the air by using the muscles in your legs and feet.

D: What vertical dimensionality do you require?

time) you may get a completely different result. In my experience, it is quite surprising that a different set of answers, taking a completely different tack, often come to the same conclusion.

However, occasionally you can follow a different path and come to a completely different root cause. Of course, this is actually a good thing as it allows you to find different approaches to a problem, or even completely different problems.

Often asking the 5 Whys can reveal flaws in the client's understanding. The previous example may have instead gone like this:

C: We want to ask and record why the operator has chosen a recovery operation.
D: Why?
C: Because operators are selecting the operation when it is inappropriate.
D: Why?
C: Because it is too easy to select the operation.
D: Why?
C: Because it is presented as the first option on the sign-in menu.
D: Why?
C: Because that is the obvious place to put it.
D: Why?
C: We don't want people to sign in and use the system in a bad state.
D: Why?
C: That causes data corruption whence we have to restore the previous day's database.
D: That is not the purpose of the "recovery". How often do you have to restore the database?
C: It has recently been happening almost daily - on average once every 4 months per site.
D: We need to look into the reason for this!

Conclusion

As a developer you need to work with your customer(s) to find out what they are trying to achieve. Ask questions, take an interest in their business, and try to see things from their point of view (or even their customer's point of view - see The Customer's Customer in Customer Management).

Don't let the customer dictate the solution to a problem. Hopefully, you are much more familiar with different techniques and technologies, as well as any technical limitations. On the other hand, you may need to understand more about the customer's business in order to choose the best solution.

Conversely, the customer needs to better understand the development process (see What Every Customer Should Know in Customer Management). Some customers may need persuading to work with you in a partnership, but you will probably be surprised how amenable most customers are to developing a mutually beneficial relationship with their suppliers.

Above all, be frank with the customer, even at the risk of offending them (for example, by pointing out their mistakes due to lack of understanding). Of course, this should be handled tactfully, but it is in everyone's best interests to ensure the customer understands the truth.

Equally, you should not hide your own mistakes. This will help to build a relationship based on trust and mutual respect. All parties should be honest and open to work together to come up with the most satisfactory solution.