Production Line Mentality (and Silo Syndrome)

At the start of this month I mentioned, in passing, what I called Production Line Mentality. This is the propensity for individuals working in a large organization to concentrate solely on what they have been told to do, ie, their "job specification". This behavior is promulgated by a multi-layered, controlling management structure.

Two people pointed out that what I was describing already had a name - Functional Silo Syndrome or simply Silo Syndrome - a term ascribed to Phil Ensor. Though Production Line Mentality and Silo Syndrome are very similar I don't believe they are exactly the same as I discuss later.

Since the discussion of Production Line Mentality created so much interest I thought I should elucidate and mention a few things that I did not have room for last time. Later on I will talk about how the Agile approach addresses it.

Production Line Deficiencies

Some say the production line is the most important invention of the industrial revolution. However, it has well-documented problems such as being boring, demoralizing, demeaning and, if the truth be known, tends to generate product of low quality. Luckily, in manufacturing the traditional production line has almost vanished in the last few decades due to studies (eg, see next) demonstrating better approaches. Unfortunately, in the software industry elements of the production line remain.

To show the problem, and the solution, I will first mention a study of a manufacturing production line. This occurred at a Swedish washing machine maker in the 1980's. The company switched one production line to a system where different teams assembled the machines from start to finish.

Originally with a production line of about 20 people (six machines per day), they  reorganized into four teams of five people. (Of course, there were costs involved in re-tooling, training etc.) Each worker was given the skills and knowledge to assemble a major component or even the whole machine.

In brief, they found that productivity quickly increased to about 7-8 machines per day (and even more over time). Morale improved and less supervision was needed as individuals were motivated to get the work done and even suggest improvements that boosted their own efficiency. Importantly, the product had a marked improvement in quality. There were many other documented advantages such as less conflict and much greater flexibility when someone was absent.

Production Line Mentality in Software

OK, so the production line has problems, but what has that got to do with software development? Actually, quite a lot.

First, many developers are quite happy to potter along on their own problem with little or no interaction with others. By focusing on a small narrow task they lose sight of the bigger picture. To be productive all team members need to communicate and understand the organization's objectives.

Second, traditional software development methodologies encourage this behavior The Waterfall model divides the development process into a series of phases (much like a production line). The work is divided into specialized areas: business analysts, system analysts, designers, coders, various flavors of testers, technical writers, etc.

Finally, many large IT organizations are placed under a large amount of pressure by senior management. This often leads to a blaming culture and a controlling management style which is probably the largest contributing factor, as I discuss below.

Silo Syndrome

Although the Silo Syndrome has essentially the same causes and effects as the production line mentality it is usually described in terms of the behavior of departments or "groups" within a larger organization rather than individuals. In the interests of self-preservation a group becomes protectionist of their knowledge and uncooperative with other parts of the organization.

The original article on Functional Silo Syndrome by Phil Ensor beautifully explains this (see Functional Silo Syndrome). The following discussion of causes and effects of production line mentality generally also apply to the silo syndrome.

How Does It Happen?

The behavior starts as a response to an autocratic, controlling management style. As a defense mechanism the individual (or group) withdraws from any activity that involves risk. Some also restrict access to information, since knowledge is power. Of course, this is detrimental to the organization as a whole by stifling creativity, knowledge sharing, and collaboration.

There are many reasons for this poor management style. It could simply be due to an individual manager. More often it is due to the culture of the organization -- coming from the top. Autocratic managers are generally attracted to an organization with such a culture (but they can pop-up anywhere).

I can think of a few different scenarios:

Egotism

A common problem is a technically gifted individual who has been promoted to a position of power and attempts to micromanage all subordinates. Often they believe they know better than anyone working for them how to do the work. They only have people working for them because they don't have enough time to do everything themselves.

Their enormous ego leads them to believe it, but they generally do not know better than those actually doing the work. Even if it were true it is still demoralizing and demeaning to be micromanaged in this way.

Insecurity

A related problem is a manager that is insecure in their own ability. Surprisingly this may occur in concert with the above since the manager may project confidence but are worried that eventually others will see through the façade. By controlling information they can more easily cover up their mistakes or find someone to blame.

Self-preservation

One of the most basic instincts is for self-preservation. If a manager gives up control then their role may be seen to be less important or even redundant. By embedding themselves in the work processes they make themselves an essential cog of the machine. Also, as we saw above, an insecure manager tightly controls information to hide their own ineptitude.

This is perhaps the biggest impediment to addressing the problem. In a top-down, authoritarian structure, many layers of management are required to monitor and control everyone. If the people who actually do the work are empowered to make decisions then many of the layers can be reduced or even eliminated. Unsurprisingly, there is a huge amount of resistance to this sort of change.

Transference

If a culture of fear emanates from the top then it will permeate the entire organization. There is no advantage, and significant disadvantage and difficulty, in treating your staff well if you are treated badly yourself. In other words a manager cannot empower their subordinates if they are not themselves empowered.

Megalomania

It is rare but some managers simply enjoy wielding power to the point that it is almost sadistic. Luckily, I have only ever encountered one such manager who would deliberately inconvenience or even humiliate those that worked for him.

Ignorance

Finally, there are those managers who don't know any better. Either, they are inexperienced and are following the lead of others or this is the way they have always worked and are not open to any other options.

How Does It Work?

In most organizations, production line mentality results in a large hierarchical management structure. Each manager is given strict control of a group which is split into sub-groups each with their own manager, and so on. This hierarchy can contain many layers until you get down to the people at the bottom, who actually do the work.

Information flows vertically. If you need information from someone in another section this is done through your supervisor. Information is distributed by a manager on a "need to know" basis.

Rewards are doled out by management based on the perceived value of each individual. Inevitably this fosters competition and rivalry, which kills any sort of collaborative spirit. Individuals expend effort making themselves look good (and even try to make their co-workers look bad) to their boss, rather than focusing on the goals of the organization.

When something goes wrong it's easy to point the finger at the person responsible. However, the root cause of the problem is usually that the person who made the mistake was not given enough information or assistance to do their job properly.

Negative reinforcement (ie, punishment for mistakes) is the primary motivator, rather than positive reinforcement. This leads to an aversion to risk where nobody wants to take responsibility for anything beyond their small area of influence.

Indicators

Many people working in this sort of environment are not even aware of it or just accept it. This is often because they have have never worked for a more enlightened company. Here are some possible clues based on the work environment:

•  there are more chiefs than Indians - ie more managers than workers
•  monetary rewards are given based on individual performance reviews
•  each position has a detailed written job description
•  you often have to sign legal documents such as non-disclosure agreements
•  requirements documents must be signed off by several people
•  developers are not allowed to talk to customer representatives

Here are some clues that your boss is the problem:

•  you don't really know what your boss does
•  you can never find your boss or arrange a meeting
•  you are not given enough information to do your work properly
•  you have to ask for a (deserved) pay rise
•  you cannot send emails to the client without clearing it with your boss
•  your boss threatens to fire you when you express any sort of dissent
•  you get into trouble for swapping tasks with a co-worker
•  when you book a meeting with your boss she postpones or is late
•  if your boss books a meeting with you and you are late she hits the roof
•  all emails you send must be CC'd to your boss
•  you cover up a problem rather than tell your boss
•  you discover there was a meeting (on a subject crucial to your work) to which you were not invited
•  you are asked to work overtime when there is no reason to do so
•  your boss keeps reminding you about the poor job market
•  your boss never admits they made a mistake
•  your boss keeps reminding you of your own past mistakes

Here are a few general clues about an organization with a poor culture:

•  you have never met anyone who uses the software that you write
•  everyone knows about a problem that needs fixing but nobody is willing to tackle it
•  there is often conflict in the workplace
•  many problems occur - the same problems happen over and over
•  people in the organization have stress-related health problems
•  you are regularly given large design documents about projects you have never heard of
•  two developers discover that they have both fixed the same bug

If you are a boss, here are some clues that you might be the problem:

•  you get the feeling that you don't know the full story about what is happening
•  the people who work for you show no initiative
•  the people who work for you are incompetent
•  the people who work for you are risk-averse
•  you don't know the name of everyone who works for you
•  there are people that you have not talked to in over a month
•  you have an "open door" policy but nobody takes advantage of it
•  the client asks for a simple change to which you give an estimate, but are shocked when the team cannot deliver

Consequences

By now I hope you are convinced of the detrimental effects of production line mentality (and silo syndrome). To reiterate it makes for an organization with little collaboration and creativity to solve problems. Insular thinking results in inefficiencies such as poor processes, redundant processes and bad decision making.

The effect on morale results in poor employee engagement, absenteeism and sub-optimal productivity. Moreover, because managers build themselves into the system they become a single point of failure which affects flexibility and resilience.

Prevention

So how do we prevent this from happening? The simplest way is to follow an Agile methodology. For example, in Scrum the team is self-organizing, so resist the urge to "help" them get organized. Another piece of good advice comes from XP (Extreme Programming) to "eliminate fear".

The most important point is for managers to empower their charges. Don't assume they will get it wrong. If they don't do it the way you would do it, it may actually be a better way. If you trust people they will usually honor that trust.

Some simple tips are

• encourage spontaneous feedback, eg don't react badly to a mistake or oversight.
• reward the team rather than individual to encourage collaboration
• eliminate formality & the need to "go through channels"
• have more than an open door policy - go out and talk to people
• use instant messaging - but also talk face to face sometimes
• encourage people to interact as necessary with whoever they need to

Does Scrum Promote Silos?

Scrum is great for counteracting "production line mentality". Allowing the team to self-organize promotes interaction (within the team) which results in better organization, processes and products. It is also good for morale due to team members having a feeling of making a contribution and working together toward a common goal. In particular, the daily standup allows everyone to know what everyone else is doing, with improved coordination and less mistakes.

But does Scrum inhibit collaboration between teams? The simple answer is - it shouldn't but it can.

When different groups do not collaborate this is known as the "silo syndrome" as we saw above. Sometimes Scrum teams have this problem due to a misunderstanding of the roles on the product owner and scrum master. Let's first look at the problems these roles are designed to overcome.

Product Owner

Even in (or especially in) an open, collaborative environment there can be a problem of conflicting and volatile requirements. There are often many people (users and other client representatives, managers, business analysts, etc) with input into how the software is to be modified. One manager might insist on all bugs being fixed as a high priority, another might want to sneak in their favorite feature or improve an existing feature -- you have probably seen this sort of thing yourself. Different people have different ideas and priorities; and these ideas and priorities tend to change often.

In Scrum, having a product owner and sprint backlog avoids this problem by having all requirements come through one person. This ensures that the developers are not pulled in many different directions at once. Further, because the sprint backlog is unchanging they know that things will not suddenly change, at least until the end of the sprint.

However, having a product owner does not in any way preclude the team from talking to people outside the team. In fact this sort of collaboration is encouraged. It simply means that all requirements come through the PO and are prioritized in the backlog.

Scrum Master

The scrum master is in many ways an interface between the team and higher-level management. The main task of the scrum master, outside the team, is to remove roadblocks. This involves them using their influence to obtain resources, help or information. Some scrum masters take it upon themselves to intercept interference from outside the team, such as a senior manager who wants to control the team. Their are always managers who think they can organize the team better than it can organize itself.

In summary, an over-zealous scrum master can turn a scrum team into a "silo".

Finally, a Scrum Team can be unfairly perceived as a silo. A team should be composed of all the persons required to accomplish their tasks from start to finish. This means that they may not require a lot of communication with people outside the team.


Conclusion

Production line mentality (and silo syndrome) begins with a culture where personal interest (or the interests of a group within the organization) is not aligned with the organization's best interests. Fear prevails and individuals are driven by self-protection (or protection of their department) rather than desire to do the best for the organization as a whole.

This culture is characterized by many layers of bureaucracy which is a classic example of the DIRE principle. Employees show no initiative and just wait to be told what to do. Risk-taking is discouraged since mistakes are punished.

The solution to this problem is to empower employees to take control of their own destiny. Encourage collaboration and discourage rivalry by rewarding the team and not individuals. Share the vision of the organization so that everyone is working towards the same end.

Generally, following the ideas behind Agile will encourage collaboration and work against the production line mentality (and silo syndrome).

[Note: I was also going to talk about the closely related problem of vendor mentality, but that will have to wait for next week.]

Agile Roles

Back in May I wrote about how software design is done in Agile. I received some feedback which I promised to respond to in a future post (this one). Sorry for the delay.

Someone asked about the role of designers in Agile. They thought I was implying that specialist designers were not needed. Well, just to be clear, I will say it outright: No dedicated designers! All or most of the team should be involved in the design though typically some will have more input than others.

"No dedicated designers!"

When moving to Scrum people who just do "design" need to stop it. Designers traditionally are either technical (sometimes called architects) or business-oriented (sometimes called analyst/designers). I talk below about how these (and other) roles may change when moving to agile but in brief designers can move in two different directions - either to more development work or to more of a PO type role. Architects often become more hands-on - spending more time coding and reviewing. Analyst/designers can move back to development or if not from a technical background will need to move into a role involving mostly acceptance testing and customer interaction.

Another comment was: "If design, programming (and testing) are to be done together then should they all be done by the same person?"  In essence: YES. But before I get into that I need to talk a little about roles in Agile teams.


Multi-skilled Teams

Everyone agrees that Agile development uses multi-skilled (or cross-functional) teams. The problem is nobody really agrees (or even pretends to know) exactly what that means. Does it mean each of the team members have the skills to do all, or at least many, of the required tasks? Or does it mean that altogether the team has the required skills but each specializes in their own skill-set?

In my experience most Agile "experts" use the latter definition. For example, I was recently involved in the implementation of Scrum in a large company, and one of the things that the Scrum consultants emphasized is that a Scrum team (ie, "pigs" not "chickens") is made up of members with a wide variety of skills. However, I get the strong impression that this is mainly to placate those employees who feel that they may be redundant in the new order of things.

In reality, in an organization suffering from production line mentality (see below) adoption of an Agile methodology will mean that there will be less need for some existing roles. Less BDUF (see my post on Agile Design) means that there is no need for dedicated designers as design is performed continuously by the developers. Also, developers take more responsibility for testing so there is less need for dedicated testers doing black-box testing (like regression tests). This means that developers have more responsibilities (rather than just coding), but this is a good thing as it promotes DIRE (see DIRE) but also makes the work more rewarding and interesting.

If you need less people for other roles, such as designers, does this mean you need more developers? The simple answer is no. In my experience the same developers can actually be more productive, so you may even need less developers too!

Here I should note that I am talking about using Scrum in a pure software development environment.

I know that Scrum has been used successfully in other environments and that is great. However, whatever the environment all team members must have enough shared context to understand what everyone else in the team is doing.

Scrum Team Composition

Ideally all members of a Scrum team should be developers. That is, team members (apart from the Product Owner and perhaps the Scrum Master) should have the ability to write code. To understand why I will first recap on the advantages of Scrum.

Scrum (and Agile in general) has many benefits such as promoting team-work, working towards a common goal, quick response to change, etc. This is achieved by good communication and empowering the team to make their own decisions. The daily standup is particularly important as it allows everyone to know what everyone else is doing.

When everyone in the team can (and does) understand what everyone else is doing this avoids issues like:

• differences in understanding of what is to be done
• poor division of work
• duplicating work - such as two developers fixing the same bug
• forces pulling in different directions and introducing conflicting changes
• not realizing when someone needs help
• not being able to have someone else take over a task due to illness
• not being able to have the whole team focus on a critical problem
• inability to find another team member who understands what you are doing well enough to perform a review

Unless the team members are all developers they cannot really understand what everyone else in the team is doing (and how, why, etc). This is essential for Scrum to work (see Why Scrum Fails).

Of course, different individuals will have different levels of experience and different aptitudes. Over time, team members become familiar with each other's abilities. It is easy and natural for team members to take each other's skills into account when allocating tasks during the sprint planning.


Production Line Mentality

The production line was a major invention in the history of industrialization, since even unskilled workers can become very good at a simple repetitive task. However, its drawbacks were quickly discovered. Workers suffer from frustration and alienation. Once a task is mastered boredom sets in and mistakes are made, even for simple things.

Quality and productivity are greatly improved when workers are given a variety of challenging tasks. Performing different jobs as part of a small team working towards the same goal can be a tremendous motivator.

Another problem with production line mentality is that it usually results in post-production testing - ie someone inspecting the product coming off the end of the line. 

This sort of "Black-box testing" is not a good way to add quality particularly for software (see Unit Tests - White Box Testing).

What I call "Production Line Mentality" is where everybody has a specific well-defined job specification. This is common in an organization with a top-down management style where a worker's purpose is to impress their boss rather than work with others. In other words it does not encourage people to work together to accomplish their team's goals.

It is caused by a controlling (rather than empowering) management style where each role is kept simple to be easily understood and hence controlled. However, it results in a large number of specialized positions and a multi-layered management structure to be able supervise everybody in the organization.

This discussion actually ties in nicely with some of my recent posts. First, multi-layered management is a classic example of the Layer Anti-Pattern which is a pattern of bad software design which I have found can be applied in other domains.

Moreover, these layers form an excellent example of the DIRE principle. First, the managers at the top of the hierarchy are typically completely out of touch with the people actually doing the real work in the organization as they are isolated from them by all the layers. Perhaps even worse is that it encourages the workers themselves to concentrate on their own specific task without regard for the goals of the broader organization.

In an Agile development environment the developers are empowered to do whatever they need to do (within their bounds) to finish their tasks by the end of the sprint. For a traditional controlling manager it may be hard to deal with an environment where you can't even understand what everyone is doing let alone control it.

A controlling manager can be a real problem for the Agile approach. On the other hand, having developers who are happy to (or used to) being controlled like this is another major obstacle for the adoption of Agile. Years of manipulation (and even threats) means that many are happy simply to be given work as outlined in their job spec.


Traditional Roles

Let's look at traditional roles and how they change in an Agile world.

Team Leader

Scrum books simply say that the team leader role is no longer necessary. The team self-manages so the need for a manager within the team ceases to exist.

I disagree with this. In fact most of the same books describe a role for the Scrum Master which is almost exactly what I have always thought a team leader should be. (Perhaps they like the different title to emphasize that many team leaders are using a bad style.)

The team leader is now called the Scrum Master. Many team leaders also write code. That's OK too - you can have a part time scrum master who also codes.

However, if you think a team leader/scrum master is there to tell the team members what to do, then forget it. One of the main advantages of Agile is that teams are self-organizing and self-managing (within the bounds set for them by the PO and higher management). Small self-organized teams generally work much more efficiently and harmoniously than "centrally-organized" teams.

Designer

There is no longer any room for analysts/designers who spend weeks or months understanding a problem and trying (unsuccessfully, of course) to come up with the perfect, well-documented design. In an Agile world analysis and design is performed "just in time" (see JIT). The developers design, code (and test) all at once. This is not to say that what is designed is left to the whim of the developers - in fact how the design evolves is more closely controlled through the customer representative (product owner).

Traditional designers can move toward a role with industry knowledge, in particular how the software will help the business (or even the customers of the business). Perhaps they will be the actual PO (product owner), though I prefer someone even closer to the customer than that. Probably they will become a "business analyst" assisting the PO. Depending on the software (and despite the Agile preference of JIT design) there may be a need to mock up designs slightly ahead of time to garner feedback from the PO and/or customer.

Alternatively, or also, they may also do a lot of Acceptance Testing (see below). Whatever they do, their main focus (as for any Scrum team member) should be on completing the sprint backlog before the end of the current sprint.

Architect

There are just two points I want to mention about software architects using quotes from the excellent book 97 Things Every Software Architect Should Know.

" [the architect] should not sit in an ivory tower dictating the way forward but should be on the ground working with the team "

- from 19. Architects must be hands on by John Davies

"As well as being prone to incidental complexity and inertia, designing large systems upfront means larger projects, which are more likely to fail, more likely to be un-testable, more likely to be fragile, more likely to have unneeded and unused parts, more likely to be expensive, and more likely to have a negative political dimension. ...

"Design the smallest system you can, help deliver it, and let it evolve towards the grand vision. Although this might feel like giving up control, or even shirking your responsibilities, ultimately your stakeholders will thank you for it."

- from 97. Great software is not built, it is grown by Bill de hOra.

In brief: architects work with the team to evolve the design, spending most of their time coding and reviewing.

Programmer

In an Agile environment programmers now do everything that is required of the team (analyze, design, code, test, etc). They do whatever it takes to complete the sprint tasks by the end of the sprint.

This is in stark contrast with some traditional environments where programmers have just been coders - they take a design from the analyst/designer and code it for "QA" to test. This is demeaning for programmers who are usually capable of a lot more. Even if you care not for the morale of developers giving the programmers free reign to think about the design and even experiment at little will produce a much better design than any analyst/designer (no matter how clever) can invent in their head.

Of course, this does not mean the programmers can do whatever they want. They are closely guided by the product owner who, after all, prioritizes the backlog effectively deciding on the sprint task(s) that are the focus for the team.

Tester

If waterfall development is disempowering to programmers it is much worse for testers. As compensation, testers are often given the impressive title of quality assurance experts, despite the fact that quality assurance has little to do with testing.

You may not believe this but using Agile development (especially Unit Tests) there is a greatly reduced need for dedicated testers. How great this reduction is depends to a large extent on how much testing the developers do already. At one extreme I recently worked in an environment where there were twice as many testers as developers! The software contractor had there own tester(s) who worked closely with the coders. Then the customer had several testers, plus an "offshore" test team to perform regression testing, as well as an automated testing team.

There are many different names for the types of testing (and testers) that are used in the software industry. In Agile there is just two general types of testing:

1. Unit Tests: automated tests written by the developers of the code to test individual (low and high level) modules
2. Acceptance Tests: performed by the PO, customers, or anyone who wants to try the software

Note that Unit Tests are always automated, but acceptance tests may be automated too. Often the developers will add a simple scripting system to make it easy for the PO and others to create test scripts.

This is useful for testing the current changes. But don't go overboard and add lots of tests for existing functionality as they will most likely fail when changes are made (and even act as a disincentive to change). Leave the regression testing to the Unit Tests.

To understand this better consider the testing that an aircraft manufacturer does. First, individual parts are developed (or purchased). The developers of these parts (for example, spark plugs) will design, build and test them themselves and provide a hopefully reliable product. Some parts may be made from a combination of other lower-level parts, but it is still up to the developer of any part to thoroughly test it. Of course, larger, more complicated, parts may require more elaborate testing - testing of an aircraft engine may require a huge test rig. In the end it is always up to the maker of a part to ensure it performs to the requirements of the plane manufacturer. Testing of parts is the equivalent in software development of Unit Testing.

When the first plane (or a prototype) is finally built it is then up to a test pilot to perform various flights and manoeuvres to ensure that the plane performs as expected. This is the equivalent in software development of Acceptance Testing. This analogy is not perfect though, since Acceptance Testing of software should usually be performed much earlier than test flights.

If a Scrum team requires a dedicated tester at all then perhaps one or two members (at most) could be required for Acceptance Testing on behalf of the PO. They should spend much of their time understanding what the software is to do by discussing it with PO, business analyst, customer and any other stakeholders - something that an analyst/designer traditionally does. They might also attempt to automate much of their work using acceptance test scripts.

Whatever, they do their main focus should be producing working software at the end of the sprint. This means giving the users what they need as opposed to testing for conformance to any written specification.


Conclusion

If you are about to, or have been trying to, move toward an Agile development environment then you should ignore the advice given by many Agile coaches and experts that most roles will not change. There is no longer any need for software designers and less need for testers. Instead the whole team is involved in analysis, design, coding and testing - and this is all done "just in time" (see JIT). (Of course, if they are willing to adapt a role can be found for anyone -- it may be more challenging but will probably involves better use of their abilities.)

Most importantly, managers have less control of the team, which is now self-organizing. The major benefit is simply that a creative team can organize themselves much better than any manager. It also makes for a more rewarding and enjoyable working environment. The developers concentrate on creating good software and overcoming hurdles rather than making themselves look good for their annual performance review or trying to shift the blame when something goes wrong.

It may be hard to overcome the unthinking production line mentality (I do what the boss tells me to do) or vendor mentality (I do what the customer tells me to do). But please try!

Agile's Fifth Element

You have probably heard of the "Agile Manifesto". This was designed to enunciate Agile's fundamental approach by those who coined the term "Agile" software development (see Agile Manifesto).

• Favor individuals and interactions over processes and tools
• Favor working software over comprehensive documentation
• Favor customer collaboration over contract negotiation
• Favor responding to change over following a plan

Elements, Principles, ???

I'm never sure what to call the Agile Manifesto's 4 main statements. They can't be principles as there are also 12 Agile Principles (see Agile Principles).

I have seen various names, such a guidelines, values or statements. I think Elements is the logical choice as it emphasizes their fundamental importance.

For a long time now I have felt there is one thing missing from the above. It is at least as important as a couple of them. I call this Agile's Fifth Element.

• Favor simple design over reusability and generality

What is an Agile Element?

To see why we it should be added to the other four we need to decide how a statement qualifies to be an Agile Element. In my opinion an Agile Element needs to:

• challenge, or at least de-emphasize, (previously accepted) best practice
• be fundamental to the Agile approach
• be important enough not to simply be considered an Agile Principle

There are many books on software project management typical of the Waterfall era (mainly written from the 1960's to the 1980's) that emphasize the importance of following strict procedures, having everything well-documented, and having a water-tight contract with the customer. Most of all they talk about the importance of planning a project down to the smallest detail.

I won't go into why these ideas don't work (again), though the authors make them look very good on paper. Luckily Agile came along and turned all of this on its head. The four elements of the Agile Manifesto clearly highlight what was (and often still is) wrong with what was considered the correct way to develop software.

How Does it Qualify as an Agile Element?

Let's look at how the proposed fifth element satisfies the above criteria by looking at each criterion in turn.

Does it challenge accepted best practice?

The big paradigm to come out of the 1980's was the emphasis on reusability. Many in the Agile world now think that this emphasis was a mistake as I previously explained in Reusability Futility.

In brief, experience has shown that too much software is designed to be flexible and reusable for no purpose. This leads to increased complexity to the detriment of other attributes such as understandability, verifiability, efficiency, etc. The worst effect is that software becomes harder to maintain and cannot adapt to change.

OK, but why is it fundamental to the Agile approach?

Keeping things simple is part of the agile approach. In fact the 10th Agile Principle says:

Simplicity -- the art of maximizing the amount of work not done -- is essential.    

Simplicity is necessary in order to create small, frequent software releases - a major part of the Agile approach. Further, keeping things as simple as possible allows the software to more easily evolve, since unnecessary constraints interfere with continual refactoring -another fundamental feature of Agile development.

designing for flexibility
adds complexity

Of course, you may ask: can't you design software to be both simple and flexible? Sometimes you can find a design that is simpler and more flexible than any alternative but usually there is a trade-off, particularly as modules become larger. The overwhelming general experience is that designing for flexibility adds unnecessary complexity.

  

Is it really important enough to be the Fifth Element rather than just a principle?
  
Yes, it is! It counters one of the biggest problems currently facing software development - the tendency of developers to over-design and add unnecessary code. There are lots of reasons this occurs such as:

• temptation to add something now which is "sure to be needed later"
• desire to use some new design pattern that one has recently been read about
• hope that a general purpose module can have use in other projects
• use of particular new technology to have on your CV
• desire to avoid future code modifications

These reasons, and their consequences, are further elaborated in Gas Factory Anti-Pattern.

If you're still not convinced I recommend reading Chapter 18 of the excellent book 97 Things Every Software Architect Should Know. It is written by Kevlin Henney and entitled Simplicity Before Generality, Use Before Reuse. It says exactly what I have been trying to get people to understand for many years.

Why Is It Missing?

You would have to ask the authors of the Agile Manifesto why they left it out. I suspect that its importance was simply not appreciated at the time.

I guess the first four elements highlight problems with best practice that have been promulgated since the 1960's. The proposed fifth element highlights the problem of something that only became considered best practice later (1980's) - that is designing software for reusability.

Another theory is that many of the creators of the Agile Manifesto were heavily involved in OO (Object-Oriented) technologies. To de-emphasize reusability would be anathema to them, since reusability is a fundamental part of OO design.

A Different Fifth Element Proposal

As I write this post I have discovered that our dear old friend Uncle Bob (Martin) has also proposed an additional element for the Agile Manifesto:

• Favor craftsmanship over execution

In essence he is saying that many developers are only concerned with getting the code working without regard to the quality of the code they produce. I agree with Uncle Bob that this is a serious problem that affects much, if not most, software development.

But Uncle Bob's proposal, though eminently worthwhile, does not qualify as an Agile Element for the reasons mentioned above (see What is an Agile Element?).

First, it does not question accepted best practice since software quality experts have been emphasizing the importance of various quality attributes of software (such as maintainability) for decades. I also do not believe it is fundamental to the Agile approach, though many Agile practices, which could be considered as promoting craftmanship (such as pair programming), will improve code quality.

Summary

My proposal is for an additional element to be added to the existing four elements of the Agile Manifesto. I consider this new element to be a least as important as a couple of the big four - personally I would rank it third. I believe it to be fundamental to the Agile approach as it is practiced today. For example, many practitioners of XP (Extreme Programming) talk about the importance of YAGNI and simple design (see Simplicity is the Key). Similarly in Scrum, developers focus on solving current customer needs by concentrating solely on the most important backlog items.

... software is designed
for general purpose
and, ironically,
ends up being fit
for nothing at all!

The problem of unnecessary generality and reusability is largely unrecognized, but significant and growing. I believe it is mainly due to the training and mentality of developers, particularly software architects. Perhaps with the best intent, software is designed for general purpose and, ironically, ends up being fit for nothing at all!

Far better is to start with a simple design that can be easily modified (with the assistance of Unit Tests) into whatever the future requires. Without the burden of unnecessary complexity the software may be able to evolve forever or at least not become a "legacy" application almost immediately as most software tends to do.

So do you think this is a worthwhile proposal? I would be interested to hear your comments.

Agile Design

There is one aspect of Agile which people have trouble coming to grips with. I talked a little about this recently when I mentioned JIT design (see JIT). You may have heard it called other names like evolutionary design, incremental design, emergent design, continuous design, etc. I will use the term Agile Design or JIT Design.

Some people are adamant that designing like this is fraught with danger or just doesn't work. However, there are many, many projects where it has worked better than anyone expected and certainly better than the Waterfall approach of Big Design Up Front (BDUF). For example, see Continuous Design by Jim Shore.

The opposition is simply due to not really understanding how design is done in Agile development. In particular, using JIT design without using other aspects of Agile methodologies such as regular feedback, collaboration, continuous integration, collective code ownership, etc, will doom it to failure. And as usual I will mention that Unit Tests are crucial (see Unit Tests).

How it works as part of an Agile methodology is hard to understand so I will use an analogy. Hopefully, this allows you to understand my point.

An Analogy

I will use the analogy of designing a house. I chose this analogy as you probably already have an idea on how that's done.

Houses are built from a plan. Often an architect has spent many weeks or months creating a design on paper. Even the worst houses are built from some sort of floor plan. (In fact, in my country, government regulations preclude building without a detailed plan that has been approved by the local council.) The point is that for any house, a lot of time is spent in design and planning before construction can even begin.

Over many years (even millenia) this has been refined to be the best way to build a house. In fact this is essentially the same way that all complex physical devices and structures are built - some sort of designer, architect, or engineer (or team thereof) creates a blueprint which is carefully perused, verified, and approved before the final product is actually built.

If you just went and bought some materials and started building a house do you think it would be anything other than a disaster? Of course not!

So why do I think that designing and building software should be any different? The simple fact is that you can't build a house or software in the conventional manner without a fairly detailed design. However, if you change the way you build the software things are different. This is why Agile software development is considered to be a methodology - it only works when you understand and embrace it, in toto.

Let's see how we might build a house in the Agile manner. Note that, in the discussion below, D stands for developer (the house creator) and C stands for client (the house user).

From this ...

D: What is your most basic requirement for the house?
C: I want something that stops me getting wet when it rains.
D: Try this. [A piece of corrugated iron on four posts.]
C: I keep bumping my head on it.
D: How's this? [The corner posts have been extended.]
C: Better. What about something to stop the wind.
D: Try this. [Four walls attached to the posts.]
C: It's less breezy but I can't get out.
D: I could add a door. ...

to this!

This continues until C has their dream house (or they can't afford to pay for any more changes).

Software is different

I hope you realize that I am not advocating this way of building houses! This would be a particularly slow and expensive way to do so. However, when building software this can be a better way (and I would argue almost always is). Building software is different for these reasons:

1. Modifying physical objects (like extending the above-mentioned corner posts) can be time-consuming and expensive. Software is far more malleable. To make something larger may be a simple matter of changing a constant and recompiling.

2. Continually changing the design of anything of any complexity quickly becomes a nightmare of managing and controlling the interactions. With software this is a little easier because you do not have to worry about physical interactions (such as the location of pipes and wiring). However, it is still important to decouple different parts of the software whenever there are no major drawbacks to doing so (see DIRE). Crucially, Unit Tests can be used to ensure nothing is broken as the software is continuously redesigned and refactored.

3. House design is well-understood and there are many standards and conventions that are well-known, proven and followed. Software design is far more variable. Moreover, the client does not have a very good idea of what they want. Even if they did the developers do not have a very good idea of how to give it to them until they try to build it.

4. For practical reasons design constraints for houses do not change much if at all. For example, the government is not going to one day say that all electrical wiring in all houses is to be changed immediately. (Even if there were an urgent safety issue this would be logistically impossible.) However, these sort of major changes are regularly thrown at software developers. BDUF cannot easily cope with change the way continuous design does.

The Argument For BDUF

I have often read the blog of Joel Spolsky, whom I respect and admire greatly. However, many years ago I read a post which saddened and disappointed me. Part of Joel's blog is even quoted on the BDUF page on Wikipedia as an argument in favour of BDUF.

However, if you carefully read the full paragraph from which the quote is taken, Joel's example is actually an excellent argument for Agile Design. Here is the full paragraph:

"Many times, thinking things out in advance saved us serious development headaches later on. When I wrote the first draft of this spec, I had a more complicated flowchart that assumed that either party (helper or victim) could initiate the process. This became extremely complicated and convoluted for various reasons. As I worked through the screens that would be needed to allow either party to initiate the process, I realized that Aardvark would be just as useful, and radically simpler, if the helper was required to start the whole process. Making this change in the spec took an hour or two. If we had made this change in code, it would have added weeks to the schedule. I can’t tell you how strongly I believe in Big Design Up Front, which the proponents of Extreme Programming consider anathema. I have consistently saved time and made better products by using BDUF and I’m proud to use it, no matter what the XP fanatics claim. They’re just wrong on this point and I can’t be any clearer than that."

- Joel Spolsky, 2006

The first thing that hit me is the mention of the first draft of the spec with the "complicated flowchart". It is obvious that the first draft is already suffering from the problems commonly associated with BDUF. In particular, it is clearly making unnecessary assumptions.

Flowcharts      

Does anyone use flowcharts anymore? Back in 1979 I was told by my lecturer to avoid flow charts (probably due to their association with the evil goto). Personally, I have found UML activity diagrams (and sequence diagrams) to be more useful.

But using Agile Design obviates the need for lots of diagrams.

Luckily (for Joel) he had the experience and insight to be able to identify a problem in the spec and fix it before it made it into the code. Apparently in his development environment code was hard to modify since a change to the spec that takes an hour or two would require weeks of code changes. (Note that this is another symptom of not using Agile techniques such as Unit Tests.)

Now imagine if the designer(s) of the software had less time, less experience or less insight than Joel. They may have gone with the first draft of the spec with the consequent later weeks of recoding that this entailed. This is plausible as I doubt many designers are as experienced and insightful as Joel. So rather than demonstrating why BDUF is better than Agile, Joel's example clearly shows its weakness.

Now consider the Agile approach, which is to first create a minimal working version of the software. In this initial version it might not have even been obvious who (helper or victim) could initiate the process. Users would try the software and provide feedback which is used to evolve it into what they need. In this way the conundrum of who should initiate the process would probably not even have arisen.

Another thing to note is how Joel simulated the use of the software in his own mind by "working through the screens". Not many people can do this effectively. It's always far more effective to have working software than try to imagine how the software would work. (Note that this is another advantage of Agile element 2: Favour working software over comprehensive documentation.)

Finally there is one thing I need to be absolutely clear on: thinking things out in advance is not the only way to avoid serious headaches later on. And most of the time it's not even the best way.


Problems of BDUF

The plain fact is that for more than 50 years BDUF has been tried and it consistently fails. Even for simple, well-understood projects, no matter how hard you try, it's impossible to produce anything approaching a good design without attempting to implement that design. I have seen this time and again. Trust me on this.

"What?" I hear you say. "You can't say that without an explanation." Well, I can if I want to, so there! Actually, I don't need to, since it has been discussed elsewhere. For example, see the Wikipedia page.

I have also touched on the subject many times in previous posts in this blog. For example, it is the principle cause of the Gas Factory Anti-pattern. More recently, I have talked about it in the series of posts on Unit Tests and specifically in the Cost of Rework section in JIT.

But just because you asked so nicely, here is a summary of recognized problems with creating a detailed specification up-front:

• the customer often does not really know what they want, at least till they try it
• it does not adapt to changing requirements
• it tries to anticipate future changes, adding unnecessary complexity
• it tries to document every known "fact" (of which many are usually incorrect)
• it makes unnecessary assumptions about how the design will be implemented
• it is very hard to foresee problems
• many things (often major things) are overlooked
• it takes too long to create - stakeholders become worried when there is little sign of progress
• it separates design from coding, even though they are very closely related (see DIRE)
• it results in a contorted implementation - it's often easier to change the code than fix the spec
• the document is not updated when the inevitable changes are made

There are probably more, but those are the ones that occur to me now.

However, now that I think about it there is one major drawback that I have not seen discussed. This is that developers don't really come to a full understanding of what the software is supposed to do until they actually try to implement it. It's only then that logical inconsistencies, omissions and other mistakes in the design become obvious. Further, limitations (unknown to the designer and often even to the developers) only become apparent when you attempt to write the code.

Possibly most significantly, only when the developers implement the design do better ways to do things occur to them. I have lost count of the times with BDUF when I have finally completed the code, only to then think of a much better solution (simpler, more efficient, more usable, more maintainable, etc).

Agile Design

The Agile approach is only to design what is needed for the next feature or iteration. This is usually the smallest possible addition to the system. The designer should only concentrate on the current iteration and never consider any possible future changes. This is a lot harder than it sounds. (I do concede that occasionally there are exceptions to this rule which I will discuss later.)

Since past iterations did not look to the future, a particular new feature may completely invalidate a large part of the existing design. Don't panic! This is normal. Traditionally such frequent refactoring would create chaos, since every iteration would have the distinct possibility of breaking existing functionality. Without complete regression testing after each iteration bugs would creep in and not be found till some time later

This is why Agile Design requires use of Unit Tests. Unit Tests allow continual redesign and refactoring to be performed without fear of breaking existing functionality.

The advantage of the Agile approach is that the closely related activities of design and coding are done consecutively and, to some extent, simultaneously. As I discussed previously in DIRE separating the design and coding leads to the major problems of Waterfall development.

The one huge benefit that I have found is that, after developing earlier features, the developers are very familiar with how the software works. This greatly improves the chances of a better design being developed for the next iteration. Mistakes and omissions are far less likely and a good (simple, DRY, etc) solution is much more likely.

B2DB Syndrome

Every programmer experiences the B2DB (Back To Drawing Board) syndrome when they are first learning to program. This is where one tries to write some code but in doing so obtains a better understanding of how to do it. The consequence is one has to go back to the drawing board, often many times. (I discussed this in more detail in JIT Testing).

For many this is a very painful experience and they believe there must be a better way, especially (as is often the case) they have already been trained in another field (such as a branch of engineering) where it is essential to do up-front design.

I believe this is the real reason for much opposition to Agile Design - it feels like the bad old B2DB syndrome.

It's true that there is a lot of refactoring going on in Agile Design, but this is actually an advantage. Of course, more experienced developers are going to have a better initial design, but even the best will see better ways to do things once they fully understand what they are trying to do. Also (as all the Agile experts emphasize) it makes the process much more amenable to changing requirements.

Finally, at the risk of sounding like a broken record, I will mention that Unit Tests make the refactoring safer and easier. Rather than having the feel of dread of B2DB, you can actually learn to enjoy refactoring and the process of creating the best design you can find.

Summary

• only make the simplest possible enhancement that will work
• forget the large design document with lots of UML diagrams
• use the code (especially Unit Tests) as the "documentation"
• don't design for the future, concentrate on what is needed now
• don't make the code flexible, just write for the specific task at hand
• try to forget all the extra (irrelevant) information that has been given to you
• don't be afraid to refactor existing code
• use Unit Tests to allow changes to be made without fear of breaking anything

DIRE

The most useful ideas in software design can be applied in different ways and in different settings. They can save a lot of time and trouble if you can learn to recognize when they are appropriate. Otherwise, you can try to keep them in mind when designing and building software.

One such idea is the DRY principle (see the DRY section in Software Design). It simply says that keeping an eye out for duplication (in its many subtle forms), and trying to eliminate it, will save you a lot of time and worry in the long run. The main advantage is that it is easy to remember and, with practice, easy to identify and apply. The DRY acronym was invented by Thomas and Hunt (see The Pragmatic Programmer). Of course, the idea behind it has been around a long time but Thomas and Hunt elucidated and promoted it well.

Unfortunately DIRE is a 4 letter acronym. I know the fashion nowadays is for TLA's but this was the best I could do.

DIRE is an acronym that I invented for another idea. It stands for Don't Isolate Related Entities. It basically means that you should make every effort to keep things together that belong together. There is a close relationship between DRY and DIRE -- DRY is really a special case of DIRE where the "related entities" are so closely related that they can be partially or totally combined.

DIRE may seem obvious but many things conspire to defeat it. It is not always apparent how or why "entities" are closely related. Often related entities are separated for technical (and even non-technical) reasons, or simply by accident. I give examples below to show the broad range of areas and how you might apply it.

Origins

I guess everyone has had the experience of being interrupted in a sequence of tasks. When you come back later you've forgotten most of the context. Continuing where you left off is tedious and error-prone. In particular, interrupting developers a lot is a great way to generate bugs.

Like everyone I quickly discovered this when I started programming professionally (about 3 decades ago). However, it was not until about 2 decades ago that I began to think of this as an example of a more general principle. This was precipitated by my first encounter with JIT.

I discussed JIT in detail last week but in brief JIT is a QA technique where things are done "Just In Time". JIT is particularly useful in software development to:

1. Increase visibility making it easier to find problems in the development process
2. Reduce costs of delay (such as interruptions as discussed above)
3. Reduce costs of rework (such as redoing design and testing)

SPACE vs TIME

JIT is an example of not isolating related activities in time. I guess in that case DIRE actually stands for Don't Isolate Related Events.

There are also many examples of not isolating related things physically (ie in space) - from the order of lines in source code to the seating arrangements of developers.

The cost of rework has by far been the biggest problems in traditional software development. For example, a large up-front design can mean a lot of rework as the design needs to change. But typically, to avoid the cost, we somehow find a way to live with a bad design, with many undesirable consequences which I explained a few months ago (see Change).

JIT
is an example
of DIRE

Gradually developers are realizing the advantages of JIT, and hence DIRE, since JIT is an example of DIRE.  For example, many Agile practices (such as JIT-design and TDD) avoids the costs of rework by applying the idea of JIT, and performing related activities more closely in time.

Examples

Coding

A good example of DIRE as applied to code is the declaration of variables. Separating the declaration of a variable from its use is a common source of bugs (as I mentioned in point 8 of this Code Project Article). This is why C++ improves on C by allowing variables to be declared anywhere not just at the start of a compound statement.

Many Coding Standards work against the advantages of DIRE, by prescribing:

• order of declarations in a header file
• order of functions in a source file
• order of members in a class

Design

Most design practices and patterns do not intentionally contravene DIRE. However, they almost invariably promote decoupling in some form or another. Decoupling by it's nature tends to separate things. Though essential in almost every design, decoupling can be overused or misapplied.

Often a better design can decouple in a different way and not isolate closely related things - a good example is provided by the Layer Anti-Pattern. Sometimes a particular design pattern is appropriate but a cumbersome interface can make communication difficult between two closely related objects. Finally, sometimes complicated design patterns are simply not necessary but are used due to a misunderstanding or zealous desire to try something new.

Management

Managers essentially do two things:

1. help employees to achieve the goals of the organization
2. show that the work is being done

The first of these is very much about using the DIRE principle; but the second can work against DIRE.

First, helping employees is mostly about removing roadblocks. Roadblocks take many forms but are essentially things that prevent people doing their job efficiently and effectively. Removing roadblocks simply means getting rid of something that isolates related entities - such as a developer and the information they require to do their job.

Another part of helping employees is to get them working towards a common goal. This is achieved by fostering communication between them - another good example of DIRE where the related entities are the team members.

The problems occur when a manager allocates tasks and tries to monitor the work being performed. To do this a project must be divided into smaller and smaller tasks. The problem is that dividing tasks, by definition, creates divisions. If the way that tasks are divided is badly done then related tasks are separated in time or done by different people when they should not be. The classic example of this is the Waterfall development methodology as discussed in the next section.

Finally, some managers directly contravene DIRE for their own self-interest. This is why developers are not allowed to communicate directly with users or why they are only given just enough information to do their job. By controlling access and knowledge managers retain power which reduces the likelihood that it will be discovered that they are unnecessary, or not as important, to getting the work done as they like to believe. Of course, this sort of behavior is bad (but still common in the software industry) since it has DIRE consequences.

Process

Development methodologies are one of my pet topics (and the reason for this blog). Here also there are good examples of the benefits of DIRE.

As I mentioned above the best (or worst, depending on how you look at it) example of this is the Waterfall development methodology, which tries to control and monitor the development process, but in the process isolates closely related tasks. The processes of designing, coding and testing of a particular features or piece of software are very closely related. The Waterfall methodology separates them (and not just in time) by splitting the project into phases. This contravention of DIRE is the reason for all the problems with the Waterfall methodology and the main reason for the invention of Agile (as explained below).

the Layer Anti-Pattern in design is analogous to the Waterfall model of development

It's only when you think of these things in terms of DIRE that an interesting analogy between software design and the development process becomes apparent. Splitting development tasks in the wrong way causes DIRE problems. In software design, as we saw above, splitting the design into modules in a bad way causes similar problems. A specific example is the analogy between the Layer Anti-Pattern in software design and the Waterfall model of development - splitting a design into a stack of layers is like splitting the work into a series of phases. In both cases the separation causes problems by isolating closely related entities.

How Do Related Entities become Isolated?

The whole point of DIRE is to bring together things that should never have been separated. But was there a reason for this separation in the first place? Sometimes things are separated by accident but usually it is done for a reason. Often there is a trade-off, where DIRE is sacrificed for some other perceived benefit such as reduced complexity (as in the Layer anti-pattern) or improved monitoring/control (as in the Waterfall methodology).

Divide and Conquer

The most common reason for isolating things is to reduce complexity, using the principle of divide and conquer. This is fundamental to software design and expressed in such terms as modularity, information hiding, decoupling, etc (see Software Complexity).

Although important, it can be used improperly. For example, how software is divided into modules is often done poorly. On the other hand a split *may* be appropriate but the method with which a module communicates (ie, its interface) may be too limiting. And sometimes different parts of code are separated prematurely or for no good reason at all.

Planning

Talk to almost any manager and they will say "You can't achieve anything without a plan". In fact many will believe this with every ounce of their being. I guess this is natural as planning is their raison d'etre. However, it is an exaggeration at best. In fact, there have been many worthwhile achievements in human history which had no plan.

Don't get me wrong - I think a little bit of planning can be a good thing. However, the tendency is to divide a project into many small tasks. Each such task has a well-defined deliverable so that the manager can closely monitor progress to determine if the project is behind schedule.

Unfortunately, detailed planning can interfere with actually achieving team goals. By dividing the project into tasks the implication is that each task can be completed independently, which can result in closely related activities being isolated. Even at a high level dividing a project into stages means that work stops after each major milestone, then gradually builds up before the next deadline. This stop-start work-flow isolates related tasks and leads to poor continuity.

Playing Safe

A common reason for separating related entities is avoiding risk. JIT is a perfect example -- when JIT is used in manufacturing the process becomes crucially dependent on input parts arriving on time. Hoarding inventory is safer but introduces the problems I discussed last week.

In software development a good example of the disadvantages of playing safe is demonstrated in the traditional way of designing software. Here the design is supposed to be fully specified and signed off by the customer before coding is begun. This allows for the developers to protect themselves if  when things go wrong and the customer becomes unhappy. Their usual refrain is "It's not our fault, you signed off on the spec."


Much better results are invariably obtained using the Agile approach of JIT-design (see below). Doing things the right way usually involves more risk which is the reason XP promotes the value of courage (see Extreme Programming). 

Retaining Control

Another, more insidious, reason that managers like to isolate, is to retain control. If developers start talking directly to users then the work may begin to get done without even involving the manager. And since knowledge is power then information is doled out on a "need to know" basis. If you have a boss like this then my advice is simply to look for a new job.

Over-design

Similar to the problem of over-planning is the problem of over-design, exemplified by the Gas Factory Anti-Pattern. Martin Fowler gives a very reasoned account of this in Is Design Dead?

Accidental

Sometimes things are separated for no good, or apparent, reason. This is a problem to especially watch out for as a design evolves.

Agile

DIRE  is at the core of the Agile methodology. Consider the four elements of the Agile Manifesto and how they exemplify DIRE...

1. Favor Individuals and Interactions over processes and tools.

I have always found this rule a bit vague, but I think it is about getting the team working together, rather than following some safe, bureaucratic procedure. Breaking down barriers and encouraging communication and brainstorming is a great example of DIRE.

2. Favor working software over comprehensive documentation.

This is a critical part of Agile and nowhere is it better demonstrated than in the use of Unit Tests. Unit Tests are working software. Moreover, they can work as a substitute for massive amounts of design documentation. In fact they have several advantages over design documentation (see the Documentation section of Unit Test Advantages).

How does this relate to DIRE? Well the problem with a design document is that it is isolated from the "real" design - ie the actual implementation. The only link between the two is through the mind of the developer(s). This has forever been a major source of problems since people misinterpret and misunderstand.

Unit Tests avoid this problem. Using Unit Tests as a substitute for design documentation allows you to take advantage of the DIRE principle. As documentation Unit Tests are directly connected to the code they are designed to document and test.

3. Favor customer collaboration over contract negotiation.

This is similar to rule 1. It is about getting developers to communicate and work with the users. As I said above, contract negotiation is symptomatic of playing safe.

4. Favor responding to change over following a plan.

Responding to change is at the heart of Agile development. Most Agile principles and techniques use DIRE to quickly and more easily respond to change:

• JIT-design brings the design process and corresponding coding together
• continuous integration is about bringing different modules together early to detect problems earlier
• TDD brings coding and corresponding testing together to find bugs faster
• getting developers and users/customer to work more closely together allows quick response to change
• giving users working software earlier finds problems faster

Disadvantages of DIRE

There are no inherent disadvantages of DIRE that occur to me. The main problem is that it may be a matter of opinion the best way to split up a task or problem, such as the design of some software.

For a large problem, we necessarily have to split it up to reduce complexity. Using divide and conquer we can usually isolate and solve sub-problems one at a time until the whole problem is solved. This is the most fundamental principle in software design, and in most technical problems.  See my first post.

As an example, consider the Waterfall development methodology, which splits the process of creating software into phases (design, coding, etc). There are obvious advantages into doing the design of all the parts of the software at the same time, then all the coding together, etc. Doing all the testing at the end should avoid a lot of regression testing. However, experience (and awareness of DIRE) tells us that isolating the design of particular part of the software from the coding, and isolating the coding from the testing causes many problems.

Waterfall Methodology

The Agile approach is to divide the problem up in a very different way. First, quickly get a minimal working version of the software, then add to it over time. I have previously discussed the advantages of this evolutionary approach, ad nauseum, so I won't repeat them here.

Agile Methodology

The disadvantage of the Agile approach is the separation of related activities that occurs.

It's easier to design all the features at the same time since when you go back to design the next part you may have forgotten a lot of the original design. Similarly, most programmers will tell you that it is faster to implement Features 1a and 1b (of the above diagram) together. Finally, there will be a great deal of testing rework, since testing Feature 1b will probably entail retesting feature 1a and the core system again. 

Unit Testing

Fortunately Agile techniques, particularly Unit Testing, come to the rescue in alleviating all these disadvantages:

• Unit Tests act as design documentation making it easy to do design and re-design at any time
• Unit Tests make it is easy to add new features without fear of breaking existing behaviour
• Much testing is automated with Unit Tests which means the re-testing is automated

Summary

DIRE is a simple principle that helps to create better software. It may seem obvious but keeping it in mind allows you to find better approaches to many problems. For example, DIRE is behind most of the principles of Agile.

Using a divide and conquer approach is natural and essential to most tasks and problems, but sometimes how something is divided may cause undesirable consequences. However, by careful application of the DIRE principle it is often possible to reduce the "isolating" effects of the division. Further, looking out for things that should not be separated may allow you to see a different way to do the division. A good example of this is the Layer Anti-Pattern.