Agile Software Development

In this chapter, I will define what agile software development stands for and look at the key practices of agile development from a manager’s perspective. I’ll tie these practices to the motivations introduced in the previous chapter.

Definitions

Let’s take a look at the agile lingo before we dive into the practices of agile software development.

What Is Agile?

An agile methodology is a framework for software engineering that embraces change. For example, software development is often complex, and requirements are, especially in the beginning of a project, unknown or ambiguous. Therefore, an agile framework must have built-in mechanisms to allow the project to tackle and reduce these uncertainties. These mechanisms are listed as the key practices covered in this chapter. If only one of these key practices is left out, an agile project is incomplete.

But agile also means that the framework itself is flexible and adapts to any situation. That means that the same framework applied to two projects will have two different interpretations. The agile approach is best described as empirical because the process itself must be adapted to its environment, which is not necessarily the organization but could be the individual project. Agile is not a “one size fits all” solution.

Agile Processes

The following agile processes have been successfully adopted in the past in a wide range of industries. Publications, training courses, and a substantial user community exist for each of the processes listed. During this introductory part of the book, written for managers, I’ll provide a quick definition of each of the popular agile processes and information about their origin. Later, in Part III of this book, we’ll take an in-depth look at Scrum as an example of a popular agile project management process.

Extreme Programming

Extreme Programming (XP) was developed by Kent Beck, Ward Cunningham, and Ron Jeffries during the 1990s as a set of dynamic programming practices. Today, XP is the most often adopted agile methodology in the high-technology industry. The most noticeable practices of XP are pair programming and test-driven development, which we will discuss in more detail later in this chapter. Although XP provides planning practices for project management, it is often seen as an agile engineering process.

Scrum

Ken Schwaber and Jeff Sutherland, who developed Scrum in the 1990s, define it as a framework for agile project management rather than an agile process. Scrum has its origins in lean manufacturing (which is described later in this section), iterative-incremental development, and the Smalltalk engineering tools. Scrum provides a simple set of rules. Aside from these simple rules, Scrum is extremely flexible and adaptive to emerging situations. Scrum is a term used in rugby and not an acronym. In a nutshell, Scrum is quite different from existing project management practices. First, the role of a traditional project manager is shared among three different roles: the product owner, the scrum master, and the team. Second, two different backlogs are used to manage scope: the product backlog, which captures the scope of the product, and the sprint backlog, which contains the detail work for the current iteration. A sprint, which is the Scrum synonym for an iteration, is four weeks long. The entire Scrum team meets daily for 15 minutes so that each member can give other team members a quick update. Scrum is very popular in the agile industry. For that reason, I have dedicated Chapter 11 to providing a more detailed overview of the framework—in particular, when it is applied in the context of agile portfolio management.

Dynamic Systems Development Method

Also developed in the mid-1990s, the Dynamic Systems Development Method (DSDM) has its roots in Rapid Application Development (RAD), an iterative-incremental process model that uses prototypes at each stage of development. Compared with agile development, which strives for working software at the end of each iteration, the prototypes might be incomplete and not functioning. The prototypes do provide, however, a great way of including all stakeholders early in the requirements work, because the prototypes might be enough to get feedback—for example, from end users. Prototypes can be created for all aspects of the system, including its architecture. In reality, they are often used with graphical user interfaces. DSDM consists of the following nine principles:

• Active user involvement

• Addressing business needs

• Baselining of high-level scope

• Communication and collaboration among all stakeholders

• Frequent delivery

• Team decision making

• Integrated testing

• Iterative-incremental development

• Reversible changes throughout development

Lean Development

Lean development, originated by Bob Charette, applies the principles of lean manufacturing to software development. The result is a kit of 22 tools. The names of these tools still reflect their manufacturing origin—for example, “eliminate waste.” Mary and Tom Poppendieck are leading advocates of lean development in the agile software development industry, having spoken and written extensively about it.

Unified Process

Listing the Unified Process (UP) here as an agile development process is not entirely correct. Compared with the other processes, the Unified Process is a descriptive process rather than an empirical one. That means the UP describes in text, like a hyperlinked version of a book, what particular roles are required to do, when they do it, and how they do it. As with any book, changes to it are more challenging to redistribute. Although the Unified Process is based on architecture-centric, iterative-incremental development principles, the project phases, disciplines, and relationships between roles and responsibilities provide less flexibility. On the other hand, it is exactly the somewhat fixed description of the process that appeals to large organizations that need to comply with a variety of standards and need to outline, for example, a companywide policy for a software development process. Nonetheless, the UP provides a tremendous step forward from the traditional waterfall process (large, separated, and sequenced software engineering phases) and is often an intermediate step between traditional processes and agile processes. There are two noteworthy flavors of the Unified Process:

• IBM Rational Unified Process (RUP). The RUP is a result of merging three methodologies (Booch, Objectory, and OMT) during the early and mid-1990s into one unified approach. After Rational was acquired by IBM in 2003, an eclipse-based^[1] process authoring tool called the IBM Rational Method Composer (RMC) was developed. The eclipse framework provides a consistent platform for all toolmakers to develop and deploy software products. By using such a framework, developers can organize different tools under one umbrella and view them through perspectives. Therefore, RUP can easily be modified using the RMC.

• OpenUP. OpenUP is an open-source process released in October 2006. A year earlier, IBM Rational donated a significant amount of its “RUP for Small Projects” to the eclipse community (which you can learn more about at http://www.eclipse.org/epf) with the goal of developing a simple iterative-incremental process framework for small projects. Like the commercial version of RUP, OpenUP comes with an eclipse-based free process authoring tool called the Eclipse Process Framework (EPF).

Crystal Clear

The father of the crystal clear process is Alistair Cockburn. Crystal clear, like the other agile processes, is rooted in iterative-incremental development. Alistair’s work is also heavily focused on humans, invention, and the idea of developing software as a corporate game. Interesting in this process is the emphasis of product and methodology fine-tuning at the beginning and in the middle of the iteration. This approach enables every project to evolve not only the system deliverables but also the chosen process itself. The ceremony of project documentation is also delegated to the project level.

Crystal clear also requires the team members to be in close proximity, the identification of real users, and that the team use basic code-versioning tooling. A major differentiator from other processes is the consideration of project communication. The larger the team, the more communication has to be factored in. Whereas a team of 5 or 10 members might easily fit into a team room with very effective communication channels, this setup might not be applicable for a team of 50 or more members. Based on the size of the team, different flavors of the crystal clear family can be adopted.

Agile Manifesto

The agile manifesto (which you can read at http://www.agilemanifesto.org) is the result of a meeting at the Snowbird ski resort in Utah in 2001. Prior to that date, the individual agile processes were referred to as lightweight. I think it was a good idea renaming it to agile because lightweight could have given the impression that it is easy to do and that heavy things were left out. Lightweight could also lead someone to believe that it was incomplete. Once you implement agile development practices, you will see that doing so can actually be difficult and that agile practices are not incomplete. The word agile, however, presents a challenge for the community. Nobody likes to admit that they are not agile, so some developers who call themselves agile are, in reality, not agile in our definition of the practices.

Despite any small misgivings about the name, all 17 participants at the ski resort defined the process and signed the manifesto, which was to become the measure of agility in the years to come. I remember the release of the manifesto, which immediately gave the industry a tangible definition of agile and ground rules for adding new ideas in the future. Still today, the manifesto provides clear direction and is used to discuss and compare agile methodologies, including those in this book. More important in my opinion, the manifesto provides one common roof for all agilists, whatever their favorite agile methodology might be. Here are the core values of the manifesto:

• Individuals and interaction take precedence over processes and tools.

• Working software takes precedence over comprehensive documentation.

• Customer collaboration takes precedence over contract negotiation.

• Responding to change takes precedence over following a plan.

Please note that the left side of each statement is valued more than the right side. What is important and often misunderstood is that the manifesto does not recommend neglecting the values of the right side—for example, project documentation. It simply means that the values on the left are valued more highly. Every agile project team has to find the right balance as a team, but also they must find balance within the organization. Figure 2-1 illustrates a value system for a sample project or organization when it interprets the agile manifesto. In Figure 2-1, the arrows indicate how strictly or loosely the values on the left are balanced compared with the values on the right. But even if the arrow is placed toward the right end, it does not imply that the values on the left are overruled. It means that the organization needs to consider other elements as well.

Figure 2-1. Example of a value system for an agile project

By the end of 2007, more than 4,700 professionals across the information technology (IT) industry had agreed to and signed the manifesto. Among the 17 authors of the manifesto were representatives from the Scrum, Extreme Programming, DSDM, and crystal clear methodologies.

Agile Alliance

The goals of the Agile Alliance (which you can read more about at www.agilealliance.org) are to promote agile development in the software industry. With approximately 4,000 members in December 2007, the Agile Alliance is the largest nonprofit community of agile professionals in the industry. The alliance promotes agile methodologies that comply with the agile manifesto, offers funds to members who promote agile development, offers a library of agile publications, and announces local events in the industry. The flagship of the Agile Alliance is the yearly Agile conference. This five-day conference offers several programs, depending on your interest, and a variety of speakers. The interest in this conference seems boundless. Attendance increased from 675 participants in 2005 to 1,100 in 2006 and to more than 1,600 in 2007. Every year, the conference is sold out, and it seems the topics create a degree of interest limited only by the size of the venue. I agree, large does not necessarily mean better quality, but I believe the attendance records demonstrate that agile development has become a mainstream approach.

Agile Project Leadership Network

Just as the Agile Alliance represents the agile manifesto, so does the Agile Project Leadership Network (APLN) represent the project management declaration of interdependence (PMDOI). The core values of the PMDOI are as follows:

• We increase return on investment by making continuous flow of value our focus.

• We deliver reliable results by engaging customers in frequent interactions and shared ownership.

• We expect uncertainty and manage for it through iterations, anticipation, and adaptation.

• We unleash creativity and innovation by recognizing that individuals are the ultimate source of value and creating an environment where they can make a difference.

• We boost performance through group accountability for results and shared responsibility for team effectiveness.

• We improve effectiveness and reliability through situation-specific strategies, processes, and practices.

The APLN is a nonprofit organization consisting of a community of project leaders organized into local chapters. During the remainder of this book, I’ll connect the principles of the agile manifesto with the principles of the PMDOI. Furthermore, I’ll apply these principles in the context of portfolio management in Part II.