The Agile Balanced Scorecard

We recently published a nice article in the online journal "Projects@Work" on frameworks for Agile metrics. An excerpt is below.

Leading and Lagging Indicators
There are a number of frameworks for measuring performance on software development projects, including leading/lagging indicators, efficiency/effectiveness, and the Balanced Scorecard. Is it possible to combine the best of these approaches into a unified system that can be implemented on larger Agile projects?

The U.S. Federal Reserve Board keeps close tabs on two high-level measures of economic performance: GDP and inflation. While growing GDP and simultaneously keeping inflation in check are the end goals, the FRB looks at many other measures as well. That is because GDP and inflation are “lagging indicators” of economic performance - they confirm longer-term trends but cannot predict them. For example, it took macro economists a full year to figure out that we were already in the current recession. So it would be foolish to focus on only these numbers because by the time they indicate that a change is required, it is already too late.

Enter “the leading indicators.” Leading indicators are measures that often give early signals as to where the economy is heading. By looking at such measures as housing starts, durable goods orders, warehouse inventories, unemployment rates, etc., the FRB can see early trends in where GDP and inflation may be headed and decide what actions to take.

What does all of this have to do with Agile and software development? Managers in corporate settings also need to think about leading and lagging indicators of software development performance. They are expected to develop some foresight into what is to come and take proactive measures as appropriate. What metrics can act as early warning indicators of future software development performance - and which can tell us whether or not we were ultimately successful? We will discuss some potential leading and lagging indicators that may be used to assess software development success, particularly in the Agile development space.

Effectiveness vs. Efficiency

Yet another way to think about software development metrics is to consider the sometimes-opposing forces of effectiveness and efficiency. Efficiency measures are focused on how well we planned and used our resources to deliver our product. Budgets, schedules, labor hours, average time per software release, average spend per software release, SLOC, etc., are common examples of software efficiency metrics - here are many more. Effectiveness measures, on the other hand, indicate whether or not our IT solutions meet our business needs. Examples include customer acquisition and retention rates, customer satisfaction, revenue or cost savings, and profitability.

In developing a measurement system for IT projects, we need to think about how to assess the following two questions: “Did we build the right thing?” - our effectiveness - and “Did we build it the right way?” - our efficiency. If you build the right product at the right time, you will have a high likelihood of success and can then work to improve your efficiencies. But if you build a product that few customers want, you can be highly efficient and still suffer a massive loss.

IT has a history of putting more emphasis on efficiency metrics than on effectiveness metrics - but I believe that effectiveness trumps efficiency almost every time.

The Balanced Scorecard

Robert Kaplan and David Norton introduced the Balanced Scorecard approach to organizational strategy design and measurement in 1992. As the name suggests, the idea is that organizations need to balance their strategies and their measurement systems across a spectrum of parameters to achieve long-term success. Since then, the framework has grown into a strategic management system that can be scaled up and down the organization and also used for holistic organizational measurement across the following four basic categories.

Financial - Measures of financial performance. At the corporate level, these might be revenues, profit, earnings per share, etc.

Customer - Measures of customer satisfaction including customer growth, retention, satisfaction, equity, and others.

Innovation and Learning - Measures of internal skills and capabilities. Examples here might be the percentage of revenue from new products, level of education in our workforce, number of training hours delivered to employees, etc.

Internal Business Processes - Measures of processes usually in terms of efficiency such as time-to-market, inventory, throughput, defect rates, etc.

The Balanced Scorecard was designed to be scalable across the organization. That is, high-level corporate measures along these four categories could be pushed down to the business-unit level and functional levels, and also rolled up and integrated organizationwide. In this system, individual departments could contribute to financial performance, customer satisfaction, internal business process improvement and innovation and learning. These four categories are just suggestions, and organizations should modify them to fit their particular needs.

An Agile Balanced Scorecard

All three of these frameworks - leading and lagging indicators, efficiency and effectiveness, and the Balanced Scorecard - have merit. But is it possible to achieve a holistic view across all of these important measurements and make it work at the program and project level where most of the work actually gets done?

The Agile Balanced Scorecard takes the best of these approaches to form a unified measurement system at the project and program level. This is done by modifying its traditional four categories to better align with the way we typically think about projects. Our categories are:

Product: Are we building high-value products that are in demand by our customers?

Financial: Are we achieving planned-for financial results?

Team:Are we working well together as a team and addressing the needs of the various stakeholders?

Schedule:Are we on track to deliver within the schedule commitments?

You could stick with the original four categories of the scorecard, use these modified categories, or define your own when setting up your own scorecard.

For more details and some sample metrics, see the article at Projects@Work.

How Not to Do Resource Management

Every now and then, we will get the interesting question of how to do "resource management" in an enterprise that is adopting Agile. "How can I allocate people 100% to a single team for so long when we have so many projects going on?"   I like to reply that this is actually the wrong question.  Instead of answering the question of how to do resource management, we like to say that the right question is "How can I not do resource management?"  How can I avoid it altogether or at least greatly reduce it?

There are 2 problems in the question above.   The first is the assumption that we need to move people around to staff projects.  The second is related to the number of simultaneous projects.

Moving People Around

There is a tendency to think of projects as the primary unit of organization.   In this view, the project is static and we move people and resources to the project in order to staff it.  When the project is finished, we return the people and resources to the central pool and make them available for the next project.   There are several issues here mostly related to the pe

ople side of things.   Treating people as individual pieces that can be moved around like pieces on a chessboard does not go a long way towards creating high performance teams.  You will recall that teams go through a life cycle of maturity:  forming, storming, norming, performing, and all of that, and this maturation process takes time.  By the time the team is really performing, usually after quite a few months, the project begins to wind down and we break the team up and send staff off to other projects where they have to start the process all over again.  As soon as we get a high performing team we break it up!

In the Agile world, it is common to think of the team as the central unit of organization and we move projects to the team.   In this model, teams go through the maturation process but once they achieve a high performing state, we keep them working together for as long as possible. The team members know each other well, they know their strengths and weaknes

ses, they have learned how to communicate and collaborate with one another and they have hopefully worked out their major differences.  Why would you want to break that up?  Instead, look at the projects that are in the queue and assign the project to the team that is in the best position to deliver from a capacity and skills perspective.

Specialists

Some projects demand the use of specialists who are in limited supply.  In those cases, you may need to share specialists across many teams and the teams will need to adjust their Agile schedules around the availability of the specialists.   Moving only the specialists around is a lot easier than shuffling everyone.  And oh, by the way, if you don't have enough specialists to serve all of the projects, then you have an obvious bottleneck that needs to be addressed.

Too Many Projects

Finally, the biggest problem in most organizations is too many projects and too little focus. Just like having too many cars on the highway, having too many projects in flight is a guarantee for SLOW DELIVERY.  There is plenty of inventory management and queueing theory to support this and we all know it intuitively.   Most IT organizations have a lot of projects going on, hence a lot of spending going on, but not a lot of delivery happening.  Each project (or car) is inching slowly along, fighting for attention and limited resources.  The solution of course is take some cars off of the highway so that the remaining cars can fly!  But that requires the political strength to say "I'm sorry but we are fully allocated and we cannot support your project right now.  We will queue it up for the next available team."  Easy to say, harder to do. This takes strong, decisive management to truly manage demand against capacity in order to keep from overloading the system.  Agile can help however.   Since Agile does require that teams spend a significant amount of time together, it is difficult for them to support many simultaneous projects.  Agile teams are better able to focus and get the project done quickly.   By moving towards Agile on a large scale in your organization, projects will have to be better prioritized and some will have to go into the queue while others get the focus they deserve.  And this is a great step towards achieving throughput and delivery instead of high-spend and never-ending projects.

Mistake Proofing and The Role of QA

Mistake Proofing

Sometimes, we find that we've been doing something for so long, that the way that we do it becomes taken as as a given.  Such is the relationship between software developers and QA testers.  It has unfortunately become standard practice for the developers to write the code and the testers to find the bugs.   This is a common although inefficient method of quality assurance in which defects are allowed to enter the system and then testing is used to try to discover them and have them removed.  A far more efficient and effective method is to borrow from the concept of "mistake proofing" from lean manufacturing systems.   The idea behind Mistake Proofing is to take steps to help ensure that defects never enter the system in the first place.   This leads to higher quality of course but also to lower testing times, lower testing resources, and less re-work.   

In the waterfall world with its functional silos, it is natural for development and QA to have a somewhat adversarial relationship at times.  This is part of the nature of big hand-offs across functional boundaries.  But as we all know, on Agile teams, testers are first-class members of the team who should be working very closely with developers, analysts, etc.   But what exactly is the nature of that work?   How exactly do developers and testers work closely together?   I feel that one of the primary goals should be mistake-proofing.   Mistake-proofing on a software development project involves:

a)   Having developers and testers and analysts discuss feature requirements in great detail so that they all develop a common understanding of the requirements.

b)  Discussing in some detail how exactly we are going to test each feature.  Here is where the tester can add great value to the development process and help in the mistake-proofing process.  The tester tells/discusses with the developer:

• The test scenarios that will be used
  
• The test data that will be used
  
• The boundary conditions that exist and what should happen in those cases
  
• Any use-cases or other work-flow paths that will be taken in order to get to the test-case at hand. 
  
• What the system should do in the event of an error condition
  
• Test automation needs
• etc

In discussing the requirements and testing at this level of detail, we begin to treat test cases as first class requirements that the developer must take into account.   By knowing exactly how the code will be tested, the developer can write code to pass the tests and thereby keep defects from ever entering the system.   This is like getting the questions to the final exam before the final exam;  you ought to be able to pass a test if you know the questions in advance. 

The Role of QA

In this model, the QA role transforms from finding defects to:

1. Specifying how to keep defects out of the system.   
   
2. Providing independent verification that the system works per the requirements
   

 The result; fewer functional defects.  And fewer functional defects means less time and resources spent finding them, tracking them, fixing them, and re-testing them. QA should then be able to focus more of their time on other higher value testing such as integration testing, end-to-end testing,  etc., which are often short-changed in the pursuit of functional defects.  

To Estimate or Not, that is the Question?

One popular debate today is whether estimating is worth the effort. To those experienced with Scrum, this question seems silly, how can you prioritize the Product Backlog, or identify what can fit in a Sprint, without an estimate? Aren't you just rolling the dice if you don't estimate?

This is actually a misunderstanding. If you apply kanban without fixed iterations, the focus is not on how big an effort is, but when it will be complete. As such, instead of estimating size, kanban estimates cycle time, which, in the end, is still an estimate.

The common example given is joining a line. Do you care how long (i.e. big) the line is, or do you care how long you will have to wait (i.e. the cycle time)?

The Corey Ladas Take

Corey Ladas explains the latter view in a posting on his blog, a portion of which is quoted below. I added the boldface to the last sentence:

"Rather than go through the trouble of estimating a scope of work for every iteration, just make the backlog a fixed size that will occasionally run to zero before the planning interval ends. That’s a simple calculation. It’s just the average number of things released per iteration, which in turn is just a multiple of average cycle time. So if you have 5 things in process, on average, and it takes 5 days to complete something, on average, then you’ll finish 1 thing per day, on average. If your iteration interval is two work weeks, or 10 work days, then the iteration backlog should be 10. You can add one or two for padding if you worry about running out. This might be a point that’s been lost on the Scrum community: it’s never necessary to estimate the particular sizes of things in the backlog. It’s only necessary to estimate the average size of things in the backlog."

Now, since I took the quote out of context, you may want to read his entire blog posting, or attend his upcoming class on June 11th in Atlanta, to find out more.

The Cases for and Against Estimating

Many reasons are cited against creating estimates; just a few are listed:

• The estimates are never accurate, so they don’t add value.
• Even if the estimates are relatively accurate, we know we have to do the work, so why waste the time.
• The process of creating an estimate creates resentment when everyone is forced to chime in, and reach consensus, even when they don’t agree.
• The estimate itself causes people to modify their behavior in destructive ways, typically pressuring them to finish, causing low quality. In those instances where they are ahead, the estimate causes them to relax and waste time.

There are also many reasons cited for creating estimates, once again, just a few are listed:

• Determining business value depends upon an understanding of both return and cost, so you can’t determine value unless you estimate effort.
• You can’t plan a Sprint without estimates, or, if you are doing waterfall, you can’t build your schedule without estimates.
• Without estimates the team will lack discipline and will waste large blocks of time, potentially overbuilding a simple feature.

Simple Scenarios

As with most questions in respect to process, the answer is, “it depends”. But, here are some extreme examples to guide the decision on whether or not to estimate.

If you are fixing bugs in a mission critical system, and these bugs are prioritized by severity, and all high severity bugs must be fixed, then it is clear that estimating adds little value. Simply work on the highest priority bugs, and get the fixes released as quickly as possible.

On the other hand, if you are working on new development, or adding a large new feature set, and you must present a complete business case, including expected return and cost, then you will need to create some form of estimates.

Guidance for the Rest of Us

If you are somewhere in the middle of these two extremes, you need further help in deciding how to proceed.

• Remember the difference between size and effort. When you are estimating features, you are estimating size, so use a relative point scale and let the velocity calculation convert size to time. When you are estimating tasks, use an absolute measure like hours/days to represent the time required to complete a task.


• Create your estimates quickly and don’t get paralyzed by precision. Being quick may be the middle ground between estimating and not estimating.


• If you don’t have enough confidence to estimate, don't! Either execute a research spike to find out more, or simply delay creating an estimate until work on related stories or tasks give you an increased level of confidence.

• If technical uncertainty is high, and you must proceed now, allocate instead of estimate. With an allocation the business can limit the maximum effort and the team can monitor real world progress to cut short efforts that won't be finished in the time allocated.
  

• Use either a simple estimation scale, like T-Shirt sizes (S, M, L, XL), or a more complex scale that has increasingly larger gaps. Doubles (2, 4, 8, 16, 32) and fibonacci sequences (1, 2, 3, 5, 8, 13, 21) are both examples of scales with increasingly larger gaps that demonstrate that uncertainly increases along with size.


• Adjust estimates based on actual experience. The counterpoint to this is to bury your head in the sand.
  


• Involve the Product Owner and the business at large.
  • Although it should go without saying, the Product Owner should be deeply involved in the estimation process, providing insight and guidance.
  • The Product Owner should also be involved in the daily stand up meetings where progress is discussed so that he/she can gain an appreciation for the uncertainty inherent in the estimation process.
  • The Product Owner and the ScrumMaster should continually remind the business community at large about the limitations inherent in the estimation process.
  
  
• Most importantly, focus on the journey, not the destination. Sounds abstract, but instead of worrying about the exact estimate (is it 5 Story Points or 8), focus on why people vary on their opinions so that you can better understand the story and the work that will need to be done to deliver it. If you engage the whole team in the estimation process the context gained from can be more valuable than the estimate itself.

In Closing

I estimate that at least 10% of the people reading this post would find it useful, so please send an email or comment on the post so I can see how accurate my estimate was.

COBOL Maintenance, A Model for Modern Agile Processes?

At the Lean/KanBan Conference on Thursday 5/7, Eric Willeke of Inkubook gave a unique, purely visual, well received presentation about Inkubook’s journey to Kanban. They eventually progressed to a mature Kanban approach, with a visual board, strict Work-In-Process (WIP) limits, and short lead times. As a result of a massive layoff, the business analysts were let go and the company size was cut in half. On the positive side his team started to communicate directly with marketing and the now much smaller team was able to swarm, and instinctively keep WIP low, without using the physical Kanban board or setting explicit WIP limits. His talk reminded me of high performing teams of old (I presented on just this topic at the March 3rd Agile Philly Meeting), as such, this blog is about those teams and what we can extrapolate from their experience.

Historical Perspective

In my 25 year professional career, I have seen software teams that have performed at exceptionally high levels by following what I will call a continuous flow approach. For the sake of this blog posting, continuous flow software development can be defined as a software development model where highly collaborative teams work on a small number of tasks at a time, often swarming, to take work from inception to production in very short time frames.

What might be surprising to some, is that some COBOL maintenance teams were early practitioners of this modern approach, many of these teams date back to at least the early 80’s (probably earlier, but I have not personally seen or read about them).

Maintenance teams often had to understand the interaction of thousands of lines of code, just to fix a single bug, and they were frequently under pressure supporting mission critical systems. As such, the best developers were often assigned to maintenance, not to new development!

Typically, team members had many years experience working in one industry, and often for one company. As such they acquired an in depth understanding of the business they supported. They also built close relationships with business peers, meeting on an almost daily basis, often sharing lunch, and frequently meeting outside of work. In short, they were anything but isolationist computer geeks. Since the COBOL language, and the mainframe environments on which they ran, evolved slowly, the programmers were experts with the tools they used, and their many years in the industry allowed them to be experts in the domains and systems they supported.

These teams were small, and they were able to deliver into production with the only requirement being sign-off from their business peers that the system was ready. The team members could do business analysis, design, and systems testing, along with coding, while following a highly agile, continuous flow process. The team members seldom worked in isolation, they pulled in bugs to fix based on input from the business, and they often swarmed on a single bug to quickly complete it.

Practical Applications

From a practical perspective, the historical perspective on high performing teams gives insight that is still valuable today.

It further emphasizes the power of the generalizing specialist concept. A generalizing specialist is a term coined by Scott Ambler, to represent individuals who are a good at many things and a master of a few (as contrasted to a generalist who is not a master of anything). I don't discount the value of specialist, but having some generalizing specialist on the team can result in significant improvements in productivity by providing flexibility on task assignments and simplifying communications (i.e. multiple team members can speak each others language).

Along a similar vane, it demonstrates that swarming is an effective approach. In general swarming helps improve short term effectiveness by leveraging the skills of specialists and decreasing WIP, and it increases longer term effectiveness by increasing knowledge transfer. This knowledge transfer can over time, result in the generalizing specialist as defined above.

Finally, it provides further clarification that keeping WIP to a minimum, pulling work when the team is ready (instead of having it pushed) and emphasizing fast turn around is a natural and effective way to develop software.

WIP limits are core to Kanban, but they also apply to Scrum. In fact, a story point limit on a Sprint Backlog is a form of a WIP limit on what enters a Sprint. You can go one step further with WIP limits by placing them on features, tasks, or both, to control how work is executed within a Sprint.

As an example, you can set a task limit equal to your team size multiplied by 1.5. Thus if your team has 8 members, you would never allow more than 12 tasks in flight at any one time.

Among other things, WIP limits will decrease the thrashing that is created as individuals start a task, then move to another before it is complete. The WIP limit will also make it easier to track progress within a Sprint as your burndown charts will be more meaningful earlier.

You can start applying the concepts immediately by evaluating current WIP and providing input to the team if the number of in-process features or tasks exceeds a reasonable limit, such as more than 2 open features (obviously dependent on the context of team and feature size) or more than 2 open tasks per team member.

Closing

Much of what is new, is an extension of what has proven to work in the past, so if you are not already applying them, consider the concepts of generalizing specialists, swarming and WIP limits to improve the effectiveness of your teams.

PODCAST: Bud Phillips on Agile and Lean at Capital One

One of the best podcast interviews I've heard is the one that Bob Payne conducted with Bud Phillips in 2006 at the Agile conference.  Bud, then a business-side VP of Decisioning Services at Capital One, really brought out some of the best aspects of Agile methods: 

• Taking the Lean perspective on Agile;
  
• Making the initial transition from Waterfall to Lean-Agile;
• The positive impact on team and associate satisfaction, "associates blossom....it's a positive virtuously driven cycle";
  
• The positive impact on customers, including a shared definition of what's valuable, "not functional perfection, but the total outcome of all the work put together" and a closer collaboration with IT teams;
• The "enormous human dimension" of Agile -- how people start to work in a different, trusting way;
• The ability to change incrementally more easily because Agile in this group was business-side driven as  (as opposed to being a purely IT-driven initiative); and
• How Agile challenges leaders, and how they work, allow for room and inspire.

I strongly recommend it.  Here's a link to it: http://agiletoolkit.libsyn.com/index.php?post_id=116686.  

Do let me know what you think of it, too!

Tele-seminar on June 18: Agile PMO

On June 18th, 2009 I'll be presenting a tele-seminar on scaling Scrum beyond individual projects with an Agile PMO.   Check this link for more details: http://www.mmpubs.com/catalog/18-june-2009-the-agile-pmo-teleseminar-p-297.html.