PM Tips

While some might argue that these shifts from the plan on any project are enough reason to abandon traditional project planning altogether, that we need to be agile, in truth it is this very planning, if we understand its true utility, that makes us most agile of all.

First off, we need a reasonable plan, which means that the participants have been involved in developing an understanding of how they intend to work together to solve the problem at hand. This is a far cry from someone dropping a standard WBS into Project and cramming numbers into the tasks so that the overall schedule fits the constraint someone promised. That approach is management suicide.

Once we have that reasonable plan, if we are watching things carefully, we are best equipped to know at the soonest possible moment that we are deviating from that plan. Rather than being disappointed, though, we need to recognize that our system is working and actually be please that we have that visibility. This gives us the greatest possible flexibility or making adjustments so that we can still achieve our overall goals.

Note that this critical step is dependent on one factor, regardless of whether we are working a traditional project network or an agile project. Every team member needs to be completely frank about their progress, and the moment there is a chance that their task will not go as planned, the team needs to be called into action. We wouldn't wait until we were up to our armpits in quicksand before we called for help, but many of us seem to think that if we are falling behind, we need to try to catch up on our own. Nothing could be further from the truth. It is far more of an imposition to the team to not let them know that something is amiss as soon as possible.

With a plan in place and the team working together to deal with issues, we can kick into action. For many issues, it may be sufficient to not take action at all, or to merely keep an eye on the problem. For others, we will need to do something, before that 'do something' means we need to ask for more time or money from the client.

Done well, a plan provides the right backdrop for helping the team make creative decisions that can keep the project moving forward to meet original expectations of budget, scope, time and quality. This is not the same as 'on track', as our goal is not to follow a deterministic path to completion, but to get to a successful completion as we had previously defined. We did define what success would look like, didn't we?

If we can wean ourselves from the expectation that we will follow our planned path to completion, then we have a great tool at our disposal. Our initial view of the network of activities that we had put together, while it showed one possible view of how we would interact to get things done, most probably contained a number of assumptions that could be optimized. Just as a first design for a product is not necessarily the best, just as solution ideas in the analysis stage are one way we could solve the problem, our project network likely has plenty of potential optimization within it.

For many schedules, we can usually optimize our first cut at the relationships to shave 25% to 50% off of the scheduled duration. We can break dependencies between activities, sometimes simply working them in parallel, sometimes at a cost of adding some sort of bridging task to enable that parallel work. We can add resources to some of the tasks, being careful to recognize the potential overhead that we need to add to make this work. We could cut some scope and still maximize value produced or even dumb down the quality of some features if we have a handle on our original expectations in these areas.

This is a program that throws a lot of information at the group over a 3 month period, on a full-time basis. There is certainly a lot to cover, to be sure, and the intent is to give the students a practical grounding in project management principles (as opposed to how to pass a 4-hour multiple guess exam) in the context of group work around a case-study that was provided. The teams evolve through all the project phases from chartering to close-out, and we get to throw some fun curves at them along the way as we track their project schedules to partial completion (so they have to figure out what is going on and decide what to do about it).

What we see is very much like a reflection of the real world. Actually, a bit more mature than many real-world projects where there isn't even a credible idea of how the project will get to closure, other than that time marches on and the project will hopefully ship sometime. The groups have a strong handle on the construction of a reasonable project schedule from a WBS, and can credibly talk to the progress of the project using Earned Value terminology. With so much emphasis and effort going into the mechanics of MS Project and Earned Value, it appears difficult to take the insights to the next level.

That next level is what real project management is all about. It is one thing to be able to tell me what the numbers say from Project, that's something I can do myself should I choose to crack open the MPP file. What excellent project managers do well is to take that data and translate it into information. Why is the project progressing in this manner? What are the options we have at our disposal to remedy any concerns? Which elements of our project are moving along nicely, and is there anything we can reinforce to be sure things stay that way?

No project scheduling software gives us these insights. This is why we have project leaders.

If we recognize the limitations of project scheduling tools (whether they be software-based or post-its on the wall), we can focus on how our gray matter can improve our ability to bring more projects through to completion successfully. With a well-developed schedule and reasonable tracking practices, we can understand why some things are falling behind (or why things are going surprisingly well), distill this down to root causes, and make adjustments quickly to keep things on track. We need to go beyond stating that the project will be late or cost more than expected, then simply asking for more time or budget.

Our insight into the critical path is a critical source of flexibility in how we wrangle our project to completion. Beyond going back to the trough for more resources, the team should be able to adjust their practices to meet overall expectations without any sacrifices. For any reasonably sized project, there can by upwards of 50% or more of adjustments that can be made beyond the original expectation of timelines. Most projects never take advantage of these opportunities.

Another neglected component of most projects is the human element. With the emphasis on which tasks blew their expectations and how much later we expect to be (as reported from the raw schedule data), we often overlook the fact that many of these delays are a result of breakdowns in communication between team members. How well we work together, how effectively we communicate, how engaged we are in the success of the project are critical factors to manage, not just results of how lucky we were in getting things done. Again, no scheduling approach can deal with this at all.

We were working with a number of teams recently in the context of project management training, and one of their early assignments was to craft up a team agreement that could be used to ensure that the interactions between the group were positive and constructive. The team assignments were a significant portion of the work to be completed, and it was critical for overall success that the team be able to function in a cooperative manner.

As is often the case, the teams start the session unaware of what's in store for them, and they can quickly feel that they have been 'dropped into the deep end'. For one group, the overwhelming amount of work drove them to the decision to divide and conquer - a couple of people work on putting the team agreement together, while the others work on the project charter. On the surface, this was a good tactic, as both assignments were completed on time. The true ramifications of this, though, didn't take long to manifest themselves.

I've noted before that it can be deceptively easy for a subset of the team to produce an artifact and assume that it is good enough to use as a basis for moving forward. In requirements training, I intentionally get everyone in the class to independently craft a vision statement in 5 minutes, and the consensus is that it is a relatively easy exercise. When we're the only one involved in filling out a template, it's easy. There is rarely any internal debate about what goes into which field, we have a strong tendency to simply use the first piece of information that fits. We come at it from our own self-consistent perspective, and the job is done quite efficiently. This happens when we develop a charter, or use cases, or any other artifact. We have produced the deliverable, but we have not nearly solved the problem.

That's why we need to consider all of these artifacts as tools for collaboration. Their content is intended to focus and drive the discussion in a specific direction, and to pull out all those differences of opinion that need to be accommodated. The structure of the artifact should be seen as a template for extracting and exploring all the alternative interpretations the team can think of, and resolving these differences now, rather than only thinking of one approach later when one of is tasked with dealing with the issue on our own. This is true for most of the information we produce, but I think it is most essential when we are producing a team agreement. Indeed, if we are careful to work through all our differences in the context of crafting this agreement, then even if we do take a divide and conquer approach for later information on the project, our differences are less likely to destroy the team.

This was not the case with this group. The agreement covered many of the typical things we would deal with in a team agreement, but in a superficial way. As it turns out, one of the points that got them into trouble was the decision to fall back to a majority vote when there was a disagreement about any particular issue. The problem here was that the group (I've used 'group' rather than 'team' intentionally) was made up of individuals with a broad range of experience, and an even broader range of cultural backgrounds. The different experiences had led to different rates of learning, and one of the people in the group quickly grew impatient as others struggled to keep up. As a result, when frustrated with not being able to quickly explain a particular point, this person would call for a majority vote (as per the team agreement). The others in the group weren't equipped to vote in a considered manner. A downward spiral, which quickly resulted in collapse of the group, backed by the argument that they were just following the team agreement.

The range of backgrounds and experience was a problem with this group only because they didn't manage to gel as a team that appreciated and harnessed these distinctions to their advantage. Had they been able to understand each other and see their differences as a collective team strength, they may have made it past this point. Given the context of training over a constrained time period, all we could do was break up the group and adjust the makeup of the teams, despite the fact that we walked away from what I believe could have been one of the richest learning opportunities the group could have experienced. Had this been a real project with a real company, the cost of this breakdown would have made it worthwhile to work through these issues. In a real-life situation, a well crafted team agreement is a critical component for success, even more so as the pressure rises.

A lot of companies think that the same practices that sustained them in the early days are good enough to take them to the next level. More often than not, though, those very practices are contributing to a great deal of risk and uncertainty, and are actually holding them back from making that leap. For something as important as a breakthrough project, you will likely need to consider a breakthrough approach to deal with the scaled complexity and risk, and likely larger team that you are getting into.

Independent of any specific approach, there are a number of questions you need to seriously ask yourself, and be able to answer about your new product as specifically as you can. While these can be useful for the run-of-the-mill things you have been doing, these are absolutely critical when taking on something so new and important. Indeed, you should answer these questions first, then use this understanding as a means of selecting which approach would be most effective to drive the project, not the other way around.

The first cluster of questions helps you understand the bounding box of the proposed product. Set your goals so you have a shot at meeting them. Can you identify a specific vision for the product, including who the primary beneficiary is, the key needs you intend to address, and how this product is differentiated from the competition? Can you quantify the expected value to the end-user or purchaser of the product, as well as the value you expect to gain from developing the product? Can you put a clear box around the product and specifically identify the external entities the system will need to talk to? Are there any constraints in terms of time, resources, scope or quality that you need to adhere to for this project? Perhaps more importantly, as almost no group agrees on this, which of those four dimensions is most important for you, and what are your degrees of freedom in the other areas? What are the risks that could impact this project, and the opportunities that could make life easier for you?

Now you can start planning how to gain an understanding of what you need to build. Can you identify all the different classes of stakeholders that will either influence or be influenced by this product? Here, think about the internal development teams and external regulatory bodies as well as the traditional user-stakeholder. Can you identify someone to reasonably represent every one of these stakeholders, as well as someone to represent those external systems the product will be talking to? Are there specific technologies that you intend to include, and are you sure you are not over constraining the solution by selecting these technologies so early? Looking at the proposed product, can you break it down to identify which analysis techniques are the most appropriate to understand the specific elements, and which people needs to be involved in these analyses? You will probably want to dive into use cases for some elements, GUI and report mockups for all user interface elements, a deep data analysis of the essential information moving within the system and across to outside systems, and likely a host of other approaches to complement these as well.

If you have done all this, you understand the shape of the product and the effort needed to tackle the analysis to really understand the product, but you have not yet done that analysis, and you are not nearly ready to focus on the implementation. While there may have been spot prototypes to validate concepts, to dive more deeply into implementation if you have not used any opportunity for deeper understanding is to forge ahead with higher risk, and usually higher cost. The effort to this point is strategic thinking that will take on the order of hours, and to forge ahead without being able to express the answers to these questions is foolhardy. Unless, of course, the product is not that important to your business.

Now, take all those analysis techniques and the people required and actually perform those analyses. Make all the decisions you can with the right people present, rather than leaving it up to individuals downstream, which would come at a much higher cost. There are more questions here. Are you working to understand the quality of the system you intend to build along with the functionality, all in quantified terms? Are there gaps that have been identified in the analysis that might require different approaches or prototypes for you to get your answer? Are you looking at the software component on its own, or the bigger picture that might include hardware, training, support and maintenance? Are you checking all of these analyses against each for consistency and to spot other holes? If you are in an outsourcing situation or have a team that doesn’t know the domain well, should you drill down to detailed functional requirements? Have you done enough analysis so that the risk of diving into implementation is lower than the risk of running out of time because of too much analysis? If you look at your history and take into account all the downstream effort because of weak analysis in the past, you will likely find that more analysis makes a lot of sense.

Just a few more questions to close out. Are all your original assumptions about that first set of questions still valid? Do you have an arrangement so that as changes occur downstream, you can reflect against all this information you have gathered to confirm that the change makes sense, and keep all this information current and consistent? Have you captured all of this information in a form that allows anyone involved with the project to locate it, and debate change based on that common line in the sand? Finally, after you have done all this and recognized that it is not all that onerous compared to the challenges you likely face on your projects without dealing with these questions, doesn’t it make sense to do this for all projects?

I ran another class from the same company through the training last week, and as they say, the plot thickens. A blend of the previous two results, and a little more insight into where to focus.

As always, the first two attempts go as expected. Horrible failure the first time around, and enticingly close to success next time around. The groups in this class then put together a couple of network diagrams that looked quite similar to the previous ones, identifying many of the same optimizations that are necessary to refine the overall time for construction (the device lends itself – guided somewhat by the instructions – to being built in one reasonable manner).

In the final attempt at construction, the class did quite admirably, and was very close to completion (and within a few seconds of the time predicted by all of the schedules that had been developed in the past) when the wheels essentially fell off. For want of a couple of connections between some of the component parts, the group would have set a new record. Instead, they fumbled about with these very few problems and blew out their overall time by more than 50%.

Watching from the outside (one of the best parts of being an instructor), they fell into the same trap that I have seen with a lot of software teams over the years.

Each individual task was pretty well thought out, with a good understanding of who was going to do what, and when. The challenges came in the handoffs, the interfaces between the components.

When we have something to do, it is usually pretty clear to ourselves how we are going to go about doing it and what completion looks like, even if that understanding evolves along the way. At any point in time, our single viewpoint is self-consistent, almost by definition. If there are uncertainties, even those are consistent (if not, then we have additional challenges). At the end, then, completion of the task is as expected at that time. We rarely disappoint ourselves at the finish line, because we have successfully tempered our own expectations throughout the activity.

Trouble is, though, that in large projects with a lot of interdependent tasks, something that we finish is going to be used by someone else. Even if we have a perfect common understanding with the recipient of our product initially, expectations will diverge as we build our product if they are not in lock step with us. That common understanding is never perfect, though, and any gaps can potentially widen considerably over time.

What we see as completion for our activities is not nearly as important as what the recipient sees. Only the recipient is qualified to assess the ‘doneness’ of our tasks (if you have a spouse, you have likely learned this for yourself). Any imperfections in that common understanding up front means we need to stay in lock-step with the recipient throughout the process to prevent any divergence. This is true with the end customer, of course, but it is the little misunderstandings along the way that build up to the large customer disappointments.

The handoffs are where things go wrong. Most relay races are fumbled at the exchange of the baton, and even with this simple, clear exchange there is a lot of effort in practicing a successful handoff. In software, the handoffs are far more complex, are ill-defined up front, and if you look back, are the source of many of the problems we face on our projects.

Have you ever felt like an outsider on a project? Like you are being operated on rather than participating with everyone else?

I’m in the final stages of writing a book, a project that has already taken 18 months. For the first 17 months it has been, for the most part, a lonely journey. Long periods of time as the sole participant, with periodic interjections of review feedback and a few contacts with the editorial staff to ensure I was on schedule (or at least close to it). Then, a couple of weeks ago, the publisher had what they call their Release to Production meeting: the point where they determine that it’s time for them to drive, to take over the schedule and bring the book to market.

Initially, this felt great, almost like they finally believed this book would come to fruition, and they could commit resources without risk. One morning, though, I woke up with a strange feeling, a sort of deja vu. It took a while, but I realized where I had that sensation before. It was a few years earlier, when I went into the hospital for surgery.

The ailment had been bothering me for quite some time (looking back, the flashes of pain eerily corresponded to the review feedback for the book), but the remedy was still something that our strained medical system categorized as elective surgery, so there was no commitment to schedule. I was to be fit into the queue when there was an opening, or more likely when the situation escalated beyond the 'elective' stage. Finally the day arrived, what I would call the equivalent of a Release to Production point.

The system took over. I was poked and prodded Suddenly I was surrounded by more people than I could imagine. Forms to fill, blood tests and interviews. It was more than 18 hours between when I entered the hospital and when I was in for surgery, so there was time to think and wonder about what was going on, how many people were actually involved behind the scenes. The questions I asked were given perfunctory responses, deeper probes for information fell into a black hole. They were all busy, to be sure, but the important thing was that they were all busy dealing with far more cases than mine. I was a minor part in their day, another routine case with nothing particularly interesting to elevate it above all the others. Regardless of how I felt about the issues, or how personal this was for me.

The same thing is going on now with the book. After having only a couple points of contact with the publisher, I now have the names of almost twenty people with all kinds of roles, and I know there are many more beyond that. More forms to fill out, and some of my queries seem to get lost in the shuffle. I know they have many books in production at any point in time, and have no illusions that my title is anyone’s core focus for any length of time. I have learned to be assertive in my queries, and to do what I can to ensure I still play an active role in this project. There is no morphine or general anesthetic here, but fortunately there isn’t the same kind of pain in this case, either.

In both cases, it is interesting to watch the factory in motion. Efficient systems built to process high volumes of product, it would be horribly inefficient to take the time to focus on any one case. It is a different level of perspective, and though individuals being processed through the system may disagree at times, it does tend to achieve its goals. I recovered from surgery, I know the book will be completed as well (mark your calendars for early November – as Jerry Weinberg told me recently, it is never too early to start marketing...).

It is still difficult, though, to put so much effort into a project and reach a point where control has been taken away. It is not the loss of control so much (frankly, I’m glad someone else was wielding that scalpel), but the loss of visibility into what is happening. I like to know what is going on, both to better understand how the system works, and for the possibility that I might actually be able to contribute in some way if I possess that understanding.

One more tie between the medical world and projects in general.

I think we have a great family doctor. While the diplomas on the wall of his office don’t tell me if he was first or last in his graduating class, what sets him apart is that when I’m in his office, his focus is on me. His empathy is spectacular. If one of the kids is sick, he will call in the evening to see how things are going. Regardless of how many people are waiting, he takes the time to explain what is going on, and he doesn’t treat me like an imbecile. We have a two-way discussion, I don’t get a lecture.

As a result of all this, he benefits as well. I would guess that this focus and empathy makes the job more interesting than writing prescriptions or taking vitals, though that same empathy can make the job more painful at times. More importantly for me, the more informed I am about what is going on, the better equipped I am to provide him with precise information so that his diagnosis is accurate. The better I understand the tradeoffs between different modes of treatment, the more likely we are to agree on an approach that meets all of our needs. Everybody wins.

Back to projects in general. Even when you are working on a project where you are ‘the expert’ and it would appear expedient to simply forge ahead, don’t ignore the concerns or insights of others around you. There is value in empathizing with the needs of others, and often you will find that deeper collaboration will provide stronger ties, better insights, and a more rounded solution. Others will not feel left out, and I would suggest that some of the most effective experts around are those that have the greatest capacity for soliciting and processing a broad range of perspectives.

Apparently, Dwight D. Eisenhower once said "In preparing for battle, I have always found that plans are useless, but planning is indispensable."

I collect quotes like this when it appears they might have some relevance in the work I do. I’m actually dubious about whether Mr. Eisenhower really said this in his life, as I am about all the things attributed to Mark Twain, Albert Einstein, and all the others that are the source of so much wisdom. Whether or not they actually said these things, it is always interesting to run into a situation where the expressed wisdom holds true.

I ran a couple of project management training sessions in the past few months. Both with the same material, for the same organization, but different groups. Throughout the session, we run a simulation where the students are tasked with physically building a small but complex device, with several components finally being integrated into the final product. Initially they try to build with virtually no planning, with results as you might expect. They get another crack at it with this experience behind them, and generally they do a bit better.

We then dive into a discussion of project management ‘hard skills’, culminating in an extensive exercise where the teams are tasked with constructing a detailed project schedule for building the device, just before they try one last time to lower their actual times to build the device. The goal is to have a completed project network, fleshed out with estimates for each activity, an understanding of the critical path, and an expectation of how long it will take them to do the job.

Through all of this, there is the time pressure to build their device as quickly as possible, injected by competition across the groups - "the other group got to this point in this much time – can you beat that"? All great fun.

With the first group, this network planning session was an extremely chaotic time as people scurried about the room, generating all manner of side discussions. It was difficult to discern any progress with the team. Several people voiced some frustration during the exercise, and indeed, nobody in the group managed to put together a complete network. Nobody could express which tasks were on the critical path, and nobody had a defendable expectation of how long the overall project would take.

For the second group, there was far more visible organization. The teams disseminated and shared their knowledge very effectively. At the end of the planning session, each of the four component teams produced schedules that looked very similar, with critical paths and durations that all came in within 20% of each other. Every one was a credible plan of attack for building the device.

With both groups, everyone was fully engaged throughout. Indeed, as I called for breaks leading up to the planning exercise, nobody would leave the room, as they were busy strategizing what do do when they had a final crack at building the device. Both groups managed to identify all of the same optimizations that would bring the overall project duration down, and had a good idea of how to resource the project effectively.

The results when they were unleashed to build the device one last time, armed with this planning experience? The first team, that had been so chaotic in the planning stages and never did generate a complete schedule, ran like a fine tuned engine and completed the project in record time. The second group, despite building a credible schedule of activities and reasonable expectations for duration, ran out of time and didn’t quite complete.

The real kicker here is that the group that finished in record time, did so in a time that was within 10% of what was expected from the other group’s schedules! It appears that while one group did a great job of building a schedule, the other group did a great job of planning how to build the device. Perhaps there is a distinction to be made between project planning and organizing to build a plan.

In the real world, clients couldn’t care less how pretty the schedule is if you never manage to deliver the product. While one group was busy figuring out how the tasks would fit together on the network, the other group was busy ironing out the details of who needed what from whom, and when they would get it. They worked out all the handoffs, everyone knew their role to play, and the actual building went without a hitch. That chaos during the planning activity was the necessary walking through of how they would coordinate to get the job done.

This is not so cut and dried, to be sure. In hindsight, the second group suffered significantly as they never had an opportunity to try to integrate the components of the device in either of their first two attempts, and this threw them for quite a loop in their final attempt. They did progress significantly as a result of their planning efforts.

So it really isn’t the plan that is important, but the activity of planning. The discussions and debate, sometimes heated arguments, can go a long way towards helping the team arrive at a common understanding of what needs to be done, regardless of whether a nice Gantt chart pops out the end of the exercise. I’ve seen many stale Gantt charts up on walls over the years, and many projects have succeeded without a Gantt chart to work with. To my mind, I’d happily lean towards the deeper discussions of how we’re going to coordinate to achieve our goals.

Inevitably, if we build our project schedule based on reasonable expectations of how long it will take to do the work and an initial cut at dependencies between activities, the resulting overall duration will be well beyond our hopes, goals, and promises. This is a challenge that will not go away, and there are different ways to deal with the issue.

Quite often, because this happens time and again, some organizations throw their hands up in frustration and give up on the scheduling approach altogether, in favour of the 'make crazy promises and code like hell' approach to getting things done.

Still others practices the 'work backwards from the deadline and cram everything in' approach, where estimates are simply dividing the available time into enough chunks to allocate to the various tasks on the schedule. Shortchanging our tasks forces us to cut corners, make mistakes, and end up taking longer than it would have if we took the time to do it right in the first place. This fallacy quickly degrades into the 'code like hell' approach, but with the backdrop of a pretty, outdated Gantt chart that provides the façade of reasonable management.

The reasonable approach is to recognize that the expectations that have been set and the original schedule of interconnecting activities are starting points to solving the problem. Depending on the context and the shape of the project, initial expectations can be tempered, scope can be cut, different resources can be applied, quality can be consciously reduced (in rare situations), and dependencies can be broken to bring activities parallel and reduce the overall duration. From most initial schedules, I have seen that the overall duration can be reduced by 50% or more without cutting scope or compromising the estimates of effort required to get the work done.

All this, though, only gets us to an initial schedule that fits expectations. Experience tells us that there will be changes along the way: added scope, missed deliveries ,things we hadn’t considered, task overruns, defects to address. A schedule with known work that fits our constraints is not enough.

We have to practice Safe Project Management.

On most projects, there are external dependencies that we rely on to get our work done. Outside contractors, equipment to be delivered, information we need to get. If we have less control or experience with these outside entities, the uncertainty that they can deliver increases. It is only reasonable that we should place a buffer of time between when we set expectations for getting that external stuff and when we need it on our schedule. The more uncertainty, the greater the buffer. We can reduce the uncertainty by taking the effort in-house, by working with a trusted source, or by emphasizing closer communication, but we can’t ignore it in the schedule. We actually can, of course, and we quite often do, but we get burned by it over and over.

Even for the tasks that we have internally, there is some uncertainty in how long it will take to get the work done. We address this in two ways.

First, we have to get away from scheduling based on optimistic expectations that everything will go perfectly. It never does, but we are still surprised by projects that don’t come in on time when we plan this way. If we recognize that estimates are probability curves, we can build schedules based on most-likely times. Experience tells us what is reasonable (excessive padding is just as bad as being overly aggressive), especially if we track our actual effort against original expectations.

And no, it does not cost us too much time to do this tracking of actual effort.

Secondly, for any string of activities, there will be a corresponding pile of uncertainty across these tasks. It’s a good practice to add a buffer of time at the end of this string to accommodate this uncertainty. This is the basis of the Critical Chain approach (why do we have to name these things?...), and these buffers need to be ruthlessly protected by the project manager, rather than be consumed at the whim of the team.

As we get started in projects in a new domain, there will be a number of activities that we know we’ll need to do, but can’t reasonably estimate how long they will take for this project. Typically in software projects, these are things like ongoing scope change, final test and bug fixing along the way. It’s good to capture these with your best guess for the schedule (and be careful to not whittle the down when other tasks overrun), but it is more important to measure how long they actually do take as the project progresses. If you have done several similar projects, or if you are on the n’th release of an evolving project, you should have a very good handle on how long these activities will take, even if you don’t know the exact tasks in advance. Having the data will help you defend the time estimates and protect these buffers as well.

All these buffers may appear daunting, but what they are is a way of capturing what we know happens on any project anyways. It is important to set firm expectations about why they are there with all stakeholders – they are not simply random padding of estimates with no basis in reality. If you are tracking your numbers reasonably, after a few projects you will be able to point to precisely why the buffers are there.

If we take all this into account in our original scheduling and ongoing maintenance of our schedule (our current expectation of how the project will play out), we have a much higher chance of meeting our expectations, and we have done nothing to whittle down our original estimates. If we can’t make adjustments to meet expectations after these buffers are in place, this is information that we can’t ignore. It is far better to know this in the early stages of a project than at the 11th hour, after we have pulled a few weeks of all-nighters in another attempt to perform a miracle.

Layered on top of this, of course, is the ongoing diligence to ensure that the schedule is adjusted as new information comes in. Lots of work, to be sure. It is effort that we can identify in advance, and it can be frightening, so it becomes easy to not do it. Compared to the effort involved in digging a project out of trouble, it is far less work and far less stressful.

Way back, about 20 years ago (OK, perhaps a bit further back...), I started my career working on embedded systems. At one point, I recall debugging a serial protocol between two devices. There was one particularly nasty problem, one that for quite a while simply refused to be solved. More about that nasty bug and the lessons learned next week.

The strongest memory of that experience was not the technical issue itself, but the interaction with my manager during that period. It seemed that despite the crazy hours we put in to resolve our challenges, no matter what time it was, we could feel our manager’s breath on the back of our necks, and his deep intonations of “When will you be done?”

Think James Earl Jones’ booming voice here. Behind closed doors, we called our manager Darth. In hushed tones, of course.

More recently, while running a simulated project where we set teams off to build a contraption in a finite amount of time, we observe similar results. In the first attempt, there is little chance for the management team to coordinate the group’s efforts and to break the problem down into the right approach. As the team frantically tries to build, management will weakly wander about, simply reminding the team of the time remaining.

Granted, that’s the only thing they can report on, but doing so actually increases the stress on the group, rather than decreasing it. In these same simulations, I have yet to hear management asking the team a simple question: "What could we do to facilitate your getting the job done faster?"

While this is a simulation, it frighteningly reflects the real world that I first saw 20 years ago and still see today. I’m staggered at how often managers seem to believe that asking, begging, admonishing the team to go faster will make a positive difference. Dig deeper, and we see that this is often all they are equipped to do. I have learned over time is that when all a manager can do is ask when you will be done, or tell you how late you are, the team is in serious trouble.

So what to do about these situations? How do we get that monkey, often thinly disguised as a pointy-haired boss, off our backs?

Management is best equipped to express the problem, the team members are best equipped to determine how to solve the problem. Good management recognizes the necessity of the team collaborating to decide on the best approach overall. Management’s primary value then becomes clearing the roadblocks that might prevent this from happening.

What we need, then, to allow us to simply forge ahead with our work, are three things:

• A clear, current understanding of the activities and deliverables that everyone on the team will do and provide to achieve your project goals. With this context, you can discuss alternatives rationally. Management can track progress without having to resort to invasive tactics that make you cringe whenever they are in the area. Most of your activities should point to this clear understanding of work. This is the schedule for the project, and often is reflected in a network that was hastily constructed initially, but has since fallen horribly out of date. Trust me, the value in doing this initially and keeping it current is incredible, especially on tight schedules.
• A list, beyond what you do know, of the things you currently don’t know and the things that might hit the project. These are the things to focus on, to narrow the scope of managing status. These represent the uncertainties that make your estimates too wide to be comfortable, and the risks that could potentially hurt the project. These are the things that should be addressed first, to avoid that "and then a miracle happens" cloud of uncertainty and stress that hits many projects towards the target delivery date.
• A concrete list of things that management can do that will make your tasks easier to accomplish. Few developers have the presence of mind to ask for this, but most management will gladly accept this when offered and do what they can to clear those roadblocks. You don’t ask, you don’t get. If you continue to try to push your way though a difficult problem without asking for help, this area is where you are falling down. When this occurs, even the most carefully planned project can fall off the rails.

Looking back, we had gaps in all three aspects. All of these things need to be built by the team, and kept current by the team. This is the infrastructure around which the team communicates, and ensures that everyone on the team contributes most effectively towards the common goals. Call it the Force.

Monitoring progress becomes straightforward, rather than a never-ending scavenger hunt for hints of progress. The team can focus on the difficult issues, the risky parts, the surprises that will arise. Management has the tools to support the team in the most effective way possible, rather than nervously watching the clock.

For some, a project is done when the task they have been working on has been completed to their satisfaction. Their work is tossed over the wall to someone else, at least until it is tossed back to them.

For others, completion means that the product has been built and is running well enough to ship. Or at least that date on the calendar has arrived.

Others recognize the surrounding collateral, whether it is sales or training materials, a deployment engagement, packaging, or a variety of other items that need to be put together to support the product itself. Completion isn’t done until the whole product is in the customer’s hands.

Still others are aware of the remainder of the lifecycle past these points, where the customer may find defects, request upgrades, or ask for more of the product.

Truly wise teams know that completion comprises all of these elements, and more. A project is not completed until the team has put it to bed properly, with a retrospective of the project. More than a session to complain about the problems that are finally done (for now), a retrospective is the best opportunity for the team to learn from their experience and prevent all these problems from happening to them again next time around.

Few project teams consistently run effective retrospectives. For most, the term post-mortem may actually be a more accurate assessment of the project outcome, but even in medical post-mortems, there is something that has been learned at the end of the day. An effective retrospective is one where the team follows through with changes to their behavior for the next project cycle.

Usually, though, teams will stop short of this goal. Whether they don’t take the time (as they are already late on their new project from of delays on the project they just completed), or simply don’t recognize or understand the value or approach of an effective review of what just transpired, they are missing a huge opportunity for improvement.

For a retrospective to work well, all participants need to participate with an attitude that they are looking for ways to get better, not to expose anyone for blame on the project. It is all about the group learning from its mistakes and reinforcing its strengths. With that in mind, it is almost essential for a strong moderator to drive the event in all but the most disciplined of teams.

Good project teams will start the proceedings off with a review of the measures of the project. Original expectations of schedule, scope, cost and quality, compared to how the project actually played out. Known drivers for any of the major discrepancies are identified and discussed, and everyone has an equal voice.

It can be quite useful for everyone to know that a retrospective is coming as a project commences, so they can collect information throughout the lifecycle that may be relevant at the end. Perhaps it is a key feature that is taken in late in the game and disrupts the architecture, maybe a team member that goes well beyond the call of duty at one point. Both the good and the bad, collecting them along the way is far easier than trying to recall these events after the battle is over.

Another way of building the group memory is to build a timeline of the project on the wall, where everyone can add different colored post-its to represent different perspectives. Visible cues of high and low points, surprises and other key elements can help the team recognize when things started to sway from the happy path.

A good facilitator can effectively bring out the voice of those that tend to sit back and watch the parade go by. Everyone should have a chance to identify the things that worked well on the project, and the things that didn’t go so well. For particularly challenging projects where people quit during the project or the client is particularly dissatisfied, bringing these parties in for participation can add tremendous value – and regain some of the lost alignment.

Even if all this is done, though, the value still has yet to been gained. With all this information, the analysis now begins. Usually, most of the issues can be distilled to one or two key root causes, which will lead to specific things that should be changed for the next project cycle, along with practices that have been beneficial and need to be reinforced.

More often than not, these changes will be in the areas of planning, scope definition, change management, or team communication. These areas tend to be where challenges hit most projects, and there’s nothing like tying the shortcomings of the early stages of a project to the pains felt towards the end to bring the point home to the group.

I was working with a team in a technology company this week, going over some of the soft skills and hard practices of project management. This is a team from a variety of disciplines that works together on projects. They had recently completed a project that people in the organization characterized as a ‘train wreck’. On the surface, one would think there would be opportunities to improve on their performance.

There was a reluctance to discuss that last project, except to say that someone who is no longer with the company handed down the decision that ended up scuttling any hopes of success. With a little digging, it turns out that the issue that burned them in that past project still remains unresolved, and is one of the greatest risks to success on their current project. This current project is already late, only a third of the way through the original schedule, again because of apparently external circumstances.

I sensed a pattern developing here.

What really threw me for a loop, though, was a comment made in response to the variety of techniques recommended for keeping a project on schedule: "But that’s all theoretical stuff..."

I would say I agree, only in that these remain theoretical practices until actually applied to a real project situation. Often, when that happens, people are surprised to find that these practices actually make a difference. Ann Landers once said "Opportunities are usually disguised as hard work, so most people don’t recognize them."

Managing a project well requires a unique combination of creativity, communication, paranoia and diligence, leveraged with a strong understanding of how to apply all that 'theoretical stuff'.

First and foremost is the need to understand all the stakeholders involved in the project. This includes those who use the product, those who build the product, and those who influence how and when the product can be built and used. All these communities, both internal and external, need to be heard and represented on the project. The communication starts here, and must continue throughout the entire lifecycle. Strong relationships go a long way towards solving problems.

Next, when putting together an initial intent of how the project will unfold, we bring in creativity and paranoia.

On almost any project, the first network of activities we put together will typically be at least 50% longer than the time we have available, sometimes two or three times as long. Rather than arbitrarily reducing our estimates of duration to make the activities fit (which is a sure prescription for failure), or cutting features to squeeze things in (which is a sure prescription for disappointment), look for creative approaches to making the paths more concurrent. Break up the bottlenecks into smaller chunks, re-balance resources to optimize your approach, even consider consciously relaxing the quality standards if that is a reasonable option. It is often the case that a project schedule can be reworked to reduce the duration dramatically from that initial network that is laid out.

Lean on the paranoia by recognizing that most of the issues on projects arise at the interfaces, and that there is a critical path that will allow you to focus contingency on a narrow slice of the overall network of activities. Whenever there is something needed from a stakeholder outside of the project team, add a buffer that is sized according to the likelihood of that external group delivering on time. History will show for most projects that failing to buffer here inevitably leads to trouble.

For all the internally driven activities, buffer at the end of the critical paths as a separate entity before project-level milestones, again based on the risk associated with the tasks along the path before it. Guard this contingency with your life as a project manager, as one person noted recently "only to be released on the provision of a fine single-malt". Never buffer against individual activities, as this padding will almost always be consumed.

Even with all these buffers at various points, creativity in constructing your network will often allow you to fit within your constraints, without having to resort to cutting features or whittling away at the original estimates. You can produce an aggressive schedule, while remaining conservative at the same time.

We all know, though, that schedules are obsolete by the time they are posted, and this is where the diligence comes in. The project has to be carefully monitored, particularly along the critical path, along those other paths that may expand to become critical themselves, and to ensure that expectations from externals can be met. There will often be times when the group needs to get together to resolve problems, and creativity continues here to provide novel ways around these challenges.

Well-run projects will have a very different but well articulated path that has led to success. Poorly run projects often have that original intended, now obsolete path, then an amorphous cloud of stuff that was done along the way in an unmanageable fashion, possibly resulting in something, often a disappointment, coming out the end.

Performed well, far more projects can be built and managed to deliver on original expectations than currently do. It requires ongoing effort, to be sure, but is well worth it for all stakeholders involved.

When it comes to getting a great deal of work done, there’s nothing like defining a project as a means of collecting your thoughts, organizing the activities and understanding the expected outcomes for your efforts. This is true if you are building a fence or a software product, but it also applies when the end result is far less tangible as well. The same approach can be very effective in building a competitive position in the marketplace, or systematically evolving your perspective on a topic.

Often, even in organizations that appreciate the value of effective project management, much of the team’s time is spent in activities that are not associated with an identified task on a specific project. Especially in consulting firms, where billable time is essential to track, client projects can be closely monitored while the rest of the time fits into a generic ‘other’ category. Harnessing this otherwise unmanaged activity can be a huge strategic advantage, allowing you to build valuable intellectual property or improve efficiencies in a planned manner.

Recognizing a broader interpretation of projects and applying a more comprehensive structure around shaping projects can reap huge benefits. Beyond the building of your products, the approach makes sense for determining where your career path should go or which direction your business should take (depending on your perspective, I see them as being very similar), putting structure around what can be a chaotic set of home improvement activities, or extracting greater value out of a family vacation.

First and foremost, don’t get caught in the trap of going with the first idea that comes to mind. Resist the temptation to simply dive in when an opportunity arises, think of it as a starting point for brainstorming alternatives. From the perspective of this broader list, the initial idea may not be the most attractive, and often turns out to look like an awful idea after all.

Ideally, this exercise is done in the context of a clear, consistent set of overall goals and values. If these don’t exist, there is a good chance it seems that you (or your organization) is wandering aimlessly. Step back, consider what is important to the organization beyond revenue, and to individuals beyond having a job.

Once you have a broad range of potential projects (recognizing that those that fail to materialize this time around could still be candidates for later), you need a mechanism for paring down the list. Done correctly, this needs to take into account a broad range of considerations, that take into account a range of perspectives from all stakeholders:

• How do these projects play to my strengths? Certainly, if you have built up a differentiating area of expertise, it is natural to focus in this direction.
• Which of these provide the most significant direct value as a result? Here, we look at the value to us, often based on the value we provide to other stakeholders that will make the offering more compelling for them to use.
• Which has the highest uncertainty? In some ways the inverse of playing to our strengths, we need to recognize that no future plays out with certainty. A project that more reliably can provide value is a safer bet.
• Which provides the most improved position? Whether this involves a growth in knowledge, capabilities, or more prominent stature in the marketplace, some projects can serve as stepping stones to make future projects more straightforward to tackle. There may be limited direct return, but many opened opportunities. The product you are building in many projects may simply be your improved expertise that can then open up additional opportunities later.
• Which of these opportunities are manageable now? Given existing resources and reasonable expectations, it is just not feasible to take on projects that are too significant, or involve too much risk. This, again, needs to be balanced against value gained concerns. For software projects, is this a leap we have never taken before? For businesses, is this an entirely new direction that will require significant investment, and do we have the credibility to build a convincing story for investors? Would we be fooling ourselves?

While looking at all of our opportunities against all of these considerations, we need to remember that we will revisit this exercise periodically. The insights we gain here will be valuable to retain for next time around. Dovetailing a number of potential projects into a sequence rather than thinking either/or can help us build a strategic roadmap of activities well into the future. The details several years out will be hazy, to be sure, but this drives us to look further ahead and think about our long-term goals (remember, this is the context we are working from anyways).

The scenario usually plays out in a manner that serves nobody well. The team commits to doing something that they all knew from the start would take an absolute miracle to achieve. They fail miserably, end up taking far more time than allocated, people are disappointed and the results are shoddy.

What started this whole dance, though, is not an estimate. Perhaps capitulation or surrender, maybe a bit of wrangling or fudging, but by no means an estimate.

All this, and the information required to put together a credible estimate was right in front of the team the whole time.

Most of the time, when we say we are estimating, the product of our efforts is a number, or a simple affirmative response. "That should take me two weeks". "Sure, I can get that done by the end of the week". Then off to work we go. A single point number is a target or line in the sand, but not an estimate.

Especially in the early stages of a project, the key component of an estimate is not that line in the sand, but an expression of the uncertainty associated with that number. This uncertainty reflects the unknowns in the information you have been provided. The less you know, the greater the magnitude and importance of that uncertainty. Indeed, early-stage estimates are usually best represented only as a range, with no commitment to a specific single number at all.

Jon Bentley’s Programming Pearls has a great exercise that expresses this concept. You are asked to come up with an estimate for a variety of items, some of which you would know very little about. How many signers were there for the Declaration of Independence? How far apart are the two main towers of the Golden Gate Bridge? The goal is to come up with a range, two numbers far enough apart (based on your knowledge of the problem) such that the chances of the actual number will fall into this range with a 90% probability.

This is the essence of estimation.

People can have absolutely no knowledge of US history and still have a good estimate. People can know nothing more than the fact that the Golden Gate Bridge is a famous bridge and come up with a good estimate. Their ranges will be far greater than a US Historian or a Civil Engineer from San Francisco, but they will still be able to provide a good estimate.

Thinking this way about estimation is counter to what we have been acclimatized to expect. Actually doing well in this can be very difficult initially, but your skills will improve with practice (most people score less than 30% on this quiz the first time through.

It is the width of the range that drives our next decision. Is the range narrow enough for us to proceed with a tolerable level of risk? Is it so wide that we need to dig deeper into the problem and break it down into more precisely estimable chunks?

This digging deeper is one of the primary reasons for expanding our understanding of the system into requirements, architecture and design. Having clarified our understanding of the problem, we have an opportunity to revisit our initial estimates and narrow that uncertainty accordingly. There will be times when you dig down to reveal insights that invalidate your initial assumptions, and your more precise understanding will indicate that you actually can’t get the task completed in the time you had hoped.

The discipline of having to choose a range makes you actually think about what you know about the problem, which is the key. Discussing the challenge in terms of uncertainty allows you to debate rationally, rather than the traditional Name That Tune style of commitment that usually takes place. Acknowledging that uncertainty points you in the right direction for discovering the important issues early and solving the problem in an effective manner.

There was little overlap in attendance between my two talks, so my blatant reuse wasn’t a big issue. Whether or not that lack of overlap is an indication of the quality of my first talk is a separate mystery.

Another mystery is that after all this time, most people in software are still surprised that they spend at least that much time in rework.

During one presentation, there was some healthy debate about exactly how to define rework, with some of the usual concerns. Do we count the time we spend fixing our stuff before we pass it out to others for consumption? What about the time spent in refactoring (never mind the debate about whether we are actually performing healthy refactoring or just politely describing post-implementation design)? How about all that stuff that comes back in maintenance after deploying the system, that’s never charged back to the original development project in the first place?

Someone approached me after the talk to chat about rework. Based on the presentation, he was aware that I had written a few articles for magazines, and he asked me how much time I had spent in rework on those articles. I was about to answer, but I had to stop myself, as I had 2 very different answers battling for dominance. I was now looking at the question of rework from a different perspective.

My response was to pull out the old favorite of consultants everywhere. "It depends".

Let me explain.

The first time I submitted an article for a rigorous review, the feedback I got absolutely floored me. All sorts of comments about how to restructure the paper, questioning some of my key assertions, suggestions that some areas were irrelevant to the topic I was writing about, and some things that just didn’t appear to make any sense to the reviewer.

Time to crack open a six-pack. I was devastated.

It took me the better part of two weeks to get back to reviewing the comments again, and only then because the next deadline was looming. This time around, though, all these comments that were originally demoralizing now actually seemed constructive. They helped make the article flow better, focus more clearly on the topic and more precisely defended my assertions. The review process in writing became a very powerful, sought-after approach to writing, and I have come to appreciate that I learn a great deal and produce a better product because of it.

The same thing had occurred in the software space much earlier (including the six-pack), when I was first exposed to the results of rigorous and effective peer review of my software work products. A thorough review from an external observer cannot help but generate a great deal of potential for refinements to a work product, and this is something that we need to learn to value in a team environment.

Even though I found that both of these first exposures were great learning experiences, they both resulted in what I would call massive rework. The effort working on both of these products more than doubled what I had invested for these first drafts, and – this is the key point – the extra effort caught me off guard and was not adequately scheduled for. Some other things would have to give in order for me to deal with the feedback.

These days, when I put together the first draft for a magazine article or conference and submit it for review, my attitudes about what happens next have changed dramatically. When I’m publishing an article, I don’t have expectations of being done until my reviewers and I are all happy with the results, and this is a long way from the point where I have submitted my first draft. Indeed, the first draft may only be 20-30% of the overall effort expended. All this effort is not rework, this is converging on a solution. It is the same overall effort as the first experiences (minus the angst, perhaps), but a healthier and more reasonable perspective. Given that past experience, I can even estimate and plan for that effort.

Given these experiences, I would define rework as follows. Any time you hand off a work product to be consumed elsewhere and experience a sense of disappointment when that work product comes back to haunt you, that’s the stuff we want to minimize. If you have passed along your work for peer review or test or further implementation and don’t expect, or wish for, or allocate time for dealing with the feedback that may come your way, you are setting your self up for rework. Any time you touch that product again in the future, you time is being spent in the rework category. By this definition, there is no ‘good rework’.

In software, we’re often caught building something on our own and effectively shipping this work product for others to consume, either internally or externally. We’re then disappointed when it comes back to haunt us with problems others have found, and we have to steal cycles from our otherwise planned work to fix the problem. Wrong.

Converging on a shared understanding of what constitutes rework needs to take into consideration what our expectations were in the first place. I’d say that if you find yourself surprised and disappointed that you are back working on something that you hadn’t anticipated revisiting, you are in the midst of rework. Regardless of whether you naively underestimated the content and value of review and constructive feedback. It is this rework that you want to minimize, either by cleaning up your expectations of the effort it reasonably takes to get the job done and scheduling accordingly, or by actually doing this effort so that problems don’t come back to bite you later. Or both.

There were 12 students from a wide range of backgrounds, from across North America. With 2 guides, we set out from Tofino with the gear and food we needed for 5 days on the ocean. For those in the group who had never been in a kayak before, this step was indeed a big one.

The guides provided some initial support, but only enough to ensure that we were reasonably safe and adequately prepared. A practice wet-exit demonstrated to us what we were up against with the Pacific – enough confidence to help us understand we could get out of the kayak when we had to, enough cold to convince us that we would be more comfortable staying upright during the trip.

What the guides did from that point was absolutely wonderful. Rather than directing us during the week, they simply suggested final destination for the trip that the group could agree to. From that point on, they merely acted as technical resources for the group as required. Weather reports from their radio and spot training on different techniques when desired were their major contributions.

All decisions – where to head towards for the day, what to eat and where to stop for meals and bio-breaks – were group decisions, based on all of the information at hand. We considered our progress against our overall objective, the current and projected state of the water and the weather, the obstacles in our path, and the condition of each individual in the group. We consciously worked to ensure that all had a voice in these decisions. We were all free to offer up suggestions for debate at all times, and there were times of disagreement, but we always resolved these differences to forge ahead as a cohesive group.

For each stage of the journey we identified a lead, who would make tactical decisions such as when to rest, when to cross open water, what formation we should take. More often than not, even these decisions were made on a consensus basis. At all times we had agreement on the day’s objective, as well as a couple of contingency plans should something come up. To a person, if the situation arose where it did not make sense to continue to work towards our goal, we would have adjusted our overall plan accordingly.

Once a decision was made, the group worked together efficiently to get the job done. People fell naturally into roles where they could contribute, based on strengths, comfort levels, or interest in trying something new.

We had setbacks along the way. After making it to a certain point toward our destination with time to spare (finishing off the day’s ride in significant swells), the weather forecast indicated gale-force winds for the next day. With one of our overall guiding principles being group safety, we decided to backtrack the next day and take a leeward route that would add another 12 nautical miles to our journey, while still retaining our overall objective. Even with this backtrack, we had a couple of spills that we had to deal with before we got to calmer waters.

By that time, after a few days of cold water and damp conditions, our overall destination at Hot Springs Cove was something we definitely did not want to miss. We forged ahead, and as it turns out, the campsite that we otherwise would have missed became one of the highlights of the trip.

Overall, 14 people with a wide range of experience accomplished more than most of us expected, and any setbacks we had were really blessings in disguise. There were no Gantt charts, no dogmatic status reports, no decisions handed down to the group, no targets that were clearly not achievable. We were all clear on the goal, we made tactical and strategic decisions in a collaborative fashion based on a wealth of current information at hand. We pulled together when we had to, and still let individuals stretch themselves when the opportunity arose. We worked hard and enjoyed ourselves immensely, and were all fully engaged participants. While we didn’t completely gel as a mature team in that brief period, we did have instances of storming that we managed well.