by Ross Pettit - 22 October 2008
When we plan IT projects, we forecast what will be done, when we expect it will be done, and how much it will cost. While we may project a range of dates and costs, rarely do we critically assess a project’s volatility.
Volatility is a measure of the distribution of returns. The greater the volatility, the greater the spread of returns, the lower the expected value of the investment.
In finance, volatility is expressed in terms of price. In IT projects, we can express volatility in terms of time, specifically the length of time that a project may take. Time is a good proxy for price: the later an IT asset is delivered, the later business benefit will be realised, the lower the yield of the investment.
Using Monte Carlo simulation we can create a probability distribution of when we should expect a project to be completed. DeMarco and Lister’s Riskology tool calculates a probability curve of future delivery dates. It allows us to assess the impact of a number of factors such as staff turnover, scope change, and productivity on how long it will take to deliver. In addition to projecting the probability of completion dates, the Riskology tool will also calculate the probability of project cancellation. The inclusion of fat tail losses makes this a reasonably thorough analysis.
Monte Carlo simulation gives us both an average number of days to complete a project and a variance. Using these, we can calculate the coefficient of variation to get an indicator of project variability.
Average (days) to complete the project
The coefficient of variation is a metric of project variance. The higher the CV, the greater the volatility, the greater the risk there is with the investment.
Historical project data allows us to put a project's CV in context. Suppose we have metrics on prior projects that show the projects IT does for Accounting have an average variance of 1 day late for every 10 days of plan, with a variance of 0.5 days. Suppose further that a new project expected to last 6 months is forecast to be 6.5 days late with a variance of 3 days. Somewhat akin to assessing the Beta (although it is not calculated entirely from historical data), this comparison allows the IT project investor to ask piercing questions. What makes us think this project will be different from others we have done? Why do we think the risk we face today is so much different than it has been in the past?
It is appealing to think that by supplying some indication of the volatility and Beta we’re giving investors in IT projects a complete picture, but there are limits to what this tells us. Our risk assessment is only as good as our model, and our model will be laden with assumptions about people, capabilities, requirements and opportunity. What if we fail to completely identify and properly anticipate both probability and impact of the risks that we are so carefully modeling? Come to think of it, isn’t this what got Wall Street into so much trouble recently?
Risk management is a primitive practice in IT. All too often, it is little more than an issue list with high/medium/low impact ratings. Typically, risks are neither quantified nor stated in terms of impact to returns. Worse still, because they’re simply lists in spreadsheets, they’re often ignored or denied. This means that the IT project manager is focused on internal risks, such as technology challenges or days lost due to illness. They don't pay much attention to external risk factors such as a regionally strong tech economy that threatens to lure away top talent, or a supplier teetering on the edge of receivership. By quantifying the impact of risks – both individually and collectively – the project manager comes face-to-face with the broader environment on a regular basis.
Focusing on volatility also makes it obvious that the “on time and on budget” goal is achieved not so much by what goes right as much as by what doesn’t go wrong. That, in turn, strongly suggests that being "on time and on budget" is a misdirected goal in the first place. It also suggests that traditional project management has less to do with project success than we might think.
Let’s consider why. Traditional IT manages to the expectation that everything can and should go according to plan. Changes in scope, changes in staff, and even mistakes in execution are treated as exceptions. Traditional IT management looks to reconcile all variations to plan and will attempt to do so even when the sum of the variations – 100% staff turnover, 100% requirements change – creates a completely new project. The traditional IT project manager is trying to maximise effort for available budget: “the project will be on time, what might go wrong to make it late, and what must we do to compensate.”
By comparison, Agile project management expects change. It focuses on reducing volatility by making deliveries early and often. The Agile project manager is trying to maximise yield given available time: “the project will be late, what can we do to bring it forward, and what can we do to reprioritise.” This is a subtle - but critical - shift in thinking that moves risks from the fringe to the center of the frame.
Risk analyses are supplements to, not replacements of, good management. Risk models are abstractions. No risk model will capture every last exposure. Assumptions in underlying numbers will be wrong or arrived at inexpertly. Worse still, they can be gamed by participants to tell a particular story. All this can lead to poor investment decisions and ultimately investment failure. Still, while working in blind adherence to risk models is hubris, working without them is reckless. Investment managers (or in IT, project managers) are paid for their professional judgment. Investors (IT project sponsors) are responsible to audit and challenge those entrusted with their capital. What managers and sponsors do pre-emptively to ensure project success is far more complete when risk analyses are brought to bear.
The higher the expected yield of an IT project, the more important it is to be aware of volatility. If volatility isn't quantified, the project lacks transparency, and its investors have no means by which to assess risk. This leads to a misplaced confidence in IT projects because long-term success is assumed. If the recent financial turmoil has taught us anything, it's that we can assume nothing about our investments.
No comments:
Post a Comment