Writing an agile contract, without changing your procurement process, is like forcing a square peg into a round hole. As an industry, we’ve tried to do so for more than decade, and we have to accept it doesn’t work. We need a revised procurement process.
This post outlines what I believe such a process should look like. It leverages decades-old techniques to provide trustworthy rigour. It aligns with the realities of agile projects and provides significant protection to both parties.
The flavour of agile assumed here is “Target-Driven Agile”. Target-Driven Agile is the flavour that sits most comfortably with corporate purchasers, but its also the hardest to create contracts for. See my earlier post on flavours of agile for background.
This post is long. I didn’t have time to make it shorter. Sorry ;-)
Step A: Business case for initiation
This first step in the procurement process is entirely internal to the purchasing organisation. Do we have a rough idea for a project? And do we know, very roughly indeed, what it might cost?
At this stage, it’s important not to lock in a budget. Doing so would make a mockery of the pricing and estimation steps that follow. Nevertheless, we still need to arrive at a very rough idea of what the project might cost, in order to decide whether it’s worth proceeding any further. The best answer I know of is Reference Class Forecasting: don’t set the price of this project, but instead find the costs of other similar projects that have already been completed. Reference Class Forecasting is:
- Recognised as the safest approach to early-stage estimation
- A clear signal to everyone that, while we have prices of other projects that are similar, we have not finalized the price of this project yet.
- Arguably more achievable, at this stage, than any other approach to cost estimation. Other approaches require detail, and personnel, that the project doesn’t have yet.
Where you do get the data from? Some customer organisations might have already completed similar projects in-house; while others might be able to ask non-competing organisations who have done similar work. The latter is particularly likely to be useful to public-sector organisations, since they don’t compete with their counterparts in other jurisdictions.
If you can’t find any data on previous projects, then it becomes more difficult. Instead of Reference Class Forecasting you may have to resort to other approaches, such as expert judgement from external consultants. (Take care not to use consultants who may want to bid for the actual work.) On the other hand, if you can find some data, but not enough for any reliable statistical analysis, my personal view is that you’ve probably got enough. Yes, with a limited dataset you can’t use all parts of the Reference Class Forecasting technique, but I suspect you’ll still get enough information to decide whether to proceed to the next steps, and that’s all you need.
It’s also important not to lock in the scope at this stage. Go for a very brief high-level description of the scope rather than traditional “requirements definition”.
Step B: Vendor Selection
The second step is about finding a vendor, and agreeing a contract with them. This is where this suggested process differs most from old-school procurement. Here vendor selection is broken down into several sub-steps to allow discovery and collaboration in the definition of scope, price and contract. There are three benefits of the discovery and collaboration: start as we mean to go on (in terms of agile culture), produce a wisely-defined contract, and reduce project risks to acceptable levels.
Step B-1: Use QBS to create a vendor shortlist
Use Qualifications-based Selection (QBS) to create a ranked short-list of vendors best able to deliver your project. QBS is a rigorous process, well proven in architecture and civil/structural engineering. Under QBS you’re allowed to ask any question except price. See my IITP Newsline article for an explanation of QBS.
Following the QBS process, proceed to step B2 with the top-ranked vendor. Do this without discussing price. At this point, don’t even mention the results of your Reference Class Forecasting, for fear of warping the process by the anchoring effect.
Step B-2: Reality check before investing in scoping
If your organisation is an experienced purchaser of software, and you found good data in your Reference Class Forecasting, I suggest you skip this step and proceed to Step B-3. For you, this step B-2 won’t add any value, and may warp or complicate the process.
But if you weren’t able to find much data in your Reference Class Forecasting, or if your organisation is inexperienced in procuring software, then before proceeding any further I think it’s wise to make sure that your cost expectations are not totally unrealistic. It’s important to make this check in a way that stays true to the spirit of QBS. So keep the numbers vague and only do this after you’ve identified the top-ranked vendor. Ask them just one question, phrased something like this: “We want to make sure our expectations are realistic. Based on what you know so far, do you think this project will be in the tens of thousands, hundreds of thousands, or millions?” The vendor may answer, “Probably in the hundreds of thousands or low millions”. If that’s in line with your Reference Class Forecast, then proceed to the next step without any further ado. But if you thought it was going to cost only 20k, then you’ve discovered useful information! Someone’s expectations are out of whack! Find out whether it’s you, or the vendor, by discussing your respective expectations and the reasons for them. Following these discussions, you’ll need to decide whether to proceed to the next step; to cancel the project; or, following the rules of QBS, to irreversibly abandon your first-choice vendor and try your second choice. But remember, you’re not negotiating price here, you’re just detecting whether it’s sensible to invest in the next step.
Step B-3: Engage with preferred vendor to define scope
This is about having both parties involved in the collaborative work of defining project scope. By involving the vendor, you’re able to leverage their expertise. And after all, isn’t leveraging external expertise one reason why you outsourced in the first place! Involving the vendor also increases their understanding of the problem at hand, setting the scene for success in the stages that follow.
Collaboratively defining the scope is not trivial. So the vendor should not be expected to do it for free. Instead, I’d suggest Time and Materials at a heavily discounted rate. Why the discount? Because the vendor will make their fair profit later, in the main project, and from the customer’s point of view, the discount minimises financial exposure at this stage.
During this step, as the vendor learns more about the customer’s business, and the customer learns more about what the vendor can do, both parties should be encouraged to suggest changes to the scope – especially simplifications.
Scope definition should be at an appropriate level for agile projects, as I described here. The outputs of this Step B-2 are as follows. They can all be incorporated in, or referenced by, the final contract.
- Scope, documented as a simple list
- Sizing points assigned to all scope items. ( The points here are simply for future reference in this project. They do not, and cannot, form any baseline for comparison to other projects because agile points are not comparable between projects.)
- Brief documentation of the likely architectural approach
Step B-4: Negotiate contract with preferred vendor
Having been involved in defining the scope, the vendor now has a good understanding on which to base a price. They should proceed to do so now, preparing an estimated price for the scope that was itemized in the previous step.
Once they have done so, they should share that price with the customer. If the vendor’s price is higher than the customer is comfortable with, the customer should negotiate with the vendor to seek agreement. The customer may:
- Share, at last, the results of their own Reference Class Forecasting. Using relatively objective data, such as this, tends to be a good negotiating tool.
- Ask the vendor to seek more cost-effective design options.
- Agree with the vendor to drop specific lower-priority elements from scope.
In addition to negotiating price, you also need to agree all the normal parts of a contract, such as ownership of intellectual property etc. Five things deserve special attention, because they need to be different from traditional waterfall contracts
- A target cost pricing mechanism. Agree on a form of target-cost pricing. Why target cost? Because, it gives both parties the security of overall parameters and boundaries, while still allowing a necessary degree of flexibility to support agility.
- Timing of payments. Closely relating to the target cost mechanism, is deciding the timing of payments. Will the timing be based on calendar dates, effort expended by the vendor, or progress made? If you choose the latter, I would strongly recommend that it should be measured by percentage of total scope completed (measured in “story points”), rather than old-school milestones. Milestones pull the project in a waterfall direction, whereas paying by percentage of total scope completed retains the necessary agile freedom to adjust the order of development.
- Progress reporting using Agile-style Earned Value. I suggest this should be mandated in the contract. See Step C for how it should be used.
- External advice. I suggest this should be mandated in the contract. See Step C for how it should be used.
- Scope management approach. Active and continuous scope management is key to the success of an agile project. The team should actively thin features, and occasionally “fatten” others, based on what they learn during the project. These discussions and decisions cause numerous small changes to the scope. Without shared expectations of how these changes will be handled, this can become a contentious issue. Whatever your chosen approach, it’s necessary to retain enough flexibility and empowerment to allow the agile team to be agile, but also to have enough control to provide good project management and governance. I’d suggest the contract should specify that:
- Feature thinning is to be a goal of the entire team, and that the entire team is to look for opportunities to do so in all their discussions with users.
- If the revised features still align with the scope as documented (in the above-mentioned Simple List) then no further discussion or approval is required. But, in any instances where cost may be materially increased OR where thinned/dropped features will not meet the scope as (concisely) documented:
- The team must seek agreement from one designated senior customer-side employee (the Product Owner). Verbal agreement is enough for small changes; email for big ones. If agreement is given, the team may proceed immediately with the change.
- AND each month the Product Owner must provide a summary of the material changes to the Project Steering Committee (or its equivalent governance group). The Steering Committee may request further discussion or justification of the changes, and in rare cases may ask the Product Owner to reverse their decision (even if the team has already started, or finished!). Yes, there is a cost to such reversals. But, as long as you have a competent Product Owner, I contend that there would be a greater cost in having the team always wait to hear back from the Steering Committee before proceeding.
- Work will be sequenced such that lower-benefit features will be left until the end. If time becomes scarce at the end of the project, they may be dropped, if the drop is approved through above-mentioned Product Owner and Steering Committee process.
Step B-5: End of Vendor Selection
If everything is OK so far, sign the contract with the preferred vendor. If no suitable agreement can be reached, re-start at step B2 with the second choice vendor. This option, to fall back to the second-choice vendor, is a key element of the QBS process. It gives the first-choice vendor an incentive to be reasonable, and gives the customer protection against getting stuck with a dud vendor.
Falling back to the second-choice vendor should be rare, thanks to Steps B-1 and B-2. But, if it does happen, most of step B-3 will have to be repeated with the new vendor. That’s just the way it is. The new vendor needs that engagement to reach sufficient understanding. You can’t short-circuit it by asking them to simply read documents created by the first vendor. Besides, you don’t like the first vendor any more! So best not to ask the second one to rely on their work!
Step C. Execute the project, tracking always and correcting when necessary
Now that you have your contract, go ahead and execute the project. As discussed earlier in this series, one of the key protections in the agile process is rapid detection and resolution of issues. So you need to make sure that’s happening. I suggest two key mechanisms:
- Earned-value-style burn charts. The burn chart should cover the whole project, not the current sprint only. (You can have a separate sprint chart too, if you like). The whole-of-project chart gives a lot of information to allow detection of problems. It should be updated regularly, e.g. weekly, and shared with both the team and the customer.
- Independent expert review and advice. This is based on a suggestion from Phillip Lee (thanks Phillip!). Phillip’s original suggestion applied to very large government projects. For private-sector projects, and even public ones up to a few million dollars in size, I’d suggest something like this: select a person (or very small group) from outside both organisations to act an external advisor. Have them visit the project at least monthly, look for the following things, and report back to both customer and vendor:
- Earned value. Is the aforementioned whole-of-project burn chart being prepared in such a way that it fairly and correctly depicts the project?
- Order of work, with regard to business value. Is the order of work in the project appropriate for on-going agile scope management?
- Order of work, with regard to risk smoothing. Are risky items being appropriately timed, during approximately the first 2/3rds of the planned duration?
- Business/User Acceptance Testing. If concurrent BAT/UAT is possible, is it being performed and are the tests sufficiently representative of real-world usage?
- Deployments. The team should be regularly deploying with a realistic deployment process to a realistic environment (even if not production yet). Is that the case?
- Is feature-thinning happening?
- Meaningful retrospectives. To what extent are issues raised in retrospectives being dealt with? What is the advisor’s impression of whether the team feels free to raise concerns and break bad news, in retrospectives and/or in more private channels?
If the external advisors are paid by the customer and turn up wearing suits, odds are no-one on the vendor side will trust them! So, I’d suggest that vendor and customer should split the advisor bill 50/50, and let that fact be known to the team. I’d also suggest that the advisors should act, and probably dress, relatively informally. Ideally, the team should be as open with the advisors as they are with each other. After all, everyone is aiming for the same thing: a successful project.
It’s important that no-one is forced to comply with the advisors’ recommendations. They’re just recommendations. I recall being part of a very successful project that used external advisors. We found that they identified very relevant problems, but that often the right solution, in our particular context, was somewhat different from what they suggested.
Any agile contract needs to allow these four types of learning to continue during the project:
- What should we build?
- How (technically) should we build it?
- What will it cost?
- How should we work together?
A traditional procurement process basically forces the first three of those questions to be answered before the project even begins, because they have to be answered in order to write a traditional contract. The answers then get frozen into the contract, discouraging or even preventing on-going learning after that point. (Or at least, discouraging learning from being acted upon.)
Contrast that with the process I’ve proposed here. It puts some of the learning up front, in the collaborative engagement at step B-3, but it does so in a way that is workable for both parties and is compatible with Target-Driven agile. The early collaboration produces answers sufficient to serve as the basis for a fair and realistic contract. Following the signing of the contract, learning continues. The Target Cost contract balances financial protection with on-going learning about “what will it cost”. And the contractually-agreed procedures for agile scope management allow on-going learning about “what should we build”. The mandatory external advice makes sure the learning is happening.
The proposed process draws on Reference Class Forecasting, Qualifications-Based Selection, and Earned Value Management. Each of these techniques is the best available approach to the problem it solves. Each is backed by significant research and practical experience. Each comes from outside the agile community – giving an impartial flavour to the proposed process.
Yes, the proposed process is new and may seem complicated at first. But, if any easy answer existed, it would have been all over the internet by now! In the absence of easy fixes, we need the courage to try difficult ones. If we don’t, we’ll be stuck with old contracts ruining new Agile.
[This is the final post in my series on contracts for agile projects]