August 12, 2017 | admin [This is an edited version of a post originally entitled Creating an Agile Contract] Writing an agile contract, without changing your procurement process, is like forcing a square peg into a round hole. As an industry, we’ve tried to do so for more than decade, and we have to accept it doesn’t work. We need a revised procurement process. This post outlines what I believe such a process should look like, based on my experience of about 10 years’ worth of agile projects. The recommendations here leverage decades-old techniques to provide trustworthy rigour. They align with the realities of agile projects and provide significant protection to both parties. The flavour of agile assumed here is “Target-Driven Agile”. Target-Driven Agile is the flavour that sits most comfortably with corporate purchasers — more so than the better known “Exploratory” flavour — but its also the hardest to create contracts for. Hence this post 😉 Step A: Business case for initiation This first step in the procurement process is entirely internal to the purchasing organisation. Do we have a rough idea for a project? And do we know, very roughly indeed, what it might cost? At this stage, it’s important not to lock in a budget. Doing so would make a mockery of the pricing and estimation steps that follow. Nevertheless, we still need to arrive at a very rough idea of what the project might cost, in order to decide whether it’s worth proceeding any further. The best answer I know of is Reference Class Forecasting: don’t set the price of this project, but instead find the costs of other similar projects that have already been completed. Reference Class Forecasting is: Recognised as the safest approach to early-stage estimation A clear signal to everyone that, while we have prices of other projects that are similar, we have not finalized the price of this project yet. Arguably more achievable, at this stage, than any other approach to cost estimation. Other approaches require detail, and personnel, that the project doesn’t have yet. Where you do get the data from? Some customer organisations might have already completed similar projects in-house; while others might be able to ask non-competing organisations who have done similar work. The latter is particularly likely to be useful to public-sector organisations, since they don’t compete with their counterparts in other jurisdictions. It’s also important not to lock in the scope at this stage. Go for a very brief high-level description of the scope rather than traditional “requirements definition”. Step B: Vendor Selection Step B-1: Use QBS to create a vendor shortlist Use Qualifications-based Selection (QBS) to create a ranked short-list of vendors best able to deliver your project. QBS is a rigorous process, well proven in architecture and civil/structural engineering. Under QBS you’re allowed to ask any question except price. See my popular ITP Newsline article for an explanation of QBS. Step B-2: Reality check before investing in scoping (You might skip this step if you are experienced in software procurement and found relevant Reference Class projects.) If you weren’t able to find much data in your Reference Class Forecasting, or if your organisation is inexperienced in procuring software, then before proceeding further I think it’s wise to make sure that your cost expectations are not totally unrealistic. It’s important to make this check in a way that stays true to the spirit of QBS. So keep the numbers vague and only do this after you’ve identified the top-ranked vendor. Do not share your own pricing expectations (since doing so warps the process due to the anchoring effect.) Instead, ask them just one question, phrased something like this: “We want to make sure our expectations are realistic. Based on what you know so far, do you think this project will be in the tens of thousands, hundreds of thousands, or millions?” The vendor may answer, “Probably in the hundreds of thousands or low millions”. If that’s in line with your Reference Class Forecast, then proceed to the next step without any further ado. But if you thought it was going to cost only, say, $20k, then discuss your respective expectations and the reasons for them. Then decide whether to proceed to the next step; to cancel the project; or, following the rules of QBS, to irreversibly abandon your first-choice vendor and try your second choice. But remember, you’re not negotiating price here, you’re just detecting whether it’s sensible to invest in the next step. Step B-3: Engage with preferred vendor to define scope By involving the vendor in scope definition, you’re able to leverage their expertise. And after all, isn’t leveraging external expertise one reason why you outsourced in the first place! Involving the vendor also increases their understanding of the problem at hand, setting the scene for success in the stages that follow. Collaboratively defining the scope is not trivial. So the vendor should not be expected to do it for free. Instead, I’d suggest Time and Materials at a heavily discounted rate. Why the discount? Because the vendor will make their fair profit later, in the main project, and from the customer’s point of view, the discount minimises financial exposure at this stage. The outputs of this stage are as follows. They can all be incorporated in, or referenced by, the final contract. (For details, see here.) Scope, documented as a simple list Sizing points assigned to all scope items. ( The points here are simply for future reference in this project. They do not, and cannot, form any baseline for comparison to other projects because agile points are not comparable between projects.) Brief documentation of the likely architectural approach Step B-4: Negotiate contract with preferred vendor Having been involved in defining the scope, the vendor now has a good understanding on which to base a price. The vendor should estimate the price for the scope that was itemized in the previous step. Once they have done so, they should share that price with the customer. If the vendor’s price is higher than the customer is comfortable with, the customer should negotiate with the vendor to seek agreement. The customer may: Share, at last, the results of their own Reference Class Forecasting. Using relatively objective data, such as this, tends to be a good negotiating tool. Ask the vendor to seek more cost-effective design options. Agree with the vendor to drop specific lower-priority elements from scope. In addition to negotiating price, you also need to agree all the normal parts of a contract, such as ownership of intellectual property etc. Five things deserve special attention: Pricing model. While not essential for agile projects, I believe target-cost pricing is best. It gives both parties the security of overall parameters and boundaries, while still allowing a degree of flexibility to support agility. Timing of payments. Closely relating to the target cost mechanism, is deciding the timing of payments. Will the timing be based on calendar dates; effort expended by the vendor; or progress made? If you choose the latter, I would strongly recommend that it should be measured by percentage of total scope completed (measured in “story points”), rather than old-school milestones. Progress reporting using Agile-style Earned Value. I suggest this should be mandated in the contract. External advice. I suggest that the contract should mandate that the team obtains (non-binding) external review of their process and architecture. See below for more on External Advice. Scope management approach. An agile team should actively thin features, and occasionally “fatten” others, based on what they learn during the project. These discussions and decisions cause numerous small changes to the scope. Without shared expectations of how these changes will be handled, this can become a contentious issue. I’d suggest the contract should specify that: Feature thinning is to be a goal of the entire team (both customer and vendor staff). If the revised features still align with the scope as documented (in the above-mentioned Simple List) and there’s no material effect on total cost, then no further discussion or approval is required. But if the features don’t align OR the cost may be materially increased, then: The team must seek agreement from one designated senior customer-side employee (the Product Owner). Verbal agreement is enough for small changes; email for big ones. AND each month the Product Owner must provide a summary of the material changes to the Project Steering Committee (or its equivalent governance group). The Steering Committee may request further discussion or justification of the changes, and in rare cases may ask the Product Owner to reverse their decision (even if the team has already started, or finished!). Yes, there is a cost to such reversals. But, as long as you have a competent Product Owner, I contend that there would be a greater cost in having the team always wait to hear back from the Steering Committee before proceeding. Work will be sequenced such that lower-benefit features will be left until the end. If time becomes scarce at the end of the project, they may be dropped, if the drop is approved through above-mentioned Product Owner and Steering Committee process. The Product Owner should report proposed drops to the Steering Committee as soon any such drops appear likely. Don’t surprise the Committee at the end, if you can break the bad news earlier. Step B-5: End of Vendor Selection If both parties are happy with the contract, then sign the contract with the preferred vendor. If no suitable agreement can be reached, re-start at step B2 with the second choice vendor. This option, to fall back to the second-choice vendor, is a key element of the QBS process. It gives the first-choice vendor an incentive to be reasonable, and gives the customer protection against getting stuck with a dud vendor. Step C. Execute the project, tracking always and correcting when necessary Now that you have your contract, go ahead and execute the project. As discussed earlier in this series, one of the key protections in the agile process is rapid detection and resolution of issues. I suggest two key mechanisms for detecting issues: Earned-value-style burn charts. The burn chart should cover the whole project, not the current sprint only. (You can have a separate sprint chart too, if you like). The whole-of-project chart gives a lot of information to allow detection of problems. It should be updated regularly, at least weekly, and shared with both the team and the customer. Independent expert review and advice. This is based on a suggestion from Phillip Lee (thanks Phillip!). Phillip’s original suggestion applied to very large government projects. For private-sector projects, and even public ones up to a few million dollars in size, I’d suggest something like this: select a person (or very small group) from outside both organisations to act an external advisor. Have them visit the project at least monthly, look for the following things, and report back to both customer and vendor: Earned value charting – existence and acceptable accuracy. Order of work, with regard to business value. Order of work, with regard to risk smoothing. Business/User Acceptance Testing. If BAT/UAT can be done concurrently with on-going development, is it being performed and are the tests sufficiently representative of real-world usage? Deployments. The team should be regularly and realistically deploying, even is only to a BAT/UAT environment. Meaningful retrospectives. To what extent are issues raised in retrospectives being dealt with? What is the advisors’ impression of whether the team feels free to raise concerns and break bad news, in retrospectives and/or in more private channels? I’d suggest that vendor and customer should split the advisor bill 50/50, and let that fact be known to the team. It’s important that no-one is forced to comply with the advisors’ recommendations. They’re just recommendations. I recall being part of a very successful project that used external advisors. We found that they identified very relevant problems, but that often the right solution, in our particular context, was somewhat different from what they suggested. Conclusion Any agile contract needs to allow these four types of learning to continue during the project: What should we build? How (technically) should we build it? What will it cost? How should we work together? A traditional procurement process basically forces the first three of those questions to be answered before the project even begins, because they have to be answered in order to write a traditional contract. The answers then get frozen into the contract, discouraging or even preventing on-going learning after that point. (Or at least, discouraging learning from being acted upon.) Contrast that with the process I’ve proposed here. It puts some of the learning up front, in the collaborative engagement at step B-3, but it does so in a way that is workable for both parties and leaves plenty of opportunity for useful learning later. The proposed process draws on Reference Class Forecasting, Qualifications-Based Selection, and Earned Value Management. Each of these techniques is recognised as the best known approach to the problem it solves. Each is backed by significant research and practical experience. Each comes from outside the agile community – giving an impartial flavour to the proposed process. [This is the final post in my series on contracts for agile projects] Post
[This is an edited version of a post originally entitled Creating an Agile Contract] Writing an agile contract, without changing your procurement process, is like forcing a square peg into a round hole. As an industry, we’ve tried to do so for more than decade, and we have to accept it doesn’t work. We need a revised procurement process. This post outlines what I believe such a process should look like, based on my experience of about 10 years’ worth of agile projects. The recommendations here leverage decades-old techniques to provide trustworthy rigour. They align with the realities of agile projects and provide significant protection to both parties. The flavour of agile assumed here is “Target-Driven Agile”. Target-Driven Agile is the flavour that sits most comfortably with corporate purchasers — more so than the better known “Exploratory” flavour — but its also the hardest to create contracts for. Hence this post 😉 Step A: Business case for initiation This first step in the procurement process is entirely internal to the purchasing organisation. Do we have a rough idea for a project? And do we know, very roughly indeed, what it might cost? At this stage, it’s important not to lock in a budget. Doing so would make a mockery of the pricing and estimation steps that follow. Nevertheless, we still need to arrive at a very rough idea of what the project might cost, in order to decide whether it’s worth proceeding any further. The best answer I know of is Reference Class Forecasting: don’t set the price of this project, but instead find the costs of other similar projects that have already been completed. Reference Class Forecasting is: Recognised as the safest approach to early-stage estimation A clear signal to everyone that, while we have prices of other projects that are similar, we have not finalized the price of this project yet. Arguably more achievable, at this stage, than any other approach to cost estimation. Other approaches require detail, and personnel, that the project doesn’t have yet. Where you do get the data from? Some customer organisations might have already completed similar projects in-house; while others might be able to ask non-competing organisations who have done similar work. The latter is particularly likely to be useful to public-sector organisations, since they don’t compete with their counterparts in other jurisdictions. It’s also important not to lock in the scope at this stage. Go for a very brief high-level description of the scope rather than traditional “requirements definition”. Step B: Vendor Selection Step B-1: Use QBS to create a vendor shortlist Use Qualifications-based Selection (QBS) to create a ranked short-list of vendors best able to deliver your project. QBS is a rigorous process, well proven in architecture and civil/structural engineering. Under QBS you’re allowed to ask any question except price. See my popular ITP Newsline article for an explanation of QBS. Step B-2: Reality check before investing in scoping (You might skip this step if you are experienced in software procurement and found relevant Reference Class projects.) If you weren’t able to find much data in your Reference Class Forecasting, or if your organisation is inexperienced in procuring software, then before proceeding further I think it’s wise to make sure that your cost expectations are not totally unrealistic. It’s important to make this check in a way that stays true to the spirit of QBS. So keep the numbers vague and only do this after you’ve identified the top-ranked vendor. Do not share your own pricing expectations (since doing so warps the process due to the anchoring effect.) Instead, ask them just one question, phrased something like this: “We want to make sure our expectations are realistic. Based on what you know so far, do you think this project will be in the tens of thousands, hundreds of thousands, or millions?” The vendor may answer, “Probably in the hundreds of thousands or low millions”. If that’s in line with your Reference Class Forecast, then proceed to the next step without any further ado. But if you thought it was going to cost only, say, $20k, then discuss your respective expectations and the reasons for them. Then decide whether to proceed to the next step; to cancel the project; or, following the rules of QBS, to irreversibly abandon your first-choice vendor and try your second choice. But remember, you’re not negotiating price here, you’re just detecting whether it’s sensible to invest in the next step. Step B-3: Engage with preferred vendor to define scope By involving the vendor in scope definition, you’re able to leverage their expertise. And after all, isn’t leveraging external expertise one reason why you outsourced in the first place! Involving the vendor also increases their understanding of the problem at hand, setting the scene for success in the stages that follow. Collaboratively defining the scope is not trivial. So the vendor should not be expected to do it for free. Instead, I’d suggest Time and Materials at a heavily discounted rate. Why the discount? Because the vendor will make their fair profit later, in the main project, and from the customer’s point of view, the discount minimises financial exposure at this stage. The outputs of this stage are as follows. They can all be incorporated in, or referenced by, the final contract. (For details, see here.) Scope, documented as a simple list Sizing points assigned to all scope items. ( The points here are simply for future reference in this project. They do not, and cannot, form any baseline for comparison to other projects because agile points are not comparable between projects.) Brief documentation of the likely architectural approach Step B-4: Negotiate contract with preferred vendor Having been involved in defining the scope, the vendor now has a good understanding on which to base a price. The vendor should estimate the price for the scope that was itemized in the previous step. Once they have done so, they should share that price with the customer. If the vendor’s price is higher than the customer is comfortable with, the customer should negotiate with the vendor to seek agreement. The customer may: Share, at last, the results of their own Reference Class Forecasting. Using relatively objective data, such as this, tends to be a good negotiating tool. Ask the vendor to seek more cost-effective design options. Agree with the vendor to drop specific lower-priority elements from scope. In addition to negotiating price, you also need to agree all the normal parts of a contract, such as ownership of intellectual property etc. Five things deserve special attention: Pricing model. While not essential for agile projects, I believe target-cost pricing is best. It gives both parties the security of overall parameters and boundaries, while still allowing a degree of flexibility to support agility. Timing of payments. Closely relating to the target cost mechanism, is deciding the timing of payments. Will the timing be based on calendar dates; effort expended by the vendor; or progress made? If you choose the latter, I would strongly recommend that it should be measured by percentage of total scope completed (measured in “story points”), rather than old-school milestones. Progress reporting using Agile-style Earned Value. I suggest this should be mandated in the contract. External advice. I suggest that the contract should mandate that the team obtains (non-binding) external review of their process and architecture. See below for more on External Advice. Scope management approach. An agile team should actively thin features, and occasionally “fatten” others, based on what they learn during the project. These discussions and decisions cause numerous small changes to the scope. Without shared expectations of how these changes will be handled, this can become a contentious issue. I’d suggest the contract should specify that: Feature thinning is to be a goal of the entire team (both customer and vendor staff). If the revised features still align with the scope as documented (in the above-mentioned Simple List) and there’s no material effect on total cost, then no further discussion or approval is required. But if the features don’t align OR the cost may be materially increased, then: The team must seek agreement from one designated senior customer-side employee (the Product Owner). Verbal agreement is enough for small changes; email for big ones. AND each month the Product Owner must provide a summary of the material changes to the Project Steering Committee (or its equivalent governance group). The Steering Committee may request further discussion or justification of the changes, and in rare cases may ask the Product Owner to reverse their decision (even if the team has already started, or finished!). Yes, there is a cost to such reversals. But, as long as you have a competent Product Owner, I contend that there would be a greater cost in having the team always wait to hear back from the Steering Committee before proceeding. Work will be sequenced such that lower-benefit features will be left until the end. If time becomes scarce at the end of the project, they may be dropped, if the drop is approved through above-mentioned Product Owner and Steering Committee process. The Product Owner should report proposed drops to the Steering Committee as soon any such drops appear likely. Don’t surprise the Committee at the end, if you can break the bad news earlier. Step B-5: End of Vendor Selection If both parties are happy with the contract, then sign the contract with the preferred vendor. If no suitable agreement can be reached, re-start at step B2 with the second choice vendor. This option, to fall back to the second-choice vendor, is a key element of the QBS process. It gives the first-choice vendor an incentive to be reasonable, and gives the customer protection against getting stuck with a dud vendor. Step C. Execute the project, tracking always and correcting when necessary Now that you have your contract, go ahead and execute the project. As discussed earlier in this series, one of the key protections in the agile process is rapid detection and resolution of issues. I suggest two key mechanisms for detecting issues: Earned-value-style burn charts. The burn chart should cover the whole project, not the current sprint only. (You can have a separate sprint chart too, if you like). The whole-of-project chart gives a lot of information to allow detection of problems. It should be updated regularly, at least weekly, and shared with both the team and the customer. Independent expert review and advice. This is based on a suggestion from Phillip Lee (thanks Phillip!). Phillip’s original suggestion applied to very large government projects. For private-sector projects, and even public ones up to a few million dollars in size, I’d suggest something like this: select a person (or very small group) from outside both organisations to act an external advisor. Have them visit the project at least monthly, look for the following things, and report back to both customer and vendor: Earned value charting – existence and acceptable accuracy. Order of work, with regard to business value. Order of work, with regard to risk smoothing. Business/User Acceptance Testing. If BAT/UAT can be done concurrently with on-going development, is it being performed and are the tests sufficiently representative of real-world usage? Deployments. The team should be regularly and realistically deploying, even is only to a BAT/UAT environment. Meaningful retrospectives. To what extent are issues raised in retrospectives being dealt with? What is the advisors’ impression of whether the team feels free to raise concerns and break bad news, in retrospectives and/or in more private channels? I’d suggest that vendor and customer should split the advisor bill 50/50, and let that fact be known to the team. It’s important that no-one is forced to comply with the advisors’ recommendations. They’re just recommendations. I recall being part of a very successful project that used external advisors. We found that they identified very relevant problems, but that often the right solution, in our particular context, was somewhat different from what they suggested. Conclusion Any agile contract needs to allow these four types of learning to continue during the project: What should we build? How (technically) should we build it? What will it cost? How should we work together? A traditional procurement process basically forces the first three of those questions to be answered before the project even begins, because they have to be answered in order to write a traditional contract. The answers then get frozen into the contract, discouraging or even preventing on-going learning after that point. (Or at least, discouraging learning from being acted upon.) Contrast that with the process I’ve proposed here. It puts some of the learning up front, in the collaborative engagement at step B-3, but it does so in a way that is workable for both parties and leaves plenty of opportunity for useful learning later. The proposed process draws on Reference Class Forecasting, Qualifications-Based Selection, and Earned Value Management. Each of these techniques is recognised as the best known approach to the problem it solves. Each is backed by significant research and practical experience. Each comes from outside the agile community – giving an impartial flavour to the proposed process. [This is the final post in my series on contracts for agile projects]