John Andre (JA): Hello and thank you for joining today's Acquisition Seminar, hosted by the Federal Acquisition Institute. Today's seminar, entitled "Agile Acquisitions 101: The Means Behind the Magic", presents a look at agile acquisitions and its place in the entire agile software development process. The Federal government spends about $50 billion on Federal IT contracts, but does it receive $50 billion worth of value? The way the Federal government buys and uses technology has changed, but the Federal workforce, policies, and procedures, and how we manage vendors, has not. So what is the answer? There's a lot of talk about agile, but what is it? And why are new procurement methods and strategies needed to successfully implement it? Agile software development is a mindset that requires a completely new way of thinking about an entire development process from the moment a need is identified through execution. In order to procure for these types of services, it is important to understand what agile is and what it is not, what the risks are in the process and how they are mitigated, and what value and success look like. Traditional procurement methods for waterfall software implementations lack the flexibility to take advantage of the benefits of time, schedule, and cost that agile software development methods bring to the Federal government. For this reason, the acquisition workforce needs to make its processes agile, using innovative and creative solutions to procure IT services while maintaining compliance with the FAR and Federal law.
This seminar will identify several methods and strategies that have been and can be applied to achieve this goal. This seminar will feature examples of where agile acquisition has been successfully implemented in the Federal government. The lessons learned, the success stories, and the opportunity to question the trailblazers themselves should give everyone a great foundation to begin. To lay that foundation for our knowledge of agile, we are pleased to welcome Chris Cairns, managing director, 18F consulting under the General Services Administration, Traci Walker and Jonathan Mostowski, members of the Procurement Strike Force Team within the United States Digital Service, and Mark Naggar, the project manager for the Buyers Club in the Department of Health and Human Services. Before we begin, let me remind you that we will hold a live question and answer session at the end of today's presentation. If you have a question about anything you may hear from our presenters, we encourage you to submit it at any time using the survey link to the right of the video screen. We will collect and review your questions during the presentation, take a short break, and then return to answer as many as we can. So let's begin with Chris Cairns from the General Services Administration who will introduce us to agile and its precepts.
Christopher Cairns (CC): My name is Chris Cairns, I am the managing Director at 18F Consulting, which is within the General Services Administration, and I'm here to talk about agile software development. So first -- what is agile software development? Well really, agile software development is a philosophy and the document that best captures this way of thinking is called the "Agile Manifesto". It was written by 17 preeminent software developers, including the likes of Martin Fowler and Kent Beck, in 2001. The Manifesto includes four core values and 12 key principles for agile software development. Four of these values, which you can see here, are: individuals and interactions over processes and tools, working software over comprehensive documentation, customer collaboration over contract negotiation, and responding to change over following a plan. Now, I won't read off all of the twelve principles behind agile software development, but a few of them include: customer satisfaction by rapid delivery of useful software, welcoming changing requirements even late in development, and working software delivered frequently with a preference toward shorter timeframes, such as weeks rather than months. If you want to read the Agile Manifesto in full, and I encourage you to do so, please visit the agilemanifesto.org. Or, you can just type "Agile Manifesto" into a search engine.
So, why be agile? Well, agile helps you achieve success in three ways. From an organizational perspective, agile helps reduce costs and increase the value of software delivery, and thus increase your return on investment. Agile emphasizes shorter delivery cycles which allows teams to identify risk early in the project and if need be, to cancel the project before too much money is spent. Agile also emphasizes delivering the most valuable features first, so even if development teams cannot complete 100% of the features, at least the organization is potentially guaranteed some of the benefit from the work. Agile also welcomes change in requirements at any point in the project, so if conditions change, and they do, agile teams can adapt accordingly. From a technical standpoint, agile emphasizes the importance of quality software in terms of reliability and adaptability of the software. Quality is achieved through such practices as continuous code integration--a 100% completion of each feature before moving on to the next one. Lastly, from a personal standpoint, individual role players from developers, to managers, to business executives, typically derive a high degree of satisfaction and engagement while working on agile projects. For example, developers enjoy writing quality code, as opposed to making shortcuts, and having influence on their work estimates. Agile teams are really empowered to be self-organized and self-disciplined. Project managers enjoy being able to influence feature priorities on a continuous basis. And executives enjoy long-lasting, high business value software.
So there are a number of agile methodologies that express the 12 Agile Manifesto principles in the form of specific practices. Two of the most popular methodologies are XP and Scrum. Now, the key difference between an agile methodology and your more traditional, waterfall-driven methodology is that waterfall assumes all requirements can be known up front and people never change their mind, while agile takes the opposite approach. And, as you can see here, the scrum methodology is very simple in its format. When a Scrum project gets kicked off, it starts off with the development of a product vision. The project vision usually includes what are the goals and expected benefits of the software. That project vision is translated into a product backlog. And the product backlog is represented in the form of user stories and user stories are prioritized by cost and benefit. The cost is usually expressed in a term called "user story points". So, when a project team kicks off a sprint cycle--and the sprint cycle is usually two weeks in duration--they have a sprint timeframe meeting and look at the product backlog and decide, based on the team's capacity to complete work during that period, how many user stories can they take on. And so they pop the high priority ones out of the product backlog, and they put it on on the Sprint backlog. And that represents the work they are going to commit to completing during that two week period. So over those two weeks, they implement the user stories and at the end have a sprint review, where the product owners and customer representatives go through the acceptance criteria that's typically defined for the user stories, and then sign off on whether the feature is fully complete. And if it is complete, those stories are popped off the stack; that frees up the capacity to work on more features during the next cycle. Those sprint cycles continue until such a point that the project is deemed completed. Ideally, at the end of each sprint, you want to leave the software in a state where it can actually be releasable to users and provide immediate value.
Now, as I mentioned, these agile methodologies define a specific set of practices. XP and Scrum are very similar. They follow the same agile ceremony for the most part; they use a little bit of different terminology. The big difference between XP and Scrum is XP is a little bit more dogmatic about the technical practices that agile teams have to use. There is a very strong focus in XP on achieving high-quality software. They do this through making sure such practices as test driven development, pair programming, and re-factoring are followed very closely. Two key concepts that you need to understand for agile is that it is both iterative and incremental. The first concept, iterative development--what this means is that, when we build something, we then evaluate whether or not it's going to work for us. And based on that feedback, we make changes to it. So, we are building software expecting to change it. So, just as a painter may create several sketches for a painting and then create the underpainting showing the color and form, and then once that is fleshed out, iterating until the painting is complete. It is very similar in that fashion. By iterating, we are seeking to find the solution, and once that solution is found, to continue to build upon and improve that solution. The second concept is incremental development. What this means is that, at each increment, we are adding a little software at a time. As you can see in the Mona Lisa portrait, that means we paint a little portion on the canvas in incremental steps until it is fully complete. And we create increments to make software potentially releasable to users so they can receive immediate value. While you are iterating, you want to look for opportunities to create an incremented functionality that can be released to users.
The 'agile triangle' -- many of you may be familiar with the traditional, plan-driven 'iron triangle' where you are trying to manage to the constraints of the scope, cost and schedule. Agile takes a little different perspective. While it does recognize the importance of constraints, it puts equal emphasis on quality and value. Quality, as I referred to earlier, is that intrinsic value, where you want reliable and adaptable software, so as business conditions change, and new forms of value emerge, that the agile teams can capture that and incorporate new requirements into the software quickly and release to users. Value- the extrinsic quality aspect of it, is building software that users actually use, and delivers value to the business stakeholders.
So, what are some examples of agile acquisitions projects with the Federal government? Well, at 18F, all of our software development projects, which there are about 10 plus, use agile and other types of methods. One of the agencies within the Federal government that's really become kind of a Center of Excellence, in a sense, at least within DHS, is the US Citizenship and Immigration Services, and they use agile across all their portfolio project areas to a great degree of success. At 18F Consulting, which is the group that I lead, we work with several agencies in helping them to apply more modern software development practices to the software acquisition--a few of the speakers will be talking more in depth about that. We work with several agencies, like the Department of Labor and SSA, and you can help them craft acquisitions so that they can go out and procure services in an agile way. And we put a lot of emphasis on making sure that there are technical practices within those solicitations to ensure the Federal government gets good, quality software at the end of the day. GSA 18F is also currently in the process of developing a government-wide marketplace that's going to feature about 15-20 agile vendors who specialize in doing agile development and user-centered design. We are hoping that will be available later, toward the end of the year, so the government has access to great agile development vendors.
Traci Walker (TW): Hi, I am Traci Walker.
Jonathan Mostowski (JM): And I am Jonathan Mostowski.
TW: And we are here with the United States digital service. And we are here to talk about agile acquisitions. Now, Chris has explained what agile software development was, but we wanted to bring in the concept that you can actually do contracts in a new way for these services. Now, why does the government need to change the way it buys services? Right now, the government spends about $50 billion on Federal IT contracts, but does not receive $50 billion worth of services. That is a lot of money going out and we are not getting the full value for those dollars in the government. So, we want to change the way that the government buys technology. Basically, we want to improve the way that people are trained; we want to change the policies and processes that are in place that kind of limit our ability to actually get these Federal dollars spent in the right way. We also want to reach out to new partners and new solutions that are out in the marketplace that we have a high barrier of entry, limiting our ability to get to these kinds of companies that do these really innovative approaches. The solution is to transform the government's most important citizen-facing services to be simple, reliable, and delivered quickly on an appropriate price. And we also want to develop the institutional capacity so that this can be a lasting change. The way we do this is definitely through training mechanisms, such as FAI. We are also trying to develop programs for Contracting Officers that will train them in digital service programs as well as the groups, integrated project teams, the CORs, project owners. Everybody from management to legal--bring everybody along and say, "How do we change this for the government, and leave us in a much better place than we are right now?".
As I said, Chris introduced agile development, and in order to make it work in the Federal world, we need innovative and flexible contracting. We are here to tell you today that these types of contracts are legal and encouraged. Basically, if the FAR doesn't say you can't, then you can. We are going to provide some ideas and approaches to implementing agile contracts. However, it's not the only way to do that, and we are continually revising even the way that we do these contracts based on the fact we don't know enough about this world yet, we are still learning. And we know that we are going to change. So the key thing is making our contracts as flexible as the software development practices that we're actually trying to buy in terms of services.
So the manifesto for agile software development definitely highlights the idea that there is value to the items on the right, but what we value is the items on the left more. In traditional contracting methods, we definitely put these contracts around the ideas on the right and kind of put them together to say, "Alright, this is the only way that we are going to be able to do these. We have to have fixed processes and tools. We have to have firm contract negotiations upfront. We have to follow a plan." But it doesn't really take into account what happens when those plans go off track. We spend a lot of time working on really good contracts, and then, once we get into the actual execution of those, they tend to go off the rails and we tend to spend a lot more money fixing problems in the out years than we could have spent trying to be proactive in the beginning and putting that together. So these keystones for agile acquisitions are the summary of the points that were highlighted in the TechFAR. The TechFAR, along with the Digital Services Playbook, was released by the White House in August 2014 as best practices for buying digital services. This can be found by searching on Whitehouse.gov. These are basically strategies that can be employed for putting these contracts together. You understand that you need to buy agile software development, but how do I actually write a contract that encapsulates everything you are trying to buy from a service perspective?
So, switching around the way that we do contracts in this environment is that we understand the needs of the end-user community, and those are what set the priority, as opposed to having a huge requirements package that has been pre-done and 300 pages of requirements and that is what has to be delivered at the end of the contract. Right now, what we really want to understand are what are the end-users needs, and put those into smaller, just-in-time requirements. We also need to give the integrated project manager and government contractor teams the resources needed, inclusive of a flexible contract, and trust that they can deliver the required system. One of the big tenets of agile teams is that there is trust built-in between them. In the government world, there are sometimes barriers to building trust, and to make agile work, you really have to have a combined team in which everyone is a player, and everyone understands the roles and responsibilities, from procurement all the way to the contractors that are actually delivering it. The other really important keystone for this is the proper training on agile implementation. It is very important that everybody from the executive level all the way down understands what does agile mean, how do I get value out of an agile process, what does it actually mean to be implementing these kinds of services, and how are they different from what has been delivered under a traditional waterfall methodology? The other important individual interaction keystone in this grouping is that the government project owners and CORs must be empowered with decision making abilities. That means that they have to be able to make day-to-day decisions on priorities, and not have to run through chains and chains of government processes that kind of bog down a release strategy or bog down the ability for them to make decisions. They also have to be empowered enough, by their executives, to remove roadblocks and get these projects out the door because it's very important to have releasable, working code as the end value for these contracts. So, that leads to working software over comprehensive documentation.
As I mentioned, contract requirement documents need to drive towards overarching objectives. This is where performance-based contracting works hand-in-hand with agile software development. In a Statement of Objectives, you are announcing your objectives for what you want your contract to do. In an agile software development, your user story is the objective for what functionality you want to have done. So, those combined in this world is the perfect entity for marrying those two--the parts of the FAR, and also the agile software development process. User stories are just-in-time technical requirements, and they also include technical acceptance criteria for what that end product needs to have. So, in terms of the way we execute these contracts, we are not removing anything that's required in terms of technical acceptance requirements. We are just making them look a little different than you traditionally see today, and we provide flexibilities for you to be able to have that just-in-time requirements, so that when the new big thing comes through, or the end users have a major problem with the system you've released, they can come back in and fix that very quickly, as opposed to having to go through a contract modification to say, "I need to fix this". The most important thing about these is that the value you get out of these contracts is not necessarily that your contract was ended and it was completed, but that you have the frequent and continuous delivery of functional product and software code. We should be paying for released product based on services that are delivered and not necessarily just buying services. And documentation is still a very important part of these contracts, although it looks a little bit different than requiring management plans upfront and work breakdown structures. It becomes user stories, design documents, and continuous updates.
Customer collaboration over contract negotiation -- I think one of the very big differences in an agile contract versus a regular contract, a waterfall contract, is that the technical requirements and deliverables are not the same as the contract requirements and deliverables. And these should be defined separately. So when you are asking for functional requirements of what you want a system to do, that is no longer considered a contract requirement. It is actually a functional requirement, and your contract requirements are more in the terms of iterations. How many iterations am I going to get in a period of performance, and what do those iterations look like, and what is the process by which I am going to get those on a regular basis? The customer collaboration is very important as well, and that should include your internal customers as well as your external customers. That should have your legal, your security, your 508 compliance, privacy requirements--all of these should be baked into your agile process and your contract requirements. They are still in there; they are just being put into the process and being delivered as part of the iterative process. So instead of waiting until the very end of your contract to do a security review, you should probably get together with the entire team and figure out what are the security requirements that have to be implemented along the way and start developing those and putting those in to each iteration.
Finally, responding to change over following a plan. Your end-users are really the key component of this, in that they should be receiving the features which provide them the most value. Therefore, the priorities that you have for systems should look like "must haves, should haves, could haves", and you must allow your Contracting Officers Representatives and product owners to manage those priorities. That is very important, and the contracts that we have that we are advocating here allow for that flexibility. The other main thing is that software development doesn't end, so contracts have to take that into account by purchasing software development process, versus a delivered software system. You are continually going to getting a new iPhone update and a new Samsung update--a new version is always going to be released. There is at no point when you are working in the software world unless you want to completely get rid of that software choice you have made or develop it into a completely different platform that you are ever going to need to make changes. So, your contracts have to take that into account, and not look at the end goal of the contract as being the end goal as a delivery of a system. It can be somewhat of a minimal viable product that says, "We have a working version, and now we want to enhance on that", but there's always going to be enhancements to that software. So taking that as a tenet of these kinds of contracts is very important. We are also definitely advocating the modular contracting methods to reduce the too-big-to-fail mentality that encourage flexibility and change, and also still meeting the FAR requirements. Modular contracting methods definitely provide the ability to get out of contracts. It has exit strategies that allow you to mimic what is being done in the industry right now, that we cannot just necessarily terminate a contract by saying "We are done with you" and walking away from the table, as they can potentially in industry. But, by using modular contracting and reevaluating where you are in the process, and then moving forward and making new decisions, and revising your own internal processes and figuring out whether something worked well and you want to repeat it, or you want to scale it, or was it not working and it was really hard to manage--how do I reduce that? So, these methodologies, by going in smaller chunks, like six months, a year for a contract and period of performance, and then redoing it, re-looking at it is very effective. It also allows you to implement award terms or options effectively, and as part of your strategy in order to get the best services provided. And we also want to make sure we are dissuading being locked into proprietary solutions. I'm sure everybody has that contract that they have worked on, where the company has come in and said, "Well, we delivered it, we developed it and therefore it is ours, and we are the only ones that can work on this, so therefore if you are going to terminate this contract, then you basically have to start from scratch. What we want to do, especially with these software development contracts, is make sure that what the government buys, it owns. That's in terms of the code that gets developed for the solutions we are looking for, the platforms, even if we have to go into licensing of a commercial, off-the-shelf product, if we do any configuration or code changes, those are ours, and that we have bought the services that actually relate to those. So make sure in terms of the causes that are employed and the language you put in there that you are not locking yourself into something that is proprietary, because you want to be able to get out of this and move to a different provider if that agile process and solution that one company is providing for you is not working. Now I'm going to turn it over to Jonathan who is going to explain a little bit more of how do we actually implement these and what do they look like in terms of the solicitation process.
JM: Chris provided a nice explanation as to how agile development works on the program side and Traci provided the framework for how we've used the Digital Playbook and the TechFAR to build the constructs around our recommendations for contracting. So, now as we move to the next level, the more tactical approach of how we implement these two, I thought I'd start with Iron Triangles. As Chris gave a nice explanation as to how the Iron Triangle has shifted in the agile model, that's more referring to the programmatic approach to the Iron Triangle. But as my fellow Contracting Officers out there would want to acknowledge, the Iron Triangle very much still exists, so no one has to panic. But how we use the Iron Triangle is still going to shift a little bit from the contracting perspective. First and foremost -- both under waterfall and agile approaches, cost schedule and performance remain in the boundaries of the Iron Triangle, and those don't change. The difference is, under waterfall contracts, we want to maintain predictability through fixed delivery schedules and system requirements. Under agile contracts, however, we want to maintain flexibility for the purposes of quality and end-user priorities. In this case, we are using trade-offs made within the contractual parameters between cost schedule and performance. So you still know what your program is going to cost, you still know your schedule and when you are going to stop work, and you still maintain that overarching scope, but you have the flexibility built in there, so that the Contracting Officer is out of the business of functional changes.
Speaking of changes, I want to talk a little bit about changing mindsets. Steve Maraboli said, "Once you change mindsets, everything on the outside will change along with it", and that is what we are doing here. Under waterfall contracting, we focus on individual labor categories, fixed end products, and steady states. What this means is that new features mean new costs, and new schedule. And it is driven by detailed contract tracking mechanisms and program mechanisms such as work breakdown structures, earned value management, and formal plans. So when you make those changes to the functional space, you are talking about rebate, assigning schedules, resubmitting costing plans and updating EVM tracking data so that the reports match the work that was actually performed or will be performed in the upcoming period of performance. That's before you even get to changing the actual contract itself, depending on what level of detail the system requirements were documented. Under the agile approach, we are focused on teams. We are not so interested in the construct of the team but that the team has all the necessary skill sets to be able to deliver the process that you need. Speaking of process, we are looking at the repeated process for functional code, so think of it as a service, servjce contracting, and the way you measure whether that service was actually delivered is through a potentially showable increment, meaning what comes out of that service iteration is something you can use, and you heard about that in Chris's presentation. Now, instead of a steady state project, what we are really focusing on is future development, working out bugs that our discovered, whether they are discovered after an iteration or a blatant defect, and then also working on enhancements. In this situation, new features can take a variety of different forms. It could be more teams, meaning spinning up other development teams; it could be longer schedules, not that the iteration lengthens, but you add additional iterations; or it could simply be a matter of reprioritizing the backlog. And finally, the project is driven by a flexible document, known as the roadmap, which means we painted a vision of where we want to go. The scope remains the same, but the individual stops or milestones along that roadmap are very flexible.
So we are back to Mona Lisa. Let's say, for example, that Francisco approached Leo and said that he wanted a painting of his wife and his cat. Fortunately, Leo used agile painting, and so in his iterative approach, he went to his end-user and got feedback. Over time, as he was developing iterations of the painting, the feedback he got was the focus of the end user was on his wife's physical smile as opposed to his cat. Keeping the same schedule, time, and effort, Leonardo painted the painting of just Mona Lisa. Had he used a waterfall approach, we might all be sitting here wondering what the cat was thinking. I use this portrait as a metaphor to explain that in agile, we are really focused on flexibility, quality, and end-user preferences. But notice that the scope remained the same--it's still a portrait, it's the same shape and size. It's the same period of performance, same cost of the product, and in the end state, the end-user was much happier with what they got.
So when we talk about the agile development process or these iterations, what is going on there? This is just one example of the processes that could be occurring during an iteration. Don't necessarily take this diagram to imply that these are sequential. So, this could be a sequential process; mind you, these iterations could be two weeks, maybe as many as four or five weeks depending on the process that's being used. But, the individual activities are all being coordinated together, and that would be part of the requirement or the performance work statement and we would be backed by the offer, and we will get back into that a little later. I wanted to put that picture out there, so when we say iteration, what is going on, and this is a representation of that. A lot of times, the question I will get is that the FAR doesn't allow it or even that the FAR is completely silent on it--as Traci mentioned, if the FAR doesn't say you can't, you can, and that's absolutely true. But in this case, there are a lot of examples when the FAR actually does provide us tools that we can use for agile contracting. And here is just a list, and I am absolutely not going to go through every single one of them, but I do want to point out commercial contracting. When you think of this, a lot of times people think of FAR Part 15 and getting unique services to the government because we are focused on the specific end product. But in reality, what we are talking about now is buying that service, and this service of agile development is a commercially available service that industry has been doing for a very long time; it is where we got the idea from. So in commercial contracting, we have a lot more streamlined processes, so we can speed this up. We have a lot more competitive processes, lowering the barriers to entry for vendors to come in with their unique approaches and strategy. I also wanted to mention the firm-fixed price, as we do advocate using a fixed price per iteration. Since as I said, that iron triangle sits each one of those iterations, that actually--that scope is very predictable and well-defined. Under FAR Part 37, and specifically 37.602, performance-based service acquisitions, that really laid the foundation of what we are talking about here. We are getting out of the business of describing in great detail what it is we want, and we are talking about what our overarching objectives are, and allowing the vendors to bring their unique capabilities to use with a quality assurance plan, and making sure that we are getting quality performance, and not focusing on specific system requirements. Finally, I would be remiss if I did not point out FAR 39.1, which is the acquisition of IT, where it specifically talks about using modular approaches as Traci mentioned before, and it also talks about taking responsible risks when procuring IT. All of these FAR cites are very valuable when you are having these conversations with your fellow Contracting Officers, your leadership, and your legal counsel.
So, how to structure your contracts. Abraham Maslow said, "If the only tool you have is a hammer, you tend to see every problem as a nail." So what I'm trying to provide here, as Traci mentioned, isn't a one-size-fits-all answer to every agile acquisition problem, but rather, to show all the different possible ways you can do it, and I'm sure there are other ways out there that you could think of as well. Two main categories to consider are a single award approach, or a multiple award approach. Under a single award approach, you can use indefinite, or IDIQ contracts, what I call a 'Standard C' contract, or just your traditional contract. You can use government-wide acquisition vehicles, blanket purchase agreements and you can even use a set aside if you want to have the flexibility to learn. Maybe your agency is new to agile development, and you wanted to work with a contractor in a sole source environment, where you have the flexibility to change requirements without creating an unfair advantage to the vendor once they already have the award. This is a great way to learn and grow together. But the important thing to think of is with a single award, you have one vendor, so there are strengths and weaknesses. With one vendor, you get the opportunity to work with them over time, you have consistency in your development team, and you get to know each other, for better or for worse. The downside of that would be if the contract development process and contract are not working well, you are going to be in a situation where the modular contract will let you get out of it easily, but then you find yourself back at square one in finding the next solution. Which brings us to the multiple award approach. A multiple award approach can be used in various ways, with all the same contract approaches that I mentioned for the single award. But in this case, you could use, for example, an award term, where you are incentivizing the vendor to keep performing at a high level but at the point in which they are not succeeding or not excellent in their performance, you can recompete the next task order, the next activity to the vendors so you are never in a situation where, as I mentioned, you are at square one and have no acquisition strategy to move on to.
Now, contract types, incentives, and periods of performance. As I mentioned before, the recommended approach for Traci and I is to use the fixed price per iteration. Of course, it is not the only approach; you could use a firm-fixed price for the overarching activity. You could use a time and material or labor hour vehicle. I would caution the use of the time and material; while there's a lot of benefits to it and ease of flexibility in implementing, you are only requiring the contractor to deliver you a set level of effort, literally the labor hours you contract them for, and the labor categories you contract them for is what you are going to get. You don't have that stick at the end of the day to say, "Did I get useable, functional code?" Time and material is very advantageous if you have an existing team within your agency, and you need to augment that team with additional skill sets or resources that can make the team more effective without trying to bring in the entire process. Incentives--I mentioned performance-based service acquisitions, and quality assurance plans, and award terms--we really highly recommend using those. Of course, award fees and incentive fees can be used as your agency might determine appropriate. Just make sure they are really tied to the process and to the release of functional code. And finally, the only thing I'll say on periods of performance is just continue to use modular contracting, small periods of obligation to the government, so that you can reassess where you are and whether you want to move forward from that point in small increments. Because, at some point, you may determine you have gotten all the value out of a development cycle you need, and you don't want to spend any more, so this is a great mechanism for doing that.
Evaluation criteria. So you want to look at past performance, but you want to look at past performance in the form of how they demonstrated the agile approach that they're going to propose using, on requirements similar to what you are trying to obtain. For key personnel, you again want to ensure they have demonstrated experience applying the agile process they are talking about in similar types of work. And finally, in the process, and this is really key -- you want to make sure the vendor describes how they will manage, execute and measure the agile development to meet the end-user's objectives and maintain quality. Make sure that their processes aren't dictating anything; you still want the government to maintain control of what is coming out of each iteration.
TW: And some language of this, for examples is provided in the TechFAR, and we will be iterating and adding to the TechFAR and the Digital Services Playbook over the upcoming years in order to continually expand on the templates and these resources that have already been used and have been successful.
JM: So, require a definition of "done". This is really your acceptance criteria. A lot of times when we talk to other agencies about implementing these approaches, one of their concerns or questions that comes up is that they will come back to us and say, "OK, we get it, we are buying a process, we get that there is functional code, but what does that really mean? How do we know we actually got something"? And this is where this is really important is their definition of "done", and you can have the vendor propose their approach and what their definition of done is, that is something you would evaluate. Ultimately, what is the criteria will be used to determine done is done or is acceptable? This should be a required element, as I said, in the proposal, and the evaluation criteria should be used whether individual user stories are completed and can be accepted. At the bottom here, we have a few examples of a few definitions of "done", but certainly you want to tailor them to your specific requirements. Remember that you get what you ask for. So when you go out with your Request For Proposal, here are some things you might want to consider. I strongly recommend the use of Statement of Objectives; get out of the game of telling vendors how to solve your problem and paint that picture of the vision of where you are trying to go, where you need help moving your agency or your requirement to. Statements of Work absolutely can be used, but they should only if you are literally a master of agile implementation, and you want to implement a discrete set of known functions. Initial technical requirements--some items might include a service dictionary or a backlog, as I mentioned before, the product roadmap, and epics and themes, to the level that you can. And all of this again--these are not fixed requirements, they are living documents meant to paint a picture of your vision. And then what do you get back from the vendor? The Statement of Objectives will go away, and what you will actually use as your binding requirements document for the contract will be the performance work statement provided by the vendor. Make sure you give them the opportunity to say exactly how they do their process, and that is how you will be measuring them again for each period. You want to get their agile development management plan, which breaks down in detail how they will manage their program. An initial quality assurance surveillance plan, which you will ultimately negotiate with them in execution, but something that says how do they propose we measure them? And finally, I recommend getting an Unpriced Basis of Estimate; being that it is fixed price, there's no need to get the price for the particular labor categories or personnel that are on the contract, but having the breakdown of the individual categories and what they will be doing in the process really helps to paint a picture of what they are doing and how their process can be executed. For example, if they say they are going to have a Scrum Master and three or four engineers, but when you look at it, there's only two engineers and a Scrum Master as part of their development team, then they have to explain to you how that person is going to be able to manage their sprints or development cycles and be a contributing partner to the development team.
Performance management. When you are thinking about how you measure, you want to look at velocity or throughput. How much of the user stories are actually getting completed in each iteration? Make sure it's trending the right way, and if not, why. Defect density- how many bugs are being discovered during each sprint or iteration? If it is an increasing number of bugs, what is the cause of that? Start looking at the team. These are all things to have conversations with the vendors and figure out where do you go next. It's not always the hammer to say we stop and move on to a different approach. These are program management conversations in the beginning; they may ultimately move to that with a contractor taking a different avenue. Again, you want to look at customer satisfaction, that's what it's really all about. Team satisfaction, use the "Five Why's" to get to the root cause of which way the trends are going. And finally, the overall value of the iteration.
In a nutshell, just want to remind you to change the way you define your requirements. Use Statements of Objectives whenever possible. Choose the appropriate vehicle for your specific requirement, work closely with stakeholders and monitor effectively. Now remember, there will be a time when you believe everything is finished, and that will be the beginning. At this time, I want to turn it over to Mark, who is going to give us some real-world applications of agile development and agile program management.
Mark Naggar (MN): Hi, my name is Mark Naggar, I work at the Department of Health and Human Services within the HHS IDEA Lab. It is the innovation center at HHS. IDEA stands for Innovation, Design, Entrepreneurship in Action. Last year, we launched a program called the HHS buyers club. It is our effort to innovate and reinvent the acquisition of IT services. Several years back, you may have heard about healthcare.gov and there were a lot of challenges with the launching of that website. I think because of those challenges, you have seen a lot of focus on innovating and trying to modernize the procurement or acquisition of services. You heard Chris speak as well as Jonathan and Traci. There are a lot of groups across Federal government focused on improving the acquisition of IT services, and HHS is included. While the HHS Buyers Club is a new program, it's not the only one that is focused and I want to make sure I mention that because Chris's group at GSA at 18F , the US Digital Service, as well as many other groups are focused on this. And, as you can probably tell, by joining together with this acquisition module, we are trying to educate a lot of the Federal acquisition workforce and program workforce throughout the government and even outside of government, all stakeholders. This problem cannot be solved without everyone being involved. Now, the way that the Buyers Club was stood up initially came out of two scenarios or two unique areas. One was released by the White House -- three documents. The Innovative Contracting Case Studies and the Digital Services Playbook, as well as the TechFAR handbook. Those three documents were very profound and are likely to have a very dynamic effect on what you see during the next five to 10 years. Even though those three documents came out and provide a lot of information, you are likely to see them evolve over time and we hope they will because the more they do and more people share information, the better we are going to become at developing better solutions across the IT area.
The other area that helped launch the Buyers Club was not anecdotal data but actual research developed by the Standish Group in their Chaos Manifesto, which links are attached to the PowerPoint slide, as you can see. We found that acquisitions larger than $10 million fail, and tend to fail at high rates. Acquisitions or contracts are over budget and users are not involved the process. A myriad of other opportunities for failure exist and we are trying to focus on mitigating failure and increasing the likelihood of success. With that, we launched the Buyers Club, and I can tell you while that $10 million figure may stick out, I can point to probably dozens in the past year of contracts I have come across where a contract of a million dollars or less has failed. Often times, we have contractors that propose on and win contracts that they have no business performing some of these IT services. I think part of this goes back to the government does a somewhat poor job in vetting vendors. We tend to rely on text-based proposals and it is really hard to determine core capabilities. Having been in procurement and government for a while, I've seen plenty of proposals. And I have seen that it is hard to distinguish who really can do the work and who can't do the work. So it's incumbent on us to make these changes and constantly iterate and evolve, in even what you could call an agile manner. The way we go about acquiring services such as IT services to make sure the contract or vendor is capable, willing, and actually can deliver what we need in a suitable amount of time.
You have heard Chris, Traci and Jonathan talk about agile acquisitions, talk about the acquisition area, talk about implementation. I'm going to go into some practical scenarios. So I mentioned the problem with IT service failure and the statistics we have. On the slide here, you can see a lot of the solutions that we have outlined. So, just reengineering the process. The acquisition process is pre-award through solicitation and post-award, which is the actual implementation process. I'm going to lay out what the HHS Buyers Club has done as just one example of how we are trying to accomplish this.
Before I go any further, I want to lay out the three objectives of the Buyers Club. One is to test innovative acquisition methodologies, some that came from the White House, innovative contracting case studies, some that we have determined to be hybrid, and some we created on our own. Next is the development of newer, easier, more effective acquisition models and processes that I will get into in a moment. Lastly, the engagement of all key stakeholders with effective education and outreach because, as we know, the use of agile acquisitions in this approach, while it is not new, it has not been used. As Traci indicated, this modular contracting approach is within the FAR--it's completely approved. But most folks in the contracting field are just not aware it is able to be used, just as most folks in the program field aren't aware it exists either. When you operate within those lanes of what's easier for various reasons, you don't focus on what is innovative.
I am going to share with you now, in a moment, the acquisition diagram we have come up with. Before I go there, I do want to mention that, just as other organizations have, we have our own strategy and execution plan and we have plans to roll out to all of HHS and encourage the use and adoption of agile. Now, the Buyers Club is a unique name. We probably stole it from the Dallas Buyers Club because it was really catchy and creative and has caught on. But we are not doing any buying, so to speak. We are basically a strategic consulting arm, a very small one, actually. We are building and developing a large community of practice within HHS, and also working with our interagency colleagues to further develop best practices, common barriers and solutions, potential pitfalls, lessons learned, and so forth. Since this problem of improving acquisition of IT services is Federal-wide, we feel it is necessary to share what we are doing as well as to learn from others. So whether you are in the program end-user side or on the contracting side or somewhere in between, we have realized some other internal problems we are trying to solve with the Buyers Club. You will often hear within agile implementation about the use of teams--it's a team structured approach and we think collaboration is essential to success, regardless of what you are doing, whether it's acquisition of IT services, or anything else.
In this particular case, if you look at the diagram, you'll see there's something very unique here, but I am going to give you a breakdown of what usually happens. Typically a program or end-user will spend some time developing a Statement of Work, and it is full of requirements, as Tracy mentioned. Some can be very long, and that is with people who think they know what goes into these IT systems. Often times, a lot of those features are never used. So we are trying to determine a better way. Oftentimes, the procurement group feels that the program group will not let them in to help them with strategic planning, or develop acquisition plans or strategies. They feel the procurement can be a barrier. The program side with the end-users, they feel the same way about procurement, that they are going to be a barrier. But really, when you think about everything together, and you think about agile acquisitions from the pre-award standpoint, you want to bring in your best and brightest. This is what Pepsi does, what Coke does, what any Fortune 500 company does when they want to develop something new, when they want to ensure success. They're going to bring in their A team, and it's going to be very cross functional, but that's not what we do in the government. In the government, we have this assembly line approach, where the end-user puts together a Statement of Work and will pass it down the line. But in doing so, the procurement office probably won't see or hear about this requirement until it hits their inbox and at that point, their clock is ticking. And typically, in the third or fourth quarter of the fiscal year, there's really not much they can do to accomplish success. Part of our approach, again, when you look back at these figures and the failure rates, anecdotally and from actual peer-reviewed research, you'll notice that the more collaboration you have, the earlier you have it, the more chance you have for success . So we are asking the program officers and end-users to invite all subject matter experts as soon as it is determined you have a need for an IT system and need for some sort of software development. Because, in doing so, you are getting everyone possibly within internal and external stakeholders help develop and strengthen your Statement of Work into a Statement of Objectives and really that path will help lead you to success. At least that is the way we have seen it done, especially in the private sector, and we would like to mirror that as much as possible. So in doing this, and if you follow this diagram, which I will admit is in alpha or beta, it is very commonsense, it is a commonsense approach. You want to bring in those SMEs, you want to get IT to provide their guidance, budget and finance to get their guidance, procurement to help you do market research, to show you what is available. Since we are talking about agile acquisitions and agile implementation, if you wait until the last minute, you are really setting yourself up for a potential disaster.
Now, I want to talk about an actual use case. This is the first use case from the HHS Buyers Club and it took a lot of handholding and arm twisting and persuading to find a guinea pig to try the agile approach. And why? Because it is new, it is at least new to them. It was new to the procurement office, it was new to the program, and it took a while for us to go through and help them. From this slide, you can see the current -- we call it ASPE -- everyone loves acronyms in the Federal government. It's the Assistant Secretary for Planning and Evaluation. They gather data from all across HHS and put it on a main site. We do a lot with open data and this is one example. You can tell by looking at the site it is very updated. That really was their determination of the need. ASPE, as I'll call them from now on, had a need for a new web content management system, something more twenty-first century where they can share and engage with the public and other stakeholders. They also needed a new external facing Internet, an internal facing Intranet, and some redesigning of several databases to help accomplish their mission. There, where you see the ASPE web content management system that's underlined, that's linked to the actual solicitation, in case you'd like to look more at it. When we first started working with them, they had a 28-page kitchen sink full Statement of Work. We worked with them, and by we, I mean the procurement group, the end-user, project officer, and myself, to whittle it down from 28 pages to a little less than 10 pages -- and that was a Statement of Objectives. I'll admit, it probably wasn't a complete Statement of Objectives, but over this two month process of iterating this SOW to the Statement of Objectives, we took our time to make sure we satisfied all parties involved. Any time you stray from a traditional norm of acquisitions, you have to do it gradually with a lot of people. And that is what we did here.
So, now that we have the Statement of Objectives ready, we focus on the actual solicitation. We knew we wanted to pursue an agile acquisition landscape of how we are going to evaluate and how we are going to actually compete, award, and implement what ASPE needed. The FAR outlines very clearly that if two or more small businesses can do the work, you have to set it aside for them. We did that but we asked it to be set aside for all small businesses, not any particular one because we wanted to maximize our own opportunity for success. Within the solicitation, we did a few things. We broke it down into two stages. The first stage was asking for a cost proposal and asking for an eight-page concept paper. Why do we do this? As everyone knows, the cost and burden of submitting long, detailed proposals, especially ones that are text-heavy, take a lot of time for the technical evaluation panel on the program side to evaluate, and for the procurement group to go through and make sure we are evaluating accordingly. But for the most part, why we did this is that we knew we were dealing with small businesses and wanted to ensure success. There's no reason to submit a 30-page proposal at the very onset. If you have 30 or 40 companies do that, that's a tremendous amount of money, so we asked for something small. And really, this concept paper was derived to accomplish two things. One, we wanted to understand what their agile methodology was, and we wanted to understand what their concept was. They knew what our Statement of Objectives were, so they also had access to schematics of the current website. They had a "look under the hood", as I called it, at the current ASPE structure. And in doing so, we thought it was very important but also very easy for them to provide us with eight pages, illustrated agile, how they would go about doing it, how they had done it in the past. The other key factor we asked in there was past performance. And not past performance that you typically hear of in the typical government systems, whether they performed accurately or not. We went a step further, and you heard Jonathan mention this earlier. We wanted to know what their past performance was as it related to agile, as it related to what we were asking for in the Statement of Objectives. What web content management systems have you created prior? What websites have you created prior? We asked them to provide us with links. Because what is really difficult for technical evaluation panels to assess is really words on a paper. When you can actually go and look at website and understand and compare it to what your vision is, it really helps you to make a decision, when you couple all that together and see something tangible. That is what we did there.
From that point, and I will go back and mention that we had 24 initial submissions, which is great. 24 is a good number, and when you're dealing with 8 pages as opposed to 30 pages, it takes a lot less time to evaluate. We down-selected there, and this really amounts to two different competitive ranges. And both competitive ranges had different technical evaluation factors. In the second stage, we down-selected -- we gave five small businesses, the top five, each $10,000, in roughly about two-and-a-half weeks. And we asked for three things: we asked them for a revised cost proposal if necessary. We asked them for a performance work statement, which included utilizing the tech forums and the Digital Service Playbook, quality control plan, and other efforts for quality assurance, so that we knew more about how they were actually going to implement what we wanted. At this point, we thought it was appropriate to ask them for more information on a text basis because they passed that first hurdle. But the single primary reason of this two-stage down-select was to ask for a prototype. Now, prototypes are not necessary, but I think in this case, we really wanted to try something different and experiment, and we encourage a lot of groups to do the same. With this experiment, we asked and provided details and instructions on what we wanted on the prototype. We wanted some working code, we wanted mockup frames of the web content management system. And we did this in a way, for the $10,000 we gave them, to ensure they could actually accomplish this in the amount of time. When you break it down, that $10,000 equated to about 80 hours, give or take. And we cannot guarantee they spend 80 hours or 40 hours or 300 hours, but what we could guarantee was by this purchase order, this Fixed-Price purchase order, we could get something from them, and we could actually determine whether or not they were truly capable, and figure out who would be the best contractor to perform the work. And it worked very well. The evaluation factors were certainly on point. I think, for this particular case, this approach worked very well. I can tell you, having spent a lot of time with the technical evaluation panel and with the Contracting Officer, they always wanted to try some like this, but they both felt that if they had approached something similar to this idea, it never would have flown. But, once you get folks in the room, or start doing acquisition planning the way it should be, which is having these integrated product or project teams, a lot of things can happen from there. A lot of brainstorming, and creativity, and definitely a lot of collaboration. And that's what happened here.
In these slides, you can see three prototypes. This is prototype one. This is prototype two. This is prototype three, and the winner. I think we really lucked out here. The winner of this solicitation turned out to be really, really great for this group. Not only were the end users dynamically and really involved in the project, all the way through the acquisition process, but also to award. And one of the key things about having this prototype, once awarded, is you do not have to start off with a kickoff meeting and talk about where you're going to go and how you are going to start. With this aspect in this experiment, in this use case, we actually had working code. We could start with this website. They could go back and they could give their opinion. At that point, the agile implementation actually starts. We actually had probably, to our benefit, a great project officer, now the product owner -- and a great company that won the award. I will go through shortly and explain how often they communicate and how effective it has become. But before I do that, I kind of want to outline a little bit of the use case. Because every time we are doing a project at HHS, we are doing a full use case. That is getting involved until the project is completed, because we think it is important to share what is going on every couple of months, so that others can benefit from what we're doing, as opposed to waiting two or three years once the project is complete. In this particular case, we learned a lot of advantages. We learned a lot of "gotchas". We learned lessons learned that we could then take to put into the next acquisitions that we do, and we actually have plenty in the pipeline, and some that are very similar to this one. With that, I am going to say that one thing that had never been done -- at least, I had never done this in my procurement time -- is actually going back to the vendors. Now, contractors or vendors are always entitled to a debrief from the procurement or contracting office. It is required. But in my time, I had never actually gone back to the contractor or vendor and asked them what they thought about the solicitation. So I did that. I did that with all of the stage two contractors and with about half of the stage one contractors. Initially, I sent out a basic survey to be filled out, and I followed up with all of them to get more feedback. I can tell you that I learned a tremendous amount that I never would have realized. And this has helped us to refine our approach, to refine our solicitation, and for this particular case and for others, we now have ideas that we can put into use to make it better. I did not just interview and get feedback from the vendors. I also went to the contracting office, the Contracting Officer and specialist. I also went back to the end users and the technical evaluation panel. In doing so, we collaborated and brought all three of those together to formulate a very comprehensive use case that continues to evolve to this day.
So I have talked about agile acquisitions, pre-award solicitation, all the way up to award. Now, agile implementation, which is really, I think, where the rubber meets the road. It is where you have the most likelihood of failure, and it is where you need the most attention. And you do need strong product owners and very skilled contractors. So I think we have gotten up to this point, especially with this particular use case, and I can tell you that I have probably weekly interactions with the product owner. I can tell you from speaking with them and gathering feedback on a consistent basis that I know that the product owner and the contractor communicate daily, which astonished me. They operate in two-week sprints, even though they do communicate daily. They share a very rich technology stack, most of which can be acquired free off of the Internet. They use Scrum planning tools. They use a variety of different agile implementation methods that really helps them. It is also important to note here, even though I often reference the product owner, they are the end user, and they are tied in with the program office. In going by their office all the time, I can tell you they have whiteboards up, taxonomy up. They have really delved into this. Now, it may take them more time than is typical, but at the end of the day, once this project is complete, they are going to have rich features and better features than they would have seen in their kitchen sink Statement of Work. They are going to have a much better end deliverable and end up with a much better value. This implementation has been night and day, according to the end users. And they work hand-in-hand with the contractor to accomplish everything they are trying to do. They use desktop sharing, they use all the typical things you would think. But really what surprised me and what really opened my eyes to the agile methodologies that I, honestly, can admit I was not completely aware of a year-and-a-half-ago, there are so many tools that are out there to be used. But in the traditional waterfall method, you know, you can send a contractor off and ask for a deliverable in six months. And the train can veer off the track, so to speak. And that costs a lot of money. And if they go off the track and it costs you more time and money to bring them back on schedule, you know, you have wasted and you are heading down more of a failure path. With this particular instance, having all these tools that are all cloud-based, anyone from the product owner, anyone in the program office, and anyone in the contractor group can go on at any point in time and can interact on anything they are doing, on any given day, at any given minute. That is very helpful to them, and that is true agile implementation. Again, collaboration leads to success, and this use case has proven that.
So what is next? You know, this is one particular use case. I know, as I mentioned before, that GSA's 18F has many they can probably share, as does the U.S. Digital Service and many other organizations. We are all trying to collaborate and share what we are doing so we can share with others for their benefit. As far as HHS Buyers Club goes, we have plenty in the pipeline of projects that we are focused on. They are at different stages of pre-award solicitation and post-award. What we are trying to do is gather continuous feedback on what we are doing, interacting with our interagency colleagues, and really trying to improve, again, the acquisition of IT services. At the end of the day, it goes back to the problem, which is that so many IT service contracts fail for so many reasons. And it is very incumbent on us to mitigate that failure, and maximize success. The more that we do that, as hard and difficult as it may be, I think anyone can do it. You just need education and quality leadership to help guide you through it. Thank you.
JA: Our thanks again to Chris, Traci, Jonathan, and Mark for their presentatons on agile. We certainly hope you found today's seminar beneficial, as we are sure it gives us all a greater appreciation for what can be done to serve the American taxpayer. Stay tuned as we take a five-minute break. After, we will return with our guests to answer a few of your questions.
JA: Folks, thank you so much for helping us deal with our technical difficulties. Thank you for returning to FAI's Acquisition Seminar on Agile Acquisitions. We'd like to take this time now to answer a few of the questions you submitted during the presentations.
JM: OK, so question number one says that, "We only seem to reference agile acquisition processes in reference to IT projects. Is that the only place it should be used?" And this is a really great question, because there is a difference between agile acquisition and agile development. Agile development is a methodology for program management that specifically relates to developing IT, usually with software, but not exclusively. Agile acquisition is a term that I like to think of more in the context of providing flexibility within contracts. So, I often say, not every contract is agile development, but every contract ought to be agile acquisition. In that sense, I would like to refer you to the Digital Service Playbook, play number one, where it talks about getting to the need of the customers. As a contracting officer, it is our first role and responsibility to be a business advisor. So, in that sense, you know, you have got to get to your customers, talk to them, understand what it is they are trying to do, and then structure your contract to be as flexible as possible. Examples of that can range, but it can certainly be lowering the approval levels. It can be structuring level of effort acquisitions when you are trying to get support towards an existing team, but it can be based on whatever your end user's needs are. And, I would also say, Digital Services Playbook number five, where it talks about structuring modular contracting, is still a great option to use, regardless of what you're trying to acquire. Because in that sense, you're still setting small objectives and delivering value to the customer and not locking into long obligations. So if the government has reached the need or the funding levels are no longer what were projected at the onset, that you're still getting out of the contract with value and at a reasonable and responsible breaking point.
TW: OK, question number two talks about not being able to incrementally fund time-and-materials contracts, which appears to be an agency decision. That is not something necessarily within the FAR or without, but something agencies still have the right to set and to kind of guide how they are going to fund their contracts. But that does not mean that you can't use agile methods, and it also doesn't mean you cannot do modular contracting, as we mentioned, to try to set up smaller increments. If you need to be able to fund something for six months, you can probably get away with that in terms of the budget that you have and being able to fully obligate to those. But what we really are advocating on this is to try to tend to stay away from the time-and-material contracts for agile acquisitions because, at the end of the day, we are buying release product. And not just the best effort of somebody coming in and showing up and necessarily just not providing end products. But there are cases where it can be used. It is totally up to the decision of the agency and what works best for the program office using this. I think it is very important to start asking the questions of your budget people and your finance people to say, "We want to try to do these things, how can we manage them best within our agency?".
CC: Alright, question number 3 -- "Where does beta testing fit into the scheme of things?". So, the first the about agile is that, every iteration, which is typically two weeks, you are delivering releasable software to at least internal stakeholders, primarily to the product owner. At that point, there is going to be internal testing that is done. At some point, you may wish to release the software into more of a live environment. Beta is one type of a release that you can do to test the software out in more of a live environment and start to learn from actual users. Beta is also a way of setting expectations with users about the quality and completeness of your software. Most times when software is released with the tag of beta, people understand that this is not completely in the full version and there are still kinks to work out. Of course, it is always best to be using real users, as opposed to fake users, in beta testing.
JM: Question number 4 -- "If we do not know what the final deliverables will consist of, how can we accurately project our budget requirements at the start of the project?". This is a great question that comes up all the time. So it really goes back to changing the concept of what our requirements are, but planning your budget is still the same things. Doing government estimates is still the same thing. You are not just going to cast aside best practices in that arena. You just calculate your estimate assuming an agile process. So, if you remember, going back to the slides, we talked about how you have your overarching objective and the technical functional requirements will fluctuate as the project develops and you get end user feedback. But ultimately, the fixed price per iteration that was discussed will remain constant, regardless of what the specific inputs and outputs are. So really what you are talking about now is the length or duration of your development cycle -- or, I am sorry, your development period. So you will know, first off from an estimating standpoint, whatever you estimated the fixed price per iteration will cost and multiply that out by the total number of iterations you intended to be done. You would have your government estimate built that way. And I will talk a little bit more on that in one second. But with regard to tracking the contractor's performance, that is going to stay steady.
Once you've selected your contractor, and that price per iteration is fixed and locked into the contract, that will remain constant through the entire life life of the contract. And because of the nature of agile, you are always moving the end user's priorities towards the production cycles. So you are getting value delivered. So this second part of this question that talks about "Isn't it a big risk that a project scope expand beyond the available budget?", the answer is no. It is actually lower risk. Because you know what it will cost at the end; you're not going to just keep developing. You've invested this money, you don't have something you can use yet, so you have to spend more money just to walk away with something functional, at the point of which, when you have exhausted your performance or exhausted your funding, you stop and you have gotten the value you were going to get out of that project to that point. With regards to building the government cost estimate, if you do not have historical experience with fixed pricing agile development cycles, then you can do is assume the members that you would put into that team, or what your engineers would put into that team, the number of engineers, their rough hourly rates from a price schedule, and then build an estimate off that of how many individuals you would need and how many iterations you believe it would take. Again, it is very similar to the way you would anticipate or estimate an existing traditional project.
CC: So question number five is about end-user testing and test cases and how to handle bugs on a sprintly basis. First, testing should be done continuously throughout the agile process. It is not done towards the end of when the software is ready to be released, it is just done throughout. Ultimately, the product owner is responsible for deciding whether or not a user story is complete. As Traci and Jonathan have discussed, a product owner comes up with user acceptance criteria for each story. That criteria can serve as test case scenarios. When the development team is done implementing a story, the product owner executes those tests to determine whether or not it is fully complete. So an example of a user story is, as a user, I want to submit my credit card information to complete a purchase. And accompanying that story can be some acceptance criteria. For example, one of them could be that, if a credit card ends with an invalid expiration date, that the user is displayed with a problem or error to make that correction. So that is an example of a criteria, an acceptance test criteria that the product owner would be testing to make sure the story was done. In terms of bugs, ideally, as part of your definition of "done," which is kind of -- user acceptance criteria kind of serve as your functional criteria. The definition of " done" can serve as your quality criteria. And part of the definition of "done" is that all bugs or issues or fixes associated with a user story should be completely addressed at the end of the sprint. If, for some reason, it is not, you can reopen, you can create another user story so can they can be addressed in the next iteration. You do not extend the two-week iteration just because it did not get done during that Sprint.
Question number six -- "Since this approach is new, what challenges have we faced?" Focus around factors such as availability, experience, enthusiasm of government functional SME's. So, I would say, based on my experience, that agile is definitely a new approach. I am assuming this is coming from someone in an IT organization. It can be a challenge for the business side, as well in this new approach, in that they have to dedicate people to be sitting with the development team every day. But that is also what makes this process really successful. So absolutely look for opportunities where you can kind of pilot this approach and really follow the detailed rules of the agile methodology that you are following so you can gain experience and show how this is a more valuable approach. And with that, you will start getting more acceptance from different types of stakeholders, and you will be committing resources more effectively the whole time.
TW: And from my perspective, I have definitely seen that enthusiasm comes along with success. So as we have been implementing these on a case-by-case basis, there definitely has been some hesitation to start, but once it starts happening and it starts working and the team is engaged completely, and they are producing actual applications or functionality. And then they get to go back and change that if it is not working right away, or there are problems that start to happen. We can identify those at the contract level very quickly and we can start getting those things worked out. So, there has been a lot of acceptance, especially once -- even though it is a little fearful to get into, but I would definitely go back to what we said earlier. That is, get involved in training. There are a lot of communities out there that practice agile. There are a lot of resources out there available. So this is a new adoption for the government, but it is not a new adoption for the industry. Go out there and find people to talk to who do this. There is a lot of fun to be had in this new process and adopting it quickly. It is kind of a neat challenge to be part of in innovating and creating these new contracts. It is kind of a breath of fresh air in the government to say, "We can change things quickly and not spend a whole lot of money doing it correctly the first time and not having to fix those problems down the line. So we definitely have a lot of enthusiasm for it.
So question number seven asked if there was a web address for the Tech FAR. And we have put that up into the slide here so that is available. And if you go to playbook.CIO.gov, you can see the Digital Service Playbook, which has also been referenced during this presentation.
MN: Question number 8 -- "Is there any way to circumvent the workflow direction or at least a step if something changes the direction during your acquisition?". To answer this, the diagram that was shown, the third part of this module, it is an initial diagram, something we are still playing with. Part of what we are trying to accomplish here is to develop something that makes it easier for the integrated project teams to work together to accomplish an appropriate acquisition. I think with any model, there will be changes. We are working and experimenting as much as we can to evolve the model. So I do think there are definite possibilities to circumvent the workflow, to make adjustments. The workflow is designed to be situational, and I think as you are going through, if something does not make sense, definitely you would want to make an adjustment. As you are going through this, if you find an issue with the diagram as it stands now, if you have suggestions on ways that we can improve it, if you find that as you are going through potentially and trying to make it work with something you are doing, We, like everyone else, are looking for feedback -- your suggestions are very important to us. And I think as we can go through to refine these potential workflows, these diagrams, wizards, all of us collectively can collaborate and make much better workflows and diagrams, and be able to share with everyone as we move forward.
TW: Okay, question number 9 asks, "Why aren't more of the evaluations requiring vendors to provide a live demonstration of what they have done, along with their past performance customers present -- essentially saying live past performance with their demonstration?". This is a concept that I've never heard of before. I do not know that there is anything right or wrong with it, but I really like the idea that you could innovate in your past performance as well as your evaluations with this kind of topic. I can see definitely where if you can get previous customers to come in and have a chat, maybe not during the demonstration itself but with you at a roundtable, we're totally advocating for that. But also, what I like about this is that you are looking at ways to improve the way you do evaluations. As Mark had talked about with the survey for the vendors , post-acquisition, as well as doing more prototyping and demonstration scenarios or proposals, all of these are going to increase the way we can get to these services faster. Because a lot of the acquisition time is spent around creating documentation, and our acquisition processes should definitely take advantage of agility as much as anything else in this process. So we have agile contracts and we have agile software development, we should have agile solicitations, as well. We will definitely be working on best practices for the agencies to start working on being able to do evaluations and solicitations in a quick manner as well, to make sure we are getting the right companies, the right services, and getting the right products for the government.
MN: With question 10 -- "What was the timing impact of the two stage downselect on the delivery of the project scope?". This is not the easiest to answer, but I will take a stab at it. The project is currently ongoing. The solicitation was 'put out on the street', so to speak, in August of 2014. It was awarded in late September. The project itself, the contract, is ongoing. I can tell you, in terms of when the need was determined until the time the request proposal was released, it took a few months. The actual solicitation process from release of the RFP to award was about eight weeks. It was a very quick turnaround given fourth-quarter needs. Typically, without a lot of effort, without involvement of an integrated product team or project team, you know, this could take potentially six months to go from need to award. Sometimes even longer, depending on what you are trying to achieve. So what I want to point out here to give you an idea of the impact is if you have an integrated product or project team, depending on which language you use, which language you prefer, I think you will greatly minimize the time it takes to strengthen your statement of objectives, to strengthen your acquisition strategy. Anytime you can collaborate with all the internal and external stakeholders, the impact will be very positive, regardless of which method you choose, whether it's a two-stage downselect or another one.
In terms of delivery of the project scope, you know, the two stage downselect was definitely an agile type model, definitely a new way of doing things, although it has been done in the past, just not as often. I think in terms of actually achieving successful delivery of the scope, of the statement of objectives, I think it helps because one of the things that the program had here, that they would not have had before by just utilizing a tech-based proposal is that they were able to see a minimal viable prototype. Some coding, a wireframe mockup of the homepage, and the prototype prior to making an award. So when you have that before you actually make an award, you can start really with something. And when you start there with the award and the kickoff meeting of having something to look at and being able to evaluate that before you go down the road of doing an agile development, you already are ahead of the game. I think you already start off with a partnership where you both have agreed that the initial prototype was a good start, was successful, and, as I say, keep the train on the tracks, moving you in the right direction, and will help you to move quickly and stay on the same page.
JA: Don't forget, the Federal Acquisition Institute has recorded today's seminar, and the video, along with the presentation material you saw today, will be posted in the Media Library on FAI.gov. You should be up to access these items in a week or so. On behalf of the Federal Acquisition Institute, thank you once again for joining us.
