John Andre (JohnA): Hello, thank you for joining today's Acquisition Seminar hosted by the Federal Acquisition Institute. Today's seminar entitled, “Lifting the Curtain: Debriefings” presents a look at debriefings as a key and very useful part of the procurement process. Have you ever wondered how the Federal contracting process looks from the “other side”? Whether you’re a government acquisition professional or an industry specialist, have you ever found yourself asking, “What are those people thinking”? As a government acquisition professional, you probably think that your industry counterparts keep asking the same questions and just don't understand the answers you are providing. Likewise, industry representatives probably think that Federal acquisition professionals don't understand the private sector and its needs. In this seminar, we’ll uncover and share experiences and thoughts of the Federal contracting process from both government and industry. Presented for you are two mock debriefing engagements where we will offer a first-hand look at the Federal contracting process from both sides.
The business of government requires agency officials to balance four priorities: mission need, acquisition law, public policy, and industry. The complexity created by these intersecting, and sometimes conflicting priorities creates an environment that can be very confusing for all stakeholders, including legal officers, program officials, contracting professionals, members of industry, elected officials, and the public. The tensions often arise when these four priorities conflict, impacting the decision-making process. A better understanding of how industry does business will greatly help industry government relationships, thereby strengthening a government agency's ability to accomplish its mission. The debriefings presented during this seminar will shed light on both parties perspectives, and help you better understand your business partners. This is a rare opportunity to look behind the closed doors of debriefings and learn what the other side is really thinking. Before we begin, let me remind you that we will hold a live question-and-answer session at the end of today's debriefings. If you have a question about anything you see or hear, we encourage you to submit it at any time using the survey link to the right of the video screen. We will collect and review your questions during the debriefings, take a short break, and then return to answer as many as we can.
To lay a foundation of knowledge for today’s Seminar, we are pleased to welcome Jose Arrieta, Director of the Office of Small Disadvantaged Business Utilization for the Department of the Treasury. Mr. Arrieta, the people we are going to see today, are they simply actors reading from a script?
Jose Arrieta (JoseA): And that’s a great question, John. So the people you're going to see today are real industry partners and real agency acquisition officials, and they are not going to be reading from a script. What we’ve done is that we’ve given them pre-developed case studies that have half the information they need to execute a debriefing. We have asked them to meet with one another, get on the same page, and develop a strategy to manage the engagement. So, they are acting in roles, but they are real people, from industry, from government, that do these things on a regular basis, and they are playing a role based on information and not reading from a script.
JohnA: I am curious, though, about the content. Is it simply made up or is it from real life examples and experiences?
JoseA: That's a great question. So what we try to do with this training is create a holistic mechanism to train contracting professionals on how to do a debriefing and to train them on how to understand all the different things that impact debriefings. So, what we did to create the content is that we interviewed over 300 acquisition professionals. We did it while I was at DHS. My job before this was that I was at the Department of Homeland Security. I called in 300 private-sector industry professionals that work in the acquisition function. We asked them the question - What are your four or five pain points when you go through a debriefing? What does government not understand about industry? Based on that information, we had our Policy folks in the room and we had some Contractor support in the room. We looked into our portfolio. We found acquisitions that we had done historically, and based on old solicitations, we actually created these cases to actually train our people. So, when you see this, a lot of the work here can be attributed to the good folks at DHS who helped us build these case studies.
JohnA: Outstanding! Thank you, sir. Let's get started.
JoseA: Hello, my name is Jose Arrieta, and I am the Small Business Executive at the Department of the Treasury. Before we get started, I would like to thank all the folks from industry and government who have signed up for our event and are going to join us for the educational seminar today. We very much appreciate the time to be with us. I would also like to take a moment to thank the folks from the Federal Acquisition Institute and the Office of Federal Procurement Policy who have given us an opportunity to actually share some of the things we are working on with you. Today, we are trying to address a major challenge that we see on the horizon for the Federal acquisition workforce and the acquisition workforce within industry. 30% of the acquisition workforce within industry and 30% of the acquisition workforce within Federal government has less than five years left -- within five years time they will probably retire. 30% of the acquisition workforce within industry and government has less than five years of experience. So that's a significant challenge. We need to figure out a way to train our acquisition workforce to be smarter and faster, both in industry and government. The Federal government spends over $500 billion dollars a year and the bulk of that spending is done by people, and we have to figure out a way to develop those individuals so that they can holistically understand the function of acquisition to be able to engage in this process for the future, and at a much earlier age than those that came before them. Today, what we are going to do is share with you a methodology that we believe can train people to be smarter, faster, and we are going to do it within a specific function of the Federal acquisition workforce. We’re going to talk specifically about debriefings.
Now, if you think about work in the Federal government, whether you are in industry or whether you are in government, there are really four intersecting and conflicting priorities that you have to manage in order to be a successful acquisition professional. Many times, those authorities do not help one another. There is procurement law, there is public policy, there’s the agency’s mission, and there’s industry. If you think about training programs historically, training programs focus specifically on CON100 or specifically on Learning Capture Management and Business Development. However, there are not many training programs that focus specifically on holistically developing an individual with those four priorities in mind. And if you want to be a leader in our function, you need to have that type of experience.
Furthermore, the training programs that you enter in to are many times individuals talking to you and telling you what you must remember. They never challenge you to think critically, they never challenge you to use cognitive ability to actually solve difficult problems. Today, what we hope to present to you is the idea of how you can train individuals to think holistically about the acquisition function, think critically about difficult problems, and develop solutions in an environment where they don't have all the information, very much like you would in a real acquisition. However, in this environment, there is limited risk.
So now I want to talk a little bit about some of the tactical aspects about what you are about to witness. What we have done is very much like a Harvard Business case study. We created two case studies focused specifically on debriefings, and we have put together an industry team and a government team. The industry team is made up of small businesses around the beltway and they have a case study that they’re preparing for, that they are using to prepare for the debriefing. The government team is a team from the Bureau of Engraving and Printing (BEP) and an acquisition team from the Internal Revenue Service (IRS), and they have prepared to enter in to an actual mock debriefing.
You're going to get an opportunity to watch them enter into that debriefing and holistically we manage all aspects of that debriefing without the downside of a real briefing. We believe this is the best way to train the acquisition workforce in multiple functions throughout the acquisition lifecycle, and we’re going to pilot here within the function of debriefings. A couple of things I want you to remember, very tactically, about debriefings as you go through this. Debriefings are a positive thing. Debriefings decrease the likelihood of protest. If you do a good debriefing and you explain to an industry partner why they didn’t win, you will increase the quality of proposals from that industry partner in the future. That will enhance competition. Doing a good job with debriefings will improve your ability to meet your agency's mission and provide best value to the taxpayer. Not to mention the fact that listening to your industry partners when you go through the debriefing process will help you understand more about the challenges they face when they do business with the Federal government. It will help you understand more about the time and expense associated with putting together a Federal proposal, and hopefully, it will give you some insight so the next time you go through the acquisition process, you will be able to plan accordingly to lower those costs, and lower that time, and get higher-quality proposals.
Before I hand it off to Thomas, who is going to walk through some training for you before we go into these debriefings, I want to mention one more thing, and these are some things that have helped me throughout my career, specific to debriefings. When you are preparing for a debriefing, don't wing it. Practice. Have a strategy. Come up with a plan. When you are going through a debriefing that is based on a best value determination, explain the best value decision. Don't just tell the contractor that you lost because you weren't the best value. That is of no help to them. Don't rest your laurels on the fact that the debriefing that you gave was FAR compliant. FAR compliance is important, and I want everybody to be sure they do FAR-compliant briefings, but be a mentor to the industry partner. Be a coach to the industry partner. Give them information that will be useful for them in the future as they compete for Federal contracting opportunities. These are things that are going to make your job easier and are going to make the industrial base that supports you much better, much more dynamic.
The single most important thing for acquisition professionals in any phase of the acquisition lifecycle and anything you’re doing is communication. Whether you are dealing with a customer or whether you’re dealing with an industry partner, it is very important that you have developed your communication skills. Part of that is communicating verbally with words, and part of that is listening, and we hope that this debriefing training will give you a glimpse into an environment where contracting professionals and industry professional can actually learn from one another.
I thank you very much for tuning in to this. I am done for today and I am going to introduce you to Thomas O’Linn. Thomas O’Linn is the Director of Policy at the Department of the Treasury. Thomas O’Linn has worked in industry and in government, on acquisition teams in industry and in government, and he has developed a little bit of a training to help remind you of some important points to remember when you are preparing for a debriefing. He is also going to introduce and provide a little bit of information about the case studies. Thank you very much again for your time, and I hope you enjoy what we believe is the future of developing the acquisition workforce, both in industry and government. Thomas.
Thomas O’Linn (TO): Thank you, Jose. The goals for this training session will be, one will be refreshed on information, concepts, and principles associated with debriefings. You will be able to recognize oral debriefings as a critical and useful tool for improving vendor and industry relationships. You will also be able to view oral debriefings as a positive opportunity for providing information to the vendor on ways to improve as well as learning about how the Government can improve future acquisitions.
The intent of debriefing is to provide information to offerors, whether successful or unsuccessful, so they will gain knowledge that will assist them in improving their chances of success for future source selections. Debriefings may be performed orally, in writing, or in any other method acceptable to the Government Contracting Officer. There is no specific requirement to hold face to face or oral debriefings. Nonetheless, as a Contracting Officer responsible for making the decision as to the location and method, the needs of the offer should be considered when setting up a debriefing.
There are two types of debriefing. There is the pre-award debriefing, which is held for those excluded from competitive range or otherwise excluded from competition prior to award. There is also the post-award debriefing, which is held after award to either the unsuccessful or successful offerors. Some keys point to remember here are if a pre-award debriefing is held, then do not provide a post-award debriefing. Always ensure that all parties are aware of the conclusion of the debriefing, because the conclusion of the debriefing establishes the protest timelines. Also, always solicit offerors attendees list and inclusive of their titles, as a number of attendees and titles may aid in the development of the agenda and indicate the tone of the debriefing. Lastly, always document the debriefing and include an official summary of the debriefing in the contract file.
Here are some pre-award debriefing do's and don'ts. Do discuss an agency's evaluation of significant elements in offerors proposals. Do discuss a summary of the rationale for eliminating the offer from the competition. Do not discuss the number of offerors, content of other proposals, ranking of other proposals, nor don’t provide a point by point comparison with other offerors.
Some post-award debriefing do's and don'ts are, do discuss the offerors evaluated significant weaknesses or deficiencies. Do discuss overall ranking of all offerors, and, importantly, the rationale for the awards summary. Do not discuss a point by point comparison of the debriefed offeror’s proposal with those of other offerors. Do not divulge privileged commercial or financial information, or trade secrets. For further information regarding pre-award or post-award, go to FAR 15.505 and FAR 15.506.
In terms of best practices for the Government, always prepare and make a detailed agenda. This is important because being unprepared is the surest way to lose credibility and the confidence of the offeror. If time allows, one good way, as we will see here, is to conduct a mock debriefing, which enables you to assess the strengths and weaknesses of your team, and develop a sound, strategic agenda. Altogether, when preparing for a debriefing, try to think like the vendor and contractor. Thereby, be able to anticipate questions and anticipate and address their strategy on how they are going to present. Other best practices are reviewing all source selection documents and knowing your team and their roles. Knowing your team and their roles is important, as it is critical they understand the debriefing process, their roles and responsibilities during the debriefing, as well as the agenda for the debriefing – it is very important. Otherwise, as we clearly have understood and seen in previous experiences, a disorganized or unprepared team can jeopardize the entire debriefing and thereby source selection process. Lastly, anticipate questions and prepare for answers. Whether provided by the offeror or developed by the team.
Benefits to the Government are that debriefings deter protests. This is because they provide a clear understanding to the vendor of the agency’s evaluation process and the basis for the source selection decision. It also instills confidence in the contracting process by affirming that offerors were treated fairly. Another benefit is that it fosters a relationship of trust between the Government and the vendor. This is beneficial because unsuccessful offerors are now able to understand what the Government’s award determination process was and that they will be able to accept any negative findings in the debriefing if they perceive that the agency has acted with fairness, consistency, and objectivity, and in accordance with the evaluation criteria described in the solicitation. Other benefits are that it promotes future competition, it improves the quality of future offerors, and improves the quality of future solicitations. Lastly, it demonstrates mutual respect and concern. Here, by holding the debriefing, you recognize the time and effort that a vendor has expended towards providing a response to your solicitation.
Benefits to vendor and industry is that it reduces cost to obtain information, it enhances the quality of future proposals, and it improves the vendor decision-making capabilities because now you’ve reduced misunderstandings and a gap of knowledge. It also benefits vendor and industry because it provide an opportunity to provide feedback on the Government’s procurement process, as well as provides the ability to amplify information, ask questions, and understand the customer.
Some keys to successful debriefings are findings are supported by specific reference to solicitation/offer page, paragraph number, and where the strengths, weaknesses, deficiencies, risks, and clarifications occur. This is important regardless of the complexity or the dollar value of the solicitation.
Other keys to success are ensuring key points during the debriefing are captured and documented. Also, tone, syntax and technique are also very important, which will be seen in the mock debriefings shortly. Some key points or helpful hints from tone, syntax, and technique are maintaining a positive delivery by avoiding the extensive use of “will not” and “cannot”, balancing and controlling the meeting with dialogue and interchange, listening carefully to the offeror’s questions and verifying that you have answered them, watching body language and maintain good eye contact, and using a professional tone and refraining from being condescending, abrupt, or defensive. Lastly, focus on open and positive communication. Essentially, keep in mind, it's ok to disagree or not. You are here to be discussing the solicitation and the process. You are not here to debate, you are here to relate.
Convey confidence in the award process. This is key, as the entire solicitation process influences the debriefing process. Provide specific and honest feedback within regulations. But also, avoid addressing hypothetical questions, such as if we had proposed this, how would you have rated it? Lastly, establish debriefing ground rules upfront. For example, make sure it is clear upfront that the debriefing is not a forum for debate. Again, you’re there to relate and exchange information, not to debate.
Before we close, I just also want to give a highlight about the difference between debriefing versus explanation, because this may come up from time to time. Debriefing is associated with FAR 15 source selection procedures. It provides detailed information on the debriefed offeror’s proposal and the selection process. An explanation is associated with FAR, subpar 8.4 and is very brief and short and does not include lengthy discussion of strengths, weaknesses, deficiencies, ratings, or rankings of offerors.
This brings to conclusion my part of the overview. However, let's now move on to the mock debriefings. Please note that the actions, words, emotions, and opinions expressed by the individuals participating in these role play mock debriefing sessions are completely fictional and not representative or reflective of those involved. These role play mock debriefing sessions are for training purposes only. In the beginning, each participant will be provided an introduction and an overview of the case. At the conclusion, there will be a time for the participants to provide feedback as well as to answer questions.
The first scenario, we have that Treasury awarded a contract for technical services in support of its SharePoint site. Gray Space Corporation, one of three offerors who responded to the request for a proposal, has requested a debriefing. GSC believes the Government may have erred in its evaluation process, and is confident it answered all the discussion questions in their final proposal revision. The Government used the Trade-off process in accordance with FAR 15.101-1.
For scenario two, we will see that Treasury awarded a contract for engineering and technical services. BelAir Company, one of three offerors who responded to the request for proposal has requested a debriefing. BelAir wants to understand why the Government did not “follow the RFP and do a true best value award to them.” That last portion is a direct quote from a call between BelAir and the CO made during the set up of the debriefing. Again, the Government used the Trade-off process in accordance with FAR 15.101-1. Thank you.
John Q Public (JP): Good afternoon. Thank you all for coming. We are here to discuss the debriefing for the acquisition your company bid on for the Treasury CUBE. Appreciate you taking the time out of your schedule to come. My name is John Q Public, and I was the Contracting Officer for this acquisition. We’re going to go through brief introductions of our staff and yourself, we’re going to go through a little overview of what to expect during the debrief and the ground rules and what we expect to touch on. If you have questions as we go along, please feel free to stop me and ask questions as we go. At some points, I will probably ask you to wait until we get to the end of a certain section just so that I can get a thought out, and then we will go back and forth. With that, I would like to turn it over to Ms. Adata King.
Adata King (AK): I am Adata King, and I am the SSEB chair.
Jane Jacks (JJ): My name is Ms. Jane Jacks, and I am the Contract Cpecialist for this acquisition.
John Doe (JD): I am John Doe, The technical subject matter expert for Treasury.
JP: And I’m just going to pass over a sign-in sheet and if you don’t mind, just introduce yourselves as you are signing in and we’ll make that part of our debriefing record.
Anita Great (AG): My name is Anita Great, I am the CEO of GSC.
Anna Noah Itall (AN): I am Anna Noah Itall, and I am Counsel for GSC.
Sandy Lincoln (SL): My name is Sandy Lincoln, and I am the Proposal Manager for GSC.
Susie Jobs (SJ): Hi, I am Susie Jobs, and I am the Vice President of Technology.
JP: Thank you again for coming. So, as I said in the opening, this is a debriefing relative to the proposal your company put in. Essentially, what we are here to accomplish is, from my perspective, two things. One, to give your company a better understanding of why you were not selected for award. And secondly, to allow you the opportunity to gain greater insight into the evaluation of your proposal so that you can make improvements as you go forward on future proposals, not only with the Department of Treasury, but elsewhere within the Federal government.
We are going to be going through an evaluation of your specific evaluation weaknesses and strengths. As we go through, we will go in depth into your proposal as far as what we viewed as strengths and what we viewed as weaknesses. We will give you an overview of the winning offeror to include technical and price to allow you an understanding of why you did not win. We will not be doing what the FAR refers to as a point by point comparison, meaning where I will give you the technical score for the winning offeror, I will not give you the detailed breakdown that would be their unique technical solution to it as I am not permitted by law to do so.
I have all of the pertinent documentations that I am assuming you all have. Here’s a copy of the solicitation. We are just going to go through it at a high level, won’t go into any depth unless we need to speak to a specific factor or evaluation criteria.
Essentially, the proposal was rated against three technical or three non-cost factors: technical or management approach being the first, proposed staff being the second, and past performance being third, and then comes price, and those factors are listed in ascending order of importance and when combined are significantly more important than price.
As we are going through here, we will be speaking to different ratings that your company received. Each factor were scored under superior, good, satisfactory, and unsatisfactory for all factors, minus past performance. Past performance actually had a slightly different rating structure, one to include the normal/usual, one to note we only had superior, satisfactory, or unsatisfactory; there was no good rating as part of the solicitation, so it is slightly different evaluation methodology.
So before I get in to your overall proposal, what I would like to do is just walk you at a high level through both, the winning offeror technical rating and price and your technical rating and total evaluated price. There was not a ranking prepared for this, so there is none to offer. We received three bids. Two went on to the competitive range. Yours was one of the two, obviously the winner and your company, the unselected offeror. So for factor 1, technical and management approach, you were rated good. Your proposed staff was rated as a good. Past performance was rated as a satisfactory, and your total evaluated price was $4,257,056. The winning vendor, SSC, was rated as, under technical and management approach, as superior, proposed staff good, past performance satisfactory, and total evaluated price, which you had in your initial letter, of $4,427,340. Do you want me to repeat any of that, or are you good?
AG, AN, SL, SJ: No, we got it.
JP: Essentially, we made a Trade-off determination, which we will get into in a little bit, of taking a higher priced for a technically superior proposal, in accordance with our instructions and our evaluation criteria listed in the solicitation. If there are no overall questions, then we will get into your specific proposal and the factors for how we came up with your individual technical ratings. Does that work?
AG, AN, SL, SJ: Sounds good.
JP: So, for technical proposal, for factor one, technical and management approach, you were rated as a good. Some of the strengths that were identified, the risk mitigation strategies identified in the project management plan provided realistic and reasonable approaches to resolve problems that may arise during the implementation and deployment phases, and even identified potential pitfalls and resolutions the Government had not previously considered. Your organizational matrix was well-documented and detailed roles of each member of your organization. You also provided for detailed information on your retention rate and turnover, which you listed as 5%, which the government noted as a strength.
Overall, your proposal was rated as good, so it exceeded our expectations, but it did not rate into the superior category. Under weaknesses, one of the first weaknesses that was identified by our technical team was relative to the source code proposed by your company, and specifically your subcontractor. Has proprietary code that was recommended be inserted into the SharePoint code to increase security. Your proposal went into an explanations to why this security would be beneficial to the Government and went into a description of, why from your perspective, it represented a minimal impact to the Government based on the security that would be gained from that heightened level of security. The technical evaluation team evaluated it, they saw it as a weakness, and did not see it as a deficiency, and I think that is also important to remember as we go through here. We do not have any deficiencies in your proposal. It is seen as a weakness, so your overall rating for this proposal is still good, but this is one of the areas the technical team highlighted as a weakness, or problem from their perspective, on how that code could be used in the future and locking us into a potential sole source environment for updates relative to a SharePoint solution that should be able to be transferred between vendors on recompetition.
One of the clarification questions that was put to your company was asked to give more detailed information relative to task 2.1.4, supporting the implementation and post-implementation of SharePoint updates, including coordination with data center security, accessibility, and other CUBE directors with operational compliance responsibilities for CUBE applications and infrastructure. The only information we received relative to this was a short paragraph within your proposal and your response that essentially stated, “will support implementation and post-implementation of SharePoint sites, coordination data, documentation to support application, accessibility and data center operations will be prepared in accordance with the CUBE director requirements. Essentially, when our technical evaluation team was looking at it, you met our minimum requirements. You told us what we had stated in the statement of work. But the lack of additional detail and how you were going to accomplish that was listed as a weakness within your proposal, and a greater enhancement of how you were going to do this sort of lead to a greater understanding by the technical evaluation team as they were conducting their review. Similarly, the statement identified one important area is one Treasury message. The technical panel did not feel that you substantiated your understanding of how multiple Treasury components would interact with each other, and how the SharePoint implementation would be affected, negatively or positively, by this interaction. Again, your proposal restated our requirements with some supporting documentation but did not fully substantiate what we were looking for or some of the other areas the tech team would have wanted to see to be able to fully understand your solution.
The last identified weakness under this factor is that your company stated they would apply an organized approach for implementing SharePoint across all of CUBE. They did not expand on this approach and presented little substantiating their claim. We referenced your page 12 paragraph two as the only area where you addressed this and the area where the tech team, highlighting, “our implementation team lead will develop an all-encompassing implementation plan that addresses the need of each CUBE director and additional staff, including subject matter experts will be included as required, to ensure that if a substantial issue arises they are immediately addressed. We will conduct monthly meetings with the Government leads to review progress.
I understand you have staffing and you identified the staffing and you were going to have meetings and you were going to respond to our needs. How in which would you mechanically respond to those? Do you have any troubleshooting techniques that your company has available, the overall project management you have available to be able to go through it, things of that nature that allow us a greater understanding of how you will do what you say you will do in the proposal could have led to this not being a weakness. Those weaknesses, is in important to highlight, none of them are deficiencies, none of them are show-stoppers from the Government’s perspective, they are however areas and items that were identified as weaknesses. Your proposal did not receive a satisfactory, it received a good, it just did not receive a superior qualification. If you look at the definition rating for superior on this and the next factor, you’ll notice that any weaknesses preclude you from being rated in the superior area. Therefore, by definition, we had to go with the good rating for one or more, but these are the four that we identified. With that, I will pause for a second. I know it's a lot of information to digest. Would you like me to go through all the technical areas, and then we can discuss, or would you like me to pause after each factor so we can have a discussion on those?
AN: I do have one question. I would just like to be clear that those four items you listed as weaknesses, they were weaknesses, rated as weaknesses by you, were they significant weaknesses or just rated as weaknesses?
JP: They were rated as weaknesses.
AN: OK. With regard to the evaluation of the proprietary source code in the weaknesses assessed by your technical team, would we be able to get more of an explanation from your technical representative?
JD: Sure. It's not just that it’s proprietary. SharePoint is also proprietary application, where we wouldn’t get the source code. But, the difference between the base SharePoint COTS application and the proprietary security add-on is that we would be stuck with our corporation for any maintenance on that proprietary add-on, whereas there is a variety of vendors that can maintain just plain COTS SharePoint solutions.
SJ: When you evaluated that, did you consider the fact that when we did market research with you, you guys were really excited about the product? That is one of the reasons why we bid this proposal because you gave us the hint that this is exactly what you needed a product that helped address security risks. In this day in age, with OPM hacks and the like, it seems like a very important thing. So, we’re surprised to hear that you evaluated that as a weakness.
JP: And so, I can address that. I understand during market research, the customer organization talked to many vendors to try to gain an understanding of what the market place was or what was not. At no time was the Government's intention to give you a hint one way or the other. Our requirements are as stated in the solicitation. What we had within the solicitation is all we can evaluate against. When we went out looking for a commercial solution to what we have, tailoring of that becomes an issue when we look at the overall solution. That is why your company was rated as a weakness because it proposed a proprietary plug-in for security. That’s not the only way to meet security. What you were told prior to that, I cannot speak to, nor can anyone else, because all we can speak to is how we evaluated your proposal, based on the solicitation and your response to it.
AN: So, if I may, in the solicitation, you clearly stated that you wanted innovative and effective strategies and solutions. I didn’t see anywhere in there, and I might have missed it, so if you would, please clarify, I didn’t see anywhere where it said nothing could be proprietary to the Government or coming from us to you guys.
JP: I understand, and that’s also why it's not listed as a deficiency within your proposal. Innovated solutions are sometimes good, and are sometimes bad. The value that you perceive they add is sometimes value we understand and share with you. Or an innovative solution can be something at this time is not within the Government's risk tolerance or how we would like to proceed. We try to give you as much detail around our constraints within the solicitation and within our environment. We were open to many different types of solutions. But our role was to evaluate those solutions against the solicitation. It was not a mandatory requirement of the solicitation that it was not all open source code or a tailoring. Therefore, you could have proposed a tailoring solution that was seen as a strength. Conversely, in this case, the proprietary source code was viewed as a weakness. However, like I said, that was not a deficiency, it wasn’t a show-stopper. You still ended up receiving a good for your technical overall rating. So, still, a very strong mark from that perspective.
AN: This seems to be a pretty important factor in the rating of our proposal.
JP: Absolutely.
AN: Reviewing the questions you provided to us when we were determined to be in the competitive range, you submitted questions to, I presume, both offerors. Did you submit questions to SSC as well?
JP: We did a point by point comparison. So, from our perspective, we held what the FAI would call “meaningful discussions” with both offerors. What we submitted to the other company, I cannot speak to. What I can speak to is what we submitted to your company.
AN: I see. There was no question about the proprietary code. Did you consider asking us about that, and asking us to address that aspect?
JP: From the technical evaluation report, the information relevant to the proprietary code was discovered and learned after the discussion route had gone. So, it was something that was evaluated on your revised proposal submission.
AG: So if this was a more secure technology, why wouldn’t you follow up with a question?
AN: Open a second round of discussions?
JP: After we completed our initial round, we established competitive range. We went back out, and –we added revised proposals, a second evaluation. We thought we had a firm understanding of both offerors proposals, we had an award decision we could defend, we proceeded to award. There is no onus on us to have repeated rounds of discussions. Each proposal submission from the company needs to be their best and final offer for lack of a better term. They tell us we can’t use AFO anymore these days, but your proposal needs to be able to stand as the original solicitation stated. While we reserve that right to hold discussions, we are not required to hold discussions. In this case, we established a competitive range of you and another company, excluding another third bidder. Held one round of discussions. After that discussion, we had a firm technical evaluation, defensible source selection, and what we felt would be the best value trade-off analysis that was sustainable and moved forward with award.
SL: So, would you mind clarifying something for us? You said our strength was our risk mitigation strategy, but yet you are saying that our code, which is something innovative and providing you the security that you were requesting, is essentially a weakness. Can you help us understand how our risk mitigation strategy, which obviously a weakness refers to a risk, how that was evaluated and what process you guys went through to determine that the code and all that we were offering with respect to the code, that our risk mitigation strategy would not apply to that in such that it would become a weakness?
JP: The weaknesses identified by the technical team speaks to our ability to tailor a specific COTS product, in this case SharePoint, the software, while to my technical team lead’s point earlier, while SharePoint is not an open source system where we can have this source code, it is a COTS product. The risk or the weakness, for better term, was identified by the fact that your proposal included a proprietary piece of software to be added on to a COTS product that would then put us in a position of any future maintenance, changes, enhancements to that system would then have to not only leverage the COTS product that we bought through SharePoint, but also specifically the subcontractor bid under your proposal. It would lock us into an environment where those changes would not be forthcoming, and that is seen as a weakness by the Government. Not necessarily commenting on whether or not what the software itself does, is beneficial or not beneficial, the weakness though was identified was around the fact that it was not an open source code or was not something that could, let me rewind for a second, it was something that would lock us into a proprietary software.
JD: It’s a lifecycle cost risk that we would not have a competitive contractor base for that part of the system. And, it’s just based on my experience with other software acquisitions, I felt it was appropriate that it was a weakness in that respect, that it could be costly or we would only have one source for the maintenance or enhancement, that piece of your solution.
SL: OK, so I am having a hard time with this, quite honestly. Nowhere did we state that it was not going to be open to you. Nowhere did you state that it had to be a COTS solution in the solicitation. And, when you did the evaluation on the pricing, the lifecycle costs, obviously, our pricing was not out of line. So, the price of it is one thing. Your saying the price and the lifecycle cost is where your saying it is a risk, right? And because of your experience, it became a weakness, but, yet, nowhere do we have a COTS, a request for COTS. And, so there is a bit of an issue here.
AG: And, I think, just to be frank here, is again, we thank you, the reason we’re here is we want to learn and we want to be able to continue to bid on other opportunities, but we also want to make sure that we are providing the right solution or look at no bidding a situation. In this situation, we knew that this was the best solution, that it is actually very cost effective, and it’s proven on our past experience. So, that’s where, again, with my team and why we are here, to learn, is this something that when we’re providing sound market research to show a more secure system. And, to be able to have that in there to not only be able to show, as my proposal person said, you see it as a plus as a risk mitigation, but then as a weakness. Again, we are finding this to be slightly confusing, and something that when we had our Legal Counsel send the letter, we don't see anything until now, not even in the questions. So, in order to be able to move forward for us and with others, because we work with other Government agencies where this is not an issue at all, and to find out now, for the first time, it is a little disheartening.
JP: I understand your position. I’ll speak to the future acquisitions, then we’ll go back to the one we’re actually discussing here. One thing I would suggest, I know some vendors are hesitant to do because it reveals to others what your potential solution will be, during the question-and-answer period of any solicitation, you can go in and ask them specific questions around source code, what would be acceptable, what is your risk tolerance relative to these types of acquisitions, you don’t have to be specific on what subcontractor you are going to use or what kind of software you are going to use. But, you can ask the Government for clarity if you don’t see it within the solicitation, as far as, would a solution that encompassed something that was proprietary be acceptable?
Now, going back a bit to this specific solicitation. When you discuss the fact that your proposal, our solicitation, doesn’t speak to any issues about having a proprietary code in it, your proposal actually speaks to the fact that you understand that this could be seen as a risk, however but you believe it to be “minimal” risk to the Government based on what your solution is and how you believe it would be handled. I understand your perspective, I respect your perspective, the Government technical evaluation team had a different perspective as part of their evaluation. But, your company did identify, one that was proprietary sourced. Two, you understand that could be an issue, however, you went on in your proposal to explain that we feel the security concerns and what our software from our subcontractor would do outweigh these concerns. Which is a fair statement. However, the technical evaluation team reviewed it and had a different take of opinion that what you have, which is not uncommon when you bid a different type of solicitation or evaluation.
SL: So, two things. We did come in and meet with the staff, we met with everybody. We presented our solution and it was received very, very well. This was not just an unknown solution we just sent over as an RFP response. This was presented. We explained all of our capabilities with SharePoint, including the proprietary code that increased the level of security during your market research phase, and it was wildly accepted. Everybody really wanted it. It was very positive. We had no negative feedback on it, there was no concern about it being proprietary. So, we openly discussed that.
SJ: We have been transparent the whole time.
SL: The entire time, so now going into the proposal response. Again, I’m going back to the fact that nowhere in here did you say no proprietary code, right? And, you had led us to believe during the market research that the proprietary code was exactly what you wanted relative to getting you to where you wanted to go. And then, even though we presented that there was a minor risk, we gave you our risk mitigation plan, and you rated it us as a strength on that risk mitigation plan. So, how is it that the code itself is suddenly, if a technical team is suddenly saying it is negative because it is a risk, but our risk mitigation is a strength, there is a discrepancy there that we cannot get past.
JP: OK, so one to the market research. We hold market research with many different vendors, I caution you, as far as market research and the rest of it, what is presented there is not information that will be evaluated. What is presented there is your marketing, your technical solution, different innovations and open collaboration so that we can understand the marketplace and the different options available to the Government. When your proposal comes in, all we will evaluate is what is in your proposal. So, if you do a two-hour presentation on your capabilities and how this all works and how it mitigates all issues for one group of ten Government employees as part of market research, and you do not include the same type of information within your proposal, we cannot go back out and draw down the deck from that presentation as outside information. Nor would we do it if one of your competitors left something out of their proposal.
SL: It was included. It's there. And, as I just said, so we included the fact that it was a capability of the SharePoint and the proprietary code, we discussed that during the market research, and we included it in the proposal, and the risk associated was not seen as a negative. It was seen as a strength because of our risk mitigation strategy. And so…
JP: Your overall risk mitigation strategy relative to the technical approach and the rest of it was rated as a strength. A component of that falling under your risk mitigation for that software does not speak to your entire risk mitigation strategy. Your overall risk mitigation strategy, a different risk arises as part of the technical and management approach. The overall solution is an overarching risk management proposal that your company submitted, your risk mitigation relative to how we’ll deal with this being proprietary source code may be a portion of that but not speak to the entirety of your risk mitigation plan. Your risk mitigation plan should accompany all different aspects of your proposed solution, not just your subcontractor’s proprietary software for one component.
SL: So, what did it not include then? Because I am a little confused on you came about of this, because if we look at the fact that you said it was a strength, we’re still not getting to…
SJ: We were honest about the risk. So we clearly clarified that it was a risk there.
JP: And I understand where you were clarifying on the risk. And I understand where you were stating from your perspective it proposes what you called a minimal risk. The technical team evaluated it, the technical team identified it as a weakness. Not a deficiency [hold on one second], not a showstopper, not something that would prevent an award, just as a weakness.
SL: But the weakness that the technical team mentioned was relative to lifecycle cost.
JP: Not what…
SL: That’s what he just said.
JP: Then he misspoke, and we will get to cost as we go through. The technical evaluation dealt with the proprietary software, as I have explained.
SL: So, can you talk to us about how you actually went through that evaluation and came to the conclusion that it was a weakness and not a strength?
JP: Hang on. We went through the evaluation in accordance with the solicitation. How we evaluated with what was in the section now with what we evaluated in section M. That is how we evaluated it straight out of the solicitation. We reviewed the proprietary source code and at this point I think we’re going to move on from this and agree to disagree, because, I understand your perspective, if I were sitting in your seat, I may share your perspective, and if you were sitting in mine, you may share mine. It’s a disagreement over whether or not your subcontractor’s proposed software, being proprietary code, the fact that that would lock us into a single vendor, would take a COTS product and make it non-transferable to another vendor at a later date, was identified as a weakness.
SL: That was an assumption made on your part, though.
SJ and SL: Because you don’t have to be locked in though. You don’t have to.
JJ: Well, if you take a look at your proposal on page six, paragraphs 4 through 6, you talk specifically to the fact that this would be an inconvenience. “While this code is owned by GSC, we feel the inconvenience to CUBE is minor when compared to the benefits of thwarting external penetration from unwanted visitors and internal security threats from employees.
SJ: Inconvenience isn't the same as being locked in for life though. I mean, you could have also negotiated other arrangements with us to be able to sell you the code.
JP: I understand your perspective. However, when we’re looking at the proposal, the onus is on the vendor to answer the questions upfront, to provide us a description of how it’s going to be. The fact that I could negotiate it out, the fact that I could do “x” or I could do “y” is something that should have been explained or offered up front. If you believed it to be an area that was going to be of no concern or no issue at all, why was it highlighted? I mean, I understand from market research you believed it would not be. Our solicitation stands as is, our technical evaluation was based on that solicitation, and in accordance with section M. I don't think we are going to come to an agreement over one perspective verse the other. I fully understand where you're coming from and I think I have explained it as much as I can relative to it.
AG: And, we agree. I think we should move on as well too, especially because we’re looking, we’re trying to get a better understanding of the trade-offs. This is just one area, this is something that we all firmly believe in, and again, we’ll take issue with it not being documented in the question and that it had been, what we feel, is overlooked. But in looking at this…
AN: We would like to make one last point, for the record. You go ahead.
SL: So, you guys had requested that SharePoint be a custom application that essentially required custom code in order for it to be deployed farm through farm solutions. So, how are you intending to have that not be the case?
JP: Our custom code on COTS applications is transferable code. When you read our FAR clauses as part of the solicitation, that’s all transferable code. Right of ownership goes to the individual who paid for it. In this case, that would be the Federal Government.
JD: Because that would be code first produced…
AN: Well, actually, unlimited rights goes to the Government, not ownership.
JP: Correct, not ownership.
AN: Your vendors still own the code.
JP: But we have unlimited rights and unlimited transfer on development code that was done as part of our contract.
SL: Which is exactly what this is.
AN: Well, and I think our, our, well, we were just hoping that we would have engaged in discussions where we would have been asked.
JJ: We had a superior, a technically superior proposal very similarly priced.
SL: So, how were they technically superior?
AG: Yes, I think we need to move up to the…
JP: We’re going to go through, we’re going to go through your tech eval…
JJ: We’ll get to the award rationale.
AG: And, I think we’re highlighting this again, where it says, by offering the innovative, effective, and efficient management strategies and solutions. Again, that was…
JP: And, again, I understand that. And, I encourage, as we write in our solicitation, for innovative solutions. However, sometimes when you have an innovative solution, sometimes it will work, and sometimes it will not. By definition, innovative is a little more outside the box, or a little more creative than what the Government is typically accustomed to, and the onus is on the vendor to explain why that innovative solution will work within the confines of the solicitation and requirements statement. And in this case, we have identified the weaknesses as we have seen them. Outside of that specific weakness that we have gone in great length on, I understand your perspective, while you may not agree with the Government’s decision, at least you would understand our logic there. While you may not agree with the logic behind it, but that’s sometimes a back and forth, and that’s the way it goes. On the other factors, or the other weaknesses identified, the only other thing I would highlight is just an additional substantiation of what your claims are. Is the only that I would draw back on those as far as simply reiterating the proposal with minor comments on how you’re going to accomplish it can be seen as a weakness when you do not fully substantiate your claims. And that was, as we walk through some of the others ones underneath this factor, and that’s an area in the future where I would take an opportunity to improve on in your proposal.
AG: So, look at…Are we OK to move forward?
AN, SL, SJ: Yes.
AN: We’d like to understand how we were rated on the remaining items.
JP: Absolutely. Happy to get into that right now. Alright. So, for proposed staff. Again, your company received a rating of good on this factor. The strengths identified by the technical evaluation team, the Program Manager assigned to this task has a sense your experience with SharePoint implementations across Treasury components, including handling sensitive data and information. This breadth of experience should limit schedule delays and assist in identifying potential risks, ensuring appropriate mitigation strategies are in place. The proposed security team lead has experienced working on projects for the CUBE.
AG: And, these are the strengths?
JP: Yes, these are the strengths. As such, he has significant knowledge of the concerns CUBE has been facing with this Treasury wide deployment and is adept at navigating these challenges and mitigating them appropriately. So, those are the two strengths, and I believe the Program Manager is actually a place where your company clarified and turned something that was a strength in a previous offering into a strength. For your weaknesses, the resume for your technical team lead identified had the required seven years of implementation experience. The experience was in the implementation of CrossPoint, a SharePoint sister application. While there are many similarities between two, limited experience with SharePoint application could negatively impact the implementation and deployment. The tech team references the proposal excerpt from there. The other weakness that was identified, the implementation team lead and security team lead are also working in the U.S. Marine Corps. It is unclear how these resources will support two contracts full-time. Because these two individuals are GSC personnel, the commitment level that is required, the comment was did not address whether these resources would be leaving the Marine Corps project to support the CUBE project upon award. So, there was a concern and weakness identified for these people who have been on multiple contracts and not an explanation on how they would transition from one contract to the other and what their full-time support would look like. Those were the two weaknesses identified under that area.
SL: So, if we may address that?
JP: Sure.
SL: OK, the two individuals, the leads that are currently on the Marine Corps work, on page 14 of our response, the last sentence says that, “due to their experience on the USMC SharePoint project, they are ready to begin upon contract award. So, we clearly stated that they would be ready to go.
JP: From our technical evaluation team, they did not understand how the transition between the US Marine Corps and the current start up would go.
SL: So, where did you ask for a transition? You asked for them to be available, right? Key personnel, we provided you the key personnel, which means they’re available day one. We stated they were available day one, so where do we need a transition plan for that?
SJ: Especially off of another project that you guys don’t need to worry about.
SL: Yes.
JD: It wasn't stated whether the person that would leave the U.S. Embassy contract or still work that project part-time in addition to ours. So, the question mark was whether they would be dedicated to our project.
SL: Yeah, in the cost build-up, it’s in that, number one. And, number two, again they are key personnel. We have given you who our key personnel are, and when they're ready to be deployed and provided you with the appropriate amount of resources based off the pricing model. So, there shouldn’t be any, that shouldn’t be seen as a weakness at all.
AN: It seems as though you read something into our proposal, a proposal that clearly stated there would be key personnel available on the first day of performance. How can that be a weakness?
JP: The technical review of the proposal indicated a lack of understanding on whether or not they would be full-time available to this project day one.
AN: Then wouldn’t that be something that you would address in your questions with us? Since there was a lack of clarity and understanding against words that were clear.
AG: Yes, I mean, it says, “due to the experience on the USMC SharePoint project, they are ready to begin upon contract award.”
AN: And they are!
AG: I’m a little confused as to what we are missing.
JP: Hang on. In the language there that says they would be ready to begin upon contract award due to the experience can be viewed as ambiguous. Not that they will be available, the sentence is speaking to their qualifications deriving from the U.S. Marine Corps contract. How the technical evaluation team…the technical evaluation team’s comments, the weaknesses identified, are what we are sharing today.
AN: Wouldn’t, as a Contracting Officer, you see if an offeror proposes someone as key personnel, understanding the clauses associated with key personnel, and only a Contracting Officer, with that approval of a Contracting Officer could we deliver anybody but that team, full-time, under this contract, as our cost proposal demonstrated?
JP: Under your replacement personnel and under the clauses that are applicable, the Contracting Officer could not preclude you from removing a person. They approve the…
AN: I understand, but…
JP: Hang on. You asked me as a Contracting Officer. The clauses are applicable. As a Contracting Officer I cannot restrict the company on who they retain on the contract or who they do not retain on a contract, or whether they fire somebody or anything else. You can remove them today. All I have the authority to do is to ensure that what the FAR would refer to as equivalent or better based your original proposal, and I approve the replacement.
AN: Our original proposal isn’t asking for anything but the acceptance of these individuals as key personnel. The Government read into our proposal, made some assumptions when we’re citing the past experience that should be seen as relevant and should be seen as beneficial given the similarities that the Marine Corps is doing with Treasury
.
SL: Which is exactly what was stated. That was asked of us to do.
AG: Could you give us guidance on what words would be winning words here, because I don’t know how much more clear we can be on this.
SL: Right.
JP: OK, so I admit, I understand the points you are making. Let me take it back, let me review your entire proposal relative to that factor and see whether there is a discrepancy. That's all, that’s all that I can do at this point. My understanding and my belief is that the Government’s position on this weakness is justified. Let me take another read at it. Let me take another read at it with the technical evaluation team. We’ll provide you a response back on it, probably early tomorrow morning.
SL: OK.
JP: Whether or not it stands as written, or whether it changes on that, or if there are any corresponding changes, we’ll let you know. But, as of right now that weakness is as written, the interpretation of the technical team is as stated. Happy to have them take a look at something.
AN: OK, and we do appreciate that.
AG: And, I think one other question is, is it one person? Is it all people? I mean, we really need more clarification on that.
JP: No, no it is the two people that were identified. There is no more clarification required on that, nor will I provide it afterwards. The two people I just identified, that’s the only two people in question. Those are the two people listed full-time on one contract, you proposed them full-time here.
SL: Well, so, here is one of the concerns that I have. In the RFP, you stated, if proposed key personnel are not currently in the employment of the offeror, written agreement for potential employee work must be submitted with their resume. Ours are currently employed, so I’m just going to help you understand from a business perspective, yes, we have people on contract. They have to be on contract in order for us to stay in business, and that contract comes to an end just like your contract will come to an end at some point. So, when you ask that we provide key personnel, and you clearly state in here that it is expected that all key personnel will serve the duration of the task or until an equivalent replacement is nominated. CUBE will approve all key personnel and will reject any nominated personnel not meeting expectations or qualifications. We provided you the key personnel, very clearly stated our key personnel and very clearly stated what the benefits are of them, their experience, just as you asked. So, from a proposal perspective, how could we make it more clear to you that…?
JP: By clearly stating that upon award of this contract or lead up to award they would be transitioned off…
AG: But wouldn't those letters have satisfied that?
JP: What letters?
SL: And they were employees of our, so we didn’t need to provide letters. We did state upon contract award, we did state that.
AN: And, with that, we do appreciate you relooking at this issue. We look forward to your response.
AG: So, let’s move on.
AN: I think we’d just like to…Are there any issues with CrossPoint?
SJ: Could you, while you’re reopening the other issue, look at that? CrossPoint is a similar technology to SharePoint. In fact, it’s becoming the market leader. So, the fact that…
JP: The requirement is SharePoint. The stated requirement of the solicitation, the stated years of experience is with SharePoint. As we are talking about clear items, that is what the requirement of the solicitation spoke to. Not a sister application, not a similar, or the next version of requirements. What it stated is SharePoint experience.
SL: We understand that. We definitely understand that. And, while they are very much the same, that’s, I mean, it’s strictly a terminology issue. And they’re very similar, and we understand that.
JP: Right, and there are a lot of things that are very similar. Like Diet Pepsi, Diet Coke. Different sodas are very similar. Specific SharePoint experience is a specific requirement of the solicitation that was what was bid. As far as that goes, there is nothing to relook at.
SL: Right, and we understand that.
AG: So, we are getting your response back tomorrow?
JP: Absolutely. We will take a look at that. Please note that based the way the items are worded that you received a good. So you've already exceeded minimum satisfactory requirements. You received a good.
AG: Right, but we are not here to receive good. We’re here to clarify this.
JP: I understand that. But, please understand, the clarification that we will take a look at was not expected to change your evaluation score by definition. Because the weakness identified via the SharePoint in the superior qual, under the ratings does not allow for weakness. So, we will readdress this.
AG: So, how do you rate the difference between our good and CSS’s good?
JP: There is no difference in ratings between one rating of a good and another version of a good. A good is a good. By definition that is ratings.
JJ: The clarity that we are going to be providing is solely surrounding what was unclear, because our weakness states it is unclear how these resources will be able to support two contracts full-time. Of course, the letters of commitment were not required, so we’ll go back to the tech team and ask specifically what was unclear. But it will not change your rating.
AG: What is the difference then between good and superior? Again, you clarify one we’re still a good. Is that what you’re saying?
JP: I believe that's the definition of the ratings. As we just discussed. It’s SharePoint verse non SharePoint. What we’re going to clarify for you is, one, if there was any mistake made relative to it, because the Government…
AN: That’s an assumption you made…
JP: But the Government needs to make sure that our tech eval, we believe it to be proper, but we’re going to make sure that we go back and look at that. We’re also going to make sure that as we’re looking at it, to see what was…
SL: Can we have a minute, can we just have a minute?
JP: Sure.
[whispering]
AG: I think our technical expert has something to share.
SJ: OK, so I just wanted to point out that factor that factor two of the proposed task specifically says this project requires resources that are familiar with SharePoint and who have worked with the Government to prepare software for deployment across the agency. That software, any software, it’s not even specific. Familiarity with SharePoint is very different than expertise in SharePoint, and, in fact, I might submit that our CrossPoint expert is familiar with SharePoint, because you have to be familiar with SharePoint in order to be an expert with CrossPoint. So, so…
JP: But, the onus for demonstrating experience, familiarity, and the rest falls on the proposal from the vendor. Your familiarity with seven years of experience with a like application was viewed as a weakness. It was not viewed as a deficiency, it was not viewed as something that would not allow this individual to serve, it was viewed simply as a weakness because it did not demonstrate experience with SharePoint. And the contract was for SharePoint.
AN: So, could your technical representative please explain why familiarity with CrossPoint is not sufficient?
JD: OK, well…
JP: Hang on. We’re not going to get into an in-depth discussion over the different applications and comparison between a program that we do not use, or widely use as much as we use SharePoint within Treasury. We’re not going to get into a position where we are mischaracterizing what the application or the system that your employee has experience with. The onus, as we stated, for proposal submission is for you to demonstrate experience with SharePoint. The technical team rated it as a weakness because the experience that you did demonstrate was with a SharePoint-like application, but not SharePoint. And, the backbone of this application is SharePoint. I believe we've given all the clarification relative to that that we can at this point. And, I don’t, my technical representative is a technical expert in SharePoint, not necessarily sister applications. Nor are we going to get into a discussion about what would be best for the Government on it. We are purely here to discuss your proposal, our understanding of your proposal, how you were rated, why you are rated, to give you an opportunity to understand why you were unsuccessful, and an opportunity to expand on where you can improve on in the future. In the future, relative to something like this, if you are going to use a similar application for a similar item, my suggestion would be to fully articulate and fully explain why that experience is a one for one, meeting the Government or anyone else's needs, In this case based on the technical evaluation it did not meet it. Therefore, it was listed as a weakness for that one portion.
SL: So, I guess my question would be, um, it was not a requirement to have SharePoint. The requirement was not to have SharePoint. And, as the technical advisor, if they do not have familiarity between the two, why would you not ask a clarification question? Or, why, I mean, how did they have the capability to be making the determination about this if it is not a requirement?
JP: They have the technical expertise on what we were requesting, on the system and implementation that we were looking to contract for. What I was saying is that we are not going to get into a discussion of whether SharePoint or, I apologize… --
JD: CrossPoint.
JP: CrossPoint are better applications, how similar they are, what the coding that goes into it, how they interact with security functions within the environment...
SL: We’re not looking to get into that. We’re trying to understand the evaluation…
JJ: Well, if you go back to the factor it will specifically say that in determining the rating for this factor, “the Government will give greater consideration to the staff with the most experience and expertise with SharePoint and deployments of such.”
SL: That’s giving greater, yeah, that’s a strength.
SJ: That’s a strength, not a weakness. That would have given us points, not…
JJ: Well, we are not technical experts in CrossPoint. I don’t know if you know this, but the Government is a couple of years behind the rest of the industry. So, we hire contractors like yourself to help us to better understand what's available in the marketplace in the future, but right now we’re stuck in the dark ages with SharePoint, I guess. I've never even heard of CrossPoint until this point.
SL: We’re just saying that if it was, I mean, you're seeing it as a negative on us, right?
JP: We are seeing the experience as a weakness.
SL: It should have been stated as a requirement.
AK: The requirement as stated was that you had to have familiarity with SharePoint. The onus was on you to explain what the familiarity with SharePoint was, not to necessarily talk about CrossPoint. And even though it may be a sister application, you should have explained what that application was, why is was so similar to SharePoint, but we really have a SharePoint requirement.
AG: I believe that we did do that and, again, we also answered the question getting back to you were looking for innovative solutions. To your point, that you think the Government may be a couple years behind, and we respect that, and we’re here to serve your mission.
JP: And, so, we’re still working off Word 2007, for the record. [laughter]
AG: We want to make sure that we do offer you the most cost-effective, innovative solution. So when you put those words in an RFP, I feel like your disingenuous now saying, no let's go back to this, but we are trying to help you move forward.
JP: And, I understand that completely. And I think we also need to all around. I don't believe that any vendor ever submits a proposal for anything I contract for that does not believe they are offering the Government the best solution, and the most innovative solution to reach that product. Otherwise they shouldn’t be spending their bid and proposal costs to go after it as a capture for their business plan. The Government's job, or the buying activities job, is to evaluate those innovative plans, some of which we will agree with, and some of which we will not. And some of those solutions we will view as positive, and some of those solutions we will view as a weakness or a problem or deficiency. In this case, the technical team did not believe that we had sufficient information to understand the relevance to SharePoint, as stated in the requirements. We viewed it as a weakness. We did not view it as a deficiency. We did not rate your offer as unsatisfactory. But we did identify it as a weakness.
AN: What we’d like to ask though, is because the requirement was not experience with SharePoint but rather familiarity with SharePoint…
SJ: It’s fairly subjective evaluation.
SL: Yes, very subjective.
AN: And, we've all acknowledge that CrossPoint is essentially the sister and does...
JP: I'm characterizing it based on your application. I'm not saying it is a sister application. I’m using the terminology from your proposal as my lack of familiarity. To properly classify it I would need to go and take a look at it. But continue.
AN: Being that our personnel, the technical lead, it seems as though the fact that the words CrossPoint were in her qualifications as opposed to SharePoint were what we were rated as a weakness. Would you see that CrossPoint, by definition, you have to have familiarity with SharePoint?
JP: But by definition, your company needs to...
AN: And by having CrossPoint, it is familiarity with SharePoint, therefore, at least it’s not a weakness.
AG: Right.
JP: No, no, hang on.
AN: Would you take that back to your technical team, and ask them…
JP: No, no, I will not, and here’s why. The onus on this portion is to demonstrate why CrossPoint is similar experience to SharePoint as part of your technical proposal. That was not there.
AN: Your technical team should understand that.
SL: Yeah, if your technical team is doing the evaluation, and you have not clearly stated that SharePoint is a requirement, they should have the capability to understand what would be familiarity and what's not familiarity?
JP: We’re going to need to table it for now. I understand your position. No, it is not something I’m going to relook at as we move forward. We’re just going to handle past performance. I appreciate your concerns, and I appreciate your statement. Hopefully, you understand exactly where we are coming from as far as what the evaluation was and how it was done. But, if you could talk to past performance at a high level?
JJ: OK, great, thank you, Patrick. In the interest of everyone’s time, I’m going to go through this fairly quickly. In the area of past performance you received a satisfactory. Two of the references indicated that your company was doing a great job supporting their software implementation and stated that key personnel were responsive and well-qualified and that the work performed was of the highest quality. However, weaknesses were also noted in the area of past performance. And, I’ll draw your attention back to the evaluation ratings. You’ll notice there is no option for good. Neutral is in a situation where you do not have any past performance. Superior is just strengths. And, satisfactory is that you are allowed to have some weaknesses. So, in this instance, we did see that there were some weaknesses. And, one of your references that was returned back to us indicated that your monthly reports wwere submitted late, understanding that the progress of the project and the timely indentification of potential risks and issues is paramount to the successful implementation and deployment of the SharePoint across the CUBE organization. So, that is why that was noted as a weakness. Late reports on one of your past performance references. That is a clear weakness. I think that we can all agree that if something is late that it is a weakness. The next weakness that was noted was…
AK: Bridget, we need to stop you. Patrick, can you end the session?
JP: Sure. I apologize for having to step in for one second. So under the past performance aspect, the one profile that did have weaknesses identified with it, that the Contract Specialist has gone through, identifies those areas that caused your proposal to be rated as satisfactory. Your cost was evaluated at 4% lower than the winning offeror. So, as you look at the proposal and the actual evaluation, the trade-off analysis was done for a technically superior proposal on a factor of one, two, and three. When combined, you had similar ratings, or exact ratings, for factors two and three. The difference in the trade-off became for the top technical rated factor in descending order of importance for factor one, the difference between a superior and a good, in this case the Government and myself, as the Contracting Officer, determined that that 4% cost savings did not justify trading off for the lower-priced offeror. Our solicitation clearly stated that technical is significantly more important than cost and price. And for the 4% trade-off in price, we went with the technically superior contractor in accordance with our stated solicitation. Now, I do appreciate everyone coming in. I appreciate your comments, the passionate discussion we’ve had. Unfortunately, we’re butting a little bit up on time, and we have other debriefs to conduct today. So, I appreciate it, I look forward to proceeding from all of you in the future. I appreciate your proposal submission to the Department of Treasury. We encourage you to submit proposals in the future. We look forward to doing business with your company in the future and thank you for being a valued partner to the department.
AK: And we will send you the clarification information, and we will also give you the script that we used today so that you can look over the rest of the information that we were going to give you.
AG: Oh, good, thank you.
JP: Thanks so much, I do appreciate it.
AG, AN, SL, SJ: Thank you.
JoseA: Well, guys, thank you very much. That was a wonderful session, I could tell. We could hear you through the doorway. So, thank you very much for taking the time to prepare and do this. Just a couple of quick questions for you. This is meant to train the contracting professionals holistically and give them exposure to industry and industry’s perspective. I just wanted to feel, I wanted to hear your thoughts, do you think that it achieved that, and what would you say to the contracting folks that are watching this that you would want them to know as a small industry partner, that you would want them to know about industry when they are giving a debriefing or doing business with private-sector clientele?
AG: I think I can start. Again, I thank you for this. Not only are they learning, but we are learning as well too. And so, I just want to make sure that everyone is aware of that. I think for future, I would, the oral dialogue, having the oral debriefings is key. And I know that sometimes hearing some of the Contracting Officers and Specialists that they’re a little intimidated by that. I think that we encourage it, because we only want to make our proposals better and to be able to offer that. And I think the communication is key and I welcome it.
SL: Yeah, and I think from a market research perspective, I mean, that came up during the session. You know, as industry, when we respond to market research, we go through the process of not just the proposal side, but also all the business development associated with it, the meetings, etc.. There’s a significant cost associated with that on our side. But we’re doing it because we are really trying to understand the needs, and get to your mission, and really fulfill your requirements. And so, when we go through that process, we’re doing that and then responding to the proposal. And, we are putting the information in that response that we believe has been articulated from the Government. And, this was a perfect example of how we went, we explained it, we came to the same understanding of how fantastic the solution was, and then all of a sudden it was in left field. I really came out of this feeling like, well that was the biggest waste of money. And, very honestly, I felt as though it was one of those times that the market research was done and Government already had somebody in mind and they were just looking for to the three bids to come in. So…
SJ: I appreciated in the oral feedback, the opportunities for improvement. I think that is where we benefit the most. I think the other thing that we would always want the Government to be focused on is specificity of requirements. I think that’s clearly, in this particular case study where some of the pain points were. And, I know that when I respond to an RFP, that is often a pain point, is the requirements are not specific enough. So, it’s hard to, it’s hard to actually address the requirement in an adequate way if we don’t, if we don’t understand it at the specificity that you want to tell it to us.
AN: And, also with that, if there is a lack of, if there is an inherent…when you get the proposals and you find there is a contradiction, use the discussions as a mechanism to get that clarity. It is so frustrating to learn that a decision was made or an assumption was made about a proposal which may not be accurate. And it is important to use the tools that you have available to you to do discussions and get that clarity.
TO: During the mock debriefing, I kind of noticed a number of points where there was tension or dialogue between both parties. But then the Contracting Officer just kind of pointedly said it’s ok to disagree. As a vendor, from your perspective, do you find that as a moment of opportunity to lower the tension and reinforce that this is supposed to be a relationship here, that we’re supposed to have an open dialogue and not to be in a position of, you know, what do you think?
SL: I felt like he was placating us. I mean, really, it did not allow us to really have an open dialogue and I think that really is the most important thing. I mean, again, we are trying to work with Government. When we’re responding, we are not just trying to make money. Right? I mean, especially because we are small businesses, we are here because we want to help. And so, when we go through the whole market research portion, we’re trying to understand…when we go through the proposal response…we’re trying to understand, we’re trying to provide a good solution. And then to come out on this end, where he says, it’s OK to not agree, it’s kind of like, OK well just go on your way, that did not open up that dialogue and we do need to have that open communication so when we come down to points of the ambiguity in things, that we use kind of common sense and open that discussion and say we may not know as much as you do, can you explain this to us?
JoseA: And, guys, I think the goal of this session was to actually highlight some of those tensions and why it's important to communicate. It sounds like the example was proven for the audience. And so, I really appreciate you taking the time to do this. I know that there was a lot of misinformation and that this was the hardest of the two cases in terms of there was a lot of meat in there, and I really appreciate you taking the time to work together to pull this off. Thank you very much.
AG, AN, SL, SJ: It was great, thank you!
JohnA: So guys, thank you very much. This took a lot of time to prepare for. You guys have done a wonderful job. We could hear the activity through the doorway. So, I wanted to ask you, this was set up to give the contracting professional a holistic experience in a very safe environment, because it is a made-up acquisition. Now, what are your thoughts? What are some key takeaways you had for industry or for younger government personnel who are watching this? And, did you learn anything from going through this process?
AK: I think that everybody should go through this, because people do not do debriefs often enough. Especially because we don't really do a lot of routine buys anymore. So, I really think that everybody should go through this. One of my takeaways was that I definitely want our CLE's to do something like this, which is our career ladder employees, our interns. The other takeaway I had was that we talked yesterday about, well, we think they are going to question us on this, this, and this, and we were exactly right. So, we were sort of ready for some of the arguments they were going to give us. So, it's just having the knowledge and being aware.
JJ: I had a great time, honestly. It was so refreshing to be able to have an honest conversation without the fear of reprisal, such as a protest. I think, if we had a situation like this in real life, we could actually help the contractors to come up with better proposals and to better understand what the Government needs and to help the Government push forward into requirements that will be innovative eventually. But, right now we cannot rewrite the FAR, so, instead, the best thing to do is practice and anticipate those arguments that are going to come up. We have notes on every single thing that they said before they said it. So, I really enjoyed it, and I think that we were well prepared and Patrick did a great job addressing the concerns.
JP: Thank you! From my perspective it was up lot of fun. I enjoyed doing it. A little less realistic than it would be in normal life. We probably engaged in a little more of the topics in greater in-depth than we would have. We felt that making some mistakes here and there would actually be good for the learning process and we put it out there. A couple of things that we noted before we showed up were, you know, in normal debriefs the vendors bring their attorneys, we bring ours. In normal debriefs, we don't, if I'm running a debrief I’m going to be the only one talking. If I need to get information from other panel members or side barring to make sure that we are not talking about lifecycle costs. [laughter] The thing that I would take away from this is, and it’s important, is that we do debrief training, or at least we used to do with a lot of our folks. When you’re going through a tech eval and you’re the CO reviewing it, you need to be thinking like this and asking the questions, because the way that we explain it to our folks is, you read the tech eval, this is what you have to go on to help the contractor understand about why they did not get it. Think as they would. Go through it and challenge it, ask questions so that you fully understand it. A little frustrating that we couldn’t have that as part of our prep for this. There was stuff that we were looking at thinking, oh man we’re going to get killed on this. But, I mean, it is a good opportunity to give feedback. I probably give more feedback than most COs. And, just I feel that it is important to give industry a good understanding of what's going on while still protecting the Government’s rights and not just doing a debrief to check a box. Because I could just sit here and tell you, here are your factors and here are your weaknesses. No comments and walk out the door. And that’s not the real reason for a debrief.
JD: OK, and I would add that, as a technical evaluator, the inclination is to want to talk shop. It's good and it’s important to share technical information during the debriefing, but it also has to be balanced by a caution for what you say in a debriefing type context.
JP: Yeah, your technical folks should be saying all their talking in the technical documents that they handed to you and they should be whispering in your ear to clarify it.
JoseA: Thomas, did you have anything for the folks from IRS?
TO: Yes, as part of the mock debriefing, I noticed that you had a comment about it’s ok to disagree. Now, how did you, did you think that was an important moment, or do you think that was not necessary?
JP: When you get into a real debrief, you’re going to see emotions get raised and people argue. You’re not going to agree. When you’re sitting there telling someone they lost a $20 million project, they do not agree that they were not the best person. They think you made a mistake. At a certain point you need to stop that, agree to disagree, and move on, because they’re not going to redo the tech eval, and I am not going to be convinced that they are the right vendor. And at some point you need to move to something that is productive. It is kind of like getting into an argument with friends or family. At a certain point, you’re not going to actually win that fight. Most times I lose them. [laughter] And you need to move on, agree to disagree, in a polite way, or however you want to phrase it. But, it is important to bring closure to things.
JoseA: So, the reason we did this, guys, and I’ve said this throughout the recording, is that 30% of the acquisition workforce, eligible to retire within five years, 30% of the acquisition workforce has less than five years of experience on both the industry and government side. We think this is a mechanism to train people to be smarter, faster without the risk of making a mistake in a real-life environment. And I very much appreciate you doing this. You guys were kind of up against the wall in this one. This was by far the harder case and this was, I would say, a little bit slanted against the Government and I appreciate you taking the time to prepare and do this.
AK, JJ, JP, JD: Thank you so much, we appreciate it!
Sherean Miller (SM): That was an interesting session. My name is Sherean Miller and I am Senior Vice President at Federal Management Partners. We’re a Government contractor.
Tony Grayson (TG): And I’m Tony Grayson. I'm the Program Manager for the FAC-C program for the Federal Acquisition Institute. And we are here to talk about our observations on the debriefing that we just saw.
SM: Right, exactly, and it was quite a lively debriefing! I was sort of sitting there and I could feel the emotion in the room, particularly on the side of the industry and then also on the side of the Contracting Officer.
TG: Yeah, I think it was very lively because the Contracting Officer, when het set up the rules of engagement at the start, said that the contractor could ask questions as he went along and as a result of that, it became very lively. Because the Government, in fact, took a risk that the Contractor was going to bring things up and they weren’t going to have enough time. And so, as we saw at the end, they kind of ran out of time. So, if you open yourself up to that, you have to really realize that you may be giving part of the time to the Contractor then, and you may not get everything out that you want to get out.
SM: Right. And, on the part of the industry, they were rather sort of argumentative, and had a lot to say and had a lot to disagree with. And, I think the purpose of the debrief is really for them to learn about what they could do differently. And, at points in that conversation, I felt they were trying to challenge and go up against the Government.
TG: Exactly. They were trying to say that there proposal, whoever evaluated it, that the evaluation should be changed. Which brings up the point that one thing I saw that I don’t think I have ever seen before was that the Contracting Officer agreed to go back and reevaluate the proposal. I don’t think that would typically be what would happen, and if the Contracting Officer during the debrief thought that something came up and he had a question, then I think he would go back and ask his legal people or his management on how he should proceed. I wouldn’t tell the contractor in the middle of the debrief that, hey, I’m going to go back and reevaluate your proposal.
SM: For sure, that definitely surprised me. And I think that they were trying to throw a few curve balls into the mock session, because I don’t think that would typically happen.
TG: Yeah, typically they’re trying to get information, solicit information, and not trying to argue right there with your evaluation. If they want to argue the evaluation, they’ll do that through the protest process, I would think. The other thing, even after the Contracting Officer said that they were going to go back and re-review the evaluation, he then said that, hey, it didn’t make any difference because the evaluation criteria would have been exactly the same. They weren’t going to change. Which was probably a little surprising. The contractor was probably a little upset when they heard that.
SM: Right, for sure, and I think they probably felt like they were being placated a little bit in that, in that, by the Contracting Officer.
TG: And in my experience, it’s not just the agile over against themselves that’s important, they’re kind of a general indicator of how the contractors stack up when compared against each other. But, it’s is the strengths, weaknesses, and the deficiencies that are part of that, coming up with where they are that are actually more important. And, in fact, the source selection authority when they’re doing the trade-off, when they were doing the trade-off and assistance is going to look at, is going to look at the strengths and weaknesses and kind of roll those against each other, as well as cost and price.
SM: Right. I was sort of surprised when the industry was bringing up basically all the stuff that happened prior to the solicitation coming and basically saying, well in the market research phase we felt that we were given a hint. And, you know, that’s not really applicable in this debrief situation. It’s very much focused on what the solicitation is saying, and, you know, I think the Contracting Officer did a great job at shutting that down, and reminding folks that the pre-market research is one thing and when the solicitation is active, it’s a very different sort of conversation.
TG: And the Government quite accurately described that, hey, they are only going to evaluate what’s in the proposal, not what was said beforehand, but what was actually on the pages of the proposal itself.
SM: Right, exactly. And, it was interesting too, they came back a lot with stuff like, why didn’t you clarify this in the Q&A? You could tell they were very frustrated by that, and I can understand where they were coming from in that they were saying you interpreted it differently than what we meant. And I think the CO reminded them that, well, if you think there is any opportunity for us to misinterpret what you’re saying, then make sure that you are very clear and saying it in a very articulate way.
TG: And the contractors also stated, hey why didn’t you bring this up during discussions? If we had known that then we could have given you feedback on that or changed our proposal to reflect that. He didn’t do that and so I think the Government said hey, we had meaningful discussions. We came out and said we had meaningful discussions. So there is no basis for a protest in that instance. Of course, the contractor may feel differently.
SM: Yeah for sure, I think they definitely felt differently in that session. Let's see. I mean, there was a lot of conversation around CrossPoint and SharePoint, and, again, knowing from the industry perspective if they are not clearly articulating that in the proposal, that they are asking for SharePoint or like experience, and there arguing that CrossPoint, but I think the point that he made was show us that CrossPoint is similar to SharePoint. Make that connection so that we understand it, because we don't know what it is.
TG: And I think that’s a good point, because the Government is not going to assume anything. They’re going to literally read what you have in your proposal and they’re not going to assume that you need anything or want anything or are going to have to find anything. You have to tell them exactly what you want them to think. And, clearly they did not do that.
SM: Yes, and I will say from a…
TG: And clearly they didn’t do that.
SM: Yes, right, for sure. And you know, from a…
TG: And, I think he gave them good advice on how to do that in the future.
SM: Yes.
TG: He said, hey, you can use another software program that you think is more appropriate than, and I asked for a different one, then you got to show me why it is more appropriate.
SM: And I think, you know, there was a lot of emotion, particularly on the industry side. And I think it was then good when they said, you know, understand where we are coming from. We just want to be able to learn from this, kind of refocusing the conversation and making sure the CO understands that's where you're coming from, I think is an important piece, not only to dial down the tensions, but then to actually have some good takeaways.
TG: And he did a good job with managing his team.
SM: Yeah, for sure, so thanks for letting us be a part of this. This was great!
TG: Thank you.
Georgiana Carpouzi (GC): Good morning everyone. Welcome to the debriefing for the Progressive Structural Inspection (PSI) Processes contract. My name is Ms. Georgiana Carpouzi, I am the Contracting Officer for the Government on this procurement, and I’m going to introduce my team. To the left of me is Mr. Samuel Body, he was the subject matter expert in the technical evaluation process. To the immediate right of me is the Source Selection Authority, Ms. Nancy Reagan, and to the far end is Mr. Noah Itall. He is the Contract Specialist and while, BelAir Team, you introduce yourselves, Mr. Noah Itall will pass the sign-in sheet around to everyone please.
Noah Itall (NI): And, if you could, while you guys are signing in, just introduce yourselves as well.
Claire Stone (CS): So I’m Claire Stone, CEO of BelAir.
Christopher Pilot (CP): Christopher Pilot. I’m the Contracts Manager.
Stephanie Cloud (SC): I’m Stephanie Cloud, I’m the Program Manager for the BelAir Team.
Steven Wing (SW): I’m Steven Wing, I’m the Senior System Engineer for BelAir.
GC: Thank you. Welcome again. Before we begin this debriefing, I am going to outline the rules for the debriefing. And, we ask that you allow the Government to complete our presentation and hold all questions until the end to allow us to have a more smooth debriefing for BelAir Company.
In terms of rules of engagement, I just want to stress and reiterate that there will not be a point-by-point comparison of BelAir’s proposal to other offerors that have proposed on this procurement. The Government will not discuss or engage in any information that is either protected or is tempted by the Freedom of Information Act. There will not be any discussion of confidential or financial disclosure of information from other offerors. And, there will not be a trade secrets that will be revealed of other offerors. Are there any questions on the rules of engagement only? OK.
So now, the Government will move forward to how the evaluation factors were applied. As listed and stated in the RFP, there were technical factors in price and as far as the technical factors, there were three, there was technical capability, past performance, and management approach. They were all listed in descending order of importance and when combined were significantly more important than price.
Now I’m going to turn it over to Mr. Body who is going to walk BelAir Company through your technical proposal and how you were rated in the evaluation process.
Samuel Body (SB): OK, so for your technical capability, we assessed you a number of strengths. We said that your technical approach demonstrated a strong depth of breadth and experience with HC130-H and HC130-J, Aircraft PSIs. Your summary of PSIs completed demonstrated the ability to repair and repaint the HC130-H and HC130-J and other similar aircraft types within reasonable time and budget. You submitted GFR compliance verifications with no deficiencies. Your facility, or your proposed facility rather, was GFR compliant and meets the regulations specified in the statements of work. You will be able to conduct all repairs indoors, and in an environmentally controlled facility. The atmosphere, conditions, and aircraft skin temperatures did meet the TO 1-1-8 requirements. The quality control plan was detailed and identified quality control activities from technical implantation through deployment. And, you currently implement effective hazardous material management and pollution prevention program in accordance with the National Aerospace standard, or, in accordance with the National Aerospace Standard 41. We did not assess you any weaknesses in that regard, and we assigned you a technical capability rating of superior on that, which was the highest possible rating.
And then, for your management approach, we assessed you, we assessed you a number of strengths as well. We said that you identified a solid project management organization that you addressed the technical and organizational management needs. You also, your work plans, processes, and procedures for managing the tasks and efforts followed industry and government standards, including the International Standards Organization and American National Standards Institute. In particular, it incorporates the risk management approach, as outlined in our risk fundamentals guide. Your integrated master schedule clearly identified and explained how you would meet the 220-delivery schedule, with the potential of a high number of non-recurrent repairs. Your employee retention rate for the last 5 years was higher than average and your program manager had more than 18 years of aircraft maintenance and logistics/parts management experience. She had documented experience with HC-130 aircraft and other similar aircraft types. You also had extensive knowledge of our, and the Department of Defense, aircraft maintenance practices and procedures. The one weakness that we assessed for you was that while your Quality Assurance Manager had either years of experience with wing aircraft heavy maintenance and processes, excuse me, possesses repair experience on aircrafts of similar category and class of aircraft as the HC-130, the experience was not within the last 10 years. But, even though you did have that one weakness, we still assessed you a rating of superior, which was the highest rating overall; the highest possible rating you could get, that is.
GC: Now I’m going to move on, thank you, Mr. Body. Now I’m going to move on to BelAir’s overall evaluated price and technical rating as compared to the successful awardee. So, as Mr. Body just stated, on technical capability, BelAir was rated superior. Open Skies, the awardee, was rated good. On management approach, BelAir was rated superior. Open Skies, the awardee was rated good. And the proposed price by BelAir was $196.58 million, and the award was made in the amount of $182.672 million dollars. Now we’re going to move on to Mr. Body, who is going to discuss, in detail, BelAir’s past performance evaluation and how that was rated.
SB: So, for your past performance information, the information you submitted is relevant, less than 5 years old, and similar in scope and complexity to this solicitation. The references that we contacted stated that you were relatively confident in BelAir’s ability to fulfill performance quality requirements most of the time. One of your past performance evaluations, a past performance evaluation survey reported several minor incidents when the offeror was not able to meet the schedule. And, one reference identified deficient quality assurance procedures and insufficient time allowed to deliver monthly reports on time.
SW: Can I interject?
SB: We’re going to keep on going through it and then…
GC: We asked you to hold all questions to the end.
SW: Oh, ok.
SB: Thank you. And then, the last, and one of the last things we noted was that one of your past performance references was unreachable. And then, we assessed you, and because of that, and because there were some, two, or one of your references raised concerns about your ability to perform to conduct those, to perform those performance all of the time, you had minor performance deficiencies, we assessed you a satisfactory rating on the past performance. And that was in accordance with the evaluation criteria, which said, that, which said, that to get a rating of superior, we said, based on the offeror’s past performance record essentially some doubt exists that the offeror will successfully perform the required effort. We didn’t assess you the lowest rating, which said that you had extreme doubt, so we assessed you the middle rating for that one.
GC: And now we’re going to move on to the results of the Government’s trade-off analysis, as stated in the solicitation, the Government was going to conduct a best-value procurement where we will use trade-off analysis in determining the basis for award, which will ultimately result in the best value for the Government. And, now I’m going to turn it over to the Source Selection Authority, Ms. Reagan, who will provide more feedback, as to the basis of the selection to the awardee.
Nancy Reagan (NR): So, based on the integrated assessment and comparisons of the strengths, weaknesses, and deficiencies of the proposals, we, I as the Source Selection Official, found that the Open Skies Corporation proposal to be the best value for the Government. When looking at your proposal, yes, you had completed the PSI and the deeper level painting on both aircrafts. In the past, we note no weaknesses or deficiencies were noted on your technical capability, however looking at your past performing history, there were very moderate risks that presented on that past performance questionnaire. When looking at your, the fact that you had some issues on meeting the schedules, the fact that you have some issue on the quality assurance procedures, and which integrate a more management, a more personnel from the Federal Government to help you assess those quality assurance reports, that in itself presents a risk to the Government, because we need to make sure we meet the schedule, we need to make sure that we have those reports on time. While your quality assurance management manager also presented another weakness, whereas, yes, she has the experience and she has the eight years of experience with wing aircraft and heavy equipment, we really need to have somebody who has that experience within the last ten years to make sure that that person has the capability and the experience to provide the feedback that we need on time.
With regards to Open Skies, as said before by the Contracting Officer, Open Skies Corporation received a good rating under the technical capability, but we found that their proposed price is fair and reasonable based on the fact that part of the RFP, we require that somebody with not only the experience but that was beyond the requirement, and they are also beyond the requirement. Yes, you are superior, but they are still representing the best value to the Government.
GC: So the last thing that we will discuss is the role that price played in the award of the PSI contract. As the solicitation stated, price reasonableness would determine, will be the basis for determining whether the proposed price was fair and reasonable, and in this case there was adequate competition and price reasonableness was based on the fact that we had more than one offeror with competitive pricing. We also, as previously stated, used trade-off analysis, and how that was used in this process, while BelAir Company was rated highest technically, overall, when you look at trade-off analysis and how it is used, trade-off analysis allows the Government to provide the best value by making the decision to award to other than either the lowest priced proposal or other than the technically highest rated proposal. So, in this case, while BelAir was the highest rated technically, the Government could not justify paying an additional $13.9 million dollars to award to BelAir Company when there was another proposed solution that exceeded the minimum requirements of the solicitation, as did BelAir’s proposed solution that was 7.61% lower, and ultimately more advantageous and was in the best interest of the Government in terms of best value. We just could not justify paying an additional $13.9 million dollars to award to BelAir Company. And that was the basis for how trade-off analysis was used and applied in this procurement and also why the award went to Open Skies as opposed to BelAir Company.
BelAir Company, as we debriefed on the technical capability and management approach ranked very well. There were some strengths that were noted. There were no deficiencies noted in the technical capability approach. There were some weaknesses noted in the management approach. We had a business relationship with BelAir Company in the past, and we look forward to working together with BelAir Company in the future, and hope that you continue to submit proposals and responses for future solicitations to the U.S. Department of Treasury.
So, at this time, now, if my team doesn’t have anything else to add, we will open it up for questions for BelAir.
CS: Thank you so much for giving us the opportunity to respond to the proposal for Engineering and Technical Services and also for giving us time for coming over here and giving us the debrief face-to-face. We are really, really surprised and shocked with this evaluation. We believe that we provided the best value. We are technically the strongest. I believe that over the period of the contract we would have given you a better performance and a much better results than the one that you have chosen. We understand that best value is a pretty subjective term. We would like to hear more on how do you quantify best value? I understand Open Skies got a good technical rating and they also got good on the management approach. I don’t, I didn’t hear, what they did on the past performance side. Whether they were superior or what, or what they got there. But, how do you compare good, good, superior, with superior, superior, good on our evaluation? How do you quantify best value? We have done business with Treasury and we would like to support you in your mission, but we definitely would like to hear more on what we could have done better, you know, to sell you better? As a company, we have spent $875,000 on pursuing this opportunity on the last two years, and we are technically very strong. We have done this in the past, but we believe it would have been a very less risk to you. So, we would like to hear how we could have done better and sell you better.
CP: Um, Clair, as a Contracts Manager, one of the things that we looked at is to minimize risk to Treasury, and one of the ways we did that, given that this was a Firm Fixed Price offering, if you could see specifically on our Volume 2, we offered one assumption and only one assumption, which was that your scope was not going to change. The type of work that we typically do is from our competitors, because we have extensive competitor analysis. Where they get you is with assumptions, so we listed only one where we know our competition very well and where they make up money on Firm Fixed Price is by having a lot of assumptions, and, as the saying goes, “each assumption is a change order in the making”. So, as long as you hold to your scope, we believe, as Claire just mentioned, that over the lifetime of the program, even though our involvement is only three years, this is a program that is going to last fifteen years, we showed that ultimately you were going to be saving more money by going with us. Steven can also give you some indication from our past performance. I think something might have gone wrong in your analysis of our past performance. Steven, if you could add?
SW: Sure.
GC: If I can just interject for just one second. I heard quite a few questions, and I don’t want those questions to go unanswered. So, if you would allow the Government to address those. I think I heard at least three questions initially, and if I’ve missed any, please, you know, by all means, feel free to restate them.
I know one question I heard was what was the past performance rating of the successful awardee? And, unfortunately, per the Federal regulations we cannot discuss, the Federal Acquisition Regulation, we cannot discuss the past performance rating of any of the offerors other than the one we are debriefing. With regards to assumptions, and scope, and how does that compare with other proposals we have received, that we may have received and evaluated in the process, we can only discuss the overall rating of the successful awardee, which we shared was good and good on technical capability and management approach. We cannot get into the point-by-point comparisons.
In terms of BelAir’s mainly one assumption, it’s up to you, it’s a business decision of each offeror as to their approach to how they want to write their proposal and respond to a solicitation that has been issued by the Government. The Government is of the position, that we’ve done excessive market research, we stand firm on our market research and assessment of the market, and feel very firmly that our requirement is strongly defined, which is why we used the Firm Fixed Price type, contract type in this procurement. So, I’m a little unclear as to BelAir’s concern with what I hear is potentially scope creep, because we’re very confident in the department that’s written the…yes?
CP: I think the assumptions are highly relevant, because your explanation was that our price could not be justified. What we showed in the proposal was that over the life of your program, we showed cost savings that were well in excess of $35 million dollars to the Government that perhaps our, we can explain more, our analysis. So Firm Fixed Price, we know is only for three years, but we laid out how our approach was going to save Treasury over fifteen years and we don’t feel, I didn’t hear, that you even considered the end-to-end lifecycle as a factor. You only considered the cost for the immediate involvement of ours, where technically, and our management approach showed, that we know that a decision that you make today is going to be one that you’re going to live with for many years to come. And perhaps, well…
SW: Chris, can I elaborate on that point? So, as you have evaluated our technical approach as being superior, we’d like to think that the rationale for that is supported by the work that we’ve innovated on the commercial side of our business, and we brought those practices to this particular solution that we provided to the Government on this particular opportunity. We feel like our pricing is a very, a very appropriate price, given what we know to be the effort associated with this type of work. We have some questions as to the realism of the price that was proposed by the offeror, and I think we have some credibility in making that assertion, just based on the fact that we’ve actually, we hold patents on several of the technical approaches that we brought to the Government on this, so we know this better than anyone else in the market. So there’s some question in our mind as to how reasonable or how realistic the price was that was provided by the offeror. And, I think our track record actually bears that out. But, with regards to our past performance though…
SC: Excuse me…
SW: Yes, I’m sorry.
SC: I’d like to hear what the Government has, what analysis they have performed on cost realism.
SW: Sure.
GC: From the solicitation, the solicitation stated and reserved the right that the Government may utilize price realism and determine if price is fair, and determining whether price is fair and reasonable, however price realism was not required in and was not used in this case. And, so, it reserved the right, and stated that the Government may utilize, but it did not say that it would absolutely be used, and it was not used.
CP: So, it was not a decision factor?
GC: It was not.
NR: And, one thing that you have to keep in mind is price realism is only measuring whether a price is too low, and that is not what we have here when it comes to you. We were not measuring whether your price for now was too low, we were simply looking at the factors as they are.
CS: You mentioned that price is much less important than the technical, and we were rated superior on the technical and we were rated superior on the management approach, while the winner had good and good. How did you compare that? I mean, having few pointers that you mentioned in the PPQ’s, did you utilize any other sources of past performances to maybe mitigate that or maybe find out if we had done a good job on that kind of work? So, how did you, and you mentioned that in the statement of work that you would be utilizing other sources so that if one or two items were a little negative, did you perform additional research because technically we were very, very strong.
GC: So, I’m unclear as to your actual question…
SW: So the question is, something doesn’t add up.
GC: With regards to past performance, with regards to what specifically…we want to be clear.
SW: Past performance. So, how our past performance actually relates to our technical evaluation? In fact, we actually hand-picked the past performance references that we brought forward and we were pretty confident that we got superior assessments in terms of what was communicated back to the Government. So, there seems to be some discrepancy, and…
SC: Well, and these were my concerns. I’d just, I’d like to know who you talked to.
GC: Well, firstly, the Government cannot disclose the names of the individuals that provide a past performance on an offeror. However, we can give you the feedback in a more, and I understand, I really do understand BelAir’s concern. With any offeror, there’s a lot of time, effort, and energy that’s spent into responding to proposals, solicitations, preparing proposals. Our customers, I mean we consider the offerors to be our business partners when the contracts are awarded. We understand that our business partners have a vested interest and want to see how they can continue to improve and do better, so I really do understand the concern. However, as I stated, we cannot disclose the names of the individuals on the past performance.
Specifically, I did hear a statement made about the solicitation stated that the Government would reach out to other points of contacts. The solicitation actually stated that the Government would, if needed, reach out to other, any other point of contacts, to gain additional information on past performance. And, while we, Mr. Body mentioned that we were unable to reach a third, your third person that was provided in your questionnaire, we did try several times. And, while the past performance rating, I heard someone say, it wasn’t superior, the rating, if you refer back to the solicitation for past performance was slightly different than for technical, technical capability and management approach. The ratings for past performance were actually neutral, which that rating is satisfied when an offeror does not have any relevant past performance. We cannot, the Government cannot rate them negatively, we can only assign a neutral rating. The other rating was unsatisfactory, and the description for unsatisfactory Mr. Body provided earlier. There is…
SB: Superior.
GC: Superior and then there is good. The reason in this case that the Government assigned a satisfactory rating, while the adjective may sound like it is a negative rating, it is not, the Government still felt that BelAir could perform based on past performance. But, there were some concerns that were noted with your points of contact that…
SB: Sorry, and I actually want to interject. There was actually, was only superior, satisfactory, and unsatisfactory for past performance. There was no good rating, there was no middle ground for us to go on this. There was no middle between satisfactory and superior.
CS: How do you do it? Do you average it out? How do you say it’s satisfactory overall? I mean, how do you quantify those? Do you have a points system?
GC: It’s not quantifiable.
SB: We don’t. No, we don’t quantify. It was a judgment made based on the information we had available to us, which were those, which were the two past performance questionnaires which listed some information that indicated that you had had, that your company had had minor performance problems and that was the reason that we assigned the rating that we did.
CS: But if we’re technically so strong, and our management is strong, is that something that you could have weighed in to provide a, to make that best value decision?
GC: When you say “weigh in”, what do you…the Government can’t make any assumptions, all we can do is assess based on the information that’s provided to us.
CP: You made a…you made a selection to Open Skies…
GC: Correct.
CP: It seems to us right now, strictly based on past performance, because we have better technical capability, better management approach. You just stated that price was not an important factor. I mean…
GC: It wasn’t when combined. So the technical factors when combined with price were significantly more important. However, when applying the trade-off process, if you refer to the Federal Acquisition Regulations, specifically 15.101-1b, it actually defines the trade-off process and how it is used in the Government determining best value. So, in this case, the RFP requested for, required that offerors propose in, propose not a solution that met the minimum technical requirements, but exceeded.
CP: Which we did.
GC: And so did Open Skies.
CP: So, this is, from Industry, I hope you understand since I run all the numbers very tightly, we spent $875,000 going after this.
GC: We understand.
CP: The only thing we have been able to learn for our investment of $875,000 is that we have a slight blemish, which is questionable, on one past performance, where we didn’t have an apples-to-apples comparison, because it wasn’t Firm Fixed Price on those past performances. And maybe…
GC: And if I may interject. In terms of the past performance questionnaires and the point of contacts that were provided, that was, the Government didn’t pick the performance, the point of contacts. BelAir provided those to the Government. So, all we can do is evaluate what’s provided to us.
In terms of the price, it seems to me that price is the big factor where BelAir is having a, I guess, experiencing some challenges in understanding why the Government would not pay an additional $13.9 million. And I’m just trying to understand, better explain, when it comes to trade-off, once it has been determined which proposed solutions are technically acceptable, there are, BelAir and Open Skies both proposed solutions that exceeded the minimum requirements of the solicitation which was required. At that point, what becomes the trade-off is the technical factors with price. In this instance, BelAir’s proposed bottom-line price was 7.61%, which equates to 13, an additional $13.9 million dollars over the life of the contract. And the Government, that is not in the best interest of the Government. The Government could not justify awarding that because you have to look at the Government’s job is to protect, we are the gatekeepers of tax payer dollars, which is our tax payer dollars, everyone sitting in this room, BelAir’s tax payer dollars. So, in using trade-off we would have…
CP: We also, we also pay taxes.
GC: Exactly, and that’s what I’m saying, everyone in this room…we cannot justify…
CP: Perhaps our Program Manager can explain a little bit more of our analysis of full life-cycle cost. Stephanie, why don’t you do that because I think the Government made a decision without considering full life-cycle cost.
SC: And I understand that wasn’t part of the evaluation criteria. And this is, I really appreciate the opportunity to learn because we’re bidding on some other things with you right now. What we hope to gain from this conversation is how you evaluate certain factors so that we can understand with how to present you with our best solution. And so, what we attempted to present in our management and technical volumes was the strength, knowing it would be, and I believe that it showed up in the evaluation as superior, the overall lifecycle advantage of using our patented technology so that you could save money over the life of your full maintenance, of these vehicles. And, so, I’d just like to ask, how, when you evaluated that, what was your view? Is that a strength, is that something important for us to continue to carry into future offerings to you? And, if so, then how much financial leeway do you think you have to be able to, you know, what would that difference in performance be worth in terms of dollars?
SB: So, so…
GC: If I could just interject, because I think this is a point that needs to, that I think will provide better clarity. There’s a difference between the term price and cost. The Government conducted price reasonableness and price analysis, and not cost analysis. So, it sounds like a lot of what BelAir has concerns over relates to getting down to the cost elements and the things of the proposal and that is not how the Government evaluated any of the offerors’ proposals that were received. We conducted price analysis. And so when the Government conducts price analysis and determine how the price, and determine whether or not a price is fair or reasonable, we look at the bottom-line cost, the bottom-line price, excuse me, proposed by the period of performance for each year in the profit. We don’t get into cost elements of a contractor’s proposal. So, I believe there seems to be some discrepancy there.
CP: Not at all.
SB: But, if I understand your question correctly. Your question is that you had proposed, that you had showed us significant cost savings in the, in your proposal that would extend beyond the three years of the contract.
SC: Correct. That’s right.
SB: But you had also stated that at the beginning of your statement that that was not one of the evaluation criteria.
SC: We knew that.
SB: Yes, and that’s part of the problem. If something is not listed as an evaluation criteria in our Request for Proposals, it’s not fair to the, it wouldn’t be fair to the other offerors for us to consider that. So, that’s why we didn’t, we did not consider that. And the reason we did not consider it was because it wasn’t listed as an evaluation criteria. Once the Government starts using other evaluation criteria, other things that are not listed in the evaluation criteria, or can reasonably be inferred from the evaluation criteria, then that, then we can’t, then that’s where we can run ourselves in to problems and start, and find there will be faults found with our award. So that’s why we did not consider it. We possibly, we may consider it going forward as an evaluation factor if we have future procurements.
SC: That would be healthy.
NI: We learn from this as well.
SB: But, at this point it wasn’t in the evaluation criteria, so we could not consider it.
CS: Steven would like to mention something…
SW: Yeah, and to that point. And, I do appreciate that feedback, because that is something we can take away. And, in retrospect, and in full disclosure, we think that we may have shot ourselves in the foot, because we took a bit of risk in regard to one of the company’s that we sought to solicit a past performance from. It was a commercial company, and if I had to put a bet on it, I would suspect that they were the ones that rated us as less than satisfactory. But, I think the takeaway that we want to convey is this forum is that we are trying to bring commercial best practices to the Government and we are trying to have them validated. We use our commercial practice to kind of innovate and practice, such that once we get it down and we actually earn a patent for that particular delivery mechanism. The challenge is that we’d like for the Government to look at something like that and see that we took a risk to innovate, to refine actually refine a process to be sure that in our proposal, it will be rated by you as superior. Unfortunately, the cost to us is that perhaps that particular customer didn’t rate us to the extent that we expected. And so, it’s a win-win potentially for all of us.
CS: I would like to also ask, how did our price compare to the IGCE that you guys conducted? Where did it rate? Was it 2% above, below, how did it compare with the IGCE?
GC: So, the Government does not disclose the Independent Government Cost Estimate. We provided feedback as it pertains to BelAir’s price proposal price and it how it compared generally with the other proposals.
CS: So you don’t use that as a criteria to evaluate price? Or, do you compare the two companies, or you don’t compare it with your IGCE?
NI: The IGCE does play a role in determining fair and reasonable price, but when we have a competitive acquisition, and we had the opportunity where we received more than one price proposal, we look for the price proposals to establish our fair and reasonable price determination.
CS: But wouldn’t you consider your IGCE to be that number that will drive the price reasonableness?
NI: Yeah, they’re significantly or significantly lower than the IGCE amount. We’re obviously going to have to go back, and that’s where price realism can play a role. That’s where we might have to, we might have a material issue with the solicitation itself. We didn’t identify any of those issues with this solicitation, so those factors did not play a role.
CP: I just want to underline part of our confusion. Our technical approach was based on the fact that your program was going to last fifteen years. That’s how it was written. Our technical approach wasn’t written to what we are going to do in three years. Our management approach was written to the fact that we would be here to support you for fifteen years, not for just three years. The past performance citations that we included were for our longest running past performance, yet how can you say that all of that was superior, and yet, when it comes to price, you do not consider full lifecycle? There’s no way we could have gotten a superior technical because we showed how this is a program that is far beyond just three years. So that’s one answer that has a cost and price that is internal to BelAir, I just can’t seem to understand that the Government can say that that could not be considered when you did in fact consider full lifecycle on everything but price. Does that make sense? That for us, it’s very, very difficult to understand how you could be looking at only a three year window for pricing and look beyond three years for everything else.
SB: So, that again, goes back to following the evaluation criteria, and the evaluation criteria did not mention, did not discuss the fifteen year lifecycle cost. So, that’s why we did not lend as much weight to that as, we did not lend any weight to that, and that’s why we went the way that we did. And even though we assessed you a superior rating in both of those, with regards to the price premium that we would have to pay to get that superior service, in our view was not worth the price premium.
SC: Feels like low price technically acceptable to me.
CP: So, was this the lowest price technically acceptable?
GC: No, this was a trade-off process, as stated in the solicitation, and as explained throughout this debriefing, and how the trade-off process is defined in the FAR and how it factored in to use. And how the Government used the trade-off process was in line with how we said it would be applied.
NI: We do apologize, we know you probably have additional questions and we probably can’t address all of them in our shortened timeframe today. But, we do appreciate you coming in and allowing us to do the debrief, and we appreciate your questions and we hope, again, to work with you guys on future requirements.
NR: And one thing to keep in mind, when you’re looking at proposals, just make sure that you put the effort on what we’re asking for and not go beyond it, and stay within scope.
CS, CP, SC, SW: Thank you. Thank you very much.
GC: And this concludes the debriefing of PSI Services procurement. Thank you so much BelAir Company.
CS, CP, SC, SW: Thank you. Thank you very much.
JoseA: So guys, thank you very much. Wonderful job. To the folks in the audience who are watching, the folks from the Bureau for Engraving and Printing here did a lot of preparation to prepare for this. And, if you go online and look at the cases, what you’ll find is the information that’s given to both industry and contracting folks that are actually playing the role in these mock debriefings isn’t complete. And it isn’t complete to kind of get us focused in on specific topics. So, keep that in mind as we engage.
So, I wanted to ask you a couple questions. I wanted to ask what are your thoughts on what you just went through? Talk maybe about some of the things that you learned in preparation, some things that you were frustrated with, did you feel that industry kind of understood what you were trying to relay to them, were there any kind of communication gaps, anything that you want to highlight about what you just went through?
NR: One of the things that we did in preparation. We felt very compelled, the lack of information was very frustrating. So, we decided to go into…I figured at some given points something of this mass had to have happened at some point. So, I went into FBO and just kind of figured out somewhere, somehow the United States Coast Guard had to have bought HC-130. So, I Googled it, and there it was – there was an announcement and an award back in 2013. And I said to myself, hm, this must have made it to the Government Accountability Office, and there it was, a GAO decision. And, so did some research and just kind of read and assumed the actual case issue was a variation of this scenario itself.
NI: Yeah, it is different. Certainly different.
NR: But we just kind of decided to get more into the weeds of what happened in the real case and just read more into past performance – what’s allowable, what’s not allowable, trade-offs. We read on price realism and did more than we needed to…
JoseA: It sounds like, it sounds like as a team you spent a lot of time engaging on this to kind of prepare for this discussion.
GC, NR, NI, SB: We did.
GC: We spent a lot of time, individually, reading through the materials and understanding and dissecting the information before we came together. We also did research, as Nancy stated, online. And, we also, my approach is, you always take it back to the FAR. What does the FAR say that you must do, what can we do, what is not permissible? The FAR is always the basis from my perspective, and that was our approach in how we outlined our debriefing and just kind of worked within the framework of that. And then incorporated other research that we felt was going to be useful to us.
NR: And you know, whatever wasn’t in the FAR, we did more research. I mean, just because it’s not in the FAR doesn’t mean that you’re not allowed to do it. So, there’s some innovation and some other stuff that you can do, so we did more research in to that and what’s more permissible, and what happened in GAO cases that we can kind of reach in to it.
SB: One of the concerns that I had was that the past performance questionnaires did not go into, there was adverse information that was on the past performance questionnaires. And I thought originally that we had to have talked to the offeror and asked them, given them an opportunity to respond. And I learned from reading GAO cases on this that you do not have to do that if you are not holding discussions with them, which we did not hold discussions with them so we did not have to do that.
GC: It also came out in our discussion so that when we met as a group, we agreed on some things and some things we didn’t agree on, and so that came out about discussion, and Sam still wasn’t convinced. So he went and looked on GAO and came back and said, you know, you guys are actually correct. We did not hold discussions. We awarded without discussions. We reserved that right in the solicitation and there was nothing we received that was adverse that would raise any type of concern that would require us to go back, so we debated each other a lot.
NR: We thought a lot about “what if” scenarios.
JoseA: So one of the things in the opening remarks to this is, you know, you have to teach Contracting Officers to think, and acquisition professionals to think more holistically about their job. And historically, training has been CON100, CON200, Program Management. And you learn about a specific silo. And so, the goal of this case was to actually force you to go and do that research. To force you to go through “what if” scenarios. To give you haphazard information and that would make you feel very stressed, so that you would kind of cognitively think holistically about this and come up with a way to deal with it. So, I’m really excited to hear that you did that, because that is like the whole purpose. If you’re training people to think, then you put them I this situation. If you’re training people to check of a checklist, you send them to CON100 and you teach them every single rule that’s in the FAR, related to FAR 15 or FAR parts. So that was the goal.
NR: One thing I learned yesterday was that we can agree to disagree, and so it was very, we learned a lot from each other and that was very, very useful.
NI: We did get a little emotional talking about it, because of the stress and the limited information that we had. You know, we were getting ourselves a little worked up on it, but I mean even with it being a mock debrief…
SB: Yes, there were some things that, internally, we disagreed on whether they could be divulged or not. But we eventually came to a consensus, and even though there were dissenters, we agreed to hold the parting line on certain items.
JoseA: That’s good.
GC: Right, it was very important, especially being the Contracting Officer in this scenario, that…
NR: That we were going to support you.
GC: Right, and not that I was just supported, but that, we, the Government, this is our position, this is the reason, and here are certain things that we…here is what we can say, this is what we cannot and will not say, and things that we should stay away from and that really proved to be helpful.
SB: It would also be helpful if you were doing this in real-life with other CORs who may not be as well-versed in 1102 roles, Contracts Specialists roles. It would be helpful to talk with the COR beforehand to make sure they know other things, like, for example if something was not in the evaluation criteria, then you should not be talking about evaluating it at the debriefing, because sometimes you can have someone who might speak out of turn and say something that they shouldn’t and cause either a miscommunication or something worse.
JoseA: Before I turn it over to Thomas, one other question. The other goal of this was to put you in a no-risk environment. I mean, there’s a little bit of risk with the camera, but a no-risk environment where you could actually engage with industry. I mean, we’ve talked to people who are acquisition professionals, who, the first time they engage with industry in a debriefing is during the debriefing. I mean, they got their roles in a class, but they never had the experience. Do you think it was helpful to actually have the experience with real industry professionals?
NR: Oh, yes, yes. It’s actually very, it’s very different and a lot of the times, we were just talking about it, we do it ourselves, 1102s, we all have the knowledge of the way it supposed to work. But, industry standards, they have different rules and so they’re goal is to make profit and make my company better. And so, we also have to be cognizant of that and understand where they’re coming from. And so their professionalism, the overview of what they’re thinking. Just the fact that they went above the scope and that they were thinking fifteen years ahead. And I’m like, wait a minute, that’s not, we have an identified need now and this is what we’re looking for. And, so, a lot of things that came across that I was like, hmm, I didn’t think that they would spend that much energy in thinking what would make you feel that you’re entitled to a contract for fifteen years. But, yes, it was very, very…
NI: One observation that came from, we were talking about this as we were preparing, we’ve done these kind of mock debriefings in classroom environments before, amongst Government personnel playing the role of a contractor. And, it never fails, we’ve all had the same experience where the Government personnel playing the role of the contractor are adversarial and like, yeah, and they’re coming at the Government personnel. And, you know, that was not the experience at all. And I think in most real-world scenarios you don’t have that kind of interaction. And, so, it was nice. They certainly were on their game, and, you know, we can learn a lot from them. Like, look, it’s a conversation, it’s not a…it’s a learning opportunity for us as well. As Government personnel who give a debriefing, it’s not just for them to learn, we have to take away stuff as well. Anyway, I thought it was a good back and forth.
JoseA: Thomas, did you have anything?
TO: I know one of the questions I thought I would like to ask is that during the debriefing scenario here, we noticed there was a lot of discussion on the pricing and they tried to tie it back into their technical approach. Did you kind of discuss, maybe beforehand, just simply saying, you know, you did have a superior technical approach, but, by the way, this is our price and your price was this. And, too bad.
GC: Disclosing the Government’s IGCE?
TO: You just simply had to…your price was too high.
GC: We, and basically we stated that we could have come back and have been more direct and stated that their price was too high. But, at this phase in the debriefing, it’s our position is, it’s not appropriate to disclose the Government’s IGCE and that’s why we did not.
NI: Yeah, and also, I think based on the information that I have, and I might be wrong, but I think all three proposals were found to be all fair and reasonable in the pricing. Obviously there’s was higher, but I don’t know that we would have just wanted to blanket say, you know, we know your price was higher so we’re not going to go with you, when all of our evaluation criteria was leaning towards…you know, price is certainly an evaluated factor, but it’s not the most important and that would kind of, I think that would kind of leave them with that feeling, like what do you mean? You ultimately just came back to price.
GC: And they heard it. They knew their price was too high. Reiterating $13.9 million. That’s a significant difference. We didn’t just use percentages, we crunched numbers. That is a huge difference in…when you’re trading off, determining is it really, the value that they’re going to provide to Government is it really worth an additional $14 million dollars? In this case, given with the facts and the evaluation results and the Source Selection Authority, we just did not have the information we needed to justify paying that additional $14 million. And, yet, if you take a step back, in the, think about it in the real-world, in the real-life situation, me as a CO, I’m still going to have major concern with going back and trying to find $14 additional million dollars when I know there’s another proposed solution that also was technically acceptable. That’s not just, in my opinion, sound business judgment. We are the gatekeepers of tax payer dollars and I feel like with the limited information we had, if we had more facts, maybe I would have done $14 million, but perhaps…
NR: And that is something we discussed…
GC: And we decided to award without discussions, and so we had to make a decision based on the information that was provided. But, perhaps we could have opened up for maybe clarification and maybe have gone back to the offeror on some things as it pertained to price, but we did not, in this case.
JoseA: That’s a good answer. And that, the whole purpose of the case, was to generate thought process there and for you to have to make a decision on that. So, I’m so happy you got to that point and had a robust discussion around that. Thomas, do you have any other questions?
TO: No.
JoseA: Guys, I thought you did a wonderful job, and if we put any undo stress on you yesterday, it was on purpose, so good work!
GC, NR, NI, SB: Thank you!
JoseA: Well, first of all, thank you guys for participating in the recording of that debriefing session. I hope you, I hope it was enjoyable. I know our folks appreciated it. The folks at FAI appreciate it, and the folks Government-wide who are watching this really appreciate it. The only way that you can learn is to actually witness real professionals interacting in a real environment. That’s what we tried to do here today.
So, one of the things I wanted to do was to give you the opportunity to talk a little bit about your experience and whether it was from the perspective of the preparation, or whether it’s from actually going through the exercise that you went through. What are some key takeaways that you’d want Government acquisition professional out there that are watching this to know about, you know, the challenges you face within industry when you prepare for debriefings, or when you go through the market research process, or the acquisition planning process? So, I turn it over to you all.
CS: I think, for us, as a company the biggest thing is knowing what Government considers best value. They just define it, best value, or LPTE, but when it comes to best value, we need to totally understand what exactly best value means. How they are quantifying the subjective term, best value. So, I…
JoseA: So, in your preparation, was that something you kind of struggled with or wrestled with?
CP: Absolutely. Um, industry, when industry goes into a debrief, what we hope to get out of it is, did we learn where we missed the mark? Period. Where did we miss the mark? And, in most debriefs, that is the biggest frustration, is that we get rated superior, and this is very typical in our scenario, in that it was superior, yet it seems arbitrary that the Government made a decision, and, to industry, it seems arbitrary. And now being more forthcoming, with more openness, on best value. So, whether it’s in real-life, or just this scenario that we just displayed, industry really want to know can we improve…
SC: We want to learn. We want to learn! And, actually, kudos to this team. I thought the Contracting Officer did, first of all, it was the Contracting Officer and not a delegate. And, so, when I go to a debriefing, I want to hear from those people who made the decision and then to know that they’re on their game! That they really understand. They were familiar with the details, they knew what we had proposed and they knew why they had made the decision that they did. So, well done by the Government team.
SW: I mean, basically, I heard them say that we goal-played it. I mean, they had a sense of what they were looking to accomplish, in terms of a solution that met the program needs. And, we, in fact, went beyond that, but they weren’t willing to pay for that.
SC: Right.
SW: So, I think being able to get that feedback, and, I think it was very effectively delivered, was very useful to us. But, I do also share in the belief from the industry perspective that best value is so amorphous at times.
CP: Yes!
SW: I mean, to really be able to pin that down so that we can actually speak to that and ultimately win the business. That’s a really big challenge for us, trying to figure out how to decode that so that we meet the right threshold of proposed solution, cost reasonableness, but then ultimately earning business.
JoseA: So, then, let me ask you a question about that. How would you take, how does industry, you know, how do you guys take information from a debrief back and utilize it in the future.
CP: Ultimately, most debriefs don’t give you enough information so that you could say, we should have done this differently. Um…
SC: But, it’s data that helps.
CP: It is data. Because, also, industry…I have a small company, we ask for a debrief whether we win or whether we lose. Such that, we really want to learn what did really do well, or did we just get lucky? Because we want to replicate what we did well.
SC: So, and then how do you use that? Right, and so, especially when you win, getting that debrief and know what to replicate. What really mattered to the customer? We learn a lot about the personality, the likes, the dislikes, what they value from hearing them talk about how they do their evaluations.
CP: Yeah, and interestingly, in a couple of, I mean, not scenario, but real-life, I’ve been told by a Contracting Officer that I don’t have the time you want. We can discuss in the normal execution of the contract what you’re doing well, but we have no time to prepare a debrief. So, it’s kind of interesting.
JoseA: Thomas, did you have any questions that you wanted to ask?
TO: I was wondering, when you encountered some of the challenges, because I was watching the mock debriefing, there was communication challenges when you were trying to get across what you were intending, such as best value and I noticed it was price reasonableness, price realism, and they were kind of talking about two things. My question to you is, when you’re in those type of environments, how do you, as a team, kind of step back and realize that situation and counteract that challenge that you experienced in terms of communication…you’re not getting your message across?
SC: I’ve known cases where we’ve been in a situation like this where we’ve needed to caucus. We’ve needed to step out as an industry team and say, what is it that I’m missing?
SW: Yeah, what did we hear?
CP: Are we talking about the same proposal?
TO: Also, do you feel that sometimes there might be a little bit too much formality this type of debriefing? I noticed that the team had their books and, hey this is the FAR. Do you think there needs to be more, there needs to be much more relationship, relate-ability, or do perceive that there’s a barrier there that…
CP: Yeah, there is. From industry, what we really want to learn is, guys this is where you really missed that boat. That, as opposed to something that seems to be very unmeasurable.
SC: Well, and this team was more communicable than some.
SW: Than most.
SC: I mean, there’s been time we go to a debrief, and in essence, they’re just re-reading the letter that they sent us, and that’s not really helpful. I mean, we really shouldn’t take meeting time to go do that.
JoseA: Guys, we really appreciate your time. I hope the audience learned from your perspective and appreciate all the effort that you put in to this. For the folks in the audience, this took a lot of time to prepare for and these guys don’t all work together regularly, so for them to come together and pull this off, I mean, we think is quite an accomplishment. So, thank you very much!
CS, CP, SC, SW: Thank you!
TG: Hi, I’m Tony Grayson, from the Federal Acquisition Institute. I’m the Program Manager for the FAC-C Program and I’m glad to be here today.
Bob Namejko (BN): I’m Bob Namejko, I’m with Green Light Acquisitions. I am retired Federal employee and we are here because the past debriefing seminar was something that we are both interested in, from different perspectives. From the FAI point of view, it’s to teach the procurement community exactly what they should be looking for as far as debriefings go. From my side, I’m looking more from the private sector side and what we expect in the private sector whenever there is a debriefing. So much of what we saw in there took into effect a lot of the, a lot of the individual tasks that Contracting Officers go through in their preparation for a debriefing, and, by the same token, what the industry has to do. And, some of my observations were centered around the fact that, it appeared to me that the Government really did their homework whenever they prepared for this.
TG: Yes.
BN: They were set up in such a way that they had one voice and that one voice commanded the audience. They didn’t allow interruptions from industry and I thought it was very well orchestrated.
TG: I agree. I think the Contracting Officer was clearly chairing it, and the Contracting Officer should chair it, and they had a clear agenda. They set it up with the rules of engagement right from the front. When the contractor tried to deviate from that, they called the contractor to task right away, so I thought that was great. So, obviously, they were quite well-prepared for the debriefing.
BN: And that’s one thing that the acquisition professionals out there that saw this debriefing, this mock debriefing, should take into consideration. Nothing frustrates industry more than to go into a debriefing and not learning anything from that. Too often you have times where you go into a debriefing where a Contracting Officer, or not even a Contracting Officer, a Contract Specialist, will be giving a debriefing by just reading the letter that was sent out. What they did at this debriefing was they enumerated the facts. They went through strengths, they went through weaknesses to let industry know the reason they lost was because they evaluated proposals in such a way that the end result was a fair judgment that was awarded, they weren’t the one that was, the company that was in the debriefing did not win the award, but I think they may have gone away with idea maybe not all of their questions were answered, but I think the Government explained to the best of their ability why they didn’t win.
TG: And I think that’s important. The contractor has to understand why they lost, but they were treated fairly in the competition. And, I think the passion of the debriefing team, and it came across, that they followed a very rigorous process, in accordance with the FAR, and had made a, you know, a conscience decision. And, as a result of that, the contractor should take away form that that they were treated fairly. That they weren’t treated unfairly, or that there weren’t some inherent biased in the Source Selection and that the Government had followed the criteria that was set forth in the RFP. There wasn’t any secret criteria there that the basis of award was made on.
BN: Right. And one of the other interesting things I found was that, on the industry side, it was not an adversarial relationship. Often times, when you’re in a debriefing scenario in real-life, the contracting community within the Federal government often goes with the idea that we are facing a faux, that we have to be prepared to go to battle with these people that are coming in here. That’s not necessarily the case. Industry spends an awful lot of money preparing proposals. Industry has to wait a tremendously long time from the time an RFP is issued till the time a contract is awarded. They have to keep teams together, they have to have their upper management coming to them constantly – when is this going to be awarded, what have you learned, what’s going on with the procurement? And so, by the time the contract is awarded, and their not the winning offeror, there are numerous reports that they have to write. So they come into the debriefing maybe with a little bit of a chip on their shoulder, but they don’t come in there necessarily because they want to do, or intend to battle protest. That’s not what they’re there for. They’re there to learn and I again believe that the Government in this case did that.
TG: I think they did. I think the contractors spent $875,000 in this situation for their proposal. Somebody at the contractor, whoever the proposal manager is, is going to have to answer to management – why didn’t they win? And you need to give them the answers to those questions.
BN: Were there any parts of the justification that you found to be a little bit, not necessarily strange, but you raised your eyes and said, “why did you say that?”?
TG: The one about past performance, that they couldn’t discuss what the past performance evaluation was, I thought was unusual, from my experience.
BN: Yeah, I had the same feeling. And the other, I’m not sure that they ever came across, when they were talking about what best value really is.
TG: Yeah, and they clearly did a trade-off, but there wasn’t any discussion on the trade-off and it kind of got wrapped around that the technical criteria were much more important than price, but it came down to price was the discriminator in terms of making the best value decision. So, and I think the contractor asked questions about that, but I’m not sure they ever got a good answer.
BN: I agree, and that’s the other thing that the acquisition professionals out there are already aware of. Industry believes that every procurement is awarded LPTA, whether that’s stated in the RFP or not. We all know that’s not the case. But, the fine line between best value and Lowest Price Technically Acceptable is such a fine, fine line that it behooves you, whenever you’re doing your evaluations, that you take that into consideration. You know that question is going to come up. You know that industry is going to believe that you awarded the contract on the lowest price, regardless of what else comes out.
TG: And I think the contractor, I think the Government tried to answer that by saying that, hey, your proposal was superior but it wasn’t worth $13.9 million dollars between good and superior. But I think the contractor had some questions about that, and perhaps there wasn’t enough information in the case where the Government couldn’t respond to that.
BN: Right. And, what about the fact that they had the two superiors for the losing team and two goods for the winning team. Did that strike you at all strange?
TG: Yeah, there was quite a, well, and it may be that the superior was based on the higher price. You know, maybe that’s how you got to superior.
BN: Right, and the conversations that the contracting group had after the debriefing was over where they talked about the fact that they didn’t have discussions with the offerors and that may have led to this scenario.
TG: Exactly.
BN: It’s an important consideration for the contracting professionals out there to realize that when you’re in that sort of situation, and all things being equal, maybe having discussions is not a bad thing to do.
TG: Discussions might have helped. It may have where they could have gone to the contractor and said, we think that your price is too high for the following reasons. And that way, maybe the contractor would have looked at the proposal and maybe have taken some of the gold plating out and made their place more reasonable, and may have gotten a different decision.
BN: Right. And the other thing the contracting professionals, or the acquisition professionals, out there need to realize is that the offerors that come in for debriefings have very, very high price lawyers just waiting in the wings that may want to convince the company that they need to protest. If a protest is ever put into action, all the information that may not be coming out in a debriefing becomes discovery for the lawyers. So, you really, really need to be careful when you’re putting your files together that you make sure that everything that you want to say is, has back-up, is justified. And I believe in this situation, again, only because this was a mock debriefing and there wasn’t a lot of information, I believe the Government did a pretty good job in this case.
TG: I think so too. They were well prepared. Worked professionally, had a lot of enthusiasm too, which, I think is, leaves credence that they made a fair, unbiased Source Selection.
BN: Right. And, whenever Jose Arrieta had mentioned in his discussion with the contract group from the Bureau of Engraving and Printing that whether or not this was a good exercise for them to do to a person, they all that it was. And, one of the things that we would like to leave you all with is the fact that if you have a chance to be involved in a debriefing, even if you’re just a Contract Specialist, a GS-5, 7, 9, very low-level Contract Specialist, if you can get invited into a debriefing to just sit there and learn what’s going on, it will be invaluable. The interface between industry and the Government is only going to be successful if there is an interface between the two. You can’t work on an island. Industry wants to know what they can do to support the mission. You want to save the taxpayers’ dollars. The two of you both have a reason to want to talk, a reason to want to be smart, a reason to want to be educated, and do your best to communicate with industry, because they want you to do that. Thank you very much.
TG: Thank you.
JohnA: Our thanks again to Jose Arrieta and to all of our participants in today's mock debriefings. We certainly hope you found today's seminar beneficial, as we’re sure it gives us all a better understanding of debriefings and how to use them to greatest effect and benefit serving the American taxpayer. Stay tuned as we take a short break. After, we will return with our guests to answer a few of your questions. And, during this break, I’d like you to consider this question, “if you had the opportunity to ask Anne E. Rung, Administrator of the Office of Federal Procurement Policy, a question, what would you ask?” Well, you’re going to get your chance. The Federal Acquisition Institute will present Anne Rung in a Meet the Administrator webcast on Wednesday, August 19, 2015, so would like your help interrogating -- I mean, interviewing her. On the homepage of FAI.gov we have posted a link that will allow you to submit your question to Ms. Rung. Act fast, the deadline to submit your questions for Ms. Rung is Monday, August 3, 2015. We will be right back.
Hello, and thank you to our audience for returning to the question and answer segment of our Seminar, Lifting the Curtain – Debriefings. Our participants have returned to answer a few of your questions in the time we have remaining. Those questions we cannot get to will be included in a document that we will post to the Federal Acquisition Institute’s Media Library along with the recording of today’s seminar. So, what do we have two lead off?
JoseA: Thank you, John. So, to the folks out there still listening, we certainly appreciate it, and we’re going to have industry chime in on some of these questions, as well, as well as Government professionals, because part of this seminar was to get you exposed to industry, because we think that's very important. So, our first question is, “it seems that during the latter part of the debriefing, the meeting started turning into an argument. And, Patrick, I would love for you to take this. How do we stop a meeting from continuing down this path? And, when you’re done, if industry could kind of chime in and kind of give their opinion on this, I think that would be great.
Q&A Panel Answer: Sure. So, when we were doing the broadcast previously we were kind of going with the flow during the debriefing, and answering questions to the best of our abilities, and getting to a little bit more detail than what we would normally get into and letting it run a little bit longer than we normally would, just for good television and good learning experience in a good training environment. As you start to see things get a little contentious during debriefs, you can take a step back, you can take a two minute breather. You, as the Contracting Officer, can start to cut off questions. Remember, debrief is an opportunity to give feedback on how they did on their proposal and how they can improve in the future. You don't have to get into a debate or an argument back and forth. And it is however your personal communication style is, you just line it up, cut off the question, and move on to the next one.
JoseA: And, Carolyn, from industry’s perspective, do debriefings typically get off-target like this? And, what does industry do to try to manage the tension in the room, so to speak?
Q&A Panel Answer: Right, and I don’t think people in industry walk into a debriefing expecting to have any argument with our Government counterparts. We are there to get information about how our proposal was evaluated. However, the hours that are spent on developing a solution for the Government and the proposal that we feel meets all the criteria laid out in a solicitation, people get emotionally involved in what they have produced. And, I think, from industry’s perspective, more often you see frustration when the information being shared by Government during the debrief is not as candid or clear as you wish it could be. Or, that you’re not hearing exactly why your solution was not the selected, winning proposal. So, I think though, for the most part, during our debriefing here, the exercise that we did, the CO handled it very well. Being clear about the guidelines that we were following, and it did probably go a little further than it would in real life.
Q&A Panel Answer: We had fun with it.
Q&A Panel Answer: We did.
JoseA: Yeah, and I think the key point is, for the folks out there in the audience, share as much as you can as an acquisition professional. Industry has put a lot of time and money into this, and there will be value on both sides. So, could we flip to the next question? Oh, so here’s a question. So, can the COR or the technical monitor for contract be included in the debriefing session with the vendor? And, you know, CO, what do you think about that?
Q&A Panel Answer: And that’s fairly straightforward. We can include anyone that you feel comfortable about having in the debriefing. It helps to have somebody from a technical aspect, but at the same time, you want to make sure that the Contracting Officer is the one doing the talking because they are the ones that understand the ins and outs of what's going to get you into trouble and what’s not going to get you in trouble.
JoseA: Very good. Going to the third question here, should we always look, and Patrick, I think this is a question for you and if anybody else wants to jump in, should we always look to the Senior CO for approval to answer a question, especially the legal type questions?
Q&A Panel Answer: Debriefs are handled by a CO and their preference of how they run it. From my perspective, if I’m running a debrief, I'm the only one who is going to be talking. If I need to huddle with my team, I will do so. You want to have one point of contact, one person who is giving the message, that way you can control the information going out as far as problems that can arise from saying things the wrong way as you go through the debrief. (4:43) For purposes of this, we ended up allowing everyone to talk, because obviously we were doing a training portion, but in real life I think you defer to your CO. You go over those ground rules before you walk into the room.
JoseA: It sounds like the CO is the quarterback and the CO should develop a plan and make sure that it is executed correctly, is that what you are saying Patrick?
Q&A Panel Answer: Absolutely, and you want to make sure you have a firm plan and that you stick to it.
JoseA: So, here is a question for me. How does a new contract specialist obtain training to be proficient in their job? You know, first and foremost, as a contract specialist or a Contracting Officer in the Federal Government we have to meet certain training recommendations that are required by FAI or DAWIA. And so, you always have to look to FAI or DAWIA for training. I would encourage all the contract specialists or 1102’s out there to leverage as much of the free training as they can possibly get. The other thing that I would recommend, and Patrick, if you could jump in when I am done and say a few things? The other thing that I would recommend is partner up with a good CO or a good Program Manager – someone that you can learn from and find a mentor within your office. I really think that that will help with your development. It is a blend of training and a blend of finding a mentor within the office you work.
Q&A Panel Answer: The best experience that you’re going to get is by doing the work and doing the work from people who know how to do it. The other thing I would mention is a plug for the Treasury Acquisition Institute. TAI is a great educational center. They offer classes on all the acquisition points and I would highly encourage people to seek some of their training needs through that.
JoseA: Ok, so let's move on. We only have time a few more questions here. Should Government have legal representation from General Counsel?
Q&A Panel Answer: So for that one it’s simple, it’s agency discretion in policy. However, our practice is if the vendor is going to bring their lawyers, then you should have legal representation, and that’s from my perspective you should not go in, down that valuable asset that your Government agency can provide to you.
JoseA: OK, so I believe that we have time for kind of one more question and I want to make sure we get some feedback from both industry and government because I think that’s important and that was the purpose of this. Industry and government, everyone at the table, Thomas feel free to jump in. Have you experienced procurements where unsuccessful offerors submit a protest prior to debriefing? What is the best practice or how should such an event should be handled?
Q&A Panel Answer: I think I can go first, briefly. I would recommend that people wait to have your debriefing before you decide whether or not you’re going to protest. That’s the opportunity for you to get the feedback. And, frankly, it’s going to save you from filing an extra supplemental or an extra supplemental protest. It will save your lawyer some work if that is the route that you go. And it gives us an opportunity to fully explain it to you, and that would just be from a government perspective my recommendation.
Q&A Panel Answer: I would echo that, from an industry perspective. I think it would be a little unwise to file a protest prior to having the opportunity to get your debriefing. It will provide clarity on the Government position and their evaluation of your proposal and you have an opportunity to engage with Government, to better understand what you believe is your basis for protest. I have not ever seen a company in my experience protest prior to having the benefit of the debrief.
JoseA: Guys, one of the keys to being a good 1102 or a professional in this industry is to understand the industry base that you do business with. I hope that this gave you an opportunity to see that, live. I just want to get the panel one last chance to make comments. I believe we are out of time. Any last comments?
Q&A Panel Answer: Thank you, I had fun doing it. Great meeting everyone.
Q&A Panel Answer: It was a great meeting and a great opportunity to engage with our counterparts in Government in a scenario that was fun, but it was real. And you all saw the exchange, the emotion, the excitement that people have about defending their position on either side. And really, we are all just looking for clarity and understanding, and that is what the objective of these debriefs are.
JohnA: Don't forget, the Federal Acquisition Institute has recorded today's seminar and the video along with the presentation material you saw today will be posted in the Media Library on FAI.gov. You should be able to access these items in a week or so. On behalf of the Federal Acquisition Institute, thank you once again for joining us.
