Critiques of the defense industry are not hard to find. Claims of overpriced, underperforming systems abound. These are often the result of ignorance, but even the industry's staunchest defenders have to admit that something is deeply wrong. And I agree, but I think that the standard narrative of why this happens fundamentally misunderstands the drivers of the problem, which means that things keep getting worse and worse. I'm not sure there is actually a solution, but at the very least, a better understanding of the problem will keep us from solving the wrong one.
I should lay my cards on the table here. I'm an engineer with a major defense contractor who you have almost certainly heard of.1 Much of this is based on my experience, closing in on seven years now, in the trenches, trying to deliver products to the military. Some of the stuff I've worked on has finally fielded. To some extent, this is an introduction to a contractor's-eye view of the issues in the industry, although I have tried to be fair to both sides.
The first fundamental fact is that for most suppliers in the defense industry, there is a single buyer for their services. Almost all problems are downstream of this.2 If you run a US defense contractor (or any part thereof) and wish to continue doing so, your first priority is to avoid ticking off the government to the point where they stop buying your stuff, because in practical terms, pivoting to do something else isn't really possible. In the past, military and civilian systems weren't that different and it was possible for, say, Douglas to pivot between building DC-3 airliners to A-20 Havoc light bombers and back as the situation required, or for shipyards to take on civilian work when military contracts fell. But a modern fighter is very different from an airliner3 and the Jones Act has killed off US merchant shipbuilding, while specialized things like combat systems have no obvious civilian counterpart.
An A-20 Havoc
This is a classic monopsony, or would be if the DoD was profit-maximizing and doing all it can to squeeze the other parties in the transaction financially. But the DoD isn't profit-maximizing, which means that it squeezes defense contractors in other ways. Specifically, it optimizes for keeping the people who hand out the contracts from getting in trouble with their bosses, and squeezes the contractors to do so. Simply having something cost too much isn't really an issue, because after all, the government doesn't have to make a profit. But if the program fails in a way that can be blamed on the people who issued the contract? That's a serious issue. So things are carefully structured to make sure that if something goes wrong, it's the contractor's fault. They didn't meet the detailed requirements laid out, or ran over budget, neither of which is the government's fault.
On the contractor's side, the incentives are somewhat better-aligned, but only in the narrow sense. Sure, you want to deliver what the contract specifies, on time and on budget. But there's a strong incentive to keep the customer happy because you're going to be working with them again, which in turn means that there's little incentive to push back on anything the government wants, no matter how crazy, so long as they're willing to pay for it. And there's definitely no way for the contractor to say "this has gotten completely out of hand, we're going to go and sell to someone else" which I suspect serves as an important check on bad behavior in the civilian world.4
But things get much worse as the relationship between the two parties moves from friendly to adversarial. I suspect this usually starts on the government side, essentially for the structural reasons discussed above, although my conflict of interest is obvious. In any case, as the sides stop trusting each other and start standing on the language of the contracts, things get much worse all around. The government has to spend a lot more time and effort developing requirements, because the contractor will demand extra money if they need to change any of them later. The contractor points out that they're supposed to make a profit (if they don't, the stockholders get angry) and the government isn't going to go out of its way to be nice to them if the problem is on their end. Repeat this cycle for long enough and you get to the point where the contractor gets yelled at for being under budget on a cost-plus-fixed fee contract.5
But at this point we need to turn to the third actor in the system, the bureaucrat's bosses in politics and ultimately the public. The government is hamstrung by rules that sound great on the campaign trail, but frequently cause problems in real life. As a friend who works in a very different part of the government puts it, "if three raccoons in a trench coat claim they can do the job, I have to believe them", because disbelieving the raccoons looks like unfairness or possibly corruption, and they're likely to file a lawsuit if they lose. Any sort of legible waste is treated extremely seriously by the system, which means an ever-expanding cloud of rules to try and stamp those out, even when the system creates far greater illegible waste in the form of people sitting around and enforcing these rules.
Our latest defense contractor6
This has two effects in defense. First, the field is dominated by companies specializing in navigating this complex world, and all of the people required to do so ultimately end up getting paid for by the contract. But even beyond that, the rules tend to very much favor incumbents for sustainment work, which is where a lot of the money gets made.7 After all, they have the expertise on the system, so any competitor will need to include a big chunk of time (and money) for getting up to speed, handing the incumbent the win. Normal economics would expect the contractor to use this to extract most of the cost of spinup as profit, but this can be "solved" by having the contractor on a cost-plus basis, so the profit is capped at a reasonable but not excessive level.8 But it does provide quite a bit of protection to the contractor, who knows that they are difficult, though not impossible, to fire. The fact that they are ultimately the subordinate is very much a thing they keep in mind, but at least they can push back somewhat. The ultimate result is to have a bunch of people on both sides spend time doing paperwork about all of this, and those people cost quite a bit of money. I'm not certain that this is actually cheaper than just letting the contractor take the monopoly profits, but it certainly looks better to outsiders, which is how most of this stuff is managed.
I don't know. Maybe we should let the raccoons bid.9
This would normally be the point where I'd offer a solution, but I don't really see one. Anything that involves pointing and yelling at defense contractors might make people feel better,10 but it isn't going to actually solve the issues. It's not that Lockheed, Raytheon, General Dynamics and Boeing are bad people who like wasting taxpayer money. They do what they do for structural reasons, and changing what name is on the badges of the contractors will end up with us in the same place, probably after a lot of pain for the new contractors. The problems on the government side ultimately originate in the political system, and because "we should give defense contractors more leeway" is a terrible campaign slogan, it seems unlikely to be adopted as policy. One thing that I do think is unexplored is having the government do more work in-house. This was historically how most defense procurement was done, and while it is certainly not without its issues, it at least removes the organizational tension you get between the government and the contractor today. But fully exploring that is going to be a matter for another time.
A meeting at the Pentagon to sign the contract.
In any case, the American defense procurement system is a mess, combining many of the worst parts of government and private industry in a system that somehow manages to work, and even to produce world-class products, despite the weird and conflicting incentives it gives to those within it. Making it better defies easy solution, but I suspect that much of this is due to the poor visibility the people ultimately in charge of the process have into the day to day realities of it.
1 I am not going to say which one just to make absolutely clear that I speak only for myself and not for them. ⇑
2 Obviously, there are exceptions in some areas, such as small arms, and from what I know, those are somewhat healthier, or at least have different problems. And no, foreign sales aren't the answer. The US does about 40% of the world's defense spending, and sales to foreign governments are almost always intermediated by the US government. ⇑
3 And Lockheed is unlikely to want to repeat their last attempt to play in that space. ⇑
4 Well, there's always a price at which you'd be willing to put up with a terrible customer, but that price is higher than it would be for someone who is easy to deal with. This is probably not unrelated to high defense prices. ⇑
5 No, I am not making this up. I was on the receiving end. The government's logic was that we must have padded our estimates so the 10% fee on the original amount was more like 15% when we were done. I was in the estimating meetings. We didn't do that. We were just conservative in our estimates, and I think that may have been a new experience for them. In any case, the system is screwed up to the point where yelling at someone for being under budget almost makes sense. ⇑
6 Thanks to Eleanor Konik for the image. ⇑
7 In a lot of ways, this isn't a bad thing, because trying to get a raccoon a security clearance is a pain. ⇑
8 Cost-plus gets a bad rap in a lot of places. Yes, it has often been abused, but it's the only thing that makes sense for a lot of military development work, where there's a lot of unknowns and the potential downside is very high. Even relatively low-risk fixed-price contracts have a history of blowing up, which is why Boeing, for instance, has stopped taking them. Also, the absolute mess that is contract definition does have the advantage of closing off the "charge everything to the DoD" abuses that cost-plus is famous for enabling. ⇑
9 Thanks to Lithros for this and the following image. ⇑
10 I am at least half-convinced that a significant but unacknowledged thing defense contractors get paid for is being the one to take the blame when something goes wrong, in addition to being paid an inconvenience fee for putting up with this mess of a system. ⇑
Comments
The obvious solution would be to split monopson. I seriously think that one of the reasons for good stuff being done, by the US defense industry is that there were often multiple buyers (Navy/Air Force). Somehow, US-made aircraft (two buyers) sold abroad more often than ships or IFVs (one buyer). How about splitting Navy procurement into Atlantic/Pacific and the same for Air Force and Army?
I'm not sure that competition in procurement is always the answer. Navies have been procuring ships essentially sole-source for, oh, four centuries or so, and had systems that worked reasonably well until they were dismantled in the mid to late 20th century. My thoughts on in-house procurement are heavily influenced by my understanding of how that was done, and seeing that things have been worse recently.
I'm not sure this is good logic. All three of those markets are rather different. We buy a lot of airplanes, which drives down the price for export orders. Ships tend to be slightly more customized, and our requirements are somewhat unique. (Also, our shipbuilding industry is kind of bad.) And we do reasonably well on the IFV market, but I think in that case, the US is a big enough customer that we aren't too worried about export orders. (Contrast the French, who used to figure out what would sell on the export market, then make their own military buy that, too.)
The problem is that the Navy and Air Force need slightly different things from their planes. This is less true of the Atlantic and Pacific fleets. And then there's things like Aegis. Who gets that, and does the taxpayer have to pay for a second version? I do think that the Air Force/Navy split in airplane procurement did have some positive effects in terms of trying different ways to organize things (some of this persists to today) but at the same time, competition isn't magic. If we need a thing to do X, and it will cost Y to do once, doing it twice is very unlikely to cost less than Y, and pretty likely to cost 1.75-2.5 Y. This is not great when there's only one customer. And the team that does X is going to run into the same monopoly/monopsony issues I talk about above.
Not so much pivoting from just making military stuff to just making civilian stuff (and vice versa), but it seems like the bigger contractors should at least be able to diversify heavily into civilian businesses and form conglomerates with civilian and defense business lines. Not just Lockheed Martin trying a passenger airline*, but just outright acquiring or developing civilian technology business lines that have nothing to do with their defense business to diversify.
Granted, then you get shareholders trying to force them to spin those off, but that's where the government could step in and push back.
I've noticed most of the negativity towards cost-plus coming from space fans.
Sorry, passenger AIRPLANE, not airline.
See, I don't think that solves the problem, because this takes place at a lower level. The issue isn't that Lockheed as a whole can't pivot. It's reasonably well diversified across a lot of military lines already, so the shareholders will be fine. It's that Lockheed's divisions can't pivot because they're doing specialist military stuff. Even if Lockheed adds a toaster division, it's not like they can just move people from the F-35 division to the toaster division without too much hassle. They can't even do that between the F-35 and F-16.
Does the Pentagon selecting Sierra Nevada for the E-4 replacement at least hint at increasing competition on the supplier side?
For my money, it seems like the biggest possible improvement is putting more responsibility (but also more freedom) on the government program administrators. As it stands, it seems like they're basically immune to the substantive outcomes of the project, as long as it's procedurally compliant. You can sort of make the case for that isolation with complex new platforms where there is a lot of uncertainty (and the development periods are so long that nobody is really in charge for the duration of the program), but if you look at something like KC-46 or the VH-92, it seems less defensible.
To Bogdanov's point, I think there are two other values you get from splitting procurement: 1. More products being tendered means you stake less on each project - one of the issues now is that new platforms are basically a once in a generation event, so whoever wins NGAD or F-35 has a clean revenue stream for the next thirty years, but whoever loses is possibly out of business, or at least is going to let their fighter engineers go. Some of this is the natural result of trying to make everything Joint, and the increasing complexity of developing incremental improvements, but some of it is a policy choice. There are second order considerations around the incremental sustainment costs, but for a sufficiently large fleet I don't think that's as material as people make it out to be. 2. Clarity of requirements and single decision maker leads to better designs and avoids the design by committee problem that has struck the F-35, NH-90, and the early incarnations of the F-110 and other platforms.
Splitting the Navy geographically doesn't make sense, but you wonder if you could split it by surface combatants vs amphibs, or some other axis? The carriers and subs are already off in their own world, but for the rest of the surface fleet there is maybe some more room for innovation.
Worth noting that many defense procurement system manage first part (mess) without managing to actually work.
Not sure how much it is matter of USA system not being so bad after all and how much is solved by flooding it with money until it starts working.
I certainly agree that the Jones Act hasn't prevented the death of US merchant shipbuilding/US merchant marine, but I'm less sure that it caused their death.
In theory, the Jones Act is acting as a (very inefficient) subsidy for the US merchant marine/merchant shipbuilding. In practice this results in a pressure to use rail or trucking, or to make domestic shipping technically international (ie Alaska cruises stopping in Vancouver).
I can see the argument that this is reduces investment/jobs/etc in US ports (since we're using rail/trucking instead). It's less clear to me that it causes a reduction in the US merchant marine/merchant shipbuilding, since the counterfactual seems to be more foreign ships/foreign crews serving the domestic shipping market.
I'd love to hear more detail about your thoughts here, though!
@Lucas
I assume the part of the act that bean is talking about is, "US built ships must use 100% US built parts (some of which are wildly above rest-of-the-world prices)"
For most parts of the chain, it isn't so much a subsidy as 'the act giveth and the act takest away'
@ike
I can see that that would increase the cost of US-built vessels, which increases the cost of domestic shipping (thus pushing shipping to rail/trucking).
I don't believe that there's a similar restriction on US-built vessels serving non-domestic traffic, and there's ~0 of that. If so, we shouldn't expect there to be ~any uplift in US merchant shipping production by repealing the Jones Act, unless it would only remove that restriction and not the US assembled restrictions.
Maybe this mattered during the decline of US ship production? ie US diesel engines became uncompetitive, but US ship assembly would still have been competitive if both sides of the market were able to buy the same designs?
I'm not sure I see it. Maybe a better designed subsidy regime would have done it, but we see very broad trends in reduction in heavy industry in the US. I agree that the Jones Act is emblematic of our failures to have good economic policy, but I'm not sure that it's solely (principally?) to blame for the decline in US ship building.
Again, better historical details very welcome! I'm very aware of the limitations of my first principles analysis here.
That's socialism! You pinko commie hippie. :-p
In all seriousness this does appear to be an area where the usual market signals capitalism relies on are mostly absent so socialism might actually make sense. The weakness of the Soviet system wasn't that MiG or State Factory Number ### was government owned, but that the non-defense economy didn't have private ownership of the means of production.
Bogdanov:
A country that makes its own IFVs is much less likely to be interested in buying Bradleys than F-35s.
The more complex something is the more the US will export because there will be less countries designing their own that the US will be competing with on the export market.
@redRover
Maybe, although in fairness I suspect that was largely because the Pentagon really wanted it to be fixed-price and Boeing really didn't.
That's a pretty good suggestion, yes. The problem is that giving them more freedom is politically difficult.
@ Mateusz
Some of both. On a global basis, I'd put us middle of the pack, although some of that is working at a scale nobody else we have visibility on is, which lets us get away with things other people can't.
@Lucas
A reasonable point, but I strongly suspect it contributed to the death spiral it fell into. In the short term, it's pretty much always more profitable to concentrate on the protected niche instead of trying to innovate and compete with the wider world. But over time, you get more and more expensive and we end up where we are now, with a tiny number of new merchant ships ever year for routes that absolutely require them. In any case, the act these days is a nature reserve for passenger pigeons, and we should get rid of it. I will let someone who has studied this more argue the case.
@Anon
I do actually think that this is actually one of the (IMO rare) cases where doctrinaire anti-goverment stuff has screwed us up. For instance, I believe a lot of the RN's technical staff got purged by Thatcher for essentially ideological reasons (it didn't help that they'd won a fight over one of the escort classes vs the private shipyards).
In a lot of ways, I think it's less that they're absent and more that they point in the wrong directions. For instance, a government is going to want to have reserve production capacity sitting around, even in lean times, because when you need it you really need it, while a private contractor is going to see that as waste. (Of course this is about British vs American naval ordnance procurement in WWII. What could have made you think I was talking about artillery shells for Ukraine?)
It's cold comfort, but at least we can take some satisfaction from the fact that everyone else manages to screw it up worse. Whenever I start to feel too bad about how bad US defense procurement is, I look at the British or the Belgians. And whenever I start to feel bad about how bad NATO's defense procurement is, I look at Russia, or India.
Chinese defense procurement is such a black box that it turns into a sort of Rorschach Test. Do we look at the progress they've made with airframes and shipbuilding, and say that their procurement is good? Or do we look at their persistent weaknesses in e.g. submarines or jet engines and say that they're worse off than Western nations? And how much of a discount do we apply for corruption?
Who has the best defense procurement? Israel? France?
In addition to Israel and France, I'd also suggest that South Korea does a decent job of procurement. Turkey is also weirdly effective. Weirdly, because the rest of their economy is kind of a basket-case of mismanagement, but they still manage to manufacture some pretty good drones, and they're pretty far along with developing an indigenous fighter aircraft after being kicked out of the F-35 program.
I notice that almost every major contract greatly exceeds the cost estimate.
Suppose the government went to a fixed price contract for almost everything except the cutting edge development stuff. Wouldn't this force defense contractors to estimate costs much better?
From the government's point of view, the fact that Boeing is taking losses on the tanker program is good, because Boeing f*cked up the tanker program. And if LM gets better at making F-35s, let them keep the profit.
See, the problem is that I don't think defense contractors are particularly incentivized to estimate cost badly now. (I have no idea what Boeing's logic was on bidding the way they did for a lot of the current programs giving them trouble. Loss leaders are a thing, and that's almost certainly what was going on with the MQ-25, but I don't think they were expecting to do as badly as they did on several of these contracts.) But anything beyond "build more of this thing you are already building" is going to have a fair bit of risk involved, and the contractor isn't necessarily the right person to take that, and if you insist, they'll try to charge a premium for it.
Yes, this is true from the government's point of view in the short term. One of the theses of this post is that we shouldn't only look at the government's short-term POV, because doing stuff like this poisons relations between government and contractor and makes things worse in the long term.
The problem is that if LM (or anyone else) makes too much profit, people have a strong tendency to decide that it is doing something wrong and needs to be punished. In theory, I completely agree, but in practice, if I were LockMart's CEO, I'd be very skeptical that the result wouldn't be to paint a big target on my back during the next election.
(That said, I am very much in favor of doing this for specific things. Specifically, missiles, where I would love to call Lockheed Raytheon et al and say "here is the price for each of your missiles. We will pay you this price for each one you deliver by this date. Go forth and make lots of missiles." But that's probably in violation of like half the procurement regulations.)
Fixed price works for competitive low risk situations with lots of bidders on semi-commoditized work, but for most other situations you end up with a lot of padding to compensate the contractors for all the risk that they assume.
What is the price for which you are willing to play Russian roulette? Even in mature, extremely well-understood fields such as construction, cost overruns of a magnitude that kill the contractor are far from rare. There is a finite, market-clearing price for the contractor to risk its existence as a company, and in this schema, the buyer will pay it -- and then even (especially?) in an ideal world, where everybody estimated all costs accurately, in the large majority of cases the project will come in close to the median estimate, and the buyer will be left confused why the contractor earned apparently excessive profits.
Furthermore, project costs have a remarkably fat tail (well past 100% overruns), where the attempted risk transfer fails. By the letter of the contract, the contractor goes bust well before the project is anywhere close to being finished, and the buyer gets to decide whether to walk away with nothing, to pay another contractor to finish the job, or to rescue the first contractor.
The overall lesson is that the government, mostly by the virtue of being larger ("better-capitalized" is a strange way of putting it) is more able to deal with the cost risk of large projects -- which risk is surprisingly large even in completely mature fields.
Pick the top 20 military programs. Anyone wanna guess how many were on budget? Were within 20%?
I think you all might be missing the point. Right now we have a system where we make a decision based on a cost estimate that we know is crap, but we use it anyway. Consider the Constellation class. It's a conventional warship with no particular cutting edge features. It's currently 40% over cost, and they have not yet even delivered a ship. The KC-46 is a rather normal derivative tanker aircraft, and Boeing has spent 250% of their estimate, and it still doesn't work right. And that's how it always goes! ALWAYS
On rather normal programs like these, let the vendor take the risk. And if they raise the prices because of the risk, that's reasonable. But it doesn't actually cost the DoD because they were NEVER going to get it for the original price anyway.
But what about the vendors? They can either build to the cost estimate they provide, or go out of business. Replacing companies that cannot do the job right with companies they do ... win! And if LM gets replaced by Sierra Nevada, that's OK.
Pick one of two choices: Keep going the way you are, or make a change.
(cooling down now ...)
With so few companies a bankruptcy may mean monopoly or even leave you with no one that can do the job.
New competitors may appear, but that could take time.
KC-46 in particular seems odd, given that Boeing had already delivered the very similar KC-767 to Italy and Japan in the 2010 timeframe.
However, it also seems like because the downside risk is so large (250% over budget), you would end up paying say 200% over the nominal budget on most programs as the risk premium, which seems excessive. Or maybe you end up with companies trying to get "project insurance" on the secondary market to spread the risk, where they only charge say 150% of nominal, give 40% of it to the insurer, and then keep the rest.
Put yourself in the shoes of a stockholder in one of those vendors. This seems like a terrible plan, because you run a serious risk of your investment going to zero, because a bad program can blow the company up. Boeing's stockholders are certainly not happy about this.
(Also worth noting that the cost increase on the FFG-62 has gone hand in hand with a sharp decrease in commonality with the original FREMM design. I suspect the original bid assumed the original commonality, but then the Navy wanted to redesign everything. Going to fixed price is going to make contractors even more paranoid about locking down the contract so they can really jack up the price when things change, because there are always changes.)
See, I think is exactly the attitude I was trying to warn against when I wrote "Anything that involves pointing and yelling at defense contractors might make people feel better, but it isn’t going to actually solve the issues." I don't think that Sierra Nevada has some magic that LM doesn't, and if they do, it certainly won't survive scaling up to LM's size. A lot of the industry is shaped by structural forces, not companies being bad and needing to be replaced, so all this really does is make being a shareholder in defense stocks or an employee of a defense contractor a worse deal. That's good for the government in the short term, but I think bad in the long term, because there's a lot of institutional knowledge at the existing contractors.
Those are actually 767s with a boom installed. The US, as is its way, demanded a lot more changes. The US defense procurement system is set up to buy the best and only the best. The problem is that many times you don't need the best, and yet the system doesn't know how to do that. The continued existence of the B-52 is probably the best evidence of this. It should be fairly trivial to build something that fills the same niche at a lower operating cost (may I offer you a BKC-46?) but this basically isn't on the table because it's not what the system is set up to do.
@bean
You wonder how much practical difference it makes between the KC-767 and the KC-46. Like, are we getting a lot of functional improvements (e.g., higher transferable fuel, shorter field requirements*, etc.), or is it just gold plating that has limited actual value?
*Though how much is this actually worth? It seems like ~all of the regular air force bases are in the 9k+ runway class.
I think the solution lies in the realm of accountability. Yes, it exists to some extent on the contractor side of the equation, but is almost non-existent on the government side. And this applies to both the federal bureaucracy/military and elected politicians, who enjoy a 98+% incumbency rate. Promotion within government is almost always a matter of political fidelity rather than merit. The American voter needs to be able to fire incompetent bureaucrats, military chain of command, and politicians. But voting doesn't seem to accomplish that goal, so what is the alternative? The Soviet Union collapse under its own weight. Is that our fate as well?
To quote wiki: “The KC-767 offered for this KC-X round was based on the in-development 767-200LRF (Long Range Freighter), rather than the -200ER on which Italian and Japanese KC-767 aircraft are based, differing by combining the -200ER fuselage, -300F wing, gear, cargo door and floor, -400ER digital flightdeck and flaps, uprated engines, and “sixth-generation” fly-by-wire fuel delivery boom.” Also, I believe there was more in the way of military avionics and countermeasures than the KC-767s carried.
There’s also the issue that one man’s gold-plating is another man’s functional improvement, and the US has historically been quite bad at thinking about those kind of tradeoffs. (I think the best example is the F-111, which had a requirement for low-level supersonic flight that was enormously expensive essentially because it let the Air Force order a new plane instead of an F-105H or an A-6 variant.) Let’s say that the KC-46 has 10% higher transferable fuel than the KC-767 because of the mods (yes, I know it’s more complicated than that, but go with it). Is that gold-plating? In some cases, that won’t matter, in others, it might let us fly fewer aircraft on a given mission. And what’s it going to cost. I think we can all agree that it is a bad plan at 2x cost, but what about at 10% extra cost? What if Boeing tells us it’s only going to be 5%, but we know that this raises the possibility of major cost overruns? How do we know how much it raises it? And keep in mind that different parts carry different risks. Pretty much all of the issues I’ve heard of have come from the boom, not the wing/flaps/etc. So those were plausibly straight wins, while the fancy boom thing was worse than having a guy looking out a window at the back. (The reason to replace the window is that the required structural mods cost payload weight, which means less fuel.)
@bean
Agreed that improved fuel transfer weight is worth something, though it may be hard to quantify.
My point was specifically around mostly theoretical use cases where we end up paying a lot for capabilities that are ~never used (like short take off rolls for tankers). The one that comes immediately to mind is the C-5 landing gear - it's occasionally been used off-runway (see https://en.wikipedia.org/wiki/File:Galaxy_Antarctica.jpg ), but would we really have lost much combat capability by targeting 757 level runway pressure? All of the wars we've been part of have featured very generous air force bases, and to the extent that we're thinking about higher intensity conflicts that (fortunately!) haven't occurred, they're in places with extensive airport networks. Moreover, how likely is it that the commanders would risk the C-5 for direct to the improvised airstrip flights, vs. transferring to a C-130 or similar?
That's an entirely reasonable point, and one that is I think always going to be something of a judgement call. Obviously, we can't just say "well, if we never used it, it's a waste and we shouldn't have bought it in the first place" because sometimes the dice just happen to land one way (Iowa never fired her guns in anger in the 80s, but all three of her sisters did essentially as a result of chance) and sometimes the thing can make a significant difference itself (the nuclear arsenal springs to mind here). And there are a couple of layers to dealing with the issues raised. Rough-field capability is the sort of thing that the operators are definitely going to add to their wish list, because it sounds like a nice thing to have. But it's not really their money that they're spending, so they have a tendency to ask for the moon. Think-tanks and outside analysis can push back some on this, but even that kind of analysis is going to rely on the assumption that we know how much the capability will actually cost ahead of time, which, as the C-5 example demonstrates, isn't necessarily a good assumption. I'm not sure there's even a good way to deal with this, because it's entirely plausible that 9/10 of these minor capabilities are good/neutral, and the last one blows up the program, but you can't really predict which is the 10th ahead of time and you don't want to get rid of all 10. (The ejection seat on the T-7 is the most obvious recent example.)
@bean wrote
I don't care why they suck so much at estimating costs, and always in the same direction. For all I care, they can take their previous methods and just add 30%. But it's not small errors, and its always in the same direction.
@bean wrote
I agree it's structural forces (I would have said incentives). So I wanna change those.
And yes, post contract changes add to cost. I assume the contracts all say "changes at the government's expense +10%". If the contractor is forced to provide a price for every major change mabe DoD department managers will stop gold plating, which is another advantage!
As to whether contractors will be willing to sign fixed price contracts .... the total spend is well over $100B so someone will want that money.
I'd be interested if someone has a better idea.
The problem with this is that when they do get things right (which happens more often than you'd think) people will scream bloody murder that the contractor walked away with a 40% profit margin. I got chewed out over 15%, where the government got to keep the savings. There really isn't a way for the contractor to win here.
No, the contractor does that already. It's negotiated, not pre-planned, because nobody is sure what the government will want.
Here's another example of why dealing with the government is so aggravating. The article details how a cybersecurity lab at the University of Georgia is in legal trouble for not following certain paperwork security protocols, such as installing antivirus on endpoint machines.
Now, I don't want to defend the lab too hard. Obviously, claiming that you're following the program requirements set forth in a contract and then not actually following them is fraud. But missing from the discussion is why the program requirements were necessary in the first place. Did that lab really need to run antivirus on all of its endpoint machines? Couldn't it have put in place other mitigation measures (such as extensive use of virtual machines, etc) that would have rendered antivirus unnecessary? Furthermore, it seems like lab actually fulfilled all the deliverables of the contracts; they're only in trouble because they didn't do so while following all of the contract's stipulations regarding cybersecurity.
I strongly suspect that the reason the DoD contract specified that all endpoint machines need to run antivirus isn't due a detailed analysis of program requirements, but rather because some DoD functionary cut-and-pasting cybersecurity "best practices" without thinking about whether those requirements were appropriate for a security research lab environment, where, quite possibly the antivirus would have gotten in the way, by raising false alarms or interfering with in development software. This gets back to the point bean raised about having to accept a bid from "three raccoons in a trenchcoat". The DoD office responsible to creating these contracts is, in fact, being paid not to think. If they conduct a detailed analysis of what they're asking the lab to do, grant a variance from the cybersecurity best practices, and then there's a breach or a hack of some kind, then the program office is in trouble, because they issued contracts that didn't require "basic" cybersecurity rules to be followed. However, if they stipulate the standard set of cybersecurity rules in the contracts, and then push labs to fulfill the contracts regardless of whether the cybersecurity rules are a hindrance or not, then they have effectively transferred that risk onto the labs.
I fear that the lesson researchers will take away from these lawsuits is that there's just too much hassle and risk of BS when dealing with the government, and that they're better off dealing with private industry instead. And then, in another two or three years we'll have another set of hand-wringing seminars at think-tanks discussing why it's so difficult for the government to get good cybersecurity.
That's true of all projects, regardless of whether the project is for the government or private industry. It turns out that unknown unknowns exist, and you can't account for them by adding a fixed percentage on top of your estimate. If you could, they wouldn't be unknown unknowns, they'd be known unknowns.
I've worked on a software project that was scheduled to take six months, but which was still going five years later when I left the company. And that was a "relatively straightforward" task consisting of taking four or five different internal company portals and consolidating them to use a common framework. Project estimation is hard, and, as per my previous comment, the fact that the government will impose arbitrary "best practices" regardless of their applicability makes it even harder.
This is a great write-up of the experience and incentives from the contractor's perspective.
I'm a bit disappointed that so many of the comments seem to be focused on poor execution by individuals in government and contractors. That definitely happens, but I think it's mostly downstream of the problems with the procurement process overall.
The big-picture problem is that the government is an absolutely terrible customer. The procurement process takes years at a minimum, and the levers available to government program managers and project directors severely constrained by the narrowly-scoped conditions on each "pot of money" allocated by Congress.
From the Acquisition Talk blog[1]:
In other words, to buy ~anything substantial, the full decision has to work its way up the entire DoD hierarchy, be communicated to Congress, survive the appropriations process (a well-known habitat of the Good Idea fairy), be bid out according to FAR rules, and then finally managed and executed. There is a loss of information and context at every stage of this process.
This process has a bunch of consequences for how the work actually gets executed:
On the contractor side, you essentially need a bureaucracy that's specialized to navigating this process. "Being good at navigating the procurement process" is a mostly distinct skill tree from "being good at building planes/ships/missiles/whatever", but you need both.
Within the contractors, the individuals that are good at the procurement bureaucracy often rise to upper management (after you, you did need them around to win contracts), and then proceed to define culture for the entire organization. This is a primary reason why you see many contractors gradually "evolve" from an engineering-focused organization to a more generic "big company"/"big organization" culture.
It requires a huge amount of effort to provide all the necessary context to N layers of time-constrained decision-makers. The media often makes a big deal about how much the major defense contractors spend on lobbyists, and draw the implication that they are essentially bribing members of Congress to win projects. In reality, members of Congress are often shockingly poorly-informed about the programs they have oversight for, and are routinely wrong in critical ways about straightforward factual questions (what is this program for, what are the problems, why does the Navy want it, etc.). It takes a full-time staff of lobbyists just to educate members of Congress and their staff on these issues, and correct all the misunderstandings so that decisions can at least be made on correct information.
As bean alludes to, contracts generally need to be awarded via a standardized process, with the contract generally going to the lowest bidder that meets all technical requirements. This often leads to absurd circumstances where the DoD staff know that the bidder isn't equipped to execute on the work, but they have to award it anyway and let them fail. If you need to spend a small amount of money on something (e.g. parts or a one-off repair), then you additionally are required to award it to a small business that meets various requirements[2].
Once money is appropriated, decisions are very "locked in". Even if the contractor and the project staff in the DoD agree that some change would be beneficial to the project, the project staff might not have any "levers" available to actually make the change. The funds might have been appropriated with language that would not allow them to be spent on a different approach, or the change might trigger FAR requirements to re-bid out part of the project.
It's a serious challenge to sustain smaller contractors through the years-long journey to a Program of Record. The DoD has various pots of money to sustain contractors through this, but they all require "selling" to even more stakeholders. The primes are diversified enough to be able to carry projects through multiple stages of maturity, but it's very difficult to sell shareholders on a bet-the-company project that won't even have a signed contract until years in the future.
Essentially the entire process assumes that all acquisition decisions can be "centrally planned" years in advance by a relatively small number of people in DC. This is more than a little ironic, given that the process was put in place by politicians focused on building up the military against... Communists advocating a centrally-planned economy.
Eric Lofgren's blog Acquisition Talk is a great deep dive on the acquisition process and some of the highest-leverage opportunities. for reform: https://acquisitiontalk.com/. At a high level, we need to "push down" more responsibility to lower levels of the hierarchy that have more context about what the government needs, and enable funds to be more "fungible" across different projects. But asking Congress to relinquish fine-grained control in this way is a tough sell.
[1] https://acquisitiontalk.com/budget-reform/
[2] https://www.gsa.gov/small-business/register-your-business/explore-business-models/setasides-and-special-interest-groups
Developing a program of record is inherently high risk, but you also wonder how much money is spent on the R&D phase, vs. once the program is mature enough that they should know with some certainty what they're doing. Cost overruns during the R&D (or T&E) phase are unfortunate but somewhat understandable, whereas production or sustainment cost overruns seem less explicable.
The other part of it though is that the DFARs are in some ways very constrained, but in other ways it seems like a lot of it is cultural. The DFARs have enough exceptions that for sufficiently urgent needs they can avoid a full F-35 level program contracting effort.
To be clear, the main "risk" I'm highlighting here is that the DoD won't decide to buy your product before you run out of money. The "Valley of Death" is a well-known problem. Spending ~2 years with almost no revenue would kill most businesses.
This is in addition to the technical and product risks in the R&D project itself, which would be true in a commercial context as well.
This assumes a clean divide between the "R&D stage" and the "production" stage, which almost never exists in practice.
The reality is that:
Learning to produce something at scale is a form of R&D of its own. Some things that work when you're building a prototype in a machine shop don't work as well once you scale them up, and you need to iterate a bit to iron out all the issues.
The requirements are never actually finalized. You will always be making some improvements over time, some of which will be costly and uncertain projects in their own right. In the Navy's context, no two ships are actually identical - every new ship generally has some design modification or difference in equipment compared to the previous ship of the same class. Some of this is "pull" from the Navy (often based on experience with existing vessels), while other changes are "push" from the contractor (based on experience building the previous vessels, or new technological capabilities).
There is a feedback loop between budget and change orders. If the program is coming in under budget, then the customer will be more open to modifications that might improve capability at some additional cost. If the program is already under scrutiny because it's massively overburning the budget, then they'll mostly try to avoid these changes where possible.
This may be true, but the culture runs deep. As a contractor it doesn't matter if the customer technically could use an Other Transactions Authority to buy your widget. The problem for you is that they almost certainly will not do that. The former general counsel of DARPA gave an interesting interview about the struggle to get other agencies to use OTAs: