The Lexington Institute’s Dan Gouré says that the much-ballyhooed Third Offset “will fail unless it first defeats the DoD's acquisition system.” The department has again missed its goals for competing enough contracts. I myself have lamented how broken the acquisition system is. But as a retired Air Force general patiently countered to me last week, it may not be so broken as it once was. Perhaps, as outgoing Air Force acquisition chief Bill LaPlante told Breaking Defense in his last interview, “we used to suck, and now we don’t suck as much.” What’s his evidence? “Our net costs continue to come down,” he told National Defense; “I have all the data.” That’s encouraging, but addressing cost specifically does beg the question about how many ways one can suck, and what exactly constitutes not sucking.

Take the convenient target of the Joint Strike Fighter program. Three years ago, the always-balanced Winslow Wheeler called the JSF “the jet that ate the Pentagon.” Today, the USAF’s F-35As may still cost a lot, but they're costing less, and projected to cost yet less. Just how much less is still debatable, but the rest of LaPlante’s data (and projections) are easy enough to imagine. The current actual costs and future expected costs of current programs are decreasing. The expected costs of future programs may also be decreasing. There may be many reasons for this relative success, but I can highlight two. Bureaucrats across the department have learned lessons about the challenges of concurrency in development and production, and of trying to bundle all that capability into a single plane or program.

That lesson on concurrency, though, does point to a second classic category of procurement performance: time. To start with, as Fred Beach of the University of Texas has highlighted, time is money too. Over the past sixty years, the Defense Department has experienced a steady increase in bureaucratization and a commensurate decrease in technical proficiency of managers in its processes for weapons development. The resulting do-overs have sometimes come with big price tags. There’s an even bigger problem for the friendlies: as David Deptula once remarked, “Al Qaeda doesn’t have a JCIDS process.” Overall speed-to-fielding still looks bad: the Defense Department is now about twenty years into that JSF effort. But as the program office and Lockheed Martin are suitably quick to note, the schedule hasn’t slipped much further in the past five years. So, in addition to LaPlante’s cost data, we should also want to see the numbers on whether particular parts of the acquisition process are moving faster.

The lesson on the diseconomies of scope points to a third and thorniest problem set: quality. Wars are audits, and while it’s fortunate that the big ones don’t come so often, that also leaves the most important capabilities untested. Has Chinese Intelligence really stolen the the JSF's source code? Just how much would that matter? The maintainability of the F-35 hasn’t been demonstrated, so will it really be more affordable than the maintenance-budget-eating F-22? The newer fighter jet is also not actually invisible, so just how deadly will it be? With years of exercises involving the older plane, the USAF can have some confidence about the lethality of a stealthy fighter with AIM-120 missiles and an APG-77 or -81 radar. The trouble is that the findings are not broadly sharable, and exercises are just that, so this last hypothesis is not so testable.

None of these problems are new. Take another favorite target, as David Axe recently highlighted in Robert Farley’s findings about “what happened after Peak Battleship,” that point in 1918 “when 118 dreadnoughts served in 13 different navies.” Sixty-three were in service in 1939, and another 24 were built in the war to replace the 23 that were sunk. What few expected was that almost all of those were taken by aircraft and submarines. After 1945, the dénouement was swift; within a decade, almost none were left. In retrospect, we knew how long battleships took to build (that’s time), but the problem wasn't primarily that they took too long. We also knew pretty much about how well they could shoot (that’s quality), but the problem wasn't just that the guns couldn’t range as far as aircraft and missiles. The main problem was cost: all that steel and fuel and crews was too much to justify within the residual mission of shore bombardment.

In pounding the table about data, LaPlante laments the overuse of cliché and innuendo in criticizing procurement officials and their performance. As he focuses on cost, one could claim that he’s looking for his keys under the lamppost—measuring what’s measurable, whether or not it’s relevant. What’s encouraging—and in turn lamentable about his departure—is LaPlante's aim to be “confident [but] not cocky about anything.” So whether we were to analyze the knowable Montana-class battleships or some unknowable future F-35Ds, we should pursue the right combination of due diligence and intellectual humility. We should strive for relevant schedules, and care how knowable our costs are, but acknowledge that the unknowables of quality are what will eventually hurt us.

James Hasík is a senior fellow at the Brent Scowcroft Center on International Security, where this piece first appeared.