Politico Pro and Bloomberg have been reporting of late that the Pentagon’s Office of Test & Evaluation (OT&E) is planning a series of head-to-head close air support tests between the A-10C and the F-35A. USAF Chief of Staff General Mark Welsh had previously called that “a silly exercise”; the Lightning II, he insisted, is a combat aircraft for the “entire battlespace,” not just close air support, or CAS, for short. POGO shortly thereafter took advantage of its Google search engine to play gotcha with his “conflicting statements” on the issue. Fairly, though, the general might have used an analogy from the 1920s. In a competitive gunnery exercise between a battleship and an aircraft carrier, the battleship will win. But that doesn’t tell us to buy the battleship—just not to wind up like HMS Glorious. Unless designed carefully, such tests may not tell us much at all. So, if we were going to design such a field experiment, build a model, or hold a wargame, what would we want to know?

Start with what information is available to the pilots, whether through the F-35A’s sensors or the A-10C’s bubble canopy. Doug Birkey of the Mitchell Institute and Rebecca Grant of IRIS Research recently stressed to Defense News that close air support isn’t strictly about visually sighting and diving on targets. It’s about garnering information for precision weapons strikes. In theory, the F-35A could do that very well, so fight’s on, right? Maybe, but read on.

Because while you’re thinking about all that information, test vulnerability too. The F-35A isn’t actually invisible, and however awesome the pilots, the A-10C can only absorb so much punishment. For ground forces, reconnaissance doctrine talks about fighting for information; in the air, that can go awry very quickly. As USAF pilot Derek O’Malley and Professor Andrew Hill of the Army War College argued at War On The Rocks, anti-aircraft defenses are more lethal than they used to be, so strafing runs may be a questionable choice.

Next, handicap on cost. O’Malley and Hill followed their essay with another about how technology would solve most problems, as long as the Joint Strike Fighter was at the center of it all. In a response, Army officer Benjamin Fernandes, now a PhD student at George Mason, argued that sooner or later, sensors will get jammed, spoofed, or hacked, and a combat pilot is going to need to visually sight and shoot someone with a cannon. But except in extreme circumstances, strafing runs may be a particularly questionable choice for a hundred million dollar airplane.

Don’t discount differential training. General John Loh, a former USAF vice chief of staff, wants to retire A-10Cs, but require that new F-35A squadrons devote a fixed amount of their training time to close air support. CAS, they say, is not a plane, it’s a mission. But if it’s a mission, then training is an essential part of it. Between single-seat aircraft types, can multi-mission pilots undertake any mission as well as single-mission pilots? The Marine Corps doubts this; its single-seat F-18C squadrons specialize in air-to-air work, while its twin-seat F-18D squadrons focus on air-to-ground. Bombing in support of troops in contact can be a two-person job.

Then finally, once you’ve scienced the hell out of those tests, listen to the market. Boeing would be delighted to refurbish the A-10Cs and sell them overseas to customers closer to, say, North Korean or Russian tank battalions. With stronger language, blogger Tyler Rogoway concurs. At one point, Boeing was even discussing terms with the USAF. But is anyone buying? I’ve argued particularly that South Korea and Poland should consider buying used, but interest there hasn’t apparently been strong. Plenty of countries have armed helicopters for ground attack. Plenty of countries have turboprop-powered, fixed-wing aircraft dedicated for ground attack. Only the U.S. and Russia have fleets of manned, jet-powered, fixed-wing, ground attack aircraft, and those are designs from the 1970s. So if no one wants these surplus A-10Cs, then maybe the market knows something.

This is not to suggest that the Marines should dump the F-35B, and just talk to Insitu about weaponizing their RQ-21A Blackjacks. All the same, DARPA sometimes has crazier ideas, and last year’s wacky can seem pretty reasonable today. Long before it had such a large fleet of AH-1Z Viper (“Zulu Cobra”) helicopter gunships, the Marines' aviation leadership feared shifting away from fixed-wing fighters. In the 1950s, helicopters were pitched as the solution to avoiding concentrating the fleet for amphibious assault. At the time, the fear was nuclear bombs; today it’s supersonic cruise missiles. But the Israelis recently replaced their Cobras with missile-firing drones, so different military establishments take different views at different times on how to support troops in contact.

Even today, even the Marines aren't completely wedded to shipborne manned aircraft. The Harvest Hawk program put Hellfire missiles on Hercules tankers to turn their KC-130s into impromptu gunships. They can loiter for hours, refuel other planes, and relay communications along the way—but don’t hazard them around serious anti-aircraft threats. The point is that even a well-designed test will leave many open questions about which airplane—fighter-bomber, drone, helicopter gunship, fixed-wing gunship—will provide the strongest competition to a dedicated, armored, manned, fixed-wing attack plane in different scenarios. So choose your test carefully, OT&E. Like the USMC, the USAF needs to fill the squadrons with planes that won’t leave the guys on the ground in the doghouse, or leave General Welsh to shake a fist skyward. Curse you Red Baron!

James Hasík is a senior fellow at the Brent Scowcroft Center on International Security where this piece first appeared.