These powerful developments led to a search for answers and alternatives in Washington. But continuity proved a persistent habit within the government. An obstinate Donald Rumsfeld was a formidable obstacle to fresh thinking. Nor were the Joint Chiefs of Staff prone to devote more resources to Iraq. The commanders on the ground in Iraq, Generals George Casey and John Abizaid, advised that Washington should maintain its patience and not discard the current train-and-transfer strategy. The president’s closest foreign-policy adviser, Secretary of State Condoleezza Rice, had grown weary of the Iraq entanglement. And a highly praised bipartisan study group led by foreign-policy mandarins James Baker and Lee Hamilton had endorsed a path of recalibration and retreat. For the president to reject such a consensus would be viewed by many as an act of peculiar defiance.

Certainly, there was presidential frustration and a faltering policy. But a switch to a successful alternative requires a high-ranking official who rejects the prevailing consensus and offers a plausible counterstrategy. Stephen Hadley, Bush’s national-security adviser, assumed that role in the Bush White House. As Bush recalled later, “He knew my anxiety. He knew my intensity on the issue. He read me like a book.” Hadley did not originally devise the actual strategy, but he provided high-level sponsorship inside the administration for those interested in rethinking the policy. Under Hadley’s direction, a group of National Security Council staffers, assorted former generals and think-tank analysts began formulating a different approach. The views of the White House planning group now coincided with the perceptions of important military officers such as David Petraeus, who already were thinking of a different policy involving additional troops to bolster a new counterinsurgency strategy.

Once the new strategy was articulated, Hadley proved critical in getting it adopted. To be sure, he was responding to urgent concerns of a beleaguered but determined president. Still, he had to overcome bureaucratic barriers, determined opposition from key military commanders and U.S. diplomats, and a Democratic Congress that was losing faith in the war. Throughout this exercise, Hadley proved a pivotal figure. He could have urged the president to give the local commanders more time. He could have joined the withdrawal chorus. Instead, he opted to salvage the president’s policy.

The reigning assumptions of the previous American strategy now came crumbling down. The conflict had been seen as a battle between the central government and an insurgency seeking to displace it. But now it was considered a sectarian conflict between Sunnis and Shia, with Al Qaeda’s violence aggravating those tensions. U.S. forces were seen not necessarily as catalysts for violence, but as agents whose more active participation and patrol could mitigate the sectarian war. The new strategy recognized Iraq’s diverse landscape and tailored tactics to that diversity. Instead of emphasizing top-down reconciliation, the new approach focused on bottom-up accommodation. Population security became critical as U.S. commanders reached out to Sunni tribal leaders and even aligned with their militias against Al Qaeda. The American presence on streets and in neighborhoods increased to give protection to Iraqis and foster confidence that they could escape their predicament. This naturally entailed not just more troops, but also different commanders and ultimately a different secretary of defense.

The surge, which increased U.S. deployments in Iraq by some twenty-one thousand troops, provided an umbrella of security that allowed nascent trends to mature. In reality, between 2004 and 2006 Al Qaeda’s harsh tactics alienated Sunni tribes at times, causing them to reach out to local U.S. commanders. But subsequent efforts at cooperation failed because the United States did not have the capacity to offer the necessary protection to cement deals. The Sunni tribesmen needed arms and a degree of support in order to establish a force able to patrol their areas and protect them against Al Qaeda assaults. The new strategy allowed Sunni tribal leaders, already chafing under the threat of Al Qaeda, to defect and thus allow moderate Sunni opinion to gain greater force vis-à-vis radical elements.

The surge probably would not have succeeded without the central government of Nuri al-Maliki committing itself to declawing Shia militias, particularly the movement of Moktada al-Sadr. In March 2008, the Shia government of Iraq moved beyond its presumed sectarian affinities and launched an assault on radical Shia forces in Basra. Thus, the government proved to skeptical Sunnis that it favored national stability over sectarian empowerment.

The surge strategy emerged through a series of thoughtful exercises that assessed all options and their potential for success. However diligent the process may have been, the notion that additional forces protecting population centers were essential for stability had been around for some time. The Pentagon could have embraced that strategy at the outset of the invasion and thus provided Iraqi institutions time to establish themselves and their authority. But, because Washington wanted regime change on the cheap and with a limited footprint, it pursued an occupation policy that proved disastrous.

The future of Iraq remains uncertain. However, there is no doubt that a change in strategy salvaged the American enterprise and saved Iraq from collapsing further into a horrific civil conflict, with America caught in the conflagration. It required not only presidential anguish and leadership but also an incisive policy maker to translate concern into a new policy. The surge brought about a quick turnaround. Had the level of violence not declined as quickly as it did, George W. Bush likely would have met Nixon’s fate: a president who launched a successful counterstrategy only to run out of time.

IT IS rare for presidents to take direct command of their foreign-policy failures. Presidential frustration doesn’t necessarily yield alternatives. Lyndon Johnson indisputably was frustrated with his Vietnam strategy, as Barack Obama must be with his Iran policy. Yet change doesn’t always emerge simply because a chief executive is exasperated and appreciates the cost of failure. Nor will a bureaucracy, set in its ways, often come to his rescue. An extraordinary alignment of interests and opportunities must come together for the ship of state to change direction. Continuity in the midst of failure has been more of the norm than the exception.

The philosopher John Dewey said that “institutions tremble when a new idea appears.” In their own ways, Dean Acheson, Richard Nixon and Stephen Hadley not only transcended such fears but also actively pursued new ideas. Acheson today stands as one of the preeminent historical figures in American foreign policy primarily because, at a key historical juncture, he turned his assumptions into questions. Nixon proved the rarest of presidents when he discarded the conventional bureaucratic wisdom and played a direct role in devising a different path. The tragedy of Nixon’s presidency was that his disgrace and fall prevented his country from taking advantage of his successful strategy. Hadley was an unlikely catalyst for change, as circumspect and cautious men steeped in legal training usually don’t buck existing templates and precedents. Yet these very different men can lay a claim on history for accomplishing the rarest of achievements: turning failure into success.

Ray Takeyh is a senior fellow for Middle Eastern studies at the Council on Foreign Relations.

Image: Pullquote: The future of Iraq remains uncertain. However, there is no doubt that a change in strategy salvaged the American enterprise and saved Iraq from collapsing further into a horrific civil conflict.Essay Types: Essay