The most polarizing man in Barack Obama’s administration—besides the president himself—is calling it quits.

Attorney General Eric Holder has been at the center of some of the administration’s biggest scandals and controversies. Conservatives have long hoped he would step down—a House resolution demanding his resignation had well over 130 co-sponsors—but it never seemed likely to happen for one reason: the confirmation hearings to replace him would be a public relations debacle.

Faced with the possibility of a Republican-controlled Senate, however, Holder had two choices: get out now or remain attorney general until Obama’s second term is up. So America’s long national nightmare is almost over—but the White House’s may just be beginning.

Let us count the ways Holder has generated bad press for his boss. There is Operation Fast and Furious, a gun-running scheme that allegedly began as an elaborate sting operation to allow firearms straw purchasers to lead authorities to major gun traffickers. It ended up with the feds losing track of the guns, which were subsequently used in crimes—including murder—in both the United States and Mexico.

The death of federal agent Brian Terry blew the lid off Fast and Furious. In February, one suspect received a thirty-year-sentence for Terry’s murder. But the story remained safely marginalized in the conservative media, with the exception of dogged reporting by Sharyl Attkisson, formerly of CBS.

Despite representing the most transparent administration in history, Holder was widely accused of stonewalling congressional investigations into Fast and Furious. He was eventually held in contempt of Congress by the House, to which he responded by claiming to be a victim of partisan persecution.

Not long before news of his resignation came down, a federal judge denied a Justice Department request to delay the release of documents pertaining to Fast and Furious. The Obama administration had asserted executive privilege over the documents.

While Operation Fast and Furious was dismissed as a right-wing concern, Holder had no such luck with the Associated Press scandal. The attorney general was intimately involved in the seizure of phone records for more than twenty lines belonging to the AP. His Justice Department dug through the personal emails of Fox News’ James Rosen.

“Search warrants like these have a severe, chilling effect on the free flow of important information to the public,” First Amendment lawyer Charles Tobin told the Washington Post. “That’s a very dangerous road to go down.” The veracity of Holder’s testimony to Congress about the intention and scope of its reporter probes has been widely questioned.

The Holder Justice Department earned a reputation for being aggressive in the enforcement of laws it liked and more selective when it involved policies with which the administration disagreed (immigration restrictions, the Defense of Marriage Act).

Holder initially resisted congressional efforts to get more information about the Obama administration’s policy of extrajudicial killings for counterterrorism purposes. Did this include Americans on U.S. soil? And if so, under what circumstances? This led to the filibuster of CIA chief nominee John Brennan and a national debate on drones.

Holder, the first African-American attorney general, was far quicker than his boss, the first African-American president, to racialize national controversies. He was far less measured in his comments about the police shooting of Michael Brown in Ferguson, Missouri. He famously called America a “nation of cowards” with regard to race.

“Though this nation has proudly thought of itself as an ethnic melting pot, in things racial, we have always been and I believe continue to be, in too many ways, essentially a nation of cowards,” Holder told Justice Department employees.

Conservative columnist Michelle Malkin retorted that Holder’s proposed dialogue “means the rest of us shutting up while being subjected to lectures about our insensitivity and insufficient integration on the weekends.”

Yet Holder does deserve credit for trying to minimize some of the damage done by the federal drug war, leading the charge on reforming excessive criminal sentencing guidelines and seeking ways to ensure that young people—many of them minorities—are not permanently damaged by early decisions.

Even here, Holder’s unilateral methods could be questioned. But it is an issue where he also worked with Congress, including Tea Party Republicans like Ted Cruz, Mike Lee and Rand Paul.

When President Obama’s nominee for the next attorney general testifies before Congress, however, lawmakers can be excused for wondering if they are getting the full story. There’s good reason to doubt they ever did with Holder.

W. James Antle III is editor of the Daily Caller News Foundation and author of the book Devouring Freedom: Can Big Government Ever Be Stopped? He tweets at @jimantle.