THE MACONDO oil well blowout in the Gulf of Mexico last April has consequences far beyond BP’s balance sheet—or even the potential rate of growth of offshore oil and gas production in U.S. territorial waters. It almost certainly marked the end of any possible legislation to bolster energy and environmental policy, perhaps for the rest of Barack Obama’s first term, and probably well into a second (should there be one). Above all, it may have stymied America’s first chance in forty years to approach energy independence.

We are a far cry from the president’s inaugural address, which promised to “harness the sun and the winds and the soil.” America was to be weaned from its internal-combustion-engine addiction, and power generation was to be revolutionized via renewables—radically reducing carbon emissions and oil imports.

Obama stressed the need to drastically change transportation technologies as well as power generation and distribution, forming the case for a new technological revolution that would enable the United States to reduce dependence on petroleum and harness the entrepreneurial forces of American society. He earmarked $105 billion out of the fiscal-stimulus program for renewable-energy incentives and projects designed to accelerate the digitizing of the national power grid. These were followed by initiatives to underwrite hybrid and electric cars and the controversial $2.8 billion cash-for-clunkers program. The administration proposed that some of the measures be paid for by $30 billion in new taxes on the oil and gas industry. Yet notably absent from this long list of programs were policies dealing with coal, shale gas, nuclear energy and offshore drilling.

To create a more environmentally friendly and more self-sufficient future, hydrocarbons must be used (some of them a good deal cleaner than others), and some of these hydrocarbons need to be produced domestically. It was in this context that soon after passage of the health-care bill, the president made a surprise announcement that he was looking at the potential expansion of oil and gas exploration into the Eastern Seaboard, the eastern Gulf of Mexico and waters offshore northern Alaska, areas that had been off-limits due to various federal moratoria that expired in 2008.

As Obama said of his energy-security plan in which bans on drilling could be lifted, “[it is] part of a broader strategy that will move us from an economy that runs on fossil fuels and foreign oil to one that relies on homegrown fuels.” And to tie the drilling to a greener economy, the president added that the Interior Department could spend up to a year “studying and protecting sensitive areas in the Arctic,” implying that if Congress didn’t pass an energy bill, the moratoria might not be lifted. All of these potential places of exploration were thought to be more gas-prone than oil-prone; and thus in this decision we can see the administration’s clear preference for natural gas as a transition fuel to a lower-carbon economy. Yet even with all these measures, an important fact of life was ignored: to get to this future, oil will be needed in the interim. And deepwater oil is the critical short-to-medium-term mechanism for reducing dependence on foreign supplies.

Twenty days after Obama’s announcement, the Macondo well erupted, and even current drilling was suspended.

The policy reactions to the Macondo blowout are hampering development of this, the most critical area of both U.S. and global crude-oil production—which had, up until last April, promised to provide significant worldwide growth in non-OPEC countries for the next decade and beyond. The Gulf of Mexico, offshore Brazil and most of West Africa, pockets of the Mediterranean and East Africa, the Indian Ocean, off the coast of Australia, Indonesia and China can all yield oil. The policy setback may guarantee that markets tighten again sometime soon—that is, unless Iraqi output grows enough to offset declining production in Iran and Venezuela, key OPEC countries.

But all this is not necessarily bad news. In many respects, changes in the U.S. energy sector are likely to come not from government intervention but from the inevitable innovation brought about by market forces. Oil and natural-gas production are improving, demand for gasoline has declined, and already-in-place controls over emissions and hazardous pollutants are bringing about dramatic, stunning and transformational changes both within the country and in the global energy sector more broadly. If we can get our energy house in order, the future may look far better than the past.

THANKS TO high oil and natural-gas prices in the middle part of the last decade, the ingenuity of independent natural-gas producers led to the development of means to tap into the vast resources of in-place hydrocarbons trapped in shale rock, essentially sedimentary rocks that have within them highly concentrated quantities of natural gas and oil. With astonishing speed, the United States was transformed from a country with declining production of natural gas to one which might just end up with growing surpluses. At the beginning of the 1990s, proved and probable American gas reserves covered between 25 and 30 years of consumption. Now, natural-gas reserves are estimated to be able to provide for 100–125 years of U.S. needs. What’s more, natural gas has even become cheaper than coal, and not just at present, where “spot” gas is priced at $4 or less and coal is $5 or more for an equivalent amount; natural gas’s discount to coal is maintained in future projections through the middle of the decade.

This natural-gas abundance changed the energy landscape entirely, with supplies clearly outstripping current and foreseeable demand. And this meant that America would no longer be a major (let alone the world’s largest) market for liquefied natural gas (LNG).

Now the LNG market is self-correcting, making prices for natural gas more stable over the long term. Much of the new fuel came into the market at the same time as the Great Recession; European and U.S. natural-gas demand declined abruptly. With a dwindling market, LNG was sold on a distressed basis, severely impacting the ability of sellers of gas, including Russia and Qatar, to find enough buyers. Well into next year the LNG glut will continue. Commanding premium prices is near impossible. And so the price of natural gas is being de-linked from that of oil. This is as it should be. The main market for natural gas is the power sector, where gas competes with relatively cheap coal; the main market for oil is transportation—where competition is virtually nonexistent.

As a result of the shale revolution in the United States, spot gas prices in Europe are about 55 percent of the oil-linked price demanded by Russia and Qatar. No longer are natural-gas prices highly volatile as they have been traditionally, with enormous price differences between the so-called shoulder seasons of fall and spring, and the winter, when demand is high. Coal can now be replaced as the primary base-load supplier to electric-power utilities in America (the U.S. coal industry always defended its pride of place by intoning cost stability). And the argument that since America has vast coal resources, reliance on coal means less dependence on insecure sources of supply from abroad no longer holds. The fact that natural gas is found throughout the United States, including in the main eastern and midwestern consuming markets, means that there are fewer and fewer bottlenecks to delivering it, and greater and greater supplies reducing its seasonal price volatility.

The shale-gas resource revolution brings with it another surprising and ancillary benefit: much of the shale resources of the United States have an impact on the production of liquid hydrocarbons as well. These come in two forms—natural-gas liquids, including propane, butane and ethane that can be used for household heating and cooking, or as feedstock for petrochemicals; and crude oil embedded in similar sedimentary rock. In the midcontinent of the United States, what was once thought to be a permanent decline of oil production that set in decades ago has not only been stemmed but new production has actually reversed the downturn. In an area called the Bakken—centered in Montana and reaching east into the Dakotas, and west and north into Canada—new oil production has been growing in leaps and bounds, rising from nothing to over 300,000 barrels per day (b/d), with the potential of reaching close to 1 million b/d within this decade. Similar prospects exist in a half dozen other areas, creating the promise of U.S. domestic, lower-48-states oil production rising by nearly 2 million b/d by the end of this decade, close to the current oil production of Iraq, Kuwait and the United Arab Emirates, and more than that of Algeria or Libya. There is no doubt that with the shale the United States has, world-class resources can be developed.

THE TENSION between the two main policy objectives of the United States—promoting cleaner energy while stemming its potential environmental costs (which are characteristic of exploitation of deepwater oil, gas and shale-based resources)—means that there is little hope for developing a consensus in energy policy through new laws. For example, environmental interest groups find both good and bad in shale. Clearly, an abundance of natural-gas resources provides not only plentiful domestic cleaner-burning fuel than coal for generating electricity but also a preferable feedstock to oil for powering vehicles. On the other hand, environmentalists find two challenges in relying on natural gas—the fracturing mechanisms that have unleashed the country’s vast shale reserves depend upon fluids that might be toxic, and they can have an impact on aquifers because it takes an abundance of water to break apart shale rocks to get to the desired commodity. So, even if there are limited dangers from toxins in the fracturing fluid, there are potential dangers from loss of water supplies.

It is the very paralysis created by these contradictory forces that leads to the possibility that the recent breakneck speed of shale-resource development will continue; no one can decide whether it is good or bad, so the market carries on. And it is the same policy stagnation that leads to de facto reliance on decision making by quasi-independent regulators, on whom the direction of domestic U.S. energy and environmental policies increasingly depends.

Perhaps best exemplifying this emergence of a new set of energy-policy drivers is what is happening in the utility sector of the United States. The U.S. Congress has been deadlocked on the issue of carbon policy; it appears there is no winning coalition across party lines that can either enact a carbon cap-and-trade system or a carbon tax. Attention therefore is on what might happen through regulation rather than legislation. Here, major regulatory events that are on the immediate horizon appear likely to reinforce the economics associated with the new competitiveness of natural gas. They will give a major boost to natural-gas use in electricity generation—at the expense of coal. The EPA’s Clean Air Interstate Rule, which aimed to reduce air pollution (in the form of sulfur dioxide and nitrogen oxide) from coal-fired power plants, is about to be replaced by an even-more-onerous framework—the Clean Air Transport Rule. It will likely have even-harsher emission limits on coal-fired generating units. Another related regulatory ruling is tied to the court-mandated need to replace the Clean Air Mercury Rule (which sought to require coal-fired utilities to reduce emissions of mercury) with a new maximum achievable control technology (MACT) rule aiming to reduce mercury emissions from industrial boilers, process heaters and solid-waste incinerators by more than 50 percent.

The EPA has until March of next year to draft the new MACT, which is scheduled to be implemented on a fast track for finalizing by November 2011. The MACT rule is designed to establish a ceiling on hazardous air pollutants by limiting emissions to levels that are now attained by the best performing 12 percent of all operating units. There is a good chance that sulfur-dioxide-control equipment—coal scrubbers—will prove to be the most effective available control technology. But about one-third of the 340 gigawatts (GW) of existing coal capacity in the United States lacks these scrubbers with no plans to install them. The costs are high—if 50 percent of those plants currently without the technology became compliant, the cost would be about $40 billion; if those plants that are partially controlled also decide to upgrade their scrubbers, the price tag could rise to $120–140 billion.

As a result of the costs confronting utilities in meeting new EPA standards, the likely course will be to retire oil-coal plants, replacing them mostly with gas-burning units. Thus far, U.S. utilities have announced plans to shut down some 20 GW of coal generation between now and 2020. Given the age and inefficiency of much of this industry, more likely than not 40–50 GW of retirements will occur between now and then, opening up demand for more natural-gas-burning units. If only 50 percent of the plants lacking controls are retired and replaced by gas-generating units, the demand for natural gas would grow by more than 10 percent in the United States, making the commodity the only available bridge to any future that needs to depend on renewables and nuclear energy.

On the other hand, the EPA is also undertaking studies to review the implications of shale gas in terms of those toxic fluids that are released in the process of fractionation to unleash gas and oil, as well as in terms of the adequacy of aquifers to support the water-intensive fracturing techniques. The current study is due to be completed by the end of 2012. In the meantime, the industry will maximize its shale finds in private leases where federal jurisdictions are—at least for the time being—less important than leases on federal lands. And though any eventual EPA decision will probably result in prolonged litigation between the extraction companies and the federal government—and between environmental groups and the federal government—for now the major pressures on further growth in shale-gas and oil extraction come from the marketplace itself.

THE UNITED States has been the main source of global gasoline-demand growth since 1990. But America has become less needy and less greedy with a concomitant ability to turn oil into gasoline with ease. Oil just isn’t what it used to be.

From 1990 until the middle of this decade, U.S. petroleum-product needs went up by 4 million barrels per day. Coupled with the decline in U.S. production, total American imports grew by around 6 million b/d. The increase in U.S. imports during this period was just shy of total Chinese demand. If the American economy was the most critical factor in tightening world oil markets then, it is now on a path of leadership in the opposite direction. That’s because the combination of new Corporate Average Fuel Economy (CAFE) standards and a rising renewable-fuel mandate should reduce gasoline demand from its peak of about 9.2 million b/d in 2007 to 7.8 million b/d in 2020. That’s a very large drop indeed.

The Obama administration looks prepared to help push U.S. needs down even further. The White House directed the Department of Transportation and the EPA to investigate 3–6 percent per annum increases in CAFE standards in the period 2017–2025, which already require 2016 vehicles to have a fuel efficiency of 35.5 miles per gallon, a 40 percent increase over the 2009 vehicle standard of 25 MPG.

With gasoline comprising 57 percent of total U.S. oil needs, and with the U.S. refining system having been built to capitalize on its manufacture, the drop in American gasoline demand is already having a significant impact on the market. The decline in U.S. demand has resulted in a reduction in gasoline prices versus heating oil. And with U.S. refiners trying to maximize production of more valuable diesel and heating oil, they are producing far more gasoline than is needed. As a result, the United States started to export finished motor gasoline last year for the first time since the mid-1950s. This has enabled refiners along the Gulf of Mexico coast to take advantage of a growing gasoline appetite in South America, and the radically increased difficulties of Venezuelan refining operations in recent years, to become suppliers of transportation fuels to Latin American markets.

The refining industry as a whole is under duress, largely because declining or stabilizing demand in Japan and Korea are turning those refiners into exporters as well; and this is all going on at the same time that Chinese and Indian producers, supported by government subsidies, have developed surplus supplies. Gasoline is, in effect, becoming a waste product in the United States and around the world.

THE REMARKABLE increases in natural-gas production in the United States were—up until Macondo—being mirrored in an equally remarkable growth in oil production in the deep water, where, just before the BP blowout, output had jumped from zero to 1.65 million barrels per day over the past five years, representing 35 percent of oil from deep water globally. All this meant relatively low energy prices for consumers.

Deepwater resources have several critical features—they are robust and they exist in many areas of the world where development problems have not been related to restricted governmental access by OPEC countries but rather to a lack of costly equipment to explore, delineate and develop resources. All of the 5-million-barrels-per-day growth in the last seven years has been in non-OPEC countries like the United States and Brazil. A new find will soon be developed in West Africa off the coast of Ghana. Robust natural-gas reserves have been found in the deep waters offshore India and Israel. Other such exploration is taking place along all three African coastal areas, offshore Australia, China, Indonesia, and in the Arctic waters of North America and Europe.

Before the blowout, production from the deep water was set to double between 2010 and 2018, with the United States likely to see its 35 percent share of global production maintained. Putting it in perspective, the United States consumes a total of some 20 million barrels per day of oil (including gasoline, jet fuel, home-heating oil and so on), and is producing about 7.2 million b/d of crude and other liquids. By the end of this decade, through a combination of U.S. shale-related oil production, offshore deepwater oil and reduced gasoline consumption, the gap between domestic production and imports was likely to be closed by more than 4 million b/d—about the same amount as total Iranian oil production and more than the total domestic production of China. While full oil independence might always be out of reach, major steps were under way to significantly reduce U.S. import dependence and vulnerability.

By and large, before Macondo, the decline in energy insecurity was being accomplished in an environmentally acceptable way. And, to the degree that there had been environmental issues posed, deepwater and shale production were providing a critical bridge to a lower-carbon future. But with the deepwater moratoria in place and new rigorous safety measures being implemented, it appears that at the end of the day, the United States will be losing 18–24 months of momentum in developing these resources. Instead of new development and new discoveries, production is inevitably declining in the interim. As a result, by 2017–18, under the best of circumstances, the United States will likely produce 300,000–500,000 b/d less per year than otherwise would have been the case. That level of lost production potentially looks to be the difference between moderate and much-higher prices around the world 4–6 years from now.

MACONDO HAS led to much finger-pointing at the oil industry. BP, the other equity partners in the well, and the service companies implementing the drilling and related activities have been sparring over who bears most responsibility. But, in truth, U.S. management of deepwater resources is as much to blame. The MMS—the former Minerals Management Service, and as of June 2010, the Bureau of Ocean Energy Management, Regulation and Enforcement at the Department of the Interior (DOI)—was a disaster in the making. Reform of the MMS should have been a high priority for this administration.

The recent history of the MMS reflects activities that one normally attributes to poor, developing, resource-rich countries—it was embedded with incompetence, corruption, conflicts of interest and scandals. It was also responsible for the second-largest source of U.S. government revenue after taxes. In September 2008, in the heat of the last presidential campaign, the DOI inspector general, Earl E. Devaney, issued a report on the management of the royalty-in-kind program, a system whereby instead of paying royalties on production in federal waters, producers paid their royalties in oil. That oil was then sold to any willing buyer—trading companies and refiners—by the Department of the Interior, which during the Bush administration had developed a sort of national oil company within the bureaucracy. Last year, about $12 billion was collected from federal royalties, half of which came from these activities. The Department of the Interior had, in short, created a trading company of not inconsiderable size.

The DOI report of 2008 noted that the management of the royalty-in-kind program involved “a culture of ethical failure.” In particular, the report stated that it was “a culture of substance abuse and promiscuity”; it was one in which close ties between federal officials and industry resulted in rigging contracts and accepting gifts; and, those managing the program engaged in “illicit sexual encounters” both with other employees and with industry representatives. A year ago, Interior Secretary Ken Salazar announced the intention to end the royalty-in-kind program, and there was legislation and administration planning on the horizon to revamp the entire setup.

Thus far, the administrative structure of federal-land leasing has been playing catch-up with the more modern resource-management programs that have been adopted in other countries. The UK adjusted its offshore management in 1988 when the Piper Alpha platform exploded, killing 166 workers. Norway, Canada, Australia and others followed. In particular, they undertook operations only now being implemented in the United States, including separating the activities of leasing, safety regulation and revenue collection, and more closely monitoring rig operations, in some cases on a 24/7 basis. Brazil, where offshore development has been critical to production for a long time, has a more modern management system than the United States.

If we are ever going to get back to deepwater drilling—and do so with long-term viability—a new setup which involves hiring only people with experience and establishing procedures that bolster safety and discourage corruption is key. But we should not kid ourselves. All of these improvements will come with delays. Given the environmental consequences of the BP blowout and the way the disaster has further divided vocal interest groups both for and against deepwater leasing and development, litigation that slows down any chance of progress is also likely.

THE UNITED States is on the cusp of achieving energy-exporter status—something unthinkable but a decade ago—not just in gasoline, diesel and heating oil, but in natural gas as well. And it is the ingenuity of our industries that has made such a future possible. At home, the momentum from the unleashing of natural gas from shale, and oil from related sedimentary structures, is providing the basis for a base-load alternative and cleaner-burning fuel than coal, while also helping to foster far more competitive conditions for natural gas globally. But we should have no hope for a viable international energy regime that helps create a greener global future. Certainly, abroad, the administration has focused its main initiatives on environmental agreements, separating itself from the policies of the Bush administration by moving to constructive engagement. But legislative gridlock and the near impossibility of enacting either a carbon tax or a cap-and-trade system through Congress, combined with the inability to reach a consensus with China and other developing countries, look likely to sidetrack further developments internationally. And, when it comes to conventional hydrocarbon issues, the United States seems to have abandoned any international strategy.

For the time being, our best hope is that the government not get in the way. Macondo should not be allowed to prevent the necessary development of oil and natural gas to bridge the gap to a lower-carbon century. Business as usual might be the best way to protect U.S. interests. The market appears to be working well enough in providing adequate supplies, and recent high prices have played a fruitful role in damping demand in some parts of the world. The focus on hybrid cars, wind and solar (which are unlikely to be the answer) are confronting gridlock issues in Washington, so that too may be well and good. Washington has a terrible track record in choosing winners when it comes to energy. The Synthetic Fuels Corporation, created by Congress and President Jimmy Carter in 1980, aimed to produce 2 million barrels a day of liquids from coal by 1985, the year in which it was dismantled because of the fall in oil prices. Experiments on shale oil by government funding produced oil at a full-cost estimate of $1 million a barrel in the early 1980s. Clearly, fostering private-public partnerships by favoring technologies is a less-than-good idea; they face a difficult path when markets move in different directions.

If there were a main criticism of U.S. energy policy over the past year and a half it may lie in precisely this area—the failure to understand how markets have moved, and how to pursue goals by harnessing rather than fighting market forces.

Edward L. Morse is the managing director and head of global commodities research at Credit Suisse. He has previously served as deputy assistant secretary of state for international energy policy and was U.S. representative at the International Energy Agency.