'03 Blackout finally trips grid-hardening effort

It takes almost 11 gigawatts of electricity to keep New York City lit in the late afternoon on a hot summer day. On August 14, 2003, every volt vanished.

At 1:32 p.m. Ohio time, a 680-mega­watt FirstEnergy coal generation plant in Eastlake tripped off. A plume of ash blown from the plant settled over the surrounding neighborhood.

Ninety minutes later, FirstEnergy’s 345-kilovolt Chamberlain-Harding transmission line feeding the city of Cleveland tripped. Extra power was automatically routed through another Cleveland circuit, the 345kv Hanna-Juniper line. But by 3:32 p.m., the Hanna-Juniper wires heated to the point where they sagged into a tree and tripped. Nine minutes later, two more 345kv lines feeding northern Ohio, the Star–South Canton and Tidd–Canton Central, overloaded and tripped off.

At 4:06, voltages on the Michigan grid began to decline in order to support increasing demands in northwest Ohio. Within 10 seconds, flows on the Michigan International Transmission Co.–FirstEnergy connection jumped by 2,000mw. In less than five minutes, 12 Michigan generating plants staggering under the soaring demand took themselves offline. Ontario then tried to support both Michigan and Ohio, to no avail.

More than poles and lines 
And so it went. For the next hour, overloaded transmission lines, then entire power-generating stations, strained to the breaking point, tripped off and shut down, hopscotching through Ohio, then Michigan, New York and New Jersey. By nightfall, the largest blackout in U.S. history left 50 million customers in eight Northeastern states and southern Ontario without electricity, bringing elevators, air traffic control systems and hospital respirators to a screeching halt before emergency generators kicked in.

The incident demonstrated just how fragile the antiquated U.S. power grid actually is. Concerns arose immediately about the ease with which terrorists or hackers might exploit such a weakness.

Most people think of the grid as the lines, poles and traNSFormers cluttering every block, but there’s much more to it than that. The grid also includes its invisible automatic control and relay systems, designed to reroute power in the event of physical or electrical problems. The relays are activated by automatic sensors, because problems arise more frequently and quickly than humans can effectively respond.

An unreliable cyber backbone controls the whole electric power system, which in a real sense is the infrastructure on which all other infrastructures depend.

Sagging power lines are only one concern. Worms and viruses, which now invade personal computers and clog business networks, could conceivably emerge that are capable of crippling the nation’s power grid. Computer hackers have already found ways to block access to Yahoo and eBay, deflect orders intended for Amazon.com, and deface corporate Web sites with graffiti, including some operated by the Pentagon.

The next target may be the nation’s power companies. According to a Los Angeles Times article published two years before the 2003 blackout, hackers found a way to get inside a computer system that plays a key role in moving electrical power where it’s needed around California.

The computers belonged to the California Independent Service Operator (Cal-iso), an agency that oversees much of the state’s electricity transmission grid, including the massive complex of power plants and transmission lines.

While Cal-iso closed the hole before power delivery was compromised, the episode sent chills down Big Energy’s spine. What if the next hacker, terrorist or fast-moving virus does something that takes the entire Northeast offline for two weeks, instead of just a few hours?

“That’s a serious issue,” says Sean W. Smith, a computer science professor at Dartmouth College.

A new federal initiative
The good news is that the federal government has finally noticed how brittle and vulnerable the system is, and rehabilitation efforts are commencing. An ambitious five-year National Science Foundation project called “Trustworthy Cyber

Infrastructure for the Power Grid” (TCIP), supported by an initial $7.5 million award, has begun whose aim is to develop a computer network that can improve the power grid’s reliability and make it secure against breakdowns, whether from nature or malice.

The project, inaugurated in August 2005, is the centerpiece of NSF’s cybersecurity efforts, this award being one of several grants totaling $36 million announced last year as part of its Cyber Trust program. An indication of how grave the federal government finds the issue can be found in the fact that both the Department of Energy and the Department of Homeland Security have pledged to help NSF fund and oversee the work.

The grid-hardening project is being led by the University of Illinois at Urbana-Cham­paign, but also involves researchers at Dartmouth, Cornell and Washington State University. The goal of TCIP is to improve the way the grid’s cyber infrastructure is built and maintained, while making it more secure, reliable and safe.

“The power grid is the most critical and most basic of all infrastructures in the sense that all other systems, medical, financial, telephones, energy, depend on the power system working,” says uiuc professor of computer science William H. Sanders.

Long-term needs
He says current restoration and renovation work on the power grid can be classified in two categories: “One category is short-term fixes that definitely need to be done, but patching is not a long-term solution, even if the immediate costs are lower. The TCIP project is focusing on longer-term research that will make fundamental changes in the way the power grid’s information technology infrastructure works.”

According to Sanders, sweeping changes need to be made on every level.

More robust network controls and sensors must be developed, as well as protocols to secure peer-to-peer information exchange. The North American power grid involves nearly 3,500 individual utility companies.

One innovation Sanders sees coming out of the project is an integrated simulator that will allow researchers to simulate the computer infrastructure, the entire electric power grid itself, including the market forces that are at work within the grid, all on a single simulator. The device will allow researchers to fully test their ideas and hypotheses.

“Three pieces of the simulator built by various project team members exist already, but we’ll be combining them in a way that’s never been done before,” Sanders says.

TCIP is intended to address both the physical structure of the grid and the computer communications network that operates it. It’s gritty work.

“What we have here is a distributed system, with devices that are scattered over the entire country that must withstand every environmental condition, and that are controlled by hundreds of separate entities,” Smith says. Smith is leading the group investigating what can be done on foundational levels, starting with computing hardware itself, to increase the systems’s reliability and security.

While failures are inevitable on any system, particularly one as vast and complicated as the nationwide power grid, what’s needed are systems that tolerate failures and keep power available.

Smith says blackouts occur due to pathological system conditions, not standard conditions.

“Our work needs to go counter to conventional wisdom and optimize for the worst case, not the average one,” he says.

Monitoring the grid
Poor communication of operational data has been recognized as a major contributing factor to all recent blackouts. One solution could emerge from wsu, where researchers have been working on a system called GridStat to overcome internal communication problems. GridStat is designed to deliver status information to participants in the power grid in a much more flexible and robust manner than is possible today. GridStat has been deployed for the past two years in a demonstration project using real power grid data from Avista Utilities in Spokane, Wash., the first operational implementation of such a flexible system.

“We have been working on identifying and solving the power grid’s communication problems since 1999 and are excited to be taking GridStat to the next level,” says Dave Bakken, a WSU professor of electrical engineering and computer science.

Cornell’s share of the work will focus on determining what parts of the system are sensitive to failure. Researchers from all four universities will work with a 14-member advisory board from power companies and equipment providers to make certain that research efforts address real-world problems. Among the companies involved are Ameren, Cisco Systems, Exelon,

Honeywell, Open Systems International, pjm Interconnection, Power­World, Siemens and the Tennessee Valley Authority.