Texas Makes a New 'Map' to Decipher the Grid's Growing Complexities

By PETER BEHR of ClimateWire

In the hour before the August 2003 Northeast Power Blackout, frontline control room operators battled an electronic storm of confusion and situational blindness, never seeing the cascading grid failure bearing down on them.

Michael Legatt, a graduate student and amateur ham radio operator in White Plains, N.Y., spent the aftermath of the blackout working with emergency responders who grappled with communications breakdowns across the New York area. It changed his life plan.

Legatt made it his mission "to prevent this kind of thing from happening again," he said.

Today, Legatt, a Ph.D. specialist in human-computer interactions, has developed the "Macomber Map," a new monitoring and visualization system for control room operators at the Electric Reliability Council of Texas, or ERCOT, based in Austin, which manages the power grid in most of the state.

As smart grid technologies begin to spread around the country, flows of data across the grid will mushroom, involving the fickle nature of solar and wind power, distributed generation, demand management and high-speed measurement devices that monitoring grid conditions in mini-seconds. Grid operators are in a race to create increasingly powerful control room systems that can cope with this tide of new information.

At the same time, the grid must be reconfigured to maintain control of generation and transmission lines if key parts of the network are disabled by cyber attack or geomagnetic storms, according to industry participants in a task force study conducted by the North American Electric Reliability Corp., or NERC. A report on such extreme risks is expected next week.

"We are at a point -- in the power systems world and life in general -- of information overload," said Legatt. "There is a risk of overwhelming [grid] operators. ... We try to navigate through that by giving them information they need more quickly and efficiently."

The Macomber Map, named for the late Gary Macomber, Legatt's mentor and friend at ERCOT, gives grid operators a statewide, satellite-level view of the Texas grid, color-coding transmission lines and other equipment that are under stress or operating outside their normal ranges. It permits operators to target a county or region in the state for deeper analysis, and allows a drill-down to check conditions at the substation level.

A weapon to fight 'information overload'

"They've done a really good job," said Joseph McClelland, a reliability expert at the Federal Energy Regulatory Commission, which uses the Macomber Map to get a high-level view of grid conditions in Texas. There are other control systems with different features that impress him, too. "I would like to see some of the tools blended," he said, while repeating praise for ERCOT's system. "They've conceptualized it very well," McClelland said. NERC also uses the map to get an overview of the state.

The Aug. 4, 2003, blackout that began in eastern Ohio cut off power to an eight-state region and one Canadian province, home to 50 million people. An Energy Department inquiry pinned the "root cause" on lax vegetation control in Ohio that allowed wires to affect heavily loaded power lines, shorting the lines and causing a cascading instability in that grid section. Faulty control room equipment, software failures and poor planning were key contributors.

But human failures also played a critical part. About a half-hour before the blackout, a grid supervisor in Indiana called a network coordinator at FirstEnergy Corp. in Akron, saying his monitor showed a major line east of Cleveland had gone off-line. "I was wondering what's going on there," the supervisor asked, according to transcribed telephone records. "Something strange is happening."

"Let me take a peek here," the FirstEnergy coordinator said. "Doggone it," he said in surprise. "When did that happen?"

Just before the blackout, the supervisor again spoke to FirstEnergy: "I called you guys like 10 minutes ago, and I thought you were figuring out what was going on there."

"Well, we're trying to. Our computer is not happy," the FirstEnergy coordinator replied. "I can't get a big picture of what's going on," the supervisor protested.

A struggle to see the 'big picture'

The blackout led to new standards for grid control procedures backed up by financial penalties when called for -- something that did not exist in 2003. Control room displays and tools have been upgraded.

A review of ERCOT's system shows the breadth of the challenges, even in the post-blackout period, according to a paper by Legatt and colleagues. The Texas grid managers, overseeing more than 550 power generators and monitoring more than 40,000 miles of transmission lines, had to deal with different vendors' systems for managing energy flows and power prices. Current and next-day operations were watched over by different teams, which had to maintain close coordination.

More than 2,000 separate screen displays were available to ERCOT controllers, but only five to 10 were kept open at any one time on the four available monitors. Operators tracking a problem at one point on the grid might have to navigate across several screens, accessing multiple data systems. Many key displays required five or more keystrokes and steps to be called up.

A malfunctioning relay could send out many alerts a second, of little importance. "What happens if you have 20 irrelevant messages, and in the middle, there's a signal of a real problem?" Legatt said. "If they have scrolling garbage, they miss it. They sneeze, or blink; they miss it. That's a dust storm: Real things are missed because they are sandwiched in with things that are far less important."

The challenge was compounded by ERCOT's future plan to shift from a statewide energy market divided into five zones to a "nodal" market with several thousand pricing points at substations.

Helping Texas operators 'lasso' their problem areas

The Macomber Map system provides a range of wide to close-up views of networks, sorts alarms by severity, provides trend graphs, and enables operators to "lasso," or outline, a part of the state on a screen map for detailed scrutiny. Common problems are linked to potential solutions.

Extensive work with control room operators provided crucial feedback and built confidence in the system's performance, Legatt said. Legatt says his approach, following Macomber's lead, is based on research on how humans access and process information.

While in college, Legatt had an internship on Wall Street, and the trading phenomenon sparked an interest in psychology and human-computer interactions. That led to his doctoral studies in psychology at the Ferkauf Graduate School of Psychology at New York's Yeshiva University.

Still in graduate school, he was home on the afternoon of Aug. 4, 2003, doing laundry, when the lights went out. Legatt's emergency work as a ham operator included trying to find a portable generator to keep life support equipment going at a children's hospital outside New York City. His switch to the utility sector followed.

"More work needs to be done, obviously," said FERC's McClelland. The grid will grow more complex with two-way power flows, large-scale integration of renewable generation, demand-response strategies to curb peak power demand, and electricity-power transportation. "All of these things have to be incorporated in a manner that ensures reliable operation of the grid," he said.