Historical Incidents & Case Studies

What Happened

• FirstEnergy Corporation in Ohio failed to trim trees growing under their 345 kV transmission lines. Over the course of the afternoon, three 345 kV lines in the Chamberlin–Harding–Star corridor sagged into overgrown vegetation and tripped sequentially between 15:05 and 15:41 EDT.
• A critical alarm and logging software bug (race condition in the XA/21 energy management system) silently failed at 14:14, leaving FirstEnergy operators completely blind to the escalating emergency for over 90 minutes. No audible alarms sounded. No screen updates appeared. Operators believed the system was operating normally.
• Without situational awareness, operators failed to reduce load or re-dispatch generation. The remaining transmission lines became progressively overloaded. By 16:05, the Sammis–Star 345 kV line tripped, creating an unrecoverable cascade.
• Within 8 minutes (16:05–16:13), over 265 generating units at 508 power plants tripped offline. The cascade propagated from Ohio through Michigan, Ontario, New York, New Jersey, Connecticut, Massachusetts, Pennsylvania, and Vermont.
• 61,800 MW of generation was lost — the largest blackout in North American history at that time.

The Synchronization Challenge

The real synchronization challenge came not during the blackout itself, but during the multi-day restoration that followed. Restoring a dead grid is fundamentally a synchronization problem: every generator must be carefully reconnected to an expanding island of live buses, each reconnection requiring precise frequency and voltage matching.

• In Ontario, Hydro One operators performed over 2,000 breaker operations during the restoration sequence. Each island reconnection required synchronization checks. Black-start hydro units at Niagara and on the Ottawa River provided the initial voltage references.
• In New Jersey, PSE&G operators manually re-energized over 100 substations, many without SCADA (Supervisory Control and Data Acquisition) communication. Technicians drove to substations and performed manual synchroscope readings at each reconnection point.
• In New York, the NYISO coordinated generation restoration using a cranking path strategy, reconnecting generation islands to the backbone 345 kV network one segment at a time. Each connection point was a synchronization event.

What the Synchroscope Would Have Shown

During restoration, a technician performing manual synchronization at a New Jersey 138 kV substation would have observed the following on their portable sync panel:

• Needle rotation: Slow clockwise at approximately 0.04 Hz (one revolution every 25 seconds), but with visible wobble — the needle would speed up and slow down irregularly because the black-start sources had limited inertia and governor response was sluggish on partially loaded units.
• Voltage meters: Bus voltmeter reading low at 132 kV (95.7% of nominal 138 kV). Incoming voltmeter slightly lower at 128 kV (92.8% of nominal). Both below normal but within emergency restoration tolerances.
• Frequency meters: Bus at 59.95 Hz, incoming at 59.99 Hz. Both slightly below 60 Hz, reflecting the underfrequency condition common during restoration when load pickup is aggressive.
• Closing strategy: Wait for the needle to approach 12 o'clock at this slow rate. Anticipate the breaker closing time (~5 cycles = 83 ms on a typical 138 kV SF6 breaker) and close slightly before the needle reaches the top. Accept the voltage mismatch as within emergency authorization limits.

Sources

• U.S.–Canada Power System Outage Task Force, Final Report on the August 14, 2003 Blackout in the United States and Canada: Causes and Recommendations, April 2004
• NYISO, Interim Report on the August 14, 2003 Blackout, January 2004
• NERC, Technical Analysis of the August 14, 2003 Blackout: What Happened, Why, and What Did We Learn?, July 2004
• Energy Policy Act of 2005, Public Law 109-58, Title XII — Electricity

What Happened

• At 20:37 EDT, lightning struck a 345 kV transmission tower on the Con Edison line from Buchanan South (Indian Point nuclear station) near the town of Buchanan, New York. The line tripped on protective relay operation — a normal expected response to a lightning strike.
• The line should have reclosed automatically, but a loose locking nut on the circuit breaker at Buchanan South prevented the reclosing mechanism from operating. This single mechanical defect — a maintenance failure — kept a critical 345 kV tie out of service.
• Over the following hour, additional lightning strikes and relay operations disconnected more transmission ties. Each disconnection increased loading on remaining lines. Con Edison's system operators attempted to shed load but acted too slowly and with insufficient magnitude.
• By 21:19, the system had progressively islanded — breaking into smaller electrical islands separated from the wider New York Power Pool. Con Edison's last remaining 345 kV tie to the outside grid tripped at 21:22.
• With Con Edison fully isolated, the internal generation was insufficient to support load. Gas turbine peaking units at the Astoria generating complex were started, but the operators could not synchronize them to the collapsing system — the frequency and voltage were too unstable.
• At 21:36, the system collapsed completely. All of New York City went dark. The restoration took approximately 25 hours.

The Synchronization Challenge

The 1977 NYC blackout demonstrates a critical concept in synchronization: there is a point of no return beyond which a collapsing system cannot accept new generation, even when that generation is physically available and ready.

• Frequency collapse: As load exceeded generation, system frequency began dropping. Each 0.1 Hz drop below 60 Hz represents approximately 600 MW of generation deficit in a system the size of Con Edison's. By the time the bus frequency had fallen to 59.2 Hz, the system was short approximately 4,800 MW — well beyond recovery.
• Voltage depression: Reactive power deficiency caused bus voltages to sag from 138 kV toward 125 kV (a 9.4% depression). At these levels, induction motor loads draw increased current, worsening the collapse in a positive feedback loop known as voltage collapse.
• Escalating slip frequency: The gas turbines at Astoria maintained 60.0 Hz (their governors held rated speed since they were not yet connected). The bus frequency was 59.2 Hz and falling at approximately 0.1 Hz every 10–15 seconds. This created a slip of 0.8 Hz with additional variation of ±0.15 Hz from governor hunting and load fluctuations on the collapsing bus.

What the Synchroscope Would Have Shown

An operator at the Astoria plant synchronization panel would have seen:

• Needle rotation: Fast clockwise rotation completing approximately one full revolution every 1.25 seconds. The rotation was not steady — the needle visibly sped up and slowed down due to the ±0.15 Hz bus frequency variation caused by load swings and generator tripping on the collapsing island.
• Voltage indication: Bus voltmeter dropping steadily from 138 kV toward 125 kV over minutes. The incoming (gas turbine) voltmeter was steady at 138 kV. The growing voltage split was visually obvious — the two meter needles diverging in real time.
• Frequency meters: Bus frequency meter showing 59.2 Hz and declining. Incoming gas turbine holding steady at 60.0 Hz. The separation between the two readings was growing.
• Operator response: No responsible operator would close the breaker at these conditions. The wildly spinning synchroscope needle, falling voltage, and falling frequency all indicated a system beyond recovery. Closing would have subjected the gas turbine to severe out-of-phase torques, potentially damaging the unit without saving the system.

Sources

• Federal Power Commission, The Con Edison Power Failure of July 13 and 14, 1977, Investigation Report, 1978
• Consolidated Edison Company of New York, Post-Event Analysis and System Restoration Report, 1978
• IEEE, Analysis of the 1977 New York City Blackout, Transactions on Power Apparatus and Systems, 1979

What Happened

• At approximately 03:01 CET, the 380 kV Lukmanier line between Switzerland and Italy tripped due to a flashover to a tree. At the time, Italy was importing approximately 6,400 MW from northern neighbors through Alpine transmission lines — nearly 25% of Italy's nighttime demand.
• Swiss grid operator ETRANS attempted to restore the Lukmanier line. After a failed reclosure, they requested Italian grid operator GRTN to reduce imports by 300 MW. GRTN began the reduction, but the process was too slow.
• At 03:25:21 CET, the overloaded 380 kV San Bernardino line (the next major Swiss-Italian interconnection) tripped due to conductor sag into trees. This triggered a cascade across all remaining Alpine tie lines. Within seconds, all interconnections between Italy and the UCTE continental grid severed.
• Italy was instantaneously islanded with a 6,400 MW generation deficit. System frequency plummeted: 50.00 Hz → 49.00 Hz → 47.5 Hz in a matter of seconds. At 47.5 Hz, virtually all generators in Italy tripped on under-frequency protection (this is the standard minimum operating frequency for European generators).
• The entire Italian peninsula went dark. 56 million people lost power. Restoration took approximately 12 hours, using 24 black-start capable hydro and gas turbine units as seed generation.

The Synchronization Challenge

The synchronization aspects of the Italy blackout appear in two distinct phases:

• Pre-separation (before 03:25): After the Lukmanier line tripped, the remaining Alpine ties carried the full 6,400 MW import. These lines experienced massive power oscillations as the Italian system swung against the UCTE system. If operators had been able to rapidly reduce the power transfer and resynchronize the Lukmanier line, the cascade might have been prevented. But the slow coordination between ETRANS and GRTN consumed precious minutes.
• Restoration (03:30 onward): With the entire Italian grid dead, restoration required black-start procedures. Twenty-four hydro and gas turbine units with black-start capability were started independently, each forming a small electrical island. These islands were then expanded and interconnected through manual synchronization at dozens of substations throughout Italy. Each reconnection required precise synchroscope-guided closing.

What the Synchroscope Would Have Shown

If a synchroscope had been connected across one of the Alpine tie points during the separation event (03:25 CET), it would have shown:

• Needle rotation: Fast counter-clockwise rotation — indicating the Italian system frequency was lower than the UCTE continental frequency. The rotation rate would have accelerated from approximately 1 Hz to over 2.5 Hz in under 2 seconds, making the needle appear as a blur.
• Direction significance: Counter-clockwise (SLOW direction) is critical to recognize. In this context, fast CCW means the local system is losing frequency — demand exceeds supply and the system has insufficient generation. This is the most dangerous condition in power system operations.
• Voltage indication: Italian side depressed at approximately 370 kV versus the stable UCTE side at 400 kV. This 30 kV (7.5%) difference reflects the reactive power deficit accompanying the real power deficit.
• After separation: Once all ties opened, the synchroscope would have no incoming voltage signal. The needle would become unpowered and fall to rest — indicating a dead bus on the far side.

Sources

• UCTE, Final Report of the Investigation Committee on the 28 September 2003 Blackout in Italy, April 2004
• ENTSOE (formerly UCTE), System Disturbance on 28 September 2003: Supplementary Report, 2004
• Berizzi, A. et al., The Italian 2003 Blackout, IEEE Power Engineering Society General Meeting, 2004

What Happened

• German transmission operator E.ON Netz had planned a routine disconnection of a double-circuit 380 kV line crossing the River Ems in northern Germany. The disconnection was required to provide safe clearance for the passage of the cruise ship Norwegian Pearl, being transferred from the Meyer Werft shipyard in Papenburg to the North Sea.
• This planned outage had been coordinated weeks in advance. However, the N-1 security analysis performed by E.ON Netz was based on an outdated power flow forecast. On the day of the event, actual power flows were significantly different from the forecast due to strong wind generation in northern Germany exporting heavily southward.
• At 22:10 CET, the Ems River line was disconnected as planned. The resulting power redistribution overloaded adjacent 380 kV lines. Within 19 seconds, a cascading series of line trips split the entire UCTE synchronous grid (serving 450 million people) into three separate islands:
  • Western island (France, Spain, Portugal, western Germany, Benelux, part of Italy): generation deficit, frequency dropped to 49.0 Hz
  • Northeastern island (eastern Germany, Poland, Czech Republic, Austria, Hungary, Balkans): generation surplus, frequency rose to 51.4 Hz
  • Southeastern island (southeastern Balkans, Greece): approximately balanced
• 15 million households lost power, primarily in France, Germany, Belgium, Italy, and Spain, due to automatic under-frequency load shedding in the western island.
• Remarkably, trained operators at interconnection substations resynchronized the three islands in just 38 minutes — an extraordinary achievement given the scale of the disturbance.

The Synchronization Challenge

The 2006 European grid split is the best modern example of successful large-scale resynchronization. The three islands had different frequencies (the western island at ~49.0 Hz, the northeastern island at ~51.4 Hz), meaning they could not simply be reconnected — the frequency and phase angle differences had to be brought within acceptable limits first.

• Frequency stabilization: Operators in each island first stabilized their own frequency by balancing generation and load. The western island started emergency gas turbines and shed additional load. The northeastern island tripped excess generation. Within approximately 20 minutes, all three islands were near 50 Hz.
• Reconnection sequence: The western and northeastern islands were reconnected first at a pre-designated tie-point substation. Operators monitored synchroscopes at the tie-point breaker, waiting for conditions to fall within acceptable limits: voltage within 5%, slip frequency below 0.2 Hz, closing at the in-phase moment.
• Final reconnection: The southeastern island was reconnected last, completing restoration of the unified UCTE grid approximately 38 minutes after the initial split.

What the Synchroscope Would Have Shown

At the reconnection tie-point substation, the synchronization panel would have displayed:

• Needle rotation: Moderate clockwise rotation at approximately 0.15 Hz — one revolution every 6.7 seconds. This is a comfortable speed for an experienced operator. The rotation was relatively steady because both islands had been stabilized by this point, with governors regulating frequency in each island independently.
• Voltage meters: Western bus at approximately 378 kV, northeastern incoming at approximately 405 kV. The 27 kV difference (6.75%) was slightly above the normal 5% limit, but operators were authorized to proceed under emergency reconnection authority given the systemic importance of restoring the unified grid.
• Closing technique: The operator would have watched the synchroscope needle approach 12 o'clock from the "FAST" (clockwise) side, anticipating the breaker closing time. At 0.15 Hz slip, the in-phase window is approximately ±5° for about 280 ms — achievable with proper anticipation and a fast breaker.
• Post-closing transient: Upon closing, both islands experienced a power surge as generation redistributed. Frequency oscillations of approximately ±0.3 Hz persisted for 30–60 seconds as governors in both islands responded to the new unified system.

Sources

• UCTE, Final Report: System Disturbance on 4 November 2006, January 2007
• ENTSOE, Annual Report 2006: Continental Europe Synchronous Area, 2007
• Bialek, J.W., Why Has the November 4, 2006 Blackout in Europe Happened?, IEEE Power Engineering Society, 2007

What Happened

• On a hot August afternoon, high ambient temperatures caused heavy air conditioning loads across the Western US. Transmission lines were operating near their thermal limits. The 500 kV Keeler–Allston line near Hillsboro, Oregon sagged into untrimmed trees and tripped at 15:42 PDT.
• The loss of this heavily loaded line redistributed power to adjacent paths, overloading them. The 500 kV Ross–Lexington line and other parallel paths in the Portland area tripped within minutes. Each trip increased loading on remaining lines in a classic cascading pattern.
• A critical failure occurred at the Captain Jack substation in southern Oregon, where protective relay miscoordination prevented proper automatic reclosure of a key 500 kV line. The relay scheme was set for a transmission configuration that had since been modified, but the relay settings had not been updated to match.
• Without the Captain Jack reclosure, the Pacific AC Intertie — the primary north-south power highway of the Western Interconnection — became overloaded. Power oscillations developed between the Pacific Northwest (hydro-dominated) and the Desert Southwest (thermal-dominated), swinging at approximately 0.25 Hz with growing amplitude.
• At 15:48, these undamped oscillations caused the Western Interconnection to fragment into four electrical islands:
  • Northern island (Pacific Northwest, British Columbia): generation surplus
  • Northern California: approximately balanced
  • Southern California & Desert Southwest: generation deficit
  • Rocky Mountain region: approximately balanced
• 7.5 million customers lost power, and 28,000 MW of generation tripped. The total load interrupted was approximately 30,390 MW.

The Synchronization Challenge

Restoring a four-island fragmented grid is a complex multi-step synchronization problem. The Western Electricity Coordinating Council (WECC, formerly WSCC) had pre-established Interconnection Restoration Procedures that governed the process:

• Island stabilization: Each island's Reliability Coordinator first ensured internal frequency and voltage stability. This required generation redispatch and load shedding within each island.
• Reconnection sequencing: The Reliability Coordinator determined the optimal reconnection sequence, typically reconnecting the largest islands first at pre-designated tie points. Each reconnection required explicit Reliability Coordinator approval — no operator could independently close a tie-point breaker.
• Synchronization at tie points: At each reconnection breaker, operators monitored synchroscopes and waited for acceptable conditions. Given the large generating units in the Western Interconnection (hydro units up to 700 MW each), governor hunting caused noticeable speed variations that complicated synchronization.

What the Synchroscope Would Have Shown

At a 500 kV tie-point substation during the restoration reconnection, the operator would have observed:

• Needle rotation: Slow clockwise rotation at approximately 0.08 Hz (one revolution every 12.5 seconds). However, the rotation was not perfectly uniform. Periodically (every 5–8 seconds), the needle would visibly speed up or slow down as large hydro generators in the Pacific Northwest responded to governor signals. This governor hunting is characteristic of hydro-dominated islands where the water column time constant creates oscillatory governor response.
• Voltage meters: Bus voltmeter at approximately 510 kV, incoming at approximately 495 kV. At 500 kV class, a 15 kV difference (3%) is within normal tolerance. Both readings would fluctuate slightly as automatic voltage regulators responded to changing reactive power flows within each island.
• Closing considerations: At 500 kV, the energization transient upon closing is very large. The surge impedance charging of long 500 kV transmission lines can cause significant voltage transients. Operators would ensure that reactor compensation was available and that transformer tap positions on both sides of the tie point were properly coordinated before closing.
• Post-closing: After reconnection, operators would monitor power flow across the tie point to ensure it settled within thermal limits, and watch for any resumption of inter-area oscillations. If oscillations developed, the tie would be tripped immediately to prevent repeating the original disturbance.

Sources

• WSCC, Western Systems Coordinating Council Disturbance Report for the Power System Outages that Occurred on the Western Interconnection on August 10, 1996, October 1996
• Kosterev, D.N. et al., Model Validation for the August 10, 1996 WSCC System Outage, IEEE Transactions on Power Systems, Vol. 14, No. 3, 1999
• NERC, 1996 System Disturbances: Review of Selected Electric System Disturbances in North America, 1998

What Happened

• The Aurora Project was a controlled experiment conducted by the Department of Homeland Security (DHS) in collaboration with Idaho National Laboratory (INL) to demonstrate a specific cyber vulnerability in power grid equipment. The test proved that a cyber attack on protective relay settings could be used to physically destroy a generator.
• The test subject was a 2.25 MW, 27-ton diesel generator — a surplus unit that was expendable. The generator was connected to the grid through a standard circuit breaker with a sync check relay (Device 25).
• INL researchers reprogrammed the sync check relay settings via a laptop connected to the relay's communications port. Critically, the relay was not bypassed — it was still operational, but its parameters were modified so that it would allow closing at unsafe phase angles. This is the key insight: the attack did not disable protection, it changed the definition of what protection considered safe.
• The test sequence:
  • Step 1: Generator synchronized normally and running in parallel with the grid. Synchroscope needle steady at 12 o'clock.
  • Step 2: The breaker was commanded open, disconnecting the generator from the grid. The generator immediately began to accelerate because its governor was set to produce more power than the diesel engine alone consumed.
  • Step 3: The generator accelerated, gaining speed. On a synchroscope, the needle would have begun rotating clockwise as the generator frequency exceeded the bus frequency.
  • Step 4: When the generator had drifted to approximately 180 degrees out of phase (the 6 o'clock position — the worst possible angle), the breaker was commanded closed. The modified sync check relay permitted this closing because its angle threshold had been reprogrammed to allow it.
• The result was catastrophic mechanical destruction. Closing 180° out of phase essentially connects two voltage sources in opposition. The instantaneous current surge was enormous — equivalent to a bolted three-phase fault but sustained by the generator's internal EMF. The resulting electromagnetic torque on the generator shaft was approximately 3–5 times rated torque applied as a sudden impulse.
• 14 of 16 engine cylinders in the diesel prime mover were damaged. The engine-to-generator coupling was destroyed. Rubber mounts were sheared. The unit was a total loss.
• Most critically, the damage occurred in less than one cycle (16.7 ms at 60 Hz) — far faster than any protective relay could detect the fault, make a trip decision, and open the breaker. By the time protective relays responded, the physical damage was already done.

The Synchronization Challenge

The Aurora test is not about a synchronization failure during normal operations — it is about the deliberate weaponization of the synchronization process. The test demonstrated that:

• Sync check relays can be compromised remotely. Modern digital relays have communications ports (serial, Ethernet, or both) for configuration and monitoring. If an attacker gains access to these ports — either physically or through the network — they can modify relay settings without triggering any alarm. The relay continues to operate, but with parameters that allow dangerous conditions.
• The sync check relay is the last line of defense. In the Aurora scenario, the relay was the only thing standing between normal operation and destruction. Governor controls, voltage regulators, and operator procedures are all upstream. The sync check relay is the final gate that either permits or blocks breaker closing.
• Cyber security of protection systems is a synchronization issue. Prior to Aurora, most utilities viewed cyber security as an IT concern separate from power system protection engineering. Aurora proved that cyber and physical protection are inseparable — compromising a single relay setting can destroy a multi-million-dollar generator.

What the Synchroscope Would Have Shown

If a traditional analog synchroscope had been connected during the Aurora test sequence, an observer would have seen:

• Before the test: Needle steady at 12 o'clock — generator synchronized and running in parallel with the grid. This is the normal operating condition, indicating zero slip and zero phase angle difference.
• After breaker opens: The needle begins rotating clockwise as the disconnected generator accelerates above grid frequency. The rotation rate depends on how quickly the diesel engine accelerates the rotor — initially slow (perhaps 0.1 Hz), then increasing as the governor opens the fuel rack to the no-load setting.
• Approaching 6 o'clock (180°): The needle passes through the 3 o'clock position (90°), then approaches 6 o'clock. At this point, the two voltages are in maximum opposition. The voltage across the open breaker contacts is at its maximum — twice the line-to-neutral voltage.
• At 6 o'clock — breaker closes: This is the moment of destruction. The synchroscope needle is at the bottom of its arc — the exact opposite of where it should be for closing. Any trained operator would recognize this as the most dangerous position on the dial. In the Aurora test, the compromised relay allowed this closing to proceed.
• After closing: The synchroscope indication becomes meaningless as the generator is violently forced back into synchronism (if it survives) or the breaker trips on overcurrent protection. In the Aurora test, the generator was destroyed before the needle could complete another revolution.

Sources

• CNN, Mouse click could plunge city into darkness, experts say, September 2007 (first public reporting of the Aurora vulnerability)
• Department of Homeland Security / Idaho National Laboratory, Aurora Vulnerability Assessment, March 2007 (classified; declassified summary released 2014)
• Zeller, M., Myth or Reality — Does the Aurora Vulnerability Pose a Risk to My Generator?, Schweitzer Engineering Laboratories, 2010
• POWER Magazine, The Aurora Vulnerability: What It Means for Generator Protection, 2008
• Control Engineering, SCADA Security: Aurora Attack on Electric Grid, 2008
• NERC, Reliability Guideline: Generating Unit Operations During Complete and Partial Blackout Conditions, 2018

Overview

This incident, documented in a 2019 Western Protective Relay Conference (WPRC) technical paper by Barner & Klingerman, demonstrates one of the most insidious failure modes in synchronization: a synchroscope that appears correct but is actually lying due to instrument transformer wiring errors.

What Happened

Following planned maintenance at a Duke Energy generating facility, technicians replaced nearly all voltage transformer (VT) wiring and cables feeding the synchronization equipment. This was extensive work involving multiple conductor runs between the switchyard VTs and the control room instrumentation panels.

Critically, the maintenance team did not perform phasing verification tests after completing the rewiring. Phasing tests — which confirm that the voltages presented to relays and instruments correspond to the correct phases and polarities — are a standard commissioning step that was omitted.

When operators prepared to synchronize the generator back to the grid, the synchroscope indicated the 12 o'clock position (0°, apparently in-phase). All visual indications appeared normal. The operator closed the breaker with confidence.

Actual Conditions

Due to a wiring polarity error, the VT secondary connections were reversed. The synchroscope was receiving an inverted voltage signal, causing the instrument to display 0° when the actual phase difference was 180°. The synchroscope was showing the exact opposite of reality.

Upon breaker closure at 180° out of phase:

• 55 kA three-phase fault current surged through the system (4.7 per-unit of normal)
• The 87L (line differential) relay operated correctly and cleared the fault in 7 cycles (~117 ms at 60 Hz)
• CT saturation caused by the massive fault current produced false differential signals on adjacent circuits
• Sympathetic trips cascaded to neighboring protection zones, expanding the outage beyond the original generator circuit

Synchroscope Indication

Technical Analysis

At 180° phase angle, the voltage across the open breaker contacts equals twice the system voltage:

Industry Context

This was far from an isolated event. According to data compiled by the Edison Electric Institute (EEI) and the Institute of Nuclear Power Operations (INPO), at least 15 documented out-of-phase synchronization (OOPS) events occurred across the industry in a single 10-year period. These events ranged from minor power swings to catastrophic equipment destruction.

Lessons Learned

• ALWAYS perform phasing checks after any VT wiring, cable replacement, or instrument transformer work — no exceptions
• Never trust the synchroscope as the sole indication — cross-check with sync lights (dark/bright lamp method), independent voltmeters on each source, and frequency meters
• VT wiring errors make the synchroscope show the exact opposite of reality
• Modern sync-check relays (Device 25) provide independent angle verification, but they too depend on correct VT wiring
• At least 15 OOPS events in 10 years per EEI/INPO data — this is not a theoretical risk
• The 87L relay's 7-cycle clearing time prevented catastrophic generator damage, but the sympathetic trips caused extended outage

Overview

Between February 14–19, 2021, an unprecedented winter storm struck Texas, pushing the Electric Reliability Council of Texas (ERCOT) grid to the brink of complete collapse. The state came within 4 minutes and 37 seconds of a catastrophic, uncontrolled grid failure that would have required weeks of black start restoration — with every generator reconnection requiring manual synchronization.

The Crisis Unfolds

As temperatures plummeted to record lows across the state, generating units began failing en masse. Natural gas plants lost fuel supply as wellheads froze. Wind turbines iced over. Coal piles froze solid. Even nuclear units experienced cold-weather equipment failures.

• 356 generating units tripped offline or were derated — representing approximately 50% of total generation capacity
• Demand surged as electric heating systems worked overtime against the extreme cold
• The supply-demand imbalance caused grid frequency to drop rapidly

The 9-Minute Cliff

At 1:50 AM on February 15, system frequency dropped below 59.4 Hz — a critical threshold that triggered automatic under-frequency protection timers. The physics of the situation was stark:

ERCOT operators initiated emergency load shedding — ordering utilities across Texas to disconnect customers from the grid to reduce demand and stabilize frequency. The scale was staggering:

• 20,000 MW of load shed — the largest involuntary load shedding event in United States history
• 4.5 million customers lost power, many for up to 70 hours in sub-freezing conditions
• The load shedding arrested the frequency decline with only 4 minutes and 37 seconds remaining before cascading generator trips would have begun

Why ERCOT Is Uniquely Vulnerable

Texas operates as an electrical island. Unlike the Eastern and Western Interconnections, which span multiple states and provinces, the ERCOT grid covers most of Texas with only limited DC ties (approximately 1,100 MW total) to the Eastern and Western grids. There is no synchronous AC connection to neighboring systems.

This means:

• ERCOT cannot draw emergency AC power from neighboring grids during a crisis
• If the grid collapses, black start must be performed entirely within Texas
• A full black start of the ERCOT system would require hundreds of individual manual synchronization operations as each generating unit is sequentially restarted and paralleled
• Estimated restoration time for a full collapse: weeks to months, not hours

Synchroscope Relevance

Had the grid collapsed completely, every power plant in Texas would have gone dark. Restarting the grid would begin with black start units — generators capable of starting without external power (typically small gas turbines or hydroelectric units). Each subsequent generator brought online would require manual synchronization using synchroscopes, working outward from black start cranking paths.

Lessons Learned

• Grid collapse is real — Texas demonstrated that a modern, industrialized grid can come within minutes of total failure
• Manual synchronization skills are critical infrastructure — black start restoration depends entirely on operators who can manually parallel generators
• Under-frequency protection creates a cliff edge — the transition from "stressed grid" to "uncontrolled collapse" can happen in minutes once the 59.4 Hz threshold is crossed
• Know your utility's black start plan — every operator should understand the cranking path, black start units, and restoration sequence for their system
• ERCOT's island topology eliminates the safety net that interconnection provides to other US grids

Overview

On September 28, 2016, the entire state of South Australia experienced a total system black — the first statewide blackout in Australian history. The event demonstrated how rapidly a modern power system can collapse and the immense difficulty of restoring it, including the failure of the primary black start strategy.

The Storm

A severe weather system of once-in-50-year intensity struck South Australia with devastating force:

• Wind gusts up to 260 km/h (162 mph)
• 80,000 lightning strikes recorded across the state
• 22 transmission towers on the 275 kV backbone network were knocked down or severely damaged
• Multiple 275 kV transmission lines tripped on fault protection

Cascade to Collapse

The sequence from first disturbance to total blackout took approximately 2 minutes:

1. Six voltage dips occurred in rapid succession as transmission faults were cleared and reclosed
2. Nine wind farms (totaling 456 MW) tripped offline due to their low-voltage ride-through (LVRT) protection settings — the wind turbines were programmed to disconnect after a certain number of voltage disturbances in a short period, a setting that was not anticipated to be triggered by this pattern of sequential faults
3. The sudden loss of 456 MW of wind generation caused massive power flow increase on the Heywood Interconnector to Victoria, the single AC tie connecting South Australia to the National Electricity Market
4. Heywood Interconnector overloaded and tripped on overcurrent protection
5. South Australia islanded instantaneously with a severe generation deficit
6. Frequency collapsed below 47 Hz (nominal: 50 Hz) within seconds
7. Under-frequency load shedding could not arrest the decline — the deficit was too large
8. Total system black — 850,000 customers lost power, the entire state went dark

Black Start — First Attempt Failed

South Australia's primary black start plan called for Quarantine Power Station (small gas turbines) to bootstrap the larger Torrens Island Power Station. This strategy failed on the first attempt — a sobering reminder that black start plans that look sound on paper may not work in practice under real emergency conditions.

Instead, the grid was restored using the Heywood Interconnector from Victoria via a dead bus energization procedure: the Victorian side energized the interconnector and pushed voltage into the de-energized South Australian network. This is a fundamentally different operation from synchronization — since the SA side was completely dead, there was no frequency or phase to match. However, the operator must verify the bus is truly dead before performing dead bus closing, as closing onto an undetected energized source would be catastrophic.

Synchroscope Behavior

Restoration Timeline

• Initial restoration (Adelaide metro): several hours via Heywood Interconnector
• Full system restoration: 13 days to restore all customers and repair damaged infrastructure
• Wholesale electricity market suspended for 13 days
• Total economic cost estimated in the hundreds of millions of dollars

Lessons Learned

• Complete system collapse is possible — even in a modern, well-operated grid
• Black start plans may fail on the first attempt — always have backup restoration strategies
• Dead bus closing requires no synchronization but demands verification that the bus is truly de-energized
• Wind generation ride-through settings are critical — the wind farms that tripped were protecting themselves but destabilized the grid
• A single interconnector as the only tie to the rest of the grid creates a single point of failure
• Two minutes from first disturbance to total blackout — there is no time for deliberation during cascading failures

Overview

On August 9, 2019, a lightning strike on a transmission circuit in Great Britain triggered a sequence of generation losses that caused the first activation of Low Frequency Demand Disconnection (LFDD) in over a decade. The event demonstrated how quickly the loss of just two large generators can threaten grid stability, even in a well-operated system.

Sequence of Events

The chain of events unfolded in minutes:

1. Lightning struck a 400 kV transmission line, causing a voltage dip across the network
2. Hornsea One offshore wind farm lost 737 MW of output — the voltage disturbance triggered a control system fault that caused turbines to trip sequentially
3. Little Barford gas-fired power station (combined cycle) tripped, losing 244 MW
4. Additional embedded generation losses brought the total to approximately 2,100 MW lost within seconds
5. System frequency dropped sharply from 50.0 Hz to a nadir of 48.8 Hz

System Response

The grid operator, National Grid ESO, deployed all available reserves:

• 1,000 MW of frequency response was activated, including 472 MW from battery energy storage systems (BESS) — demonstrating the fast-acting capability of batteries
• However, the response was insufficient to arrest the frequency decline above the LFDD threshold
• LFDD activated automatically at 48.8 Hz, disconnecting 892 MW of customer demand in blocks
• Total customers affected: 1.1 million
• Frequency recovered to normal within approximately 5 minutes
• Full power restoration completed within 45 minutes
• Rail services were severely disrupted for hours as electric traction power was lost and trains required manual reset procedures

Frequency Protection in Great Britain

Synchroscope Indication

Lessons Learned

• One or two generator trips can threaten stability — a single lightning strike cascaded into a 2,100 MW deficit
• UFLS/LFDD is automatic but disruptive — once activated, customers lose power with no warning
• Battery storage responds extremely fast but may not have sufficient capacity to cover large generation losses
• Frequency below 49.5 Hz (UK) is a serious emergency — operators must act decisively
• Embedded generation (distributed resources) can contribute to unexpected loss magnitudes
• Transportation infrastructure (rail) is particularly vulnerable to power disruptions due to complex restart procedures

Overview

On January 8, 2021, a single incorrect frequency relay setting at a substation in Croatia triggered a cascade that split the entire Continental European synchronous grid into two separate electrical islands. The event affected 400+ million people across 25+ countries and required a carefully coordinated resynchronization operation to restore the unified grid.

The Trigger

At the Ernestinovo substation in Croatia, a bus coupler circuit breaker tripped unexpectedly. Investigation revealed the cause: a frequency relay protecting the bus coupler had an incorrect setting. The relay interpreted a normal system transient as an abnormal frequency condition and opened the bus coupler, splitting the substation.

This seemingly minor substation event — a single bus coupler opening — caused power flows to redistribute across the European network. The redistributed flows overloaded other transmission elements, which tripped on their own protection settings. Within seconds, a cascading series of line trips propagated across southeastern Europe.

The Continental Split

The cascade resulted in the Continental European grid separating into two synchronous areas:

• Northwest area (including France, Germany, Spain, Italy, Benelux, Scandinavia via HVDC): slightly over-frequency due to generation surplus — this area had more generation than load after the split
• Southeast area (including Croatia, Romania, Greece, Turkey, parts of the Balkans): under-frequency, dropping to 49.74 Hz due to generation deficit

Emergency actions taken:

• ~200 MW of interruptible industrial load shed in France (to reduce over-frequency in NW area)
• ~1,000 MW of load disconnected in Romania and Turkey (to arrest under-frequency in SE area)
• HVDC interconnectors between the two areas had their setpoints adjusted to help rebalance

Resynchronization

Reconnecting two halves of a continental grid is one of the most delicate operations in power system engineering. The two areas had drifted to slightly different frequencies, and their phase angles were no longer aligned. Resynchronization required:

• Multi-country coordination between transmission system operators (TSOs) across Europe
• Careful adjustment of generation in both areas to bring frequencies close together
• Monitoring of phase angle difference at the reconnection point
• Closing the interconnecting breakers when conditions were within sync limits

The grid was successfully resynchronized within approximately one hour of the split.

Synchroscope at the Reconnection Point

Lessons Learned

• A single incorrect relay setting can split an entire continent — protection settings must be rigorously verified and periodically reviewed
• The European grid splits more often than most people realize — smaller separations have occurred in 2003 (Italy), 2006 (UCTE), and multiple partial separations since
• Resynchronization of large systems requires multi-country coordination — no single operator can manage it alone
• The same synchroscope principles apply whether synchronizing a 50 MW diesel generator or reconnecting two halves of a continental grid
• Protection relay settings are as critical as the equipment they protect

Overview

This incident, documented in NRC Event Report ML003698812, involved the out-of-phase paralleling of a Division 3 Emergency Diesel Generator (EDG) to a nuclear plant safety bus. Unlike utility-scale generator OOPS events where the grid can absorb some of the shock, EDG damage at a nuclear facility has direct nuclear safety implications because these generators are the last line of defense for powering emergency core cooling systems.

What Happened

During a routine surveillance test, operators paralleled the Division 3 EDG to the 1C1 plant electrical bus. The synchronization was performed with an excessive phase angle difference. Upon breaker closure:

• Power oscillations were immediately observed on the 1C1 electrical bus
• Bus voltage and current fluctuated severely as the EDG attempted to "snap" into synchronism with the bus
• The generator stator was damaged by the electromagnetic forces and currents produced during the out-of-phase closure
• The EDG was declared inoperable pending repair, requiring entry into a Limiting Condition for Operation (LCO) under the plant's Technical Specifications

Nuclear Safety Significance

Emergency Diesel Generators at nuclear power plants serve a safety-critical function:

• EDGs provide backup power to Emergency Core Cooling Systems (ECCS), reactor protection systems, and essential support systems
• Most nuclear plants have two or three divisions of emergency power, each with its own EDG
• Losing one EDG reduces the defense-in-depth safety margin — the plant can still cope with a loss-of-offsite-power (LOOP) event, but with reduced redundancy
• NRC regulations require that inoperable EDGs be restored within a limited time (typically 72 hours) or the plant must shut down
• An EDG must be capable of starting, accelerating to rated speed, achieving rated voltage, and connecting to its safety bus within 10 seconds of receiving an emergency start signal

Synchroscope at Closure

Industry Data on OOPS Events

This nuclear plant EDG event is part of a broader pattern documented across the electric power industry:

The 955 MVA Benchmark

For context on the severity of OOPS events at larger scale: a Westinghouse/Consumers Power study analyzed a 955 MVA generator that was synchronized at 120° out of phase. The study estimated the event caused approximately 5% loss of total machine life — from a single synchronization error. The electromagnetic torques during out-of-phase closure can reach 10–20 times rated torque, stressing the shaft, couplings, foundation, stator windings, and rotor body.

Lessons Learned

• Even "small" generators can be damaged — EDGs, distributed generation, and industrial generators are all vulnerable to OOPS events
• Nuclear safety depends on EDG availability — a damaged EDG directly reduces the safety margin for emergency core cooling
• NRC event reports are publicly available learning resources — the nuclear industry's transparency provides valuable lessons for all power system operators
• The industry experiences at least 15 documented OOPS events per decade — these are not rare occurrences
• A single OOPS event on a large generator can consume 5% or more of total machine life
• Sync-check relays (Device 25) should be properly set and tested to prevent out-of-phase closure, but they are a backup — not a substitute for proper operator technique