The blackout’s proximate cause was a software bug in the alarm system at the control room of FirstEnergy, which rendered operators unaware of the need to redistribute load after overloaded transmission lines drooped into foliage. What should have been a manageable local blackout cascaded into the collapse of much of the Northeast regional electricity distribution system.
Not a plant but another example of something that should have been small causing massive outages. From what I know talking to people who’ve worked for the province’s grid operator, it’s a massive job to keep everything going.
For a historical analogue check out what happened in Italy on 28. September 2003. One international line in Switzerland sparked to a tree, and got shut down, that caused a cascade where the other lines to France were overheating and getting shut down a few minutes after, and Italy didn’t manage to shed enough load in time to keep up their frequency internally, then everything shut off when it drooped low enough. Took them 18 hours afterwards to get the whole grid back online.
A big power plant goes down and other plants have to pick up the load. Load exceeds capacity of remaining plants and they shut down (or breakers blow, etc). Repeat.
Power plants also need energy to start up (black start), and if there’s no grid energy to power those ancillary systems, or if the power plant doesn’t have on-site auxiliary generators to provide black start capability, they’re down until they can get power again from elsewhere.
Base load plants (coal, nuclear) don’t throttle up and down quickly for changing loads. For quick response, we use peaker plants which are typically natural gas powered turbines and can respond quicker (grid batteries are, thankfully, replacing these in some cases).
That’s grossly over-simplified but it’s more or less the gist of it.
Power plants also need energy to start up (black start), and if there’s no grid energy to power those ancillary systems, or if the power plant doesn’t have on-site auxiliary generators to provide black start capability, they’re down until they can get power again from elsewhere.
This is huge, we have massive drills to make sure we can do this, and idle black start plants for just this purpose alongside almost an entire secondary grid for bootstrapping.
It trips me out that many of these plants don’t have APUs for starting themselves up, or that they were designed in such a way that they require utility power to boot up. Like I understand that black starts could have problems with frequency sync with no point of reference, but I can’t imagine that their control system circuits don’t have any form of self-powering redundancy built in to their design. Is there any reason for this?
We didn’t have digital controls when they were designed, so you couldn’t use GPS or atomic clocks to synchronize frequencies, you just needed to have a single source of coordination. Those coils were manually controlled till not long ago.
Now we should be able to use some kind of small gas turbine with a igbt rig for synchronization, much like they do with wind turbines.
People often don’t appreciate how far we’ve come over the past 2 decades, and how utterly manual and brute force we were until very, very recently.
Ah makes sense, I forget how old a lot of that infrastructure is. I do a lot of work for our local ski resort, they have a 12mw generation plant for their snowmaking system that can backfeed the utility if needed, and they’re sync setup was basically two blinking lights that you had to visually time just right to close the switch and pray you didn’t screw up lol.
BTW, there was money both in 2010 and 2021-2022 that was literally supposed to replace all those linkages with something less ancient, even for small things like yours.
My understanding is the money ran out halfway or so, maybe less, and even a lot of the big stuff didn’t get done, they kept fighting on how to do it and the details, everybody wanred standardized on their system, and in the end very little actually happened.
That is hilarious, but also I can 100% picture it in my head, and it’s not that far off from what the big machines did until recently, and pretty sure some still do.
It’s why “smart grids” are talked about a lot recently, even though it doesn’t mean much for the layperson. But for the people actually working in the industry, it matters a lot and can be a huge benefit
They could trivially save billions a year just by adding intelligent load coordination and shifting for evs, you sign up and you save a few pennies a kwh, and in exchange they can steer load away from shortages and towards surplus. It’s 1990s technology.
But management at places like Pge are just jobs programs for Newsom to sell to sew up the 2028 primary, which hilariously backfired for him (though im sure he cut a decent deal).
Most Ev loads are almost instantly dispatchable, it’s an absolute no brainer, even if we aren’t trying v2g, at least control when they charge.
Cascade failure maybe? Sudden loss causes other plants to try to pick up slack, overloading one of them, which puts even more pressure on the rest until they all fail?
It’s Cuba. They’ve been hamstringed by the US embargo for a very long time. I’m amazed they’re doing as well as they are. My understanding is that, with the exception of medicine and food, basically anything that touches American soil or American hands (literally or metaphorically) is illegal to officially export to Cuba regardless of the country it’s being sold in.
So for an example, any steel made in the US, manufactured for US companies, made using iron from the US, etc, may not be officially sold to Cuba. That doesn’t mean a Mexican company can’t buy the steel and resell it to Cuba; but it means Cuba potentially has to pay multiple tariffs and pay to have it bounced from the US to at least one other country before it can go to Cuba.
Furthermore, my understanding is that foreign companies with a US presence don’t tend to do business with Cuba because the US will put pressure on them to stop doing business with Cuba.
The embargo needs to end. Cuba could have been the US’ strange, goofy cousin with weird ideas about government and economy; because it seems like the Cuban government is making a legitimate attempt at a socialistic system. However, US capitalists ruined everything (iirc the embargo didn’t start with the cold war, it started because Cuba overthrew the banana republic previously ruling them, which pissed off fruit companies. Said companies then cried to Uncle Sam because their slaves revolted and the US said, “you gotta pay them back for all the fruit they lost, plus interest”.)
Electric grids are really tough to get back online from what I’ve read. Rolling blackouts help keep things online, but if the whole thing goes down at once it’s tricky to get all the generators on the same timing for their 60/50 Hz transmission lines.
That’s how power works, you miss your requirements by even a little and you can get cascading failures if you haven’t engineered things well and your guys aren’t sharp.
Power engineering is not something casual, it’s brutal and if you aren’t on your toes then your toes will leave black sooty marks where they were.
We spend insane amounts of money keeping things going, Cuba probably cut corners and couldn’t do upgrades due to sanctions/embargoes.
If demand exceeds supply, the whole grid will fail as the voltage/frequency drop trips all sorts of safety systems meant to protect the grid and the devices connected to them. Normally the supply and demand are continually balanced to avoid this.
If a major plant goes offline and you don’t shut down equal demand at the same time (usually by disabling entire neighborhoods) then this is the result.
I don’t get it. How does the failure of one single power plant cripple an island that large?
https://en.m.wikipedia.org/wiki/Northeast_blackout_of_2003
Not a plant but another example of something that should have been small causing massive outages. From what I know talking to people who’ve worked for the province’s grid operator, it’s a massive job to keep everything going.
For a historical analogue check out what happened in Italy on 28. September 2003. One international line in Switzerland sparked to a tree, and got shut down, that caused a cascade where the other lines to France were overheating and getting shut down a few minutes after, and Italy didn’t manage to shed enough load in time to keep up their frequency internally, then everything shut off when it drooped low enough. Took them 18 hours afterwards to get the whole grid back online.
Here’s Practical Engineering explaining how it almost happened in Texas. he also goes into why starting from a black start is so hard
Article doesn’t say, but one plausible scenario:
A big power plant goes down and other plants have to pick up the load. Load exceeds capacity of remaining plants and they shut down (or breakers blow, etc). Repeat.
Power plants also need energy to start up (black start), and if there’s no grid energy to power those ancillary systems, or if the power plant doesn’t have on-site auxiliary generators to provide black start capability, they’re down until they can get power again from elsewhere.
Base load plants (coal, nuclear) don’t throttle up and down quickly for changing loads. For quick response, we use peaker plants which are typically natural gas powered turbines and can respond quicker (grid batteries are, thankfully, replacing these in some cases).
That’s grossly over-simplified but it’s more or less the gist of it.
This is huge, we have massive drills to make sure we can do this, and idle black start plants for just this purpose alongside almost an entire secondary grid for bootstrapping.
Electricity is expensive and hard as hell.
It trips me out that many of these plants don’t have APUs for starting themselves up, or that they were designed in such a way that they require utility power to boot up. Like I understand that black starts could have problems with frequency sync with no point of reference, but I can’t imagine that their control system circuits don’t have any form of self-powering redundancy built in to their design. Is there any reason for this?
We didn’t have digital controls when they were designed, so you couldn’t use GPS or atomic clocks to synchronize frequencies, you just needed to have a single source of coordination. Those coils were manually controlled till not long ago.
Now we should be able to use some kind of small gas turbine with a igbt rig for synchronization, much like they do with wind turbines.
People often don’t appreciate how far we’ve come over the past 2 decades, and how utterly manual and brute force we were until very, very recently.
Ah makes sense, I forget how old a lot of that infrastructure is. I do a lot of work for our local ski resort, they have a 12mw generation plant for their snowmaking system that can backfeed the utility if needed, and they’re sync setup was basically two blinking lights that you had to visually time just right to close the switch and pray you didn’t screw up lol.
BTW, there was money both in 2010 and 2021-2022 that was literally supposed to replace all those linkages with something less ancient, even for small things like yours.
My understanding is the money ran out halfway or so, maybe less, and even a lot of the big stuff didn’t get done, they kept fighting on how to do it and the details, everybody wanred standardized on their system, and in the end very little actually happened.
That is hilarious, but also I can 100% picture it in my head, and it’s not that far off from what the big machines did until recently, and pretty sure some still do.
It’s why “smart grids” are talked about a lot recently, even though it doesn’t mean much for the layperson. But for the people actually working in the industry, it matters a lot and can be a huge benefit
Yeah, but the utility’s management… Holy shit.
They could trivially save billions a year just by adding intelligent load coordination and shifting for evs, you sign up and you save a few pennies a kwh, and in exchange they can steer load away from shortages and towards surplus. It’s 1990s technology.
But management at places like Pge are just jobs programs for Newsom to sell to sew up the 2028 primary, which hilariously backfired for him (though im sure he cut a decent deal).
Most Ev loads are almost instantly dispatchable, it’s an absolute no brainer, even if we aren’t trying v2g, at least control when they charge.
And where would Cuba get this tech from and how would they pay for it?
so it’s exactly like coal generators in Satisfactory
Cascade failure maybe? Sudden loss causes other plants to try to pick up slack, overloading one of them, which puts even more pressure on the rest until they all fail?
I’m amazed they don’t have redundancies in case of things like this. I guess not.
It’s Cuba. They’ve been hamstringed by the US embargo for a very long time. I’m amazed they’re doing as well as they are. My understanding is that, with the exception of medicine and food, basically anything that touches American soil or American hands (literally or metaphorically) is illegal to officially export to Cuba regardless of the country it’s being sold in.
So for an example, any steel made in the US, manufactured for US companies, made using iron from the US, etc, may not be officially sold to Cuba. That doesn’t mean a Mexican company can’t buy the steel and resell it to Cuba; but it means Cuba potentially has to pay multiple tariffs and pay to have it bounced from the US to at least one other country before it can go to Cuba.
Furthermore, my understanding is that foreign companies with a US presence don’t tend to do business with Cuba because the US will put pressure on them to stop doing business with Cuba.
The embargo needs to end. Cuba could have been the US’ strange, goofy cousin with weird ideas about government and economy; because it seems like the Cuban government is making a legitimate attempt at a socialistic system. However, US capitalists ruined everything (iirc the embargo didn’t start with the cold war, it started because Cuba overthrew the banana republic previously ruling them, which pissed off fruit companies. Said companies then cried to Uncle Sam because their slaves revolted and the US said, “you gotta pay them back for all the fruit they lost, plus interest”.)
That could never happen here.
I would have hoped things had improved in the last 20 years. Apparently they haven’t, at least in Cuba.
I’d be more amazed if they where able to build that capacity behind an American blockade
It’s an embargo, not a blockade. Plenty of other countries trade with Cuba. In fact, Canadians go to Cuban resorts on the regular.
Electric grids are really tough to get back online from what I’ve read. Rolling blackouts help keep things online, but if the whole thing goes down at once it’s tricky to get all the generators on the same timing for their 60/50 Hz transmission lines.
If you keep proactive with rolling blackouts you have a chance, but if you are slacking even a bit all the spinning plates fall at once.
That’s how power works, you miss your requirements by even a little and you can get cascading failures if you haven’t engineered things well and your guys aren’t sharp.
Power engineering is not something casual, it’s brutal and if you aren’t on your toes then your toes will leave black sooty marks where they were.
We spend insane amounts of money keeping things going, Cuba probably cut corners and couldn’t do upgrades due to sanctions/embargoes.
If demand exceeds supply, the whole grid will fail as the voltage/frequency drop trips all sorts of safety systems meant to protect the grid and the devices connected to them. Normally the supply and demand are continually balanced to avoid this.
If a major plant goes offline and you don’t shut down equal demand at the same time (usually by disabling entire neighborhoods) then this is the result.