(A true story of advance preparation, planning, developing creative solutions, risk avoidance, safety awareness, mistakes, lessons learned and luck.)
So I finally assumed the position of plant manager of a power station. Well, the fact is, I was compelled more than anything by self-preservation to take on this role. However, the experience has afforded me lessons learned, the perspective of the plant manager in a microcosm, and reinforced the need for being diligent, prepared and creative to keep running while avoiding losses. Let me explain:
The recent “freak storm” in Connecticut (the T&D entity in my region is Connecticut Light and Power (C L & P) shut down over ½ of all CT homes and businesses due to extremely wet snow and trees still essentially in full foliage. CL&P initially claimed they did not think the storm was going to be as bad as it turned out.
They have since walked this statement back as the weather reports for days ahead of the event were spot on predicting record snow falls, power outages due to heavy limbs laden with snow crashing the lines, etc. Ironically, the state went through something similar with tropical storm Irene in late AUG (it was a hurricane before it got to CT), which shut the state down with massive power outages for about a week. As Yogi would say, “Déjà vu all over again!”
“If you’re failing to plan, you’re planning to fail”
But I digress; back to my new role as electrical generation plant manager. The minute the power went off at 3:30PM on SAT OCT 29th, I knew I had to get the portable generator I had purchased some time ago for just this sort of event up and running to keep my sump pump going (lest I end up with a swimming pool in the basement). This is not an endorsement of this make/model, but it performed flawlessly.
When the storm was announced as a very probable event days ahead, I checked the generator out for oil levels, ensured it was fueled and started it to make sure it was ready. I located and staged the necessary extension cords (distribution wires?!?!) which would be required, and made sure I had (turns out I was way short) sufficient fuel on hand. I was confident that at least the emergency power station was ready if we lost primary power.
“Know your customer, manage expectations, communicate with them!”\
When the outage hit, I had to determine what the critical devices were going to be to supply power to; and make a quick calculation on the demand from this native load which I was going to serve vs. the capacity of the generator I was going to operate (3000 running watts and 3500 peak watts). I should have had this contingency plan done way ahead of this storm. So I referred to the following chart that came with the generator to estimate load conditions:
I knew that the sump pump was a must serve and was going to draw ~600 watts when running, but required 1800 watts to start. I also knew the refrigerator was going to draw 500 watts (running and starting). Not much left over for a 1500 watt space heater or a light or two (even with the fluorescent bulbs).
Clearly I had to operate in base load, but I also had to advise my customers (my wife & son … the cats were represented by my wife) that there would be must serve loads (the sump pump) while rolling blackouts would have to be employed elsewhere (i.e. run the fridge but the heater had to be off; operate the microwave but the fridge had to be turned off; etc).
“Safety and technically prudent operations and maintenance are essential.”
Having properly grounded the generator (my son positioned a fire extinguisher next to it), located it outside and away from the house and any windows (CO precaution), I next had to quickly establish my “T&D” lines (actually, just “D”) and do a rough calculation on the line loss from the long runs I was making to the objects being powered (another source of parasitic loss for my 3000W plant). Thus I decided to segregate the must run line apart from the other two service connections lest I trip the must run circuit with something as frivolous as a microwave:
Water? We’re on a well/septic, but it was clear that trying to tie the small but rugged generator I was operating to the well pump (1500 running to 2000W peak since I have a 3/4 HP well pump) was going to be a triumph of hope over reason given just the sump pump load I was dealing with. I now had to become more than just a power utility, I had become a municipal water district and devise a domestic source of grey water for the bathrooms. More about this in a minute.
“Prior planning prevents piss-poor performance”
Not knowing the duration of the CL&P outage (frankly I thought we’d be down for a day, maybe two. Ha! Big surprise!!!), I also had to determine fuel consumption while on load vs. fuel supply on hand (it is a gasoline powered generator). I quickly recognized that I was using about 10 gallons in 24 hours. If this went beyond a few days, I either had to find a fuel source from unaffected areas, or, tap into the static supply I had sitting in the driveway (the fuel in four parked cars). Siphoning is not on my list of preferred activities, so I found a source nearly 40 miles distant to fill my fuel containers (of which I did not have a sufficient quantity!).
Having properly grounded the generator, located it safely away from the house, run the lines, plugged in the necessary objects, I commenced the startup. Recognizing that the generator had to be started off load first, I disconnected all the distribution lines from it and started it. One pull and off it went, a good start! I let it run for about 3 minutes (I didn’t have too long as the sump was filling without regard to my time scale) and then first plugged the sump in.
The generator did what I expected it to do: since the sump immediately started when energized and started to pump, it was drawing its peak load and the generator sagged a bit but then recovered as the pump came to a steady run state.
Next I energized the fridge, and again, the generator sagged a bit as it experienced the 2400W starting load, until it assumed its roughly 1200W run mode. Luckily, the sump was not running at the same time the fridge started!
“Expect the unexpected, don’t rely on luck but recognize it when it greets you”
So here I was, plant manager of a base load 3.5 kW power station with a native load that had to be served but which required some rolling blackouts. My fuel supply was not interruptible, it was totally interrupted! Thus I had to secure spot fuel and keep sufficient stores on hand to keep at least two day’s worth. I had to not only find the fuel, I had to find new fuel containers to store it in. Try finding additional gas containers of sufficient size when the entire region is doing the same thing. Luck turned to my favor as I found additional proper fuel containers in two separate locations which increased my on hand supply to roughly 3 days worth.
“Know your equipment, how to set it up, how to run it, how to maintain it. Safety is everything.”
As I write this, there have been literally dozens of people funneling into emergency rooms with carbon monoxide poisoning and sadly, at least 5 people have died from this. Portable generators are marvelous devices that serve a critical function in emergencies. If you have never operated one, don’t know about where to place them to eradicate CO poisoning, understand that they produce the same deadly amperage that exists in the normal AC household circuits from the regular power company, etc then don’t even buy one or use one. From Electric Energy Online.com:
Lineman Killed By Generator Back Feed – Victim Helping Restore Power in Alabama
Alabama, July 14, 2005 – Sumter Electric Cooperative (SECO) officials report that a South Carolina lineman helping to restore power in Alabama after the damage caused by Hurricane Dennis was killed late Tuesday, reportedly by an improperly installed customer generator. According to SECO Director of Public Affairs Barry Bowman, the lineman was helping to restore power lost during Hurricane Dennis to customers in Alabama. The report Bowman received indicated the lineman was working on a power line that was supposed to be dead. It was not. “Tragically, the line this technician was working on was not dead,” said Bowman, “The line he was trying to repair had been re-energized by a customer who had improperly hooked up a generator and created a back feed of electricity from the generator into the supposedly dead line. The death of a lineman who was there to help the victims of Hurricane Dennis was the horrible result.”
“Trust your instruments…. what instruments?”
Now that I had the power plant running and serving the needs of the native load, I was constantly concerned about knowing whether the sump pump had died (it was cycling every 2 minutes at one point due to the massively wet snow outside feeding it) or whether the circuit on the generator had tripped, and while the engine was running, the distribution lines were not energized. Like the aux operator at a power plant, I began by making constant rounds to check the generator, the pump, the fridge, etc. After visualizing nightmares of waking up to a basement filled with water due to pump failure, I realized I needed a control room with DCS monitors that would enunciate alarms for any problems. Wishful thinking. However, I had a crude but effective instrument DCS analog working for me during the day and night that I only recognized belatedly.
The one light I had in my “control room” (actually my office) would dim for a brief second when the sump pump energized and would then regain its brightness. Seeing this occur for a few days, a brilliant flash of the obvious hit me: I realized that this intermittent dimming meant that the pump circuit was drawing power to start and once the light regained brightness, there was current flowing to the pump (the most parasitic draw on the system). This was also accompanied by the audible sound of the generator outside reaching its peak power output to contend with the startup of the pump, which then resumed its more normal humming. So I had two very crude control indicators: a lamp that would dim/recover and the sound of the engine outside changing its resonance for brief periods.
If the light dimmed and kept dim or fluttered, I knew I had a problem of either the fridge and the pump having energized at exactly the same time (pushing or exceeding peak load of the generator) or a distribution line (extension cord) had lost its tight connection. I’d then have to physically check the entire system (which I did on a normal basis anyway). This helped during waking hours, but what about when I tried to sleep?
I installed a small light in my sleeping quarters with the hope that should the generator trip or power from the field shut off, the difference of the light turning off would wake me. Fool proof? No, but absent an audible alarm, it was the best I had at hand.
“Be creative in developing solutions. What water supply?”
Plenty of potable water was on hand as we stocked bottled water and bottled water was readily available locally. We had started to fill the Jacuzzi tub in the basement in anticipation of the storm, but the outage occurred on its time schedule and not ours, thus, the tub did not get filled even half way.
However, you need copious amounts of water to keep the sanitary systems running (how’s that for treating toilet water politely?). After using the existing supply for a day or so, and after an abortive effort of placing snow into the tub in hopes that it would melt (it didn’t sufficiently) to replace the water, it was time for better solutions.
I had to determine how to get water to the bathroom toilets so that we could exist in at least a semi civil state. Then it occurred to me: I was pumping hundreds of gallons of clear water to the outside via the sump pump. If I could connect a transfer pump to this source, and redirect it, I could at least fill the large Jacuzzi tank in the basement and thereafter carry 5 gal buckets to be staged in the upstairs bathrooms. The local home center by this time had opened, and while they of course had no fuel cans, generators or batteries left, they did have a transfer pump!
What I had to do was shut down the sump pump, energize the transfer pump , and as the water rose, redirect that sump water to the tub. It worked!
“Required maintenance vs. deferred maintenance – you have to keep running!”
The manual on this tough little generator indicates that the oil needs to be changed every 100 operating hours. I had a decision to make: the engine was running superbly, but I knew that at some point, running any engine 24/7 would result in fatigue or failure. The problem of course was that changing the oil, although a relatively simple task, would take time (~20 minutes and the sump filled to overflowing in just a bit over 40 minutes) and any glitch (cross thread the oil plug when replacing it, fail to get the oil indicator threads back in after changing, etc) would be a fatal error as the unit would not be able to be run without a properly sealed oil supply. Result of doing the maintenance and having it go pear shaped: swimming pool in the basement. Result of the engine failing due to lack of proper maintenance: swimming pool in the basement. Operational and mission risk assessment in its most basic form.
In the normal course of running power stations, plant managers have to make similar deferral of maintenance decisions based on demand, profitability, risk, etc. These drive insurers nuts, but then again, the insurance companies don’t have to contend with the demands placed on a plant manager.
So here is what I determined; every time I had to re-fuel the generator, I would scrupulously check the oil level, add proper oil type if needed, very carefully re-thread the oil filler plug if I did add oil; but I had to forego the changing of the oil until the CL&P outage was over. I was relying on the hope that the generator manufacturer had placed enough robust engineering into the lube system that the engine was going to be more forgiving than the 100 hour recommended change period.
Thus, given the options and the possible results; I would run my plant 24/7 and would run it to refusal if the oil change out was the sole determining factor of whether to shut it down or continue running. The generator operated simply perfectly through all of this. After the power was restored, it received its much deserved but overdue oil change and all other actions necessary to ensure it is ready when called upon again.
The outage for our house lasted from 3:30 PM Saturday 29 OCT until 6:30PM Thursday NOV 2nd (unfortunately as I write this at 8:40 AM on Monday NOV 7, there are still 50,000 CL&P customers still without electricity). For the power plant manager who runs a real power station, the stakes are higher and the issues more complex, but the need for studying the lessons from other’s failures is the right path to dealing with those they might encounter. What lessons did I extract from my brief time as a 3.5 kW (peak) power plant manager?
- Be prepared for the unexpected and don’t rely upon gov’t or the industrial commercial/infrastructure to be there in the short term in an outage like this. Self-reliance and preparation is essential.
- Think of plausible, adverse scenarios that might occur and then make plans for how to successfully respond to these. In my case, this took the form of having an emergency portable generator, having fuel on hand and knowing how to set it up and safely run it.
- Plan for the worst but hope for the best: I was woefully deficient in my estimation of what amount of fuel supply to store. I also did not have sufficient containers on hand initially to even get a sufficient supply on hand. A larger generator is in the future.
- Be creative in thinking of ways to overcome the unexpected. Not everything requires a high tech solution. My light dimming instrumentation solution during the outage reminded me of a response I received years ago from a plant manager of a relatively large coal fired power station where we were doing an insurance and operational risk evaluation. He was resistive to the transition of his control room to electronic displays of water levels in the steam drum from the use of the original sight glass mirrors. When I asked about why he was so reticent to move to the electronic version he simply replied: “I never had a mirror burn out….”. Pretty basic logic that worked for him, as I can’t tell you the number of other stations I have visited where the bulbs in the enunciation panel are burned out!
- Recognize that you should operate and maintain the equipment per OEM instructions, but also recognize when you’re in an emergency or other compelling condition, that you may have to prudently make a risk determination to deviate from these. When this occurs, try to accommodate the intent of the standard instructions with other actions that supplement standard actions while continuing the mission (in my case, it was making sure I checked the oil more often than recommended).
- You can’t plan for an emergency when you’re in the middle of one. This probably is the same as #1, but it bears reiterating. My planning was too cursory.
- If you’re buying equipment for emergency purposes, buy good quality equipment and, if possible, purchase a bit more capacity & capability than you need (this could apply across the board to not just the size of a generator, but anything else you may need to rely upon).
- Safety is never a negotiable issue in terms of how you operate.
- Did I mention the fire that occurred half way through the outage in the house?
For those of you who may be interested, drop me a line and I’ll explain the fire that erupted, the root cause for this, and what I had to do to react.