Running Your Own Power Plant and the Analog of Lessons Learned

(A true story of advance preparation, planning, developing creative solutions, risk avoidance, safety awareness, mistakes, lessons learned and luck.)

So I finally assumed the position of plant manager of a power station. Well, the fact is, I was compelled more than anything by self-preservation to take on this role. However, the experience has afforded me lessons learned, the perspective of the plant manager in a microcosm, and reinforced the need for being diligent, prepared and creative to keep running while avoiding losses. Let me explain:

The recent “freak storm” in Connecticut (the T&D entity in my region is Connecticut Light and Power (C L & P) shut down over ½ of all CT homes and businesses due to extremely wet snow and trees still essentially in full foliage. CL&P initially claimed they did not think the storm was going to be as bad as it turned out.

They have since walked this statement back as the weather reports for days ahead of the event were spot on predicting record snow falls, power outages due to heavy limbs laden with snow crashing the lines, etc.  Ironically, the state went through something similar with tropical storm Irene in late AUG (it was a hurricane before it got to CT), which shut the state down with massive power outages for about a week. As Yogi would say, “Déjà vu all over again!”

“If you’re failing to plan, you’re planning to fail”

But I digress; back to my new role as electrical generation plant manager. The minute the power went off at 3:30PM on SAT OCT 29th, I knew I had to get the portable generator I had purchased some time ago for just this sort of event up and running to keep my sump pump going (lest I end up with a swimming pool in the basement). This is not an endorsement of this make/model, but it performed flawlessly.

When the storm was announced as a very probable event days ahead, I checked the generator out for oil levels, ensured it was fueled and started it to make sure it was ready.  I located and staged the necessary extension cords (distribution wires?!?!) which would be required, and made sure I had (turns out I was way short) sufficient fuel on hand.  I was confident that at least the emergency power station was ready if we lost primary power.

“Know your customer, manage expectations, communicate with them!”\

When the outage hit, I had to determine what the critical devices were going to be to supply power to; and make a quick calculation on the demand from this native load which I was going to serve vs. the capacity of the generator I was going to operate (3000 running watts and 3500 peak watts). I should have had this contingency plan done way ahead of this storm. So I referred to the following chart that came with the generator to estimate load conditions:

I knew that the sump pump was a must serve and was going to draw ~600 watts when running, but required 1800 watts to start. I also knew the refrigerator was going to draw 500 watts (running and starting). Not much left over for a 1500 watt space heater or a light or two (even with the fluorescent bulbs).

Clearly I had to operate in base load, but I also had to advise my customers (my wife & son … the cats were represented by my wife) that there would be must serve loads (the sump pump) while rolling blackouts would have to be employed elsewhere (i.e. run the fridge but the heater had to be off; operate the microwave but the fridge had to be turned off; etc).

“Safety and technically prudent operations and maintenance are essential.”

Having properly grounded the generator (my son positioned a fire extinguisher next to it), located it outside and away from the house and any windows (CO precaution), I next had to quickly establish my “T&D” lines (actually, just “D”) and do a rough calculation on the line loss from the long runs I was making to the objects being powered (another source of parasitic loss for my 3000W plant).  Thus I decided to segregate the must run line apart from the other two service connections lest I trip the must run circuit with something as frivolous as a microwave:

Water? We’re on a well/septic, but it was clear that trying to tie the small but rugged generator I was operating to the well pump (1500 running to 2000W peak since I have a 3/4 HP well pump) was going to be a triumph of hope over reason given just the sump pump load I was dealing with. I now had to become more than just a power utility, I had become a municipal water district and devise a domestic source of grey water for the bathrooms.  More about this in a minute.

“Prior planning prevents piss-poor performance”

Not knowing the duration of the CL&P outage (frankly I thought we’d be down for a day, maybe two. Ha! Big surprise!!!), I also had to determine fuel consumption while on load vs. fuel supply on hand (it is a gasoline powered generator).  I quickly recognized that I was using about 10 gallons in 24 hours. If this went beyond a few days, I either had to find a fuel source from unaffected areas, or, tap into the static supply I had sitting in the driveway (the fuel in four parked cars). Siphoning is not on my list of preferred activities, so I found a source nearly 40 miles distant to fill my fuel containers (of which I did not have a sufficient quantity!).

Having properly grounded the generator, located it safely away from the house, run the lines, plugged in the necessary objects, I commenced the startup. Recognizing that the generator had to be started off load first, I disconnected all the distribution lines from it and started it. One pull and off it went, a good start! I let it run for about 3 minutes (I didn’t have too long as the sump was filling without regard to my time scale) and then first plugged the sump in.

The generator did what I expected it to do: since the sump immediately started when energized and started to pump, it was drawing its peak load and the generator sagged a bit but then recovered as the pump came to a steady run state.

Next I energized the fridge, and again, the generator sagged a bit as it experienced the 2400W starting load, until it assumed its roughly 1200W run mode. Luckily, the sump was not running at the same time the fridge started!

“Expect the unexpected, don’t rely on luck but recognize it when it greets you”

So here I was, plant manager of a base load 3.5 kW power station with a native load that had to be served but which required some rolling blackouts. My fuel supply was not interruptible, it was totally interrupted! Thus I had to secure spot fuel and keep sufficient stores on hand to keep at least two day’s worth. I had to not only find the fuel, I had to find new fuel containers to store it in. Try finding additional gas containers of sufficient size when the entire region is doing the same thing. Luck turned to my favor as I found additional proper fuel containers in two separate locations which increased my on hand supply to roughly 3 days worth.

“Know your equipment, how to set it up, how to run it, how to maintain it.  Safety is everything.”

As I write this, there have been literally dozens of people funneling into emergency rooms with carbon monoxide poisoning and sadly, at least 5 people have died from this. Portable generators are marvelous devices that serve a critical function in emergencies. If you have never operated one, don’t know about where to place them to eradicate CO poisoning, understand that they produce the same deadly amperage that exists in the normal AC household circuits from the regular power company, etc then don’t even buy one or use one. From Electric Energy Online.com:

Lineman Killed By Generator Back Feed – Victim Helping Restore Power in Alabama

 

Alabama, July 14, 2005 – Sumter Electric Cooperative (SECO) officials report that a South Carolina lineman helping to restore power in Alabama after the damage caused by Hurricane Dennis was killed late Tuesday, reportedly by an improperly installed customer generator. 

According to SECO Director of Public Affairs Barry Bowman, the lineman was helping to restore power lost during Hurricane Dennis to customers in Alabama. The report Bowman received indicated the lineman was working on a power line that was supposed to be dead. It was not. 

“Tragically, the line this technician was working on was not dead,” said Bowman, “The line he was trying to repair had been re-energized by a customer who had improperly hooked up a generator and created a back feed of electricity from the generator into the supposedly dead line. The death of a lineman who was there to help the victims of Hurricane Dennis was the horrible result.”

 

“Trust your instruments…. what instruments?”

Now that I had the power plant running and serving the needs of the native load, I was constantly concerned about knowing whether the sump pump had died (it was cycling every 2 minutes at one point due to the massively wet snow outside feeding it) or whether the circuit on the generator had tripped, and while the engine was running, the distribution lines were not energized. Like the aux operator at a power plant, I began by making constant rounds to check the generator, the pump, the fridge, etc. After visualizing nightmares of waking up to a basement filled with water due to pump failure, I realized I needed a control room with DCS monitors that would enunciate alarms for any problems. Wishful thinking. However, I had a crude but effective instrument DCS analog working for me during the day and night that I only recognized belatedly.

The one light I had in my “control room” (actually my office) would dim for a brief second when the sump pump energized and would then regain its brightness. Seeing this occur for a few days, a brilliant flash of the obvious hit me: I realized that this intermittent dimming meant that the pump circuit was drawing power to start and once the light regained brightness, there was current flowing to the pump (the most parasitic draw on the system). This was also accompanied by the audible sound of the generator outside reaching its peak power output to contend with the startup of the pump, which then resumed its more normal humming. So I had two very crude control indicators: a lamp that would dim/recover and the sound of the engine outside changing its resonance for brief periods.

If the light dimmed and kept dim or fluttered, I knew I had a problem of either the fridge and the pump having energized at exactly the same time (pushing or exceeding peak load of the generator) or a distribution line (extension cord) had lost its tight connection. I’d then have to physically check the entire system (which I did on a normal basis anyway). This helped during waking hours, but what about when I tried to sleep?

I installed a small light in my sleeping quarters with the hope that should the generator trip or power from the field shut off, the difference of the light turning off would wake me. Fool proof? No, but absent an audible alarm, it was the best I had at hand.

“Be creative in developing solutions. What water supply?”

Plenty of potable water was on hand as we stocked bottled water and bottled water was readily available locally. We had started to fill the Jacuzzi tub in the basement in anticipation of the storm, but the outage occurred on its time schedule and not ours, thus, the tub did not get filled even half way.

However, you need copious amounts of water to keep the sanitary systems running (how’s that for treating toilet water politely?). After using the existing supply for a day or so, and after an abortive effort of placing snow into the tub in hopes that it would melt (it didn’t sufficiently) to replace the water, it was time for better solutions.

I had to determine how to get water to the bathroom toilets so that we could exist in at least a semi civil state. Then it occurred to me: I was pumping hundreds of gallons of clear water to the outside via the sump pump. If I could connect a transfer pump to this source, and redirect it, I could at least fill the large Jacuzzi tank in the basement and thereafter carry 5 gal buckets to be staged in the upstairs bathrooms. The local home center by this time had opened, and while they of course had no fuel cans, generators or batteries left, they did have a transfer pump!

What I had to do was shut down the sump pump, energize the transfer pump , and as the water rose, redirect that sump water to the tub. It worked!

“Required maintenance vs. deferred maintenance –  you have to keep running!”

The manual on this tough little generator indicates that the oil needs to be changed every 100 operating hours. I had a decision to make: the engine was running superbly, but I knew that at some point, running any engine 24/7 would result in fatigue or failure. The problem of course was that changing the oil, although a relatively simple task, would take time (~20 minutes and the sump filled to overflowing in just a bit over 40 minutes) and any glitch (cross thread the oil plug when replacing it, fail to get the oil indicator threads back in after changing, etc) would be a fatal error as the unit would not be able to be run without a properly sealed oil supply. Result of doing the maintenance and having it go pear shaped: swimming pool in the basement. Result of the engine failing due to lack of proper maintenance: swimming pool in the basement. Operational and mission risk assessment in its most basic form.

In the normal course of running power stations, plant managers have to make similar deferral of maintenance decisions based on demand, profitability, risk, etc.  These drive insurers nuts, but then again, the insurance companies don’t have to contend with the demands placed on a plant manager.

So here is what I determined; every time I had to re-fuel the generator, I would scrupulously check the oil level, add proper oil type if needed, very carefully re-thread the oil filler plug if I did add oil; but I had to forego the changing of the oil until the CL&P outage was over. I was relying on the hope that the generator manufacturer had placed enough robust engineering into the lube system that the engine was going to be more forgiving than the 100 hour recommended change period.

Thus, given the options and the possible results; I would run my plant 24/7 and would run it to refusal if the oil change out was the sole determining factor of whether to shut it down or continue running. The generator operated simply perfectly through all of this. After the power was restored, it received its much deserved but overdue oil change and all other actions necessary to ensure it is ready when called upon again.

Lessons Learned

The outage for our house lasted from 3:30 PM Saturday 29 OCT until 6:30PM Thursday NOV 2nd (unfortunately as I write this at 8:40 AM on Monday NOV 7, there are still 50,000 CL&P customers still without electricity). For the power plant manager who runs a real power station, the stakes are higher and the issues more complex, but the need for studying the lessons from other’s failures is the right path to dealing with those they might encounter. What lessons did I extract from my brief time as a 3.5 kW (peak) power plant manager?

  1. Be prepared for the unexpected and don’t rely upon gov’t or the industrial commercial/infrastructure to be there in the short term in an outage like this. Self-reliance and preparation is essential.
  2. Think of plausible, adverse scenarios that might occur and then make plans for how to successfully respond to these. In my case, this took the form of having an emergency portable generator, having fuel on hand and knowing how to set it up and safely run it.
  3. Plan for the worst but hope for the best: I was woefully deficient in my estimation of what amount of fuel supply to store. I also did not have sufficient containers on hand initially to even get a sufficient supply on hand. A larger generator is in the future.
  4. Be creative in thinking of ways to overcome the unexpected. Not everything requires a high tech solution. My light dimming instrumentation solution during the outage reminded me of a response I received years ago from a plant manager of a relatively large coal fired power station where we were doing an insurance and operational risk evaluation. He was resistive to the transition of his control room to electronic displays of water levels in the steam drum from the use of the original sight glass mirrors. When I asked about why he was so reticent to move to the electronic version he simply replied: “I never had a mirror burn out….”. Pretty basic logic that worked for him, as I can’t tell you the number of other stations I have visited where the bulbs in the enunciation panel are burned out!
  5. Recognize that you should operate and maintain the equipment per OEM instructions, but also recognize when you’re in an emergency or other compelling condition, that you may have to prudently make a risk determination to deviate from these. When this occurs, try to accommodate the intent of the standard instructions with other actions that supplement standard actions while continuing the mission (in my case, it was making sure I checked the oil more often than recommended).
  6. You can’t plan for an emergency when you’re in the middle of one. This probably is the same as #1, but it bears reiterating. My planning was too cursory.
  7. If you’re buying equipment for emergency purposes, buy good quality equipment and, if possible, purchase a bit more capacity & capability than you need (this could apply across the board to not just the size of a generator, but anything else you may need to rely upon).
  8. Safety is never a negotiable issue in terms of how you operate.
  9. Did I mention the fire that occurred half way through the outage in the house?

 

For those of you who may be interested, drop me a line and I’ll explain the fire that erupted, the root cause for this, and what I had to do to react.

 

 

Categories: Uncategorized | 1 Comment

The Sustainability Handbook – Shirley J Hansen & James W Brown

When I was asked by Shirley Hansen to write a chapter on “Green Insurance”, I felt
both intimidated and privileged to be able to place my work alongside some of the
people who are frankly the best minds in the business regarding sustainability.

The world of insurance and risk changes moment by moment, but it is a subset of
the issues management and leaders must face. Managing sustainability is indeed
the challenge of our times as we encounter new imperatives and hurdles in our
workplace regarding regulations, new codes, energy, health, etc.

As the sustainability imperative in the design, development, engineering and
construction of our buildings and workplace becomes ever more pronounced, this
handbook, created under the vision and effort of Shirley and James, is candidly a
must read for those concerned about our path forward. I urge you to check it out!
The link is on this page.
Categories: Uncategorized | Leave a comment

Welcome!

Welcome! What a time to launch a website about a business that focuses on identifying, selecting, modifying and managing risk!

As I write this (Monday August 8, 2011 @ ~2:35 east coast time) the following headline screams across the online WSJ:

 “S&P Extends Ratings Cuts, Negative Views on Insurers”

 Contributing to (and exacerbating) this atmosphere is the DOW sinking 530+ points!  Where it ends today and for the foreseeable future is anyone’s guess, but there is no shortage of media commentary offering their advice and prognostications. I have heard the market (and indeed most financial endeavors including insurance as well) aptly described as operating off of fear and greed… and fear is temporary. Today, risk and fear have joined hands.

Risk is a topic of great fascination for me personally and is of critical importance to the operation and maintenance of electric generation plants, the power generation industry, and construction projects. Risk assessment is a field of study that is like a race without a finish line in my estimation.

Risk constantly changes and morphs as time marches on and we alter activity, practices, equipment and protocols in our plants and projects. The reality of the law of unintended consequences enters the field rather vibrantly at times.  The impact of human element and management influence is alone enough to justify endowing a chair at a major university for study in this regard!

I have had the great privilege of learning from some of the best in the business by being involved in the insurance, power plant operation and construction related industries for ~30+ years, and I am humbled nearly every week when I encounter something I either didn’t know or had a poor understanding of. I pride myself on being reasonably intelligent, but the lesson to me is that the learning never stops. As an aside, I would urge readers to seek out works by Jim Chiles and Professor Charles Perrow; both have influenced my views on risk greatly and I often utilize their material in training clients.

I have come to the belief that only through a detailed, flinty determination, coupled with on-site study and scrutiny of power plants and construction projects (along with assessment of the human element factor) can one begin to properly identify and understand the risks that can disrupt operations.

However, operating power plants successfully requires a holistic approach, which, while including risk assessment, also has a myriad of critical factors to ensure success. As such, Rob Swanekamp and I have authored a book to be published soon:

 STEAMING INTO GLOBAL COMPETITION –

A TRAINING MANUAL FOR POWERPLANT LEADERS OF TODAY. . . AND TOMORROW”

The book contains chapters on how to competitively lead or manage operations, optimize maintenance, handle environmental challenges, move from supervisory to management positions, master plant financial issues and manage risk.

It starts with the premise that plants are operated by people, not simply distributed control systems. However, even an application of the best practices from the book requires that you approach the process like a total quality effort: observe, apply, check, modify, then repeat. Only then can one apply enduring elegant solutions to manage the plant while modifying risk.

Regarding risk, one must engage directly and diligently. In other words, plants and projects have to become more self reliant and informed on how to identify, select, modify and transfer the risks associated with their own endeavors. Two quotes come to mind here:

  • “If you’re not actively involved in getting what you want, you really don’t want it.” Pete Williams;
  • “Avoid the crowd. Do your own thinking independently. Be the chess player, not the chess piece.” Unknown

This is not to say that the work of others that has informed us of risk in the operation of power stations or construction projects needs to be ignored (only to be re-invented). Quite the contrary, let’s capture these best practices. But don’t put them in concrete, as we will always learn more and better practices over time.

In my opinion, the science and art of loss prevention has come to a point wherein the necessary diligence in surveying and assessing best practice has deteriorated as the costs for doing this has crowded out the premise of doing it in the first place. Stealing the essence from another saying… “If you think surveying and inspections are expensive, try losses.”

So while my firm provides risk assessment, loss prevention, claims support and surveying for our valued clients; I hope to create a vehicle to train and transfer the know how and intellectual capital such that clients can become better informed and directly engaged in the process. A lessons-learned approach has to be a big part of this.

There are many great risk professionals still left in this business. Some are in carriers and brokers, some are with third parties, some are in client organizations. However, the compelling pressure of cost vs. value has sometimes subordinated much of this expertise by restricting the time necessary to do a proper review into a cursory cut/paste effort.

There are entities which profess to perform this function for free (presumably it is embedded into their premium or exists as part of their expense ratio). “Free” engineering is sometimes worth what you pay for it. Which brings to mind another quote applicable to insurance carriers as much as it is to their clients:

“The perfume of the premium is always overcome by the stench of the loss” – (I’m not sure who uttered this first, but I have used it enough to claim partial ownership!).

Ambrose Pierce once stated rather cynically:

Insurance: An ingenious modern game of chance in which the player is permitted to enjoy the comfortable conviction that he is beating the man who keeps the table.”

I reject this cynical view; yet, understand the sentiment and premise given some history on the industry. So what is it that I hope to do with this blog? I am hopeful to engage others in a collegial fashion to seek the ideas, emerging best practices and elegant solutions that others have encountered in an effort to share these transparently with all the stakeholders in this industry.

Hopefully we can impart the lessons learned to avoid the incidents and keep all stakeholders out of the claims business (which sometimes results in the following sentiment: “We’re not happy till you’re not happy

A good friend of mine has an interesting theory on perception of risk and the effect it has on insurance (one which I am becoming somewhat convinced of). He has often told me that (paraphrasing) … you have great difficulty in getting people to purchase insurance to protect against a risk that does not exist in their conscious memory of ever having manifested itself.” So if they have never experienced loss from a flood, fire, explosion, collapse, quake, electrical or mechanical breakdown, etc, it simply does not exist in their mind. One might as well suspend history at this point.

He has become even less sanguine based on current trends of avoiding risk, sensing that it is deteriorating to a point wherein “…you can’t get people to protect against anything that hasn’t happened in the past five years.” Effectively, he is asserting that the prospect of a compelling risk may not exist in the mind of the person who has no recollection of this ever happening before, or even recently! Perfect, I hope he is wrong but my sense is he is onto something here.

OK- it is 4:16 now and the Dow is down 634 points; risk marches on and fear is still in charge of the market (but eventually, it will be crowded out by greed).

I invite you to join me and look forward to your input in the months and years ahead about this issue of assessing risk in power plants and construction projects; and how we can manage it more effectively for all stakeholders.

 

Bob Sansone

Categories: Risk, Welcome | Tags: , , , , , , , , | Leave a comment