The Calm Before the Storm

By: Dan Fanning

The sun is shining and the IT applications supporting your business are working seamlessly behind the scenes to deliver value and drive revenue. The idea of a weather-induced data center outage may be the furthest thing from your mind. While flooding, water and wind damage can be problematic, lightning storms can create electrical damage to critical infrastructure that can be devastating. Anticipation and preparation are the keys to maintaining operations during these kinds of catastrophic events. Preparing for storm season is a good way to mentally and physically sharpen the edge of your operational mindset. Over the last thirty years, we’ve compiled numerous strategies that enable a proactive approach to data center storm preparedness. With storm season approaching, it’s time to run some drills, conduct inspections and review emergency procedures and our preventative maintenance (PM) programs.

STORM WATCH PLANS – Checks and Drills
The volatility of weather systems during introduce an often-underestimated risk to data center operations. Accordingly, we are scheduling walk-throughs with our clients. Our first step is to review the lightning protection system including surge protection devices, the main circuit breaker set points, as well as the status of any critical preventative maintenance activities including recent infrared scans. The surge protection devices should not display any current alarms, and the isolation circuit breaker should be closed.

Here’s a partial list of what we routinely inspect during these storm watch preparedness cycles:

  • Check the roof leaders and drain
  • Check the pitch pockets and roof condition
  • Inspect storm sewer grates and covers for blockages and accumulated debris
  • Have your landscaping contractor identify and remove potentially problematic plantings
  • Inspect the building glazing and roofing joints for wear
  • Check the utility (water, fiber, power, sanitary) entrances and penetrations,
  • Inspect the cooling towers, external pumps, diesel generator fittings (louvers and radiators)
  • Exercise your generators and check fueling levels and your spill prevention, control and countermeasure (SPCC) plan
  • Check your ATS delays and transfer set points
  • Check the area surrounding overhead feeders, which are often subject to damage during storms
  • Inspect pad mounted transformers and vaults for flood potential and drainage issues

STORM WATCH PLAN – Participants
We always invite the IT staff, NOC operators, and the security team members to the walk-through, which presents a tremendous opportunity to educate and inform all the stakeholders that are dependent on the data center facility. The goal is to help everyone remember where critical equipment is located, where and how alarms are annunciated, and to confirm access to emergency equipment.

Most of the third-party vendors we have contacted welcome the opportunity to conduct drills with their clients. For a deeper training session, we often involve additional stakeholders from Risk Management, Safety, Audit Compliance, IT leadership and Facilities in the storm watch preparedness drills. This is a great way to invite interdepartmental exchanges with your Critical Facilities Management staff. Face-to-face interaction under drill conditions is a far better environment to meet your counterparts rather than under the stress and strain of a full-blown outage event.

Our storm preparation inventory check includes:

  • Tarps, plastic sheeting and duct tape
  • Plywood
  • Fasteners
  • Wet-dry vacuums
  • Flashlights
  • Pumps and hoses
  • Dehumidifier
  • Batteries
  • Mops and buckets
  • Plastic trash cans and liners
  • Pampers

Wait, did that say Pampers?

Diapers may seem to be an odd thing to stock, but I was shown a trick by an old-timer many years ago. The disposable diapers will hold many times their weight in water, and a drip falling onto a diaper will not splash. The adhesive tabs allow you to stick them on IT equipment, which makes them a great tool to control drips from above.

STORM WATCH PLAN – Notification, Escalation and Dispatch
Use this time to update your emergency contact lists including both internal staff and external vendors. 24/7 numbers and assignments often change over the course of a year, and it is not uncommon to discover 20% of the contact information is outdated.

STORM WATCH PLAN – Event Documentation
Documentation is critical to risk mitigation. We always work from an Incident/Illness/Injury report template (III Report) that all departments recognize as the standard reporting tool. The III Report is the document that captures the details for any event that puts the data center facility, personnel, or operations in jeopardy. Specific alarms from the critical equipment can be reported from the data center infrastructure management (DCIM) system.

STORM WATCH – Recovery
Don’t let a weather event be the first time you review and drill your Emergency Policy and Procedures (PnP) and the Methods of Procedures (MOP). Critical MOPs including Emergency Power Off (EPO), utility failures, UPS bypass procedures, main-tie-main operations, Automatic Transfer Switch (ATS) operations, and bypass modes require solid documentation and repetitive drills. Storm watch and preparedness planning presents a fantastic way to test the operational plan.

Severe weather can take even the most hardened facility offline. Mitigating risk begins with a solid operational plan, a commitment to training and continuous drilling and testing. Storm preparedness presents a great opportunity to bring the diverse stakeholders together to recommit to education and preparation. Proper use of the “calm before the storm” translates to resiliency during the storm.