How To Protect Your Infrastructure Against Power Outages

Sam Marashi

The recent massive power outage in December due to an ice storm in Toronto was another reminder of the effects of power outages on IT infrastructure. Without power many organizations remained closed or barely continued operations for days. This may be an acceptable risk for some organizations but not to all.

I was surprised to see how many of our own clients did not understand the implications of such outages. Some believed that a UPS (battery backup) was the answer for power outages.  That is true for short outages but they can’t do you much good for an extend power outage! So when I was asked to write about this I was thrilled to put in my 2 cents. I hope that at the end, the reader will have a better understanding of the various effects a wide spread outage will bring.

In the western world we are fortunate to have a fairly reliable source of electricity to run our cities and economies so as a result we don’t give much consideration to business continuity in case of power outages. Most of us believe major blackouts in North America are rare, but are they really? I did some research and found that there has been an average of approximately 1 major outage per year on the North American grid alone. This does not include local outages such as one that took place when I was writing this article which left most of the west end of Toronto in darkness for a few hours. Here are the major power outages in North America in the past few decades:

Date Duration Impact
February 2, 1971 New York, Manhattan, Bronx and Queens
July 4, 1976 Couple of hours Utah and parts of Wyoming
May 17, 1977 parts of south Florida went dark for a short period
July 13, 1977 2 days New York lost power for 2 days
September 20, 1977 several hours almost the entire province of Quebec lost power for hours
January 8, 1981 almost the entire Utah lost power
December 22, 1982 5 million people affected in California
May 17, 1985 few hours most of south Florida lost power for few hours
March 13, 1989 9 hours a geomagnetic storm knock out power in most of Quebec affecting 6 millions
October 17, 1989 1.4 million in California lost power after an earth quake knocked out a power station
July 7, 1991 nearly 1 million lost power in from Iowa to Ontario due to a wind storm
October 4, 1995 2 million customers lost power is eastern and southern US after hurricane
July 2, 1996 Couple of hours 2 million people from northwest Canada to western US due to summer heat
August 10, 1996 up to 4 days 7.5 million lost power from west coast Canada to US and northwest Mexico
January 4, 1989 up to 3 weeks or more 4 million people affected by Quebec ice storm stretching from Ontario to Nova Scotia,  northern New York and Maine
July 5, 1999 600,000 thousand customers lost power in Quebec
July 22, 2003 a severe wind storm knocked out power to over 300,00 customers in Memphis
August 14, 2003 up to couple of days entire northeast of Canada and US lost power due to a bug in an alarm system software affecting 55 million people
September 19, 2003 Hurricane knocked out power to Ontario and nine US states affecting 4.3 millions
August 31, 2005 Hurricane Katrina hits US causing 2.6 million people to lose power
August 1, 2006 1 day Laurentians, Quebec lost power affecting 450,000 customers
August 2, 2006 250,000 customers in Ontario lost power after a severe thunderstorm
December 2, 2007 up to a week Winter storm caused power outage in eastern part of Newfoundland and Labrador
December 8, 2007 7-10 days winter storm caused power outage in Midwest US affecting 1.5 million customers
February 26, 2008 a failed switch and fire caused blackouts in parts of Florida affecting 4 million people
December 11, 2008 snowfall caused  power outages in parts of Louisiana
December 11, 2008 ice storm caused power outages in Massachusetts and New Hampshire
December 26, 2008 12 hours lightning caused power outage in the entire Oahu, Hawaii
January 27, 2009 about 3 weeks ice storm caused power outage in Kentucky and Southern Indiana for 769,000 customers
February 5, 2010 several days winter blizzards caused power outage to over 200,000 customers in Baltimore and Washington
March 14, 2010 up to six days severe windstorm knocked out power to thousands of customers in Connecticut, Long Island and New Jersey
July 15, 2010 over a day 76,000 customers lost power in southern Michigan during a heavy storm
July 25, 2010 250,000 lost power in Washington during a severe storm
February 2, 2011 Forced outages at 2 power plants and high electricity demand due to cold caused rotating outages in Texas affecting 1 million
April 27, 2011 devastating tornado outbreaks knocked out power to most of northern Alabama
July 23, 2011 about an hour Most or northern Saskatchewan lost power due to a failed glass insulator
July 11, 2011 850,000 thousand customers lost power in Chicago during a windstorm
August 27, 2011 Hurricane Irene caused power outage for over 5 million people in US east coast
September 8, 2011 5 million is Southern California and Arizona lost power due to a problem in a power station
October 28, 2011 a snowstorm along east coast US caused power outage affecting  over 2 million
June 29, 2012 hurricane force winds knocked out power to more than 3.8 million people in Midwest and Mid-Atlantic US
October 29, 2012 Hurricane Sandy caused massive power outages affecting 8 million customers
February 8, 2013 650,000 customers in New England US lost power during windstorm
December 22, 2013 up to 6 days ice storm in Ontario, parts of Michigan and parts of Maritimes provinces caused power outages affecting 1.1 million customers

These events would likely increase in their severity and frequency with what appears to be more drastic weather patterns due to climate change and the aging power grid in North America.

A wide spread power outage can impact you in ways you may not have thought about. Here are some of the scenarios that can potentially paralyze your organization:

Outage on premises – You lost power in your office. Some of your IT infrastructures connected to a UPS will function for few minutes or a little more and when the small battery runs out everything will shut down.

Internet outage – Okay, so there are power outages in your area but you are one of the lucky ones, you haven’t lost power. But your Internet is not working! As you can imagine for Internet to get to your location it has to go through a lot of equipment along the way under the streets etc. This equipment generally needs power too and although some have backup generators and batteries they eventually run out. So your web-server and email-server are up and running but they don’t do much good without Internet!

Outage at the Data Centre – There have been cases of data centres that relied on backup generators and were unable to re-fuel them so they went dark too.

So what should you do to protect your organization? The answer is to understand your risk and make sure you and others understand the best and worst case scenario. If you have mission critical systems that need to be operating 24/7 and 99.997% of times, your best option is a good data centre. And not all data centres were created equally! Here are some of the things you should keep in mind when looking for one:

  1. Redundant Internet connection to major back bones is a MUST
  2. Redundant power sources. That is not only batteries and generators. Some data centre are connected to 2 different power sources
  3. Redundant facility. Some data centres do have multiple locations and their data is mirrored across all of them so if one facility is down the other ones continue serving their clients without interruption. This is especially great if you don’t have physical servers.

In addition your business continuity plan should include how these systems are accessed when your office is down. It doesn’t do you much good if applications hosted in the data centre can only be accessed from your office! In some cases a good design for remote access independent from your office infrastructure must be implemented.  At the end of the day a balanced and well thought plan can potentially save your organization a lot without putting a big dent in your IT budget.

So now this is an ideal configuration which works for some companies with larger IT budgets! However for many others with the constant budgetary challenges, there is still something that can be done to offer a reasonable level of protection.

Make sure that:

  1. At least a UPS battery backup is protecting your servers, switches and firewall/modems
  2. Ensure that your UPS’s are managed and are configured correctly to shut down and re-start equipment correctly
  3. Ensure servers and UPS’s are maintained and tested regularly and per manufacturer’s specification
  4. Have a plan of action and set up expectation accordingly

Although these measures will not protect you against the effects of an extended power outage, they will ensure your IT infrastructure and data is protected and can resume operation once power is restored; as in most cases when power outages occurs or when it is restored the power does not simply just go off or on, but rather flickers a few times which can cause permanent damage to electronic components and corrupt data.

Sam Marashi is the President and Managing Director of Tektonic Inc. With over 20 years of experience in IT, he co-founded Tektonic in 1998 to provide their clients with a complete outsourced IT department for Small and Mid-sized organizations with a special focus in Not for Profit/Charitable organizations and associations. Tektonic has been successful in bringing enterprise level support and uptime to smaller organizations using ITIL standards with in Industry Best Practices framework.