How To Protect Your Infrastructure Against Power Outages
Sam Marashi
The recent massive power outage in December due to an ice storm in Toronto was another reminder of the effects of power outages on IT infrastructure. Without power many organizations remained closed or barely continued operations for days. This may be an acceptable risk for some organizations but not to all.
I was surprised to see how many of our own clients did not understand the implications of such outages. Some believed that a UPS (battery backup) was the answer for power outages. That is true for short outages but they can’t do you much good for an extend power outage! So when I was asked to write about this I was thrilled to put in my 2 cents. I hope that at the end, the reader will have a better understanding of the various effects a wide spread outage will bring.
In the western world we are fortunate to have a fairly reliable source of electricity to run our cities and economies so as a result we don’t give much consideration to business continuity in case of power outages. Most of us believe major blackouts in North America are rare, but are they really? I did some research and found that there has been an average of approximately 1 major outage per year on the North American grid alone. This does not include local outages such as one that took place when I was writing this article which left most of the west end of Toronto in darkness for a few hours. Here are the major power outages in North America in the past few decades:
| Date | Duration | Impact |
| February 2, 1971 | New York, Manhattan, Bronx and Queens | |
| July 4, 1976 | Couple of hours | Utah and parts of Wyoming |
| May 17, 1977 | parts of south Florida went dark for a short period | |
| July 13, 1977 | 2 days | New York lost power for 2 days |
| September 20, 1977 | several hours | almost the entire province of Quebec lost power for hours |
| January 8, 1981 | almost the entire Utah lost power | |
| December 22, 1982 | 5 million people affected in California | |
| May 17, 1985 | few hours | most of south Florida lost power for few hours |
| March 13, 1989 | 9 hours | a geomagnetic storm knock out power in most of Quebec affecting 6 millions |
| October 17, 1989 | 1.4 million in California lost power after an earth quake knocked out a power station | |
| July 7, 1991 | nearly 1 million lost power in from Iowa to Ontario due to a wind storm | |
| October 4, 1995 | 2 million customers lost power is eastern and southern US after hurricane | |
| July 2, 1996 | Couple of hours | 2 million people from northwest Canada to western US due to summer heat |
| August 10, 1996 | up to 4 days | 7.5 million lost power from west coast Canada to US and northwest Mexico |
| January 4, 1989 | up to 3 weeks or more | 4 million people affected by Quebec ice storm stretching from Ontario to Nova Scotia, northern New York and Maine |
| July 5, 1999 | 600,000 thousand customers lost power in Quebec | |
| July 22, 2003 | a severe wind storm knocked out power to over 300,00 customers in Memphis | |
| August 14, 2003 | up to couple of days | entire northeast of Canada and US lost power due to a bug in an alarm system software affecting 55 million people |
| September 19, 2003 | Hurricane knocked out power to Ontario and nine US states affecting 4.3 millions | |
| August 31, 2005 | Hurricane Katrina hits US causing 2.6 million people to lose power | |
| August 1, 2006 | 1 day | Laurentians, Quebec lost power affecting 450,000 customers |
| August 2, 2006 | 250,000 customers in Ontario lost power after a severe thunderstorm | |
| December 2, 2007 | up to a week | Winter storm caused power outage in eastern part of Newfoundland and Labrador |
| December 8, 2007 | 7-10 days | winter storm caused power outage in Midwest US affecting 1.5 million customers |
| February 26, 2008 | a failed switch and fire caused blackouts in parts of Florida affecting 4 million people | |
| December 11, 2008 | snowfall caused power outages in parts of Louisiana | |
| December 11, 2008 | ice storm caused power outages in Massachusetts and New Hampshire | |
| December 26, 2008 | 12 hours | lightning caused power outage in the entire Oahu, Hawaii |
| January 27, 2009 | about 3 weeks | ice storm caused power outage in Kentucky and Southern Indiana for 769,000 customers |
| February 5, 2010 | several days | winter blizzards caused power outage to over 200,000 customers in Baltimore and Washington |
| March 14, 2010 | up to six days | severe windstorm knocked out power to thousands of customers in Connecticut, Long Island and New Jersey |
| July 15, 2010 | over a day | 76,000 customers lost power in southern Michigan during a heavy storm |
| July 25, 2010 | 250,000 lost power in Washington during a severe storm | |
| February 2, 2011 | Forced outages at 2 power plants and high electricity demand due to cold caused rotating outages in Texas affecting 1 million | |
| April 27, 2011 | devastating tornado outbreaks knocked out power to most of northern Alabama | |
| July 23, 2011 | about an hour | Most or northern Saskatchewan lost power due to a failed glass insulator |
| July 11, 2011 | 850,000 thousand customers lost power in Chicago during a windstorm | |
| August 27, 2011 | Hurricane Irene caused power outage for over 5 million people in US east coast | |
| September 8, 2011 | 5 million is Southern California and Arizona lost power due to a problem in a power station | |
| October 28, 2011 | a snowstorm along east coast US caused power outage affecting over 2 million | |
| June 29, 2012 | hurricane force winds knocked out power to more than 3.8 million people in Midwest and Mid-Atlantic US | |
| October 29, 2012 | Hurricane Sandy caused massive power outages affecting 8 million customers | |
| February 8, 2013 | 650,000 customers in New England US lost power during windstorm | |
| December 22, 2013 | up to 6 days | ice storm in Ontario, parts of Michigan and parts of Maritimes provinces caused power outages affecting 1.1 million customers |
These events would likely increase in their severity and frequency with what appears to be more drastic weather patterns due to climate change and the aging power grid in North America.
A wide spread power outage can impact you in ways you may not have thought about. Here are some of the scenarios that can potentially paralyze your organization:
Outage on premises – You lost power in your office. Some of your IT infrastructures connected to a UPS will function for few minutes or a little more and when the small battery runs out everything will shut down.
Internet outage – Okay, so there are power outages in your area but you are one of the lucky ones, you haven’t lost power. But your Internet is not working! As you can imagine for Internet to get to your location it has to go through a lot of equipment along the way under the streets etc. This equipment generally needs power too and although some have backup generators and batteries they eventually run out. So your web-server and email-server are up and running but they don’t do much good without Internet!
Outage at the Data Centre – There have been cases of data centres that relied on backup generators and were unable to re-fuel them so they went dark too.
So what should you do to protect your organization? The answer is to understand your risk and make sure you and others understand the best and worst case scenario. If you have mission critical systems that need to be operating 24/7 and 99.997% of times, your best option is a good data centre. And not all data centres were created equally! Here are some of the things you should keep in mind when looking for one:
- Redundant Internet connection to major back bones is a MUST
- Redundant power sources. That is not only batteries and generators. Some data centre are connected to 2 different power sources
- Redundant facility. Some data centres do have multiple locations and their data is mirrored across all of them so if one facility is down the other ones continue serving their clients without interruption. This is especially great if you don’t have physical servers.
In addition your business continuity plan should include how these systems are accessed when your office is down. It doesn’t do you much good if applications hosted in the data centre can only be accessed from your office! In some cases a good design for remote access independent from your office infrastructure must be implemented. At the end of the day a balanced and well thought plan can potentially save your organization a lot without putting a big dent in your IT budget.
So now this is an ideal configuration which works for some companies with larger IT budgets! However for many others with the constant budgetary challenges, there is still something that can be done to offer a reasonable level of protection.
Make sure that:
- At least a UPS battery backup is protecting your servers, switches and firewall/modems
- Ensure that your UPS’s are managed and are configured correctly to shut down and re-start equipment correctly
- Ensure servers and UPS’s are maintained and tested regularly and per manufacturer’s specification
- Have a plan of action and set up expectation accordingly
Although these measures will not protect you against the effects of an extended power outage, they will ensure your IT infrastructure and data is protected and can resume operation once power is restored; as in most cases when power outages occurs or when it is restored the power does not simply just go off or on, but rather flickers a few times which can cause permanent damage to electronic components and corrupt data.
Sam Marashi is the President and Managing Director of Tektonic Inc. With over 20 years of experience in IT, he co-founded Tektonic in 1998 to provide their clients with a complete outsourced IT department for Small and Mid-sized organizations with a special focus in Not for Profit/Charitable organizations and associations. Tektonic has been successful in bringing enterprise level support and uptime to smaller organizations using ITIL standards with in Industry Best Practices framework.
