Surviving Outages

Surviving Technology Outages

Implementing redundant and resilient solutions

How can I help my business survive technology outages?

If you missed our last article, we outlined a few reasons why outages can occur and an easy way to quickly assess the potential financial impact on your business.

We wanted to re-state that given the ever increasing complexity of technology, IT and telco related outages are unfortunately just something that every business has to try mitigate.

Even the largest banks are not immune – July this year, CBA had a 9 hour online payments outage. Imagine the impact on your business if you were unable to process payments for 9 hours!

Outages not only affect your bottom line and your employees time, but also your brand reputation and the trust you have built with your customers. So how can your business implement a mitigation strategy?

There are two distinct areas to address: the technology in place to minimise outages in the first place, and secondly, your strategy for when unforeseen issues arise.

In this article, we’ll discuss the technology and in the next one we’ll share a few ideas around testing your outage strategies and ensuring they are robust.

Vintage car - don't let an outage wreck your business

What redundant (or backup) solutions should I put in place?

If you are currently operating or considering one of our Mitel phone systems, or network solutions, there are a number of ways to assist with redundancy.

Virtualisation

If your business is using an on-premise PBX system, it is worth considering moving to a virtualised solution. This removes the physical hardware from your office and swaps it for a software based system that lives in the cloud (in one of our data centres). This has the added benefit of being easier to upgrade and maintain, and has a level of redundancy built-in via clustered Virtual Machine environments.

Resilient Controllers

In a larger organisation it is worth adding a second Mitel controller to your system. This backup system is called a Resilient Controller. If your main phone system fails for some reason, the second one instantly takes over, ensuring your communication lines stay open.

Generally you would locate the resilient controller in a separate location to minimise localised outages and have a separate set of SIP trunks dedicated to this failover scenario.

Enterprise SIP

eComms can offer various types of SIP services for your phone calls, including Enterprise SIP which offers dual circuits within the Carrier network. You effectively get two separate SIP logins and these can be setup to suit different scenarios. As Primary and Secondary SIP links, or as a load-balancer with calls shared evenly across the services.

To further enhance this, there are Session Border Controllers (SBC’s – which control call traffic flow) in each state to give you Carrier level resiliency. Or in other words if one data centre blows up, another SBC steps in and keeps your phone calls working.

Failover Data Connection

The majority of outages are related to data connections failing. This could be due to a civil contractor down the road accidentally cutting a fibre link, severe weather like the recent Australian bushfires, or hardware failures within the NBN or other networks.

The easiest and often most cost effective solution is to add a 4G backup option. We generally design our network and phone solutions with this as a standard offering. It actively monitors your network connection and will automatically flip over to 4G if an outage is detected.

Network diversity

If you have high bandwidth requirements a 4G backup may not be fast enough for your data network. Most buildings are serviced by multiple Tier 1 carriers and it is worth considering a multi-path network diverse solution. For example, you may decide to connect your business via one carrier’s fibre link and use a different provider for the failover connection.

This also helps ensure you have both redundancy as well as network diversity. Complex solutions like these require proper planning and testing within your environment and is the type of challenge our Project Managers and Network Engineers love to solve.

Routers and switches

While hardware is generally quite robust, it can fail and in mission critical environments it is worth looking at duplicating your primary routers within your network. At the very least ensure you are using business grade hardware and not those designed for less demanding consumer applications.

 

How much redundancy is enough?

Where do you stop though? What if the power fails? Then all the above systems would fail unless you had a generator, or similar, in place. What if your building is flooded? Unlikely, but do you have a business continuity plan?

This is where you need to weigh up the commercial costs and practicality versus the revenue loss. The good news is that data and SIP connections have dramatically reduced in price over the past few years and it’s quite easy to build the financial case for proper redundancy in your business.

Don’t wait until you’re off the air! Call us on 03 8542 8542 or via our online form for a free phone, network and redundant system appraisal.

Mathew Jameson
by Mat Jameson
Communications you can trust.
Enabling business transformation through effective communications.