iStockphoto/Thinkstock®

Every industry fears the disruption, and sometimes destruction, that a disaster brings.

That’s especially true in the financial services industry, where even a few thousandths of a second of disrupted data flow can lead to great loss of money.

Understandably, most people think of big-ticket natural events as the most common cause of data disruption. But they’re not always the culprits.

“Natural disasters cause the most anxiety because of connectivity disruptions,” says Eric Flick, director of disaster recovery for the Centurion suite of products and services at Symitar. “When people pick up a landline phone, they expect to hear a dial tone. Similarly, everybody expects the Internet to always be available. But what happens when a construction crew accidentally cuts through a fiber-optic cable?”

In that case, says Flick, disaster recovery specialists know how to do smaller patches as well as major recovery efforts. “We’ll provide alternative means of connectivity, such as restoring some connectivity via laptops while the severed cable is being repaired. In most cases, we can restore Internet access in less than an hour.”

He says no new type of manmade threat has emerged in the past five years. But one scenario few people consider is train derailment.

“Trains carry a huge assortment of freight, including dangerous chemicals, so you have to be ready for the threat of a toxic cloud,” Flick warns. “This means having an evacuation and business continuity plan.”

Most man-made disasters involve core system hardware crashes, says Scott Huseman, director of operations at Harland Financial Solutions’ enterprise services center. “That’s why we highly recommend that our clients have hardware maintenance contracts and that they monitor their systems. We also advise them to remember the distinction between local and regional disasters.

“A local disaster,” he continues, “may involve just one branch—a situation that’s fairly easy to work around. A regional disaster, however, may bring down your entire branch network and core processing system.”

Still, concern over man-made disasters doesn’t pre-empt concerns with natural disasters among recovery specialists.

“In the realm of natural disasters, we’ve been noticing more occurrences of unusual rain events in places where they don’t usually happen— flooding in Calgary, Alberta, and in central Wisconsin, where those areas received a month’s worth of rain in four hours,” says Flick. “Flash floods can do everything from damaging or isolating a facility to preventing employees from getting to work or members from accessing services.”

Recovery resources

Huseman says Harland Financial Solutions’ core disaster recovery offerings include:

►Disaster recovery services, featuring core system and ancillary services recovery. Basically, the company reloads a credit union’s data to get it up and running again.

►Data storage solutions, a disk-todisk encrypted data backup solution, which stores the data offsite at the company’s data center. Th is service goes hand-in-hand with disaster recovery services.

►Co-location services, where Harland Financial Solutions provides clients space in its data centers for client-owned servers and other computing hardware, enabling backup core processing and other systems.

►Enterprise resource services. “We can act as UNIX system administrators and/or as a client’s computer operations department,” Huseman explains. “This assists with disaster recovery because we know the client’s systems.”

Larger credit unions, despite their size and resources, find it’s not always necessary or feasible to do everything in-house, says Huseman.

“For example, credit unions can have a tendency not to pay UNIX system administrators at the higher end of the salary scale, so they often use younger people who leave once they’ve acquired some experience,” he says. “It’s a revolving door.

“The solution,” Huseman adds, “is to outsource with a company that has experienced, higher-paid administrators who have been on the job for years and overlap in terms of vacation and sick time.”

The main resistance Huseman encounters to outsourced disaster recovery is when a credit union wants to “own it, hold it, touch it.

“But the challenging economy and limited resources have made credit unions realize that some tasks are pretty basic and can be outsourced,” he continues. “By outsourcing they can focus on long-range planning.”

One aspect of the credit union movement gives disaster recovery specialists considerable help in quickly restoring a client’s business, says Huseman: the industry’s spirit of cooperation.

“For our outsourced clients, if single-branch Credit Union A has a disaster,” he says, “we can arrange for it to go to Credit Union B and set up a teller window that’s directly and securely connected to its proprietary database and core system. Its members can come in and conduct business as though they are at the home branch.”

Best practices

Credit unions can take several steps to prepare for future disasters, Flick says. Among them:

►Be able to accommodate staff at other locations.

►Install redundant facilities. “Say you have 12 branches—make half of them physically larger to accommodate extra staff and members,” he says.

►Use social media to communicate with members in the event of a business disruption.

Many credit unions offer mobile services to members who use tablets and smartphones. Push members to that channel during a disruption. Also, encourage members to use shared branching networks during times of distress.

Flick says some credit union clients keep a mobile unit on retainer that can be configured as a temporary branch.

“It’s a customized semi-trailer with power, air conditioning and heat, Internet access, and teller windows,” he explains. “We can have one onsite within 48 hours.”

The best-prepared credit unions are those that test their disaster recovery plans at least once a year, Flick adds.

“I use a sports metaphor to describe what that plan should do: Practice the way you plan to play,” he says. “Practice for a real-world event and don’t just do a check-the-box exercise. Expand your plan’s scope as you encounter elements you hadn’t thought about.”

Huseman agrees that even with a good test plan, a credit union should take careful note of where a plan deviates from the expected.

“It’s OK if you run into issues when you exercise a disaster recovery plan,” he says. “It’s best to find and fix a flaw during a test rather than during a real-life situation.

“Don’t be content,” Huseman adds. “Keep adding to the scenarios you test. Spell out clear roles and actions and test them at least annually.”

Flick says disaster recovery providers should be very involved when credit unions practice their plans. “We’re right there with them and can point out the gaps. We emphasize there is no such thing as a failed test—you always learn something that better prepares you for the real thing. We document restoration times and include them in letters for the credit union’s auditors and examiners.

“The fact that management has a better idea of realistic restore times is helpful,” Flick continues. “One area where there can be tension is between management and the information technology (IT) staff—management wants services restored right now and IT can only do so much if it has to spend time, say, restoring the core.”