When Disaster Strikes
CU personnel who have experienced disasters offer advice on business continuity planning.
When Louisianans think about floods, they usually think of “water falling from the sky or blown in by the wind,” says Mignhon Tourné, CEO at $300 million asset ASI Federal Credit Union in Harahan, La. As a survivor of hurricanes Katrina in 2005 and Gustav in 2008, the credit union has endured more than its share of natural disasters. But in January 2011, disaster of another sort struck ASI Federal’s corporate headquarters.
During a three-day holiday weekend, a toilet in the credit union’s main branch and corporate offices malfunc-tioned and water poured continuously onto the floor. To make matters worse, rolls of toilet paper stored on the bathroom floor floated over to the floor drain and plugged it up. When some of the credit union’s IT staff stopped by the office later in the weekend, they were greeted by nearly a foot of water on the building’s first floor.
Completely renovating the first floor took three months. “It was a major interruption to our business,” Tourné says. It’s also a reminder that disaster recovery planning must consider not only major weather-related catastrophes, but also less spectacular events.
“As much as people think about hurricanes, tornados, and other disasters, much of what we do relates to events that never make the news,” says Gary Yeager, vice president of sales at Ongoing Operations, a disaster recovery firm and CUNA Strategic Services alliance provider.
Another bit of advice you’ll receive from people who have experienced a disaster is that the recovery process teaches you how to better prepare for future calamities.
Rally and be resilient
After Katrina, ASI Federal set up a permanent disaster recovery site 150 miles away. That location unfortunately was in Hurricane Gustav’s path three years later. “That’s when we started making big changes,” Tourné says.
She ticks off a list of new backup measures:
- A call center with PSCU Financial Services;
- Core processing in Troy, Mich.;
- Servers and data for other critical applications in Dallas.
- A new disaster recovery site 400 miles north in Memphis, Tenn., where core staff could evacuate to work. The site includes temporary office space and hotel accommodations.
Overflowing toilets, however, can happen anywhere, not just in hurricane country. After the January incident, ASI Federal’s staff pumped out the water and cleaned up so they could return to work on Tuesday (Monday was Martin Luther King Day).
The branch office in another section of the first floor wasn’t flooded. Business opened there as usual on Tuesday. On the corporate office side, staff adjusted. They doubled up in second floor offices and loan officers worked elbow-to-elbow at their computers in the lunchroom. A couple of days later, Agility Recovery—a CUNA Strategic Services alliance provider—brought in temporary buildings and set them up in the parking lot. Lending and human resources staff moved there for three months.
In light of past experiences, “We have an ability to react that might be different from other folks,” Tourne says. “We know how to rally and be resilient as a team.”
Improvise as needed
At $160 million asset Pinnacle Federal Credit Union, Edison, N.J., this summer’s Hurricane Irene taught staff a critical lesson: Always be ready with Plan B. Faced with power outages, “We had to improvise,” says Harry Jacobson, president/CEO.
He’s speaking specifically of what it took to retrieve backup tapes stored in the credit union’s Fairfield branch, 40 miles north of Edison. The branch wasn’t flooded, but it was in the heart of a flood zone, where access was severely limited.
A member of the credit union’s IT staff was sent to Hagerstown, Md., to use Ongoing Operations’ disaster recovery facility. Before he left, he needed the credit union’s backup tapes. But massive traffic jams prevented him from getting to New Jersey for the tapes from his home in Chester, N.Y. “He might have made it eventually,” Jacobson says, “but we needed to move quickly.”
So Jacobson ran an obstacle course to get the backup tapes. Another credit union employee from the Fairfield branch met him in a town they both could reach by navigating around flooded roads and bridges. Then Jacobson drove to Pennsylvania to meet the member of the IT staff, who took the backup tapes with him to the disaster recovery facility in Maryland. After that experience, Pinnacle Federal decided it needs a different backup tape storage site, not yet designated. And “make sure that whoever has to go to the backup site, such as your IT person, knows how to get into the building and where to find the tapes,” Jacobson says. “It could be a building they’re not familiar with. Don’t take anything for granted.”
Next: Take care of employees
Take care of employees
Damon McGregory, facilities manager at $800 million asset Twinstar Credit Union in Lacey, Wash., offers another branch-related tip for disaster planning: “When you have branches spread all around,” he says, “you need somebody on site who can go in, look at equipment, and make a quick evaluation of what’s going on.”
That became apparent when a late 2007 windstorm struck western Washington, toppling trees and power lines. Back then, Twinstar had 20 branches located from the Portland, Ore. area to just south of Seattle. Three branches lost power. The one in Ocean Shores, Wash.—right on the coast about 80 miles from the credit union’s headquarters—was a special concern. “That entire community had no power,” McGregory says, “and it’s more isolated. It looked like that branch could be without power for days.”
Work crews cleared area roads quickly, and branch employees were able to get to work. Agility Recovery brought in generators. “We were one of the few places in town with power,” McGregory says. “We gave the appearance that it was business as usual.”
But life was stressful for employees. “They hadn’t had a hot shower or a hot meal for a long time,” McGregory says. “Sure, they showed up for work. But there was a lot going on in their personal lives due to the storm and power outage.”
That’s why Twinstar plans to bring a motor home or something similar to any future disaster-affected branch. “That way, we can provide creature comforts to employees,” McGregory says. “Next time, we’ll take care of that right away.”
Consider what if…
Tornados wreaked havoc across the south in spring 2011. DCH Credit Union, with assets of $34 million in Tuscaloosa, Ala., was left with holes in its roof and some shattered windows. Two members and all but two employees were in the credit union when the tornado struck on Wednesday, April 27. They locked the doors and headed to interior bathrooms as safe rooms. Everyone emerged without a scratch. “We were lucky,” says Kevin Maguire, CEO. “We didn’t sustain a direct hit. The strip mall right next to us was completely destroyed.”
DCH lost power and Internet access. Agility Recovery delivered generators which, because of the heavy demand for such equipment in the area, had to come from Atlanta, 200 miles away. By Friday, DCH had power, and Agility Recovery brought in satellite equipment over the weekend to provide Internet access. The credit union reopened on Monday.
In recent months, DCH has been revisiting its disaster recovery plan and considering additional disaster scenarios. What if the building had taken a direct hit? What if the computer room had been destroyed? The credit union intends to do more disaster drills with employees, and it also designated the vault as its safe room.
“Before people worried about getting trapped in there if something blocked the door,” Maguire says. “Now, everyone knows about the safety features inside, and security officers told us that the first place they look for people in a destroyed financial institution is the vault. The door will have a flag on it to make sure somebody knows we’re inside.”
Experts also stress the need for testing a disaster recovery plan. Tower Federal Credit Union, assets $2.4 billion in Laurel, Md., takes that to heart. “Our testing is continually evolving,” says Sean Zimmermann, senior vice president of operations and technology. “We test once a year at a minimum, and that’s not enough.” Tower Federal’s testing involves running operations from a disaster recovery site to see if everything works. Tabletop testing—or talking through what-if scenarios and procedures—also enters into planning.
The credit union has weathered two events that had a significant impact on its operations. In 2009, a telephone company construction crew accidentally damaged telecommunications lines into its headquarters. A year later, a hardware malfunction brought down the core system. Both times Tower Federal switched over to its disaster recovery site at Ongoing Operations.
Through these experiences, the credit union learned, “it’s extremely important to have almost all areas of your credit union involved in testing your business continuity plan,” Zimmermann says. Otherwise, he explains, staff might wrongly assume certain functions and information will be available after a disaster.
Also, retest the plan when you make system upgrades or other changes. Testing at that time could eliminate nasty surprises later. “We’ve run into that,” Zimmermann reports. “We think we’ve tested, but when we test again, lo and behold, something doesn’t work because we had made some changes.”
Finally, assess up front the impact a new system or process will have on your disaster recovery plan, Zimmermann advises. “You might be choosing among three new systems, and you have them ranked one, two, and three,” he says. “But one option could cost twice as much for business continuity because you have to replicate it for the disaster recovery site. We consider that at the front end now. It used to be an afterthought, but we’ve moved it up to the initial decision-making stage.”
Next: CUs have compassion
‘CUs have compassion
In April 2011, a tornado leveled vast sections of Tuscaloosa, Ala.
“I grew up in Oklahoma, and I’ve seen a lot of tornado damage,” says Mike Bridges, vice president of marketing and communications for the League of Southeastern Credit Unions. “But I’ve never seen anything like what I saw in Tuscaloosa. It literally was like a bomb had gone off in part of the city.”
Among the destroyed properties were some 100 homes in a Tuscaloosa Housing Authority (THA) development. THA found new housing for displaced residents. “But these people were left without one stick of furniture, one pot, or anything,” says Tommy Cobb, president/CEO of $53 million asset Tuscaloosa Credit Union. THA is in the credit union’s field of membership.
So Cobb joined forces with THA and a local Baptist church to raise money to buy everything from furniture to dishes to toothpaste, and delivered the goods right to people’s doorsteps. About a dozen credit unions from Alabama and the Florida Panhandle contributed $92,000 to make those purchases.
“Credit unions have compassion, and they trusted us to do the right thing with the money,” says Cobb. “It was that cooperative spirit in action,” adds Bridges.
A disaster recovery plan shouldn’t “sit on a shelf and gather dust,” says Paul Sullivan, vice president and general manager of Agility Recovery. Experts suggest a few steps for creating a living document:
- CUNA Strategic Services archived webinars on disaster recovery and business continuity topics:
- CUNA Strategic Services alliance providers: