Modern Middle Manager
Primarily my musings on the practical application of technology and management principles at a financial services company.
Hot Hot Hot

Thursday, January 09, 2003  

I spent today at our San Diego branch working on our hot site. Last year we decided to provide our own recovery services. After analyzing the cost of Sungard's Availability Services and comparing to what we thought it would take to create our own in-house recovery, we made the decision to keep that function in-house. While Sungard has very sophisticated services, they are also very expensive and difficult to justify in our current business environment. We decided to forego the Lexus for the Chevy.

Our broadest recovery goals are split among the following categories:

1. Wire services. Recover the ability to transmit a wire immediately. Recover the notification service within four hours.
2. Commercial banking. Recover the core accounting system almost immediately. Recover realtime connectivity to our internet banking system ASP within four hours.
3. Wealth management. Recover the core accounting system within 12-24 hours. Recover ancillary servers within 24 hours.
4. Desktop accessibility. Recover as soon as possible, not to exceed 24 hours.
5. Communications. Recover some form of communications as soon as possible, not to exceed 4 hours.

Some of these overlap. For example, desktop accessibility (Microsoft Office), access to communications (most likely e-mail using Outlook) and access to the core accounting systems will all be achieved using Citrix's Metaframe XP server. By using a single solution, recovering the Citrix server enables several goals to be met at once.

Of the funds budgeted for this project, most of the money has gone into asynchronously mirroring data from the main data center to our hot site. Our major concerns are the bank accounting data, the wire notification files, our SQL databases and Exchange information stores. The bank data will be mirrored using Lakeview Technology's Mimix between two AS/400's. That project should be finished by the end of February. The rest of the data is mirrored currently using two Network Appliance filers and their SnapMirror software. Mirroring most of the data that changes daily should allow us to recover to a point within 30 minutes of a disaster. Our trust accounting system does not provide us a means to mirror its data; however, once we've upgraded to the latest version this year we will be able to enjoy very aggressive recovery times.

Additional data communications lines are in the process of being added to the hot site for connectivity to our ASP's and we already have a second line to the Internet. When we did our analysis we figured that the communications lines would be a similar expense at either a Sungard hot site or our own, so for the purposes of our analysis they wash out.

The ancillary servers are all virtualized and their information doesn't change often, meaning we can perform quarterly backups onto DVD and restore them to our hot site. They will be running on depreciated servers that have collected over the last few years (and we'll be adding some more as we retire six servers this year). That keeps expenses low. Using Citrix servers to provide access to software means that most of our end-users can work from home during the crisis.

We will be purchasing a single SDLT tape drive to restore any tapes we need to bring back from Iron Mountain, our offsite storage vendor.

We caught a break on power and cooling issues; our San Diego building is occupied by a sister division which paid for a UPS, generator and updated air conditioning system so we're free riders.

What's the bottom line? Based on our analysis, we're going to save approximately $40K per year for the first three years and about $80K per year thereafter. I think that's well worth the effort of rolling our own site.

posted by Henry Jenkins | 1/09/2003 05:37:00 PM

Comments: Post a Comment
the author
open source