We have raised the bar on Business Continuity for Credit Unions
ESP has gone through two infrastructure iterations: starting with a service bureau model based on IBM mainframes, then we created a new FORZA3™ platform designed on standard Microsoft software, such as Microsoft SQL Server and .NET. FORZA3™ gave a modern and intuitive interface to our clients, in contrast to older mainframe-based core processors, and improved our ability to quickly deliver new third-party integrations and expand our core capacity.
Legacy IBM Mainframe
Distributed & Redundant FORZA3™
In 2017, we began planning work on a new iteration of this service bureau environment. Taking advantage of new developments in Microsoft SQL Server, new concepts in network architecture, and expanding regulatory and client expectations, we implemented new technology at the core of the FORZA3™ ecosystem.
The new generation of Service Bureau required several changes to the architecture by which we deliver core processing services, which we have implemented with our first client – Selfreliance Ukrainian American FCU – according to the following requirements:
Core components, such as FORZA3™ access for tellers or online banking for members, should not have to go down just because one server is inoperable.
Flexibility in service-offering requires us to be able to offer services in one location, with components residing in another.
Data is the life-blood of any credit union, and this data should be protected from regional disasters.
Beyond catastrophic failures, the state of a credit union’s data should be recoverable to any point in time.
Architecting the Service Bureau
We established a primary data presence in the same region as the credit union, with the following components:
- Two synchronously replicated, highly-available database servers in a fail-over cluster
- Two real-time replicated, highly-available online banking servers in a fail-over cluster
Teller connections to the core FORZA3™ platform can fail transparently between servers; in the event that one fails, the other can instantly takeover the workload, without any necessary changes on the client or member’s end.
Member connections to online banking, in the event of a failure of one online banking server, can restart their session within several seconds on the other.
Meanwhile, we configured the following in a different geographic region, to which the CU simultaneously has access:
- A third synchronously replicated database server
- A third online banking server, preconfigured for disaster recovery
- A second report retrieval and generation server
- A long-term tape archival system for daily server backups
The state of the data in the client’s primary datacenter is identical to that in the secondary datacenter; that is to say, in the event of a disaster at the primary processing location, there is no data loss when failing over to the alternate region.
Moreover, this failover is on the order of minutes, and only requires changing one piece of configuration in the teller software. Online Banking only needs to be failed-over by altering a DNS record, and we are currently working on a method to automate this process as well.
Daily backups are sent to the alternate location to be written to an encrypted tape archive, stored in a secure offsite facility that specializes in the long-term retention of critical data.
In addition to the real-time off-site replication of production data, we established a supplemental backup scheme:
- Bi-weekly full database backups
- Differential backups every 4 hours
- Transaction log backups every 15 minutes
Together, these allow us to recover the state of the database to any point-in-time. These backups are also replicated off-site and retained for a period of one week, while FORZA3™ daily backups are stored for seven years.
Minimal Migration Time
Through a carefully staged migration, we were able to keep total downtime for member-facing services to less than 90 minutes, scheduled at-night and after-hours when members were less likely to be impacted. This migration involved:
- Moving online banking ahead of time, with no downtime
- Establishment of client connectivity to primary and secondary data processing locations ahead of time
- Several weeks of regular replication of backups of all client databases
- Using SQL Server functionality to minimize the final backup/transfer size and restore time
- Triaged database restore operations and service recovery according to member impact
Improved Disaster Recovery and Business Continuity
In a disaster scenario, during which the primary data processing environment is instantly compromised with no warning, we are able to offer resumption of FORZA3™ processing in the new environment within only several minutes of an outage; services such as Online Banking can be resumed as soon as DNS propagates with new records.
By utilizing Microsoft SQL Server’s native replication functionality, we are able to allow the credit union the ability to create and run their own Crystal Reports against their own live data – and to do so against their replica server, which does not impact performance for members or for credit union personnel currently using it.
We also developed a “self service” reporting system, by which the CU can use FTP to
- Add and schedule their own custom reports for execution
- Specify reports to execute daily, weekly, or monthly
- Specify reports to output to their FORZA3™ Report Archive, or a separate private HTTPS/FTPS site
- Specify reports to output to PDF and/or Excel format
- Specify custom parameters to their Crystal Reports