Tag Archives: security

Third-Gen Service Bureau Goes Live

We have raised the bar on Business Continuity for Credit Unions

ESP has gone through two infrastructure iterations: starting with a service bureau model based on IBM mainframes, then we created a new FORZA3™ platform designed on standard Microsoft software, such as Microsoft SQL Server and .NET. FORZA3™ gave a modern and intuitive interface to our clients, in contrast to older mainframe-based core processors, and improved our ability to quickly deliver new third-party integrations and expand our core capacity.


Legacy IBM Mainframe


Modern FORZA3™


Distributed & Redundant FORZA3™

In 2017, we began planning work on a new iteration of this service bureau environment. Taking advantage of new developments in Microsoft SQL Server, new concepts in network architecture, and expanding regulatory and client expectations, we implemented new technology at the core of the FORZA3™ ecosystem.

New Architecture

The new generation of Service Bureau required several changes to the architecture by which we deliver core processing services, which we have implemented with our first client – Selfreliance Ukrainian American FCU – according to the following requirements:

High Availability

Core components, such as FORZA3™ access for tellers or online banking for members, should not have to go down just because one server is inoperable.

Geographic Distribution

Flexibility in service-offering requires us to be able to offer services in one location, with components residing in another.

Real-Time Replication

Data is the life-blood of any credit union, and this data should be protected from regional disasters.

Point-in-time Recovery

Beyond catastrophic failures, the state of a credit union’s data should be recoverable to any point in time.

Architecting the Service Bureau

High Availability

We established a primary data presence in the same region as the credit union, with the following components:

  1. Two synchronously replicated, highly-available database servers in a fail-over cluster
  2. Two real-time replicated, highly-available online banking servers in a fail-over cluster

Teller connections to the core FORZA3™ platform can fail transparently between servers; in the event that one fails, the other can instantly takeover the workload, without any necessary changes on the client or member’s end.

Member connections to online banking, in the event of a failure of one online banking server, can restart their session within several seconds on the other.

Geographic Replication

Meanwhile, we configured the following in a different geographic region, to which the CU simultaneously has access:

  • A third synchronously replicated database server
  • A third online banking server, preconfigured for disaster recovery
  • A second report retrieval and generation server
  • A long-term tape archival system for daily server backups

The state of the data in the client’s primary datacenter is identical to that in the secondary datacenter; that is to say, in the event of a disaster at the primary processing location, there is no data loss when failing over to the alternate region.

Moreover, this failover is on the order of minutes, and only requires changing one piece of configuration in the teller software. Online Banking only needs to be failed-over by altering a DNS record, and we are currently working on a method to automate this process as well.

Daily backups are sent to the alternate location to be written to an encrypted tape archive, stored in a secure offsite facility that specializes in the long-term retention of critical data.

Point-in-time Recovery

In addition to the real-time off-site replication of production data, we established a supplemental backup scheme:

  • Bi-weekly full database backups
  • Differential backups every 4 hours
  • Transaction log backups every 15 minutes

Together, these allow us to recover the state of the database to any point-in-time. These backups are also replicated off-site and retained for a period of one week, while FORZA3™ daily backups are stored for seven years.

Minimal Migration Time

Through a carefully staged migration, we were able to keep total downtime for member-facing services to less than 90 minutes, scheduled at-night and after-hours when members were less likely to be impacted. This migration involved:

  1. Moving online banking ahead of time, with no downtime
  2. Establishment of client connectivity to primary and secondary data processing locations ahead of time
  3. Several weeks of regular replication of backups of all client databases
  4. Using SQL Server functionality to minimize the final backup/transfer size and restore time
  5. Triaged database restore operations and service recovery according to member impact

Improved Disaster Recovery and Business Continuity

In a disaster scenario, during which the primary data processing environment is instantly compromised with no warning, we are able to offer resumption of FORZA3™ processing in the new environment within only several minutes of an outage; services such as Online Banking can be resumed as soon as DNS propagates with new records.

Flexible reporting

By utilizing Microsoft SQL Server’s native replication functionality, we are able to allow the credit union the ability to create and run their own Crystal Reports against their own live data – and to do so against their replica server, which does not impact performance for members or for credit union personnel currently using it.

We also developed a “self service” reporting system, by which the CU can use FTP to

  • Add and schedule their own custom reports for execution
  • Specify reports to execute daily, weekly, or monthly
  • Specify reports to output to their FORZA3™ Report Archive, or a separate private HTTPS/FTPS site
  • Specify reports to output to PDF and/or Excel format
  • Specify custom parameters to their Crystal Reports

Improve Member Confidence

Read about Extended Validation SSL here

It is our goal to ensure that your information is safe and secure, so when we see something that will improve that, we take note. When we see something that can make a positive difference in the eyes of your members, we act. For $199.00/year, we can work with you to implement the new Extended Validation SSL (EVSSL) certificate that has been proven to increase online transactions and improve customer loyalty.

What is the EVSSL?

The EVSSL will be implemented on the login page for Home Banking (A.K.A. OASIS).

“Now when shoppers visit a Website secured with an Extended Validation SSL Certificate, the latest browsers trigger the address bar to turn green and display the name of the organization listed in the certificate, as well as the certificate’s security vendor in some cases. A secure connection has been established between browser and Website, and the Website has been authenticated according to rigorous industry standards. In the example below, the browser controls the display, pulling information from the SSL Certificate and displaying it in the address and security status bar, and making it extremely difficult for phishers and counterfeiters to hijack your brand and your customers.”

 Extra, what extras?

Custom Home Banking Domain

We want to increase the transparency between credit union services and the member. The addition of a new home banking login domain will do that.

Give your members the comfort of seeing the “green” bar and your credit union name associated with all aspects of the login screen. Example below.


Let’s Increase your Member Loyalty

Let us help you increase your member loyalty and provide another tool to better manage the security between your member and the credit union.

Please note, we are not marking this solution up – you can see the price for the certificate on the webpage that we included at the top. We genuinely want to be a part of treating your member information safely and securely.

The Shellshock Exploit


Yesterday afternoon, an exploit called Shellshock came to light on a security message board .  CVE-2014-6271, as it is called by National Vulnerability Database, allows a remote user to execute arbitrary or malicious commands on a remote server, such as those that host websites; it has been given the highest impact rating, 10, for this reason .


This exploit presents a tremendous risk reminiscent of Heartbleed, because it potentially impacts many thousands of websites and other servers on the internet. A vulnerable system would allow any remote, unauthenticated user to run commands on the server, potentially allowing them to perform malicious activities: phishing, stealing data, installing a rootkit, etc.

Vulnerable Systems

Any system running Bash up to version 4.3 is potentially vulnerable, which includes Linux, Unix, and Mac OS X operating systems . The main avenue for remote exploitation of this bug is via websites served by CGI, which is a common mechanism for providing applications powered by PHP, Perl, Python, or many other languages.

Already there have been multiple reports of this exploit being used in the wild , so it is vital to examine your own environment for any unix-like systems that have Bash installed, and to apply vendor-provided patches where applicable.

You can use this tool to attempt to determine if your website is vulnerable, use the newly-available metasploit module , or run this command from a terminal on your Linux/Unix machines:

env x='() { :;}; echo vulnerable’ bash -c “echo this is a test”

If this command returns “vulnerable”, then you are vulnerable to CVE-2014-6271 and should patch your system immediately! If you run this:

env X='() { (a)=>\’ sh -c “echo date”; cat echo

And the date prints out, then you are vulnerable to CVE-2014-7169 and should attempt to patch, or sit tight until your vendor releases one, using other mitigation techniques in the short-term until such a patch is available. Currently, Debian, Ubuntu, RedHat, Gentoo, Slackware, and Suse have fixes or patches available.


Here at ESP, we’ve mostly been spared – as with Heartbleed – because our applications are primarily written in ASP.NET, and hosted in an environment that lacks any installation of Bash. Additionally, our auxiliary web servers run PHP via a module rather than CGI, which is required in most instances for remote exploitation of the bug; these servers were patched after our vendor provided updates to address the flaw.


There are several ways to mitigate this vulnerability in your own environment. Some of these avenues are listed below, along with a short explanation.

Patch pertinent systems

The easiest and most vital solution is to apply patches to any systems that you can determine are affected by Shellshock. On Debian systems, for example, this can be as simple as running “apt-get update; apt-get upgrade” in the terminal. The first available patches did no fully mitigate all attack strategies , so a second update may be required; the particular ID associated with the incomplete patch is CVE-2014-7169, and updates addressing that vulnerability have been released by several major vendors . There may be a possibility new variants of the attack, so keep checking over the course of the next several weeks and apply any superseding updates. Ask your vendors if any of their software is vulnerable and/or needs to be patched.

Use mod_php

For this vulnerability, because remote exploitation of websites requires that they be served via CGI, you could potentially mitigate Shellshock by serving your site with a module or a different form of CGI instead; if you use Apache, you can use mod_php as a temporary measure, for example. Whether socket-based CGI applications (e.g., php-fpm with nginx) are vulnerable is still being investigated , but mod_php should be safer, if you make sure that your code does not call Bash in any way that a user can alter.

Use alternate shells

Another temporary mitigation strategy is to uninstall Bash, and use a different shell as default, such as Dash .

Use firewall rules

Since this bug can affect web servers as well as OpenSSH servers, you should examine your firewall rules to make sure that only necessary sites/services are exposed to the internet. If a server is for internal use and has no business-case for being available externally, block access to it.

Further Reading

Request a DR Test On-line

In the effort to make processes for our Clients as easy as possible. You can now request a Disaster Recovery (DR) test from the website. Once you fill out the form, we will contact you and get started.

You will be able to test from a remote location if you wish. This is testing as if the CU had a disaster and had to move. All you need is an Internet connection.

You can test to your remote data as well. This test is ran in the event that there is a disaster at ESP.

If you would like to request a Disaster Recovery test time, please visit the form here.