Author Archives: Alexander Ray

Third-Gen Service Bureau Goes Live

We have raised the bar on Business Continuity for Credit Unions

ESP has gone through two infrastructure iterations: starting with a service bureau model based on IBM mainframes, then we created a new FORZA3™ platform designed on standard Microsoft software, such as Microsoft SQL Server and .NET. FORZA3™ gave a modern and intuitive interface to our clients, in contrast to older mainframe-based core processors, and improved our ability to quickly deliver new third-party integrations and expand our core capacity.


Legacy IBM Mainframe


Modern FORZA3™


Distributed & Redundant FORZA3™

In 2017, we began planning work on a new iteration of this service bureau environment. Taking advantage of new developments in Microsoft SQL Server, new concepts in network architecture, and expanding regulatory and client expectations, we implemented new technology at the core of the FORZA3™ ecosystem.

New Architecture

The new generation of Service Bureau required several changes to the architecture by which we deliver core processing services, which we have implemented with our first client – Selfreliance Ukrainian American FCU – according to the following requirements:

High Availability

Core components, such as FORZA3™ access for tellers or online banking for members, should not have to go down just because one server is inoperable.

Geographic Distribution

Flexibility in service-offering requires us to be able to offer services in one location, with components residing in another.

Real-Time Replication

Data is the life-blood of any credit union, and this data should be protected from regional disasters.

Point-in-time Recovery

Beyond catastrophic failures, the state of a credit union’s data should be recoverable to any point in time.

Architecting the Service Bureau

High Availability

We established a primary data presence in the same region as the credit union, with the following components:

  1. Two synchronously replicated, highly-available database servers in a fail-over cluster
  2. Two real-time replicated, highly-available online banking servers in a fail-over cluster

Teller connections to the core FORZA3™ platform can fail transparently between servers; in the event that one fails, the other can instantly takeover the workload, without any necessary changes on the client or member’s end.

Member connections to online banking, in the event of a failure of one online banking server, can restart their session within several seconds on the other.

Geographic Replication

Meanwhile, we configured the following in a different geographic region, to which the CU simultaneously has access:

  • A third synchronously replicated database server
  • A third online banking server, preconfigured for disaster recovery
  • A second report retrieval and generation server
  • A long-term tape archival system for daily server backups

The state of the data in the client’s primary datacenter is identical to that in the secondary datacenter; that is to say, in the event of a disaster at the primary processing location, there is no data loss when failing over to the alternate region.

Moreover, this failover is on the order of minutes, and only requires changing one piece of configuration in the teller software. Online Banking only needs to be failed-over by altering a DNS record, and we are currently working on a method to automate this process as well.

Daily backups are sent to the alternate location to be written to an encrypted tape archive, stored in a secure offsite facility that specializes in the long-term retention of critical data.

Point-in-time Recovery

In addition to the real-time off-site replication of production data, we established a supplemental backup scheme:

  • Bi-weekly full database backups
  • Differential backups every 4 hours
  • Transaction log backups every 15 minutes

Together, these allow us to recover the state of the database to any point-in-time. These backups are also replicated off-site and retained for a period of one week, while FORZA3™ daily backups are stored for seven years.

Minimal Migration Time

Through a carefully staged migration, we were able to keep total downtime for member-facing services to less than 90 minutes, scheduled at-night and after-hours when members were less likely to be impacted. This migration involved:

  1. Moving online banking ahead of time, with no downtime
  2. Establishment of client connectivity to primary and secondary data processing locations ahead of time
  3. Several weeks of regular replication of backups of all client databases
  4. Using SQL Server functionality to minimize the final backup/transfer size and restore time
  5. Triaged database restore operations and service recovery according to member impact

Improved Disaster Recovery and Business Continuity

In a disaster scenario, during which the primary data processing environment is instantly compromised with no warning, we are able to offer resumption of FORZA3™ processing in the new environment within only several minutes of an outage; services such as Online Banking can be resumed as soon as DNS propagates with new records.

Flexible reporting

By utilizing Microsoft SQL Server’s native replication functionality, we are able to allow the credit union the ability to create and run their own Crystal Reports against their own live data – and to do so against their replica server, which does not impact performance for members or for credit union personnel currently using it.

We also developed a “self service” reporting system, by which the CU can use FTP to

  • Add and schedule their own custom reports for execution
  • Specify reports to execute daily, weekly, or monthly
  • Specify reports to output to their FORZA3™ Report Archive, or a separate private HTTPS/FTPS site
  • Specify reports to output to PDF and/or Excel format
  • Specify custom parameters to their Crystal Reports

How to detect a phishing attack

Forward – Phishing attack against American Lake CU

We have recently become aware of a phishing attack against members of American Lake CU.  This attack is a variant of one which has existed since 2008, and has also targeted Chase and Bank of America customers.  It is our hope that this article will provide information both to mitigate the danger posed by this attack and by future attacks, for both members and non-members of American Lake CU.

The phishing attack has shown two variants so far:

From: American Lake <>
Date: July 11, 2017 at 9:52:42 AM EDT
To: <>
Subject: Online Verification
Dear Customer,
We’re sorry – we suspended your access to your American Lake account because of recent activity on your account.
Click Here To Activate Your Account.
Copyright © 2017 American Lake CU.
All Rights Reserved.

From: American Lake <>
Sent: Tuesday, July 11, 2017 5:02 PM
Subject: Systems Maintenance Services.

Security Alert

Dear Customer,
We are letting you know that due to an ongoing General system maintenance in our Online Banking Database its mandatory for you to Verify Your American Lake Account in order to enjoy our online banking service. We request that you complete this quick Verification process. If this is not done as urgent as possible your account might be deactivated at once.
Online Verification

This morning (7/12/2017), a second variant with a different phishing page and different email appeared. We have already gotten two phishing pages, that the emails linked to, taken down – however, variants may continue to spread. With that in mind, please read the following information closely:

What is phishing?

One of the most popular “hacking” techniques, phishing relies on vulnerabilities in people rather than in code.  Phishing campaigns take advantage of human fallibility to convince targets to voluntarily give up their sensitive information to attackers for financial gain.  The infamous “Nigerian Prince” phishing scam presents an example of this: with a sometimes convincing story, individuals are convinced to hand over personal information (bank account information, passport scans, etc) in exchange for the promise of money.  More common today are phishing attacks targeting financial institutions such as credit unions.

How does phishing work?

Phishing attacks work much like marketing campaigns, in that they operate a “funnel” – enormous numbers of phishing emails are sent out to equally enormous numbers of recipients, in hopes that some of them don’t immediately skip over it, some of the remainder open the email, some of that remainder take it seriously enough to click the link, some of those go on to enter their information, etc.

Note that while email-based phishing campaigns are most common, they can also operate through unsolicited phone calls and even traditional “snail-mail”!

How do I protect myself from phishing?

To prevent yourself from becoming a victim of phishing, it’s important to keep yourself from ‘falling down’ the funnel mentioned earlier, and to stop yourself as soon as possible in the process of becoming a victim.

Limit exposure to phishing email

While there is no fool-proof method to keep yourself from receiving phishing email, there are some tips you can use to limit the number you receive:

  1. Use an email account with spam filtering
    1. Even most free email providers offer this.
  2. Be careful where you post your email address
    1. Don’t post your email address in public comments, on public websites, etc.
    2. Try to use a different email address (or alias) for your “important” accounts, such as Online Banking, from accounts you use for online games, for example.

Recognize phishing email

When you receive an email, especially relating to your credit union account, ask the following questions to try to reduce the risk of taking a phishing email seriously:

  1. Does this pertain to me?
    1. If you are not a member of American Lake CU, and you receive an email asking for you to do something for your account there, you should ignore it.  After all, you have no account, so it couldn’t possibly be applicable to you.
  2. Does it sound professional?
    1. If the email contains strange variations in grammar, spelling, punctuation, or case, this can be an indication that it is illegitimate.  Attackers often do this to try to evade spam filters, or simply as a result of not speaking English as a first language.
  3. Is this email from who it says it is?
    1. Note that while it is trivial to spoof email addresses, these are typically more obvious to spam filters.  Many attackers will send from email addresses completely unrelated to the institution they’re phishing.  Look at the “from” address and see if it even claims to be coming from the institution.
  4. Are they asking me to give them something?
    1. Legitimate institutions will virtually never send you unsolicited email requesting that you enter personal information.  Always check with the institution to make sure such unusual requests are legitimate.

Check the address bar

While you should always try to avoid interacting with phishing emails, if you do find yourself on a website and about to enter your personal information, you should always double check the address bar to verify the “domain” of the website.  Phishing websites almost always have a different (but sometimes similar!) address to the legitimate site.

The address bar is located at the top of the window:

American Lake CU uses a technology called “EV-SSL” to provide both encryption of traffic to its website and verification of the website’s identity.  Members of the CU should check the address bar to ensure that the CU name is indicated, as well as the domain “”:

Online banking for American Lake CU looks very similar:

Note the similarities for the above two images, are compared to this phishing page:

  • The address isn’t similar
  • No “https”, no green padlock, no CU name in address bar

Some phishing scams may be look closer, such as registering “” as opposed to “”, for example.

If in doubt, call your institution!

If you think there’s a chance the email could be illegitimate, call the institution (such as your credit union) using a number you know is legitimate, and ask them about the email you received.  If it is illegitimate, they can use this as a warning for others!

When in doubt, especially if you are being asked for more or different information than normal, and especially if you were solicited to give this information via email, contact your credit union!

The Shellshock Exploit


Yesterday afternoon, an exploit called Shellshock came to light on a security message board .  CVE-2014-6271, as it is called by National Vulnerability Database, allows a remote user to execute arbitrary or malicious commands on a remote server, such as those that host websites; it has been given the highest impact rating, 10, for this reason .


This exploit presents a tremendous risk reminiscent of Heartbleed, because it potentially impacts many thousands of websites and other servers on the internet. A vulnerable system would allow any remote, unauthenticated user to run commands on the server, potentially allowing them to perform malicious activities: phishing, stealing data, installing a rootkit, etc.

Vulnerable Systems

Any system running Bash up to version 4.3 is potentially vulnerable, which includes Linux, Unix, and Mac OS X operating systems . The main avenue for remote exploitation of this bug is via websites served by CGI, which is a common mechanism for providing applications powered by PHP, Perl, Python, or many other languages.

Already there have been multiple reports of this exploit being used in the wild , so it is vital to examine your own environment for any unix-like systems that have Bash installed, and to apply vendor-provided patches where applicable.

You can use this tool to attempt to determine if your website is vulnerable, use the newly-available metasploit module , or run this command from a terminal on your Linux/Unix machines:

env x='() { :;}; echo vulnerable’ bash -c “echo this is a test”

If this command returns “vulnerable”, then you are vulnerable to CVE-2014-6271 and should patch your system immediately! If you run this:

env X='() { (a)=>\’ sh -c “echo date”; cat echo

And the date prints out, then you are vulnerable to CVE-2014-7169 and should attempt to patch, or sit tight until your vendor releases one, using other mitigation techniques in the short-term until such a patch is available. Currently, Debian, Ubuntu, RedHat, Gentoo, Slackware, and Suse have fixes or patches available.


Here at ESP, we’ve mostly been spared – as with Heartbleed – because our applications are primarily written in ASP.NET, and hosted in an environment that lacks any installation of Bash. Additionally, our auxiliary web servers run PHP via a module rather than CGI, which is required in most instances for remote exploitation of the bug; these servers were patched after our vendor provided updates to address the flaw.


There are several ways to mitigate this vulnerability in your own environment. Some of these avenues are listed below, along with a short explanation.

Patch pertinent systems

The easiest and most vital solution is to apply patches to any systems that you can determine are affected by Shellshock. On Debian systems, for example, this can be as simple as running “apt-get update; apt-get upgrade” in the terminal. The first available patches did no fully mitigate all attack strategies , so a second update may be required; the particular ID associated with the incomplete patch is CVE-2014-7169, and updates addressing that vulnerability have been released by several major vendors . There may be a possibility new variants of the attack, so keep checking over the course of the next several weeks and apply any superseding updates. Ask your vendors if any of their software is vulnerable and/or needs to be patched.

Use mod_php

For this vulnerability, because remote exploitation of websites requires that they be served via CGI, you could potentially mitigate Shellshock by serving your site with a module or a different form of CGI instead; if you use Apache, you can use mod_php as a temporary measure, for example. Whether socket-based CGI applications (e.g., php-fpm with nginx) are vulnerable is still being investigated , but mod_php should be safer, if you make sure that your code does not call Bash in any way that a user can alter.

Use alternate shells

Another temporary mitigation strategy is to uninstall Bash, and use a different shell as default, such as Dash .

Use firewall rules

Since this bug can affect web servers as well as OpenSSH servers, you should examine your firewall rules to make sure that only necessary sites/services are exposed to the internet. If a server is for internal use and has no business-case for being available externally, block access to it.

Further Reading