Author Archives: Alexander Ray

Third-Gen Service Bureau Goes Live

We have raised the bar on Business Continuity for Credit Unions

ESP has gone through two infrastructure iterations: starting with a service bureau model based on IBM mainframes, then we created a new FORZA3™ platform designed on standard Microsoft software, such as Microsoft SQL Server and .NET. FORZA3™ gave a modern and intuitive interface to our clients, in contrast to older mainframe-based core processors, and improved our ability to quickly deliver new third-party integrations and expand our core capacity.


Legacy IBM Mainframe


Modern FORZA3™


Distributed & Redundant FORZA3™

In 2017, we began planning work on a new iteration of this service bureau environment. Taking advantage of new developments in Microsoft SQL Server, new concepts in network architecture, and expanding regulatory and client expectations, we implemented new technology at the core of the FORZA3™ ecosystem.

New Architecture

The new generation of Service Bureau required several changes to the architecture by which we deliver core processing services, which we have implemented with our first client – Selfreliance Ukrainian American FCU – according to the following requirements:

High Availability

Core components, such as FORZA3™ access for tellers or online banking for members, should not have to go down just because one server is inoperable.

Geographic Distribution

Flexibility in service-offering requires us to be able to offer services in one location, with components residing in another.

Real-Time Replication

Data is the life-blood of any credit union, and this data should be protected from regional disasters.

Point-in-time Recovery

Beyond catastrophic failures, the state of a credit union’s data should be recoverable to any point in time.

Architecting the Service Bureau

High Availability

We established a primary data presence in the same region as the credit union, with the following components:

  1. Two synchronously replicated, highly-available database servers in a fail-over cluster
  2. Two real-time replicated, highly-available online banking servers in a fail-over cluster

Teller connections to the core FORZA3™ platform can fail transparently between servers; in the event that one fails, the other can instantly takeover the workload, without any necessary changes on the client or member’s end.

Member connections to online banking, in the event of a failure of one online banking server, can restart their session within several seconds on the other.

Geographic Replication

Meanwhile, we configured the following in a different geographic region, to which the CU simultaneously has access:

  • A third synchronously replicated database server
  • A third online banking server, preconfigured for disaster recovery
  • A second report retrieval and generation server
  • A long-term tape archival system for daily server backups

The state of the data in the client’s primary datacenter is identical to that in the secondary datacenter; that is to say, in the event of a disaster at the primary processing location, there is no data loss when failing over to the alternate region.

Moreover, this failover is on the order of minutes, and only requires changing one piece of configuration in the teller software. Online Banking only needs to be failed-over by altering a DNS record, and we are currently working on a method to automate this process as well.

Daily backups are sent to the alternate location to be written to an encrypted tape archive, stored in a secure offsite facility that specializes in the long-term retention of critical data.

Point-in-time Recovery

In addition to the real-time off-site replication of production data, we established a supplemental backup scheme:

  • Bi-weekly full database backups
  • Differential backups every 4 hours
  • Transaction log backups every 15 minutes

Together, these allow us to recover the state of the database to any point-in-time. These backups are also replicated off-site and retained for a period of one week, while FORZA3™ daily backups are stored for seven years.

Minimal Migration Time

Through a carefully staged migration, we were able to keep total downtime for member-facing services to less than 90 minutes, scheduled at-night and after-hours when members were less likely to be impacted. This migration involved:

  1. Moving online banking ahead of time, with no downtime
  2. Establishment of client connectivity to primary and secondary data processing locations ahead of time
  3. Several weeks of regular replication of backups of all client databases
  4. Using SQL Server functionality to minimize the final backup/transfer size and restore time
  5. Triaged database restore operations and service recovery according to member impact

Improved Disaster Recovery and Business Continuity

In a disaster scenario, during which the primary data processing environment is instantly compromised with no warning, we are able to offer resumption of FORZA3™ processing in the new environment within only several minutes of an outage; services such as Online Banking can be resumed as soon as DNS propagates with new records.

Flexible reporting

By utilizing Microsoft SQL Server’s native replication functionality, we are able to allow the credit union the ability to create and run their own Crystal Reports against their own live data – and to do so against their replica server, which does not impact performance for members or for credit union personnel currently using it.

We also developed a “self service” reporting system, by which the CU can use FTP to

  • Add and schedule their own custom reports for execution
  • Specify reports to execute daily, weekly, or monthly
  • Specify reports to output to their FORZA3™ Report Archive, or a separate private HTTPS/FTPS site
  • Specify reports to output to PDF and/or Excel format
  • Specify custom parameters to their Crystal Reports

How to detect a phishing attack

Forward – Phishing attack against American Lake CU

We have recently become aware of a phishing attack against members of American Lake CU.  This attack is a variant of one which has existed since 2008, and has also targeted Chase and Bank of America customers.  It is our hope that this article will provide information both to mitigate the danger posed by this attack and by future attacks, for both members and non-members of American Lake CU.

The phishing attack has shown two variants so far:

From: American Lake <>
Date: July 11, 2017 at 9:52:42 AM EDT
To: <>
Subject: Online Verification
Dear Customer,
We’re sorry – we suspended your access to your American Lake account because of recent activity on your account.
Click Here To Activate Your Account.
Copyright © 2017 American Lake CU.
All Rights Reserved.

From: American Lake <>
Sent: Tuesday, July 11, 2017 5:02 PM
Subject: Systems Maintenance Services.

Security Alert

Dear Customer,
We are letting you know that due to an ongoing General system maintenance in our Online Banking Database its mandatory for you to Verify Your American Lake Account in order to enjoy our online banking service. We request that you complete this quick Verification process. If this is not done as urgent as possible your account might be deactivated at once.
Online Verification

This morning (7/12/2017), a second variant with a different phishing page and different email appeared. We have already gotten two phishing pages, that the emails linked to, taken down – however, variants may continue to spread. With that in mind, please read the following information closely:

What is phishing?

One of the most popular “hacking” techniques, phishing relies on vulnerabilities in people rather than in code.  Phishing campaigns take advantage of human fallibility to convince targets to voluntarily give up their sensitive information to attackers for financial gain.  The infamous “Nigerian Prince” phishing scam presents an example of this: with a sometimes convincing story, individuals are convinced to hand over personal information (bank account information, passport scans, etc) in exchange for the promise of money.  More common today are phishing attacks targeting financial institutions such as credit unions.

How does phishing work?

Phishing attacks work much like marketing campaigns, in that they operate a “funnel” – enormous numbers of phishing emails are sent out to equally enormous numbers of recipients, in hopes that some of them don’t immediately skip over it, some of the remainder open the email, some of that remainder take it seriously enough to click the link, some of those go on to enter their information, etc.

Note that while email-based phishing campaigns are most common, they can also operate through unsolicited phone calls and even traditional “snail-mail”!

How do I protect myself from phishing?

To prevent yourself from becoming a victim of phishing, it’s important to keep yourself from ‘falling down’ the funnel mentioned earlier, and to stop yourself as soon as possible in the process of becoming a victim.

Limit exposure to phishing email

While there is no fool-proof method to keep yourself from receiving phishing email, there are some tips you can use to limit the number you receive:

  1. Use an email account with spam filtering
    1. Even most free email providers offer this.
  2. Be careful where you post your email address
    1. Don’t post your email address in public comments, on public websites, etc.
    2. Try to use a different email address (or alias) for your “important” accounts, such as Online Banking, from accounts you use for online games, for example.

Recognize phishing email

When you receive an email, especially relating to your credit union account, ask the following questions to try to reduce the risk of taking a phishing email seriously:

  1. Does this pertain to me?
    1. If you are not a member of American Lake CU, and you receive an email asking for you to do something for your account there, you should ignore it.  After all, you have no account, so it couldn’t possibly be applicable to you.
  2. Does it sound professional?
    1. If the email contains strange variations in grammar, spelling, punctuation, or case, this can be an indication that it is illegitimate.  Attackers often do this to try to evade spam filters, or simply as a result of not speaking English as a first language.
  3. Is this email from who it says it is?
    1. Note that while it is trivial to spoof email addresses, these are typically more obvious to spam filters.  Many attackers will send from email addresses completely unrelated to the institution they’re phishing.  Look at the “from” address and see if it even claims to be coming from the institution.
  4. Are they asking me to give them something?
    1. Legitimate institutions will virtually never send you unsolicited email requesting that you enter personal information.  Always check with the institution to make sure such unusual requests are legitimate.

Check the address bar

While you should always try to avoid interacting with phishing emails, if you do find yourself on a website and about to enter your personal information, you should always double check the address bar to verify the “domain” of the website.  Phishing websites almost always have a different (but sometimes similar!) address to the legitimate site.

The address bar is located at the top of the window:

American Lake CU uses a technology called “EV-SSL” to provide both encryption of traffic to its website and verification of the website’s identity.  Members of the CU should check the address bar to ensure that the CU name is indicated, as well as the domain “”:

Online banking for American Lake CU looks very similar:

Note the similarities for the above two images, are compared to this phishing page:

  • The address isn’t similar
  • No “https”, no green padlock, no CU name in address bar

Some phishing scams may be look closer, such as registering “” as opposed to “”, for example.

If in doubt, call your institution!

If you think there’s a chance the email could be illegitimate, call the institution (such as your credit union) using a number you know is legitimate, and ask them about the email you received.  If it is illegitimate, they can use this as a warning for others!

When in doubt, especially if you are being asked for more or different information than normal, and especially if you were solicited to give this information via email, contact your credit union!