Case Study

Hadoop migration improving performance and minimizing licensing costs by enhancing trade validation process

  • Facebook
  • Twitter
  • Linkedin
  • Email

The largest independent securities regulator in the United States

Capital markets

Business need addressed:
Developed scalable, cost-effective technology to streamline the trade validation process, address growing volume, and store consolidated audit trail data

Genpact solution:
Developed a multi-phase approach to port the current application to a Hadoop-based platform, which resolved capacity problems, improved performance, and saved on licensing costs

Business impact:
With a newer, more scalable solution, the client can handle larger volumes while complying with stringent timelines to maintain business as usual

Business challenge

This client relies on its audit trail application as a source of timed, sequenced-order events that contain market quotation and trading information submitted by member firms. This key application is used to verify and track member firms’ submission, reporting, and processing statistics for compliance, repair, and resubmission.

The existing technology—a platform built on a Netezza DW/BI appliance and the DataStage ETL tool—was expensive, with limited scalability options, and adversely impacted trade validation with ever-increasing volumes (consistently growing by about 45% year on year). There were frequent misses in SLA due to this growing volume—and application volume was expected to grow five-to six-fold over the next two years as additional asset classes were included. The client also needed to create a central repository to receive and store consolidated audit trail data in order for regulators to view cross-market data.

Take a copy for yourself

Download PDF

Genpact approach

At the beginning of the engagement, Genpact performed a thorough root cause analysis, using Lean and Six Sigma principles. This confirmed the need for a scalable, cost-effective platform to address rising data volumes and licensing costs.

In collaboration with the client’s developers, Genpact engaged in a proof-of-concept and evaluated new designs based on Hive, NoSQL (HBase), and in-memory-based custom Map/Reduce jobs. Based on the findings, Genpact and the developers created a multi-phased solution to migrate the trade validation process to Hadoop.

Genpact solution

To address the client’s business needs, Genpact developed a multi-phased approach to port the client’s existing Netezza/DataStage application to a Hadoop-based platform:

Phase 1: Proof-of-concept to evaluate designs based on Hadoop, HDFS, Hive, and HBase

Phase 2: Migrate from current ETL platform (DataStage) to a core Java-based ETL solution

Phase 3: Migrate from Netezza to an in-memory-based custom Map/Reduce Hadoop solution, which includes Hive and HBase

This solution enabled the client to move member firms’ trade data from the proprietary NAS system to Hadoop HDFS, along with the required reference data. The data validation engine is fully customizable, and rules are created using an XML-based template that runs on a custom-built Map/ Reduce framework.

Business impact delivered

Implementation of Genpact’s trade validation process migration to Hadoop had an immediate positive impact:

  • Resolution of capacity problems, with performance improvement over ten times
  • Lower license costs for Netezza and DataStage by using more affordable hardware for Hadoop
  • Affordable scalability potential for expected long-term audit trail data volume growth of 10–15 billion transactions per day
  • Long-term, cost-effective Tier-3 storage platform using Hadoop HDFS
  • Ability to port the new Hadoop-based solution to cloud-based systems like AWS without design changes

For more information, and visit and

Visit our capital markets page

Learn more