The largest independent securities regulator in the United States
Business need addressed:
Developed scalable, cost-effective technology to streamline the trade validation process, address growing volume, and store consolidated audit trail data
Developed a multi-phase approach to port the current application to a Hadoop-based platform, which resolved capacity problems, improved performance, and saved on licensing costs
With a newer, more scalable solution, the client can handle larger volumes while complying with stringent timelines to maintain business as usual
This client relies on its audit trail application as a source of timed, sequenced-order events that contain market quotation and trading information submitted by member firms. This key application is used to verify and track member firms’ submission, reporting, and processing statistics for compliance, repair, and resubmission.
The existing technology—a platform built on a Netezza DW/BI appliance and the DataStage ETL tool—was expensive, with limited scalability options, and adversely impacted trade validation with ever-increasing volumes (consistently growing by about 45% year on year). There were frequent misses in SLA due to this growing volume—and application volume was expected to grow five-to six-fold over the next two years as additional asset classes were included. The client also needed to create a central repository to receive and store consolidated audit trail data in order for regulators to view cross-market data.
At the beginning of the engagement, Genpact performed a thorough root cause analysis, using Lean and Six Sigma principles. This confirmed the need for a scalable, cost-effective platform to address rising data volumes and licensing costs.
In collaboration with the client’s developers, Genpact engaged in a proof-of-concept and evaluated new designs based on Hive, NoSQL (HBase), and in-memory-based custom Map/Reduce jobs. Based on the findings, Genpact and the developers created a multi-phased solution to migrate the trade validation process to Hadoop.
To address the client’s business needs, Genpact developed a multi-phased approach to port the client’s existing Netezza/DataStage application to a Hadoop-based platform:
Phase 1: Proof-of-concept to evaluate designs based on Hadoop, HDFS, Hive, and HBase
Phase 2: Migrate from current ETL platform (DataStage) to a core Java-based ETL solution
Phase 3: Migrate from Netezza to an in-memory-based custom Map/Reduce Hadoop solution, which includes Hive and HBase
This solution enabled the client to move member firms’ trade data from the proprietary NAS system to Hadoop HDFS, along with the required reference data. The data validation engine is fully customizable, and rules are created using an XML-based template that runs on a custom-built Map/ Reduce framework, as shown in figure 1.
Business impact delivered
Implementation of Genpact’s trade validation process migration to Hadoop had an immediate positive impact:
- Resolution of capacity problems, with performance improvement over ten times
- Lower license costs for Netezza and DataStage by using more affordable hardware for Hadoop
- Affordable scalability potential for expected long-term audit trail data volume growth of 10–15 billion transactions per day
- Long-term, cost-effective Tier-3 storage platform using Hadoop HDFS
- Ability to port the new Hadoop-based solution to cloud-based systems like AWS without design changes
For more information, contact, email@example.com and visit, genpact.com/what-we-do/industries/capital-markets, genpact.com/what-we-do/business-services/enterprise-application-services and genpact.com/what-we-do/capabilities/transformation-consulting