Navigating Big Data in the Cloud

FastOLA logo

Shelly Bohlin, FINRA VP Quality of Markets/Market Regulation, says the speed of our custom analytics tool, FastOLA, is a game changer for her Market Reg analysts who use the system daily and had been waiting hours at times to receive a return on a complex order lifecycle query using FINRA’s previous query tool. “There was a general frustration level, if you really wanted to see what happened, and then having to wait. You had to turn your attention away and wait for the query to come back,” Bohlin said of the system FastOLA replaced. “Now it almost immediately comes back. You stay focused."

Learn More

We sat down with Senior Director Matt Cardillo to discuss FastOLA, a custom application released in 2014 that leverages both cloud and open source technology. FastOLA, an analytics tool, compiles large amounts of market trade data from places like NASDAQ, the National Market System (NMS), Over-the-Counter (OTC) equity securities, and trades executed otherwise than on an exchange. It cuts user query time exponentially over the system it replaces by compiling data as it is received rather than compiling information on demand, reducing one five-hour query to a mere 11 seconds.


WHAT IS FastOLA?

MC: FastOLA is a custom application analytics tool that stands for Fast Order Lifecycle Assembly. Every time an order is placed for a security, there are a series of events regarding how the order is filled or eventually cancelled. FastOLA reconstructs the orders from origination through execution or cancellation by creating the linkages into an order lifecycle assembly. This gives a full picture of what happens to an order over time and helps to inform decision-making as to whether a rule violation has occurred. 

WHY DOES FAST MATTER?

MC: If you have a piece of software that takes hours to run a query, you don’t want to use it. We understand that. We are computing the Order Lifecycle Assembly (OLA) as we receive the data as opposed to computing it at the time when the user submits a query. If I run an OLA that is 500,000 rows, it will return to the user in under 30 seconds. We also use caching strategies to make subsequent runs of the same OLA even faster. Once you get an answer back, you can customize your view of the data to perform additional analytics interactively.

WHO USES FastOLA?

MC: FastOLA is used internally at FINRA by market regulation analysts and their management.

HOW DOES AN ANALYST USE FastOLA?

MC: The complexity of the market has increased significantly with advancement in technology. FastOLA helps market regulation analysts replay what has happened in the market. It does this by enabling users to follow the order flow from origination to execution or cancellation. They use FastOLA to help illuminate potential rule violations in the market and evidence of incidents. When an analyst is looking at audit trail events for a particular security traded by a particular firm, they have the ability to get the associated order lifecycle for each event. They can look at the order lifecycle for a stock that was originated through a specific firm and see how the order was divided and routed to various exchanges and ultimately filled.

WHAT TECHNOLOGY DOES FastOLA use?

MC: FastOLA is the first delivery for user analytics as part of our cloud initiative and uses both big data and cloud computing technologies. The system’s core is a large HBase cluster for rapid data access, which stores the detail data and custom indexes identifying all of the events within an OLA. We also use Amazon’s Simple Storage Service (S3) for staging incoming market data, MapReduce for calculating the OLA and preparing the data for querying, and a custom services layer that retrieves, sorts and filters the data so it is optimized to minimize the request time for the user. In addition to Hbase and Java, other open source technologies used in FastOLA include Tomcat, Spring, Angular, MySQL, and Memcached.

IS OPEN SOURCE AN IMPORTANT PART OF FASTOLA?

MC: Huge. One of the things we try to manage or minimize is vendor lock in. The speed of innovation that happens in the open source community is typically faster than what any one company is really able to accomplish. There are exceptions to this, but in general, that’s what we find. FINRA is currently integrating a leading Business Intelligence tool targeted toward the big data space into our latest application that is under development for audit trail analysis.

DID FINRA PARTNER WITH ANOTHER COMPANY WHEN DEVELOPING FastOLA?

MC: Yes, we worked closely with Cloudera. Their support has been fantastic and they were very helpful in terms of supporting us in getting FastOLA ready for launch. Additionally, Amazon has been instrumental in supporting FINRA with not only FastOLA, but also our next offering in analytics as well.

HOW MUCH DATA IS OUT THERE?

MC: FINRA has approximately three Petabytes of raw market data. We ingest billions of unique transactions each trading day. As for the order lifecycles, we have over a petabyte of data which represents two years of order lifecycles data.

WHAT'S NEXT FOR FastOLA?

MC: Visualizations. This app is screaming for visualizations to accelerate navigation of the data. Not everything needs to be displayed as a giant spreadsheet. We are also looking to enrich the application by adding more data sets.

WHAT'S BEYOND FastOLA?

FINRA’s market regulation analysts navigate through an ocean of data to explore, investigate, and identify market misbehaviors. We’re building the next generation online query tool for them to do their audit trail analysis.

FastOLA logo small

Shelly Bohlin, FINRA VP Quality of Markets/Market Regulation, says the speed of our custom analytics tool, FastOLA, is a game changer for her Market Reg analysts who use the system daily and had been waiting hours at times to receive a return on a complex order lifecycle query using FINRA’s previous query tool. “There was a general frustration level, if you really wanted to see what happened, and then having to wait. You had to turn your attention away and wait for the query to come back,” Bohlin said of the system FastOLA replaced. “Now it almost immediately comes back. You stay focused."

Learn More