At FINRA, analysts sift through billions of transactions. They’re looking for irregularities that could reveal manipulations in the market. Yet, what is a real problem and what is a red herring?
To help find out, we created the Depth of Market Tool (DOMT). This tool helps analysts look at specific trading time frames, from as small as a minute to as wide as fifteen minutes in order to investigate market activity more deeply. With graphs and charts, analysts can easily visualize the sequence of market events, all open interest on the buy and sell side at any point in time across all equity exchanges, and highlights time segments where a manipulation is most likely to occur.
Although an important tool, DOMT needed enhancements. With 3 billion records loaded daily, processing the growing amount of data became an increasing issue. Static machines struggled to keep up performance for users while also dealing with about 300 gigabytes of new data added every day.
It was obvious that increased performance was necessary for this application to provide faster response times that users needed. With much of our data already moving to the cloud, the team wanted to find a faster solution that could live there as well.
Shifting to the cloud required more than moving 2 years of cross-market data storage. Previously, the application went to a Postgres database for both storage and processing. Today, our Extract, Transform, and Load (ETL) group moves data to Amazon’s S3 bucket for storage. Then, the application moves the data to Redshift, a columnar data warehouse, to achieve faster response time for users.
To make this work, Technology Delivery Lead Janaki Parameswaran and her team had to rewrite the entire data access layer. Part of this included collaborating with the Data Management Activity Workflow, which separates metadata from the data itself. This tool not only helps DOMT with storage but also loads and transforms the data sets from S3 to Redshift. In addition, they created new data architecture optimized for performance. It may help analysts focus on small data sets, but the application sifts through petabytes of data. So, they chose tools such as Activiti Workflow, Hive on EMR, and Redshift to optimize DOMT for both cost and performance.
Rather than a headache, FINRA technologists found this exciting. “DOMT provides investigators and business users a quick way to replay the market to help detect anomalies,” Parameswaran explained, “You want response within seconds from the application. So you have to optimize for speed.”
Today, the entire program, including application servers, lives in the cloud. There’s no longer any issue with processing requests. In fact, the tool is being opened up from 200 users to the entire market regulation department, or about 600 users. It’s also being used by other applications, including Bluesheets, which helps track blue sheets requests for firms and filings requests.
This process is part of a larger strategic move for our market regulation applications. “By moving our applications and data to the cloud we are able to achieve a highly dynamic and very cost efficient platform to address the volatility in the market volumes that can vary greatly day to day,” stated Scott Donaldson, Senior Directory of Market Regulation Technology. He continued, “Before we lived in a box and moved to a bigger box when we outgrew it, which required extensive migration projects. Now we scale the applications up and down based on demand.”
Though DOMT has seen many changes in the past year, more improvements are on the way. Eight new features will be added early this year. These include additional tools for analysts including pivot position and pivot quote visualizations that further the ability of analysts to review trading behaviors.
On a larger scale, the application is going to be more closely integrated with other tools that help regulate the market such as DIVER and FastOLA. Each application will remain separate but will become, “one ecosystem” Parameswaran said. Analysts will be able to open one application from another. For instance, if an analyst sees something unusual in an audit trail while using DIVER, they will be able to open DOMT to immediately investigate the issue further.
In the end, this work isn’t just about improving analysts’ experience with applications. It should also improve investigations keeping the market safe and protecting investors.