A Reverse Engineered Solution
to Legacy Reports

When the Chicago Board Options Exchange (CBOE) gave its regulatory duties to FINRA in 2014, we needed to migrate legacy reports to our platform and be able to use data coming from exchanges like CBOE, C2 and the Options Clearing Corporation’s (OCC). Some of these reports could be shifted to some of our current applications such as WIAT, DIVER. Many, however, had to be rewritten to help regulate the options market. The CBOE Analytics team, led by Kishore Ramachandran, began to look for a solution that not only integrated these reports into FINRA’s market regulation but also improved users’ access to this information.

Reverse engineering a new solution

Converting legacy CBOE code to work with FINRA systems had its challenges. Previously, analysts would access these custom reports from various sources. Reports were created on an as needed basis from CBOE and the OCC’s raw data. It wasn’t always clear what reports were available. Accessing them was also difficult and time consuming. Discussions with analysts revealed that up to 50% of these reports weren’t used.

To start, data coming from exchanges through the Data Platforms group had to be normalized according to report usage. The CBOE reports needed custom processing in order to push data from S3 to the Redshift database which the interactive reports use. With huge amounts of source data, custom ETL request helped extract only the necessary attributes, reducing data storage costs and improving performance.

Creating new reports also had challenges since no requirements were available in the legacy system. To figure out the logic, CBOE analytics team reverse engineered the reports’ code and validated the logic to ensure it was correct in the first place. The CBOE analysts’ institutional knowledge helped the FINRA team to fine tune requirements.

We used Pentaho, a business intelligence tool, to create interactive reports. By converting existing Oracle queries to Postgres SQL queries, reports can run against our Redshift database. CBOE analytics team ensured automated provisioning, installation and deployment of application from by using tech such as AWS Code deploy, Puppet and Jenkins. This reduces the CBOE analytics team’s manual work dramatically.

Iterative for success

This work made dramatic changes to CBOE options reporting. To ensure these changes would help analysts, the team did monthly demos with the options users based in Chicago. These demos would show comparisons between the legacy and new reports. They also gave users the chance to request enhancements for a UX tailored to analyst needs. In addition, the team credits the frequent demos with helping onboard users more quickly. Analysts could see the evolution of the reports they knew and used into a new product. It made them more familiar and willing to use the new system.

From multiple applications to one dashboard

The CBOE rewrite laid the foundation for a common reporting platform called Market Interactive Reporting Summaries (MIRS). Instead of going to multiple applications for each market regulation group, the analytics team created a dashboard MIRS. It centralizes access to reports from many market regulation reporting applications. Legacy applications are now all accessible from the MIRS dashboard.

In creating MIRS, they gave both the CBOE analytics platform and DOMT database the ability to create interactive visual charts using CTOOLS. MIRS runs on a fully automated platform including provisioning, software installation, and application deployment. This automated platform also gathers reports metrics and usage, making trend analysis easy to understand.

This is just the beginning for MIRS. In the future, users will be able to navigate to other applications such as DIVER that provides the deep drill into specific data sets. Other future MIRS features include order blotter analytics, Fixed Income and Market Regulation pattern support. It will also become part of a larger enterprise desktop initiative as well. Further enhancements and product features will be added in the near future. They are also looking at other ways reports and initiatives can integrate with MIRS, improving its utility for analysts even further.