Market Order Reconstruction in HBase
Strata + Hadoop World San Jose, CA
February 19, 2015
Presented by Tigran Khrimian, VP of Data Platforms
Khrimian shows how FINRA leverages Hadoop in Amazon Web Services (AWS) to protect investors and ensure the integrity of US financial markets.
[Music Intro] Well good morning. So I'm Saman Michael Far from the Financial Industry Regulatory Authority. So I’m going to tell you a little bit about our projects and our use of AWS as it relates and so what do we do. Let's start with that. Essentially, we receive feeds of market events from exchanges in the US and firms. We get 30 billion of these market events and we create a picture what's going on in financial markets. And then run an extensive library of surveillances that we have on these to essentially look for hanky-panky going on in the markets. There are all sorts of patterns. When you've got this kind of volume there are certain things that come up. Many of you are familiar with processing at scale and it’s very different from processing not at scale. What happens is market volumes are typically very volatile. In normal trading volumes there are many factors a difference between a slow market day and a high market day. That's one aspect that makes this challenging. Financial markets are not static. There's a lot of innovation going on to attract capital, new products, new rules introduced around these. And in the middle of all of this, the market manipulators also innovate. So we need to be very agile and nimble to be able to track with these things as they change and evolve to bring new surveillances to market and new capabilities to bear to look for all of these things that are happening. So why do we choose AWS? I think that is of interest. One of them is that the level of functionality is that the right layer for us. We have significant amounts of software developers and technologists here in New York and outside of Washington DC and what we want to be able to do is innovate and develop our systems are on top of this to match to our business. And the APIs and the functionality provided gives us this ability without necessarily needing to get under the hood into the infrastructure, but having that visibility to be able to do that and tune if necessary. It’s very important for us not to get bogged in there put our dollars where our business value is, but be able to have full generality of control on what's going on. The second is the automated infrastructure deployment. That’s critical for us to operate efficiently at scale and what we get from AWS is that ability. And then the last one which is important is we're a big data processor. As you know, the big data market for solutions is very fragmented. It’s evolving. It's a new area. The winners aren’t clearly established. For us to avoid vendor lock-in, we need to be on open source. We need to use it. We need to be part of the ecosystem and contribute back to it. And AWS’ strategic commitment to that fell in line with our objectives. Then, the enterprise operations and security provided by AWS, clearly set up for enterprise support. From systems engineers that were assigned to us that we work with, the level of support we get - the security facilities—we have very stringent requirements on security that are satisfied. What are we getting out of this? Agility. Respond quickly to our business needs. Put our money where the business value is in terms of software development, speed. production application we have at the moment, for example, one of the systems in production, a typical complex query that was executed a lot was taking an hour, hour-and-a-half. Now, five to 10 seconds. And cost savings - we're projecting to save $10 to $20 million annually around this. What's next? The ability to have dynamic computing, spinning up clusters, solving business problems in a different way. This is the basis of what we're doing and what we feel is a transformative capacity for our surveillance abilities. Thank you very much.