Staying Secure
with Open Source Code

Why We Automated Tracking of Open Source Code

How We Maintained Open Source Code and Enterprise Security

Building a holistic picture of US market trading requires immense amounts of data. An average day of trading requires monitoring 30 billion financial transactions, approximately 6 terabytes of information. Dealing with this much data requires our 500 software developers to have between 100-130 apps under management, translating into over 100,000 builds.

As we began to rely more on open source code, controlling security implications became increasingly difficult. Our first attempt was a homegrown solution. Yet, managing this workflow became a full-time job for Lead Systems Engineer Marcela Carbo. “The reporting became an issue,” Carbo says. “We realized it wasn’t scalable, and we couldn’t do effective tracking.”

The Struggle Between Efficiency and Security

Balancing both efficiency and security took time. Our next solution was a commercial database that enabled tracking of open source code during different stages of workflow. It became easy to understand what code was coming in, but we struggled to know which specific components were being used.

Carbo explains, “our approval process boiled down to: as long as the dev team opened a ticket, then it moved forward. We had repos created for ‘awaiting approval’ ‘approved,’ etc… thus mapping the various repos to key workflow stages. As long as the dev team submitted a ticket, we felt we were following the approval policy. But we lacked a solution for what to do when components were rejected. We didn’t have a complete picture of who was using what and where and how.”

With so many applications and work in flux, real time understanding was critical. “In this world we live in, you pick one open source artifact and it comes with many dependencies,” says Gaitanos, Senior Director of Development Services. “If you had 30 or more dependencies, you’d end up with the task of creating and managing 30 or more tickets. This increased the administrative burden on developers.”

Because our charter includes providing training to brokers and brokerage houses, along with providing brokers with permits to trade in North American markets, our application development emphasizes high-stakes, high-volume work. Improving efficiencies in software development was critical to carry out our mission.

“Given the volume we were dealing with, we had to change the way we looked at this problem. We somehow had to shift our focus from reviewing 100 percent of artifacts introduced to our environment, to an exception-based review process,” says Gaitanos, “This way the vast majority of artifacts get vetted automatically by the process and only a select few get flagged, based on predefined conditions, to be reviewed by both our legal department and by technologists.”

Creating a Coherent and Efficient Flow of Dependencies

Combining Artifactory Pro’s ability to manage binaries and Black Duck’s OSS Logistics solution, we have reduced the legal department’s open source-related workload by 75%. With this change, we also eliminated a technology review team that vetted open source license compliance under the old system. Now all tracking of open source code is automated. We’ve even improved visibility by creating a continually-updated open source bill of materials (BOM).

“Under the old system, everything had to be tracked up front as part of the process. Now, we track open source code usage on an exception basis only, because of the BOM,” Gaitanos explains. “A human only gets involved in the event of an exception. This saves a lot of time.”

These two systems have made it easier for developers. Carbo explains: “For us, the main thing is to get out of development’s way. The old system really slowed down development, but with Black Duck, they don’t have to worry about filling out spreadsheets. Plus the legal team would have to get involved to vet each usage, and they don’t have to do this now. Changing a version doesn’t prompt all this work.”

With these new systems, developers are saving, on average, three “person days” per app. With over 100 apps, that’s 300 days of developers’ time and work.

But the savings don’t stop there, says Gaitanos. “We’ve seen a huge savings not only from a human interaction and effort perspective, but in the way folks behave towards the process. The more hurdles you put in front of developers, the more likely it is they will work around you. We’re taking hurdles out of their path and making things easier for them.” As this pilot program expands, we expect to simplify our workflows even further.

Security has also become easier. Our internal security group can now easily identify which artifacts and versions are affected by any security vulnerability. Now they can focus on the select applications impacted rather than constantly check 100+ applications. This new capability has been huge to maintain and protect not only our technology, but also all the data and services we provide.

This piece was adapted from an original Black Duck case study.