Case Study: Cisco Group Bakes Security into Web App Dev Process

“To know what is right and not do it is the worst cowardice.”

That phrase was uttered by Confucius 2,500 years ago, but reflects the spirit behind a recent revamp of a Cisco web app development process that made it more effective and secure.

“This is important as we talk about the secure software development lifecycle, because we weren’t doing what we needed to do, even though we knew what was right,” said Robert Martin, security engineer in Cisco’s Government Trust and Technology Services group.

In a nutshell, the process had fallen into a vicious cycle that pleased no one: Little communication between developers and security pros, combined with late vulnerability scans, yielded buggy software that had to be belatedly fixed, leading to missed deployment deadlines.

“We were making the same mistakes over and over again, and we weren’t making any corrections,” Martin said.

Sound familiar? This is a scenario in which countless organizations have found themselves. After years of using a linear, siloed model for creating and releasing software, organizations discover that this approach doesn’t work well in the era of rapid, agile web development and deployment.

To the credit of Martin and his group, they did something about this, instead of simply plodding along and settling for the status quo.

Buggy Apps, Delayed Releases

In the old process, the development team would work on the app with little visibility and attention to security. After completing the app, they would toss it over to the security team for a thorough review, and with a fast-approaching deadline for deployment.

“We were starting at the very end of the lifecycle, playing catch-up,” said Martin, who spoke about his group’s secure software development lifecycle during a Black Hat USA presentation and in a subsequent interview with Qualys.

Inevitably, the security check would reveal multiple vulnerabilities, configuration problems and other issues. “The dev team would come back to us and say: ‘This can’t work. We’re missing deadlines. We’ve got to get this application out,’ ” he said.

But the problems had to be fixed for multiple business, security and compliance reasons.

Robert Martin, security engineer in Cisco’s Government Trust and Technology Services, speaks at Black Hat USA 2017 about his group’s use of Qualys Web Application Scanning

Retooled Process

A major driver for revamping the process was the need to comply with industry standards and government regulations, such as NIST 800-53 and NIST 800-171 from the U.S. National Institute of Standards and Technology (NIST) and PCI DSS from the Payment Card Industry Security Standards Council.

A common theme across them is the call for thorough risk assessments, continuous monitoring, automated vulnerability and configuration scans, prioritized remediation and detailed reports.

Thus, Martin started with a risk assessment of all the web apps in scope, specifically approaching the development team with questions regarding their awareness of and compliance with these standards and regulations.

“Their answers were ‘no’,” he said. “But here’s the part that was failing on our behalf: We weren’t educating them.”

The risk assessments not only served to create the necessary awareness — Martin’s team created a bunch of new internal policies as a result — but also opened up communication channels that didn’t exist before between the security and development teams.

In order to insert security earlier into the development process, Martin’s team gave the developers access to automated security tools that were integrated into the environment. He was pleased to realize that new tools weren’t needed, but rather to make better use of tools they already had, primarily Qualys Web Application Scanning (WAS), which Martin calls the “gold standard” for web application scanning.

Martin’s team trained the developers on Qualys WAS and gave them access to it so they could start scanning throughout the lifecycle.

This created opportunities to flag security problems early and often in the process, which led to increased collaboration and coordination, more meaningful communication between the development and security teams, and on-time app deployments.

It also gave developers a visibility and an accountability into these issues that they didn’t have before, as well as a new willingness to get issues remediated promptly and correctly, with guidance and help from the security team.

With Qualys WAS’ continuous scanning, the security team and the management team were able to see if security issues were being remediated promptly. Qualys WAS is the catalyst that improved the communication between the security, development, and management teams.

“Now they know what we’re up against, they have buy-in, and we’re enabling them to see flaws on their own,” he said.

Overall, security is now embedded into the entire software development lifecycle. “We’re a cohesive part of the team instead of being the security people that always say: ‘No’.”

This improved communication has also given the security team a view not only into current application development projects but also future ones, so conversations now start at that early stage of an app’s planning.

The process has been improved not only for web apps developed in house but also for  commercial “off the shelf” (COTS) web apps Cisco buys, which in the past were often deployed before having been scanned for security issues.

Here again, Qualys WAS is being used to scan these COTS apps on a regular schedule. Now these commercial web apps are scanned first in a development environment during the “proof of concept” stage of the buying process. They’re also later scanned prior to being deployed, and then periodically in production to check for newly disclosed vulnerabilities and configuration changes. This enables Cisco to be proactive in remediating security issues. WAS is also part of the vetting process for any new COTS applications under consideration.

The end result: better, more secure and compliant apps, developed more quickly and less expensively, and delivered on time. “We haven’t missed a deployment or launch,” he said.

Cisco also took advantage of the product’s APIs to integrate it into the software development lifecycle. Specifically, Cisco is integrating Qualys WAS via API into its privileged password management solution. This has improved Cisco’s security posture and helped to satisfy controlled unclassified information (CUI) requirements.

Qualys WAS         

With Qualys WAS, organizations can find and fix security holes in web apps and APIs through  continuous web app discovery and detection of vulnerabilities and misconfigurations. It can insert security into application development and deployment environments. With WAS, organizations test for web app security issues early and often, enhance quality assurance and generate comprehensive reports. Through its tight integration with Qualys Web Application Firewall (WAF), Qualys WAS can also continuously monitor and virtually patch production web applications.

“The great thing about Qualys is that it’s as much into the development part as it is into the security side,” he said.

According to Martin, security should be baked into every product that Cisco supports or uses. By using Qualys to secure their software development lifecycle, Cisco developers are not only kept informed about new web application vulnerabilities, but also on any security issues related to their code. This enables management to track code development issues and provide training and coaching if needed.

A Source of Truth

Qualys WAS not only gave Martin’s team the visibility into web app security it had been lacking, but also what Martin refers to as a critical and fundamental “source of truth.” This means that Qualys WAS provides current and historical data on the security posture of Cisco’s software development lifecycle, Martin said.

By using Qualys WAS coupled with manual testing of web applications, Martin’s team can help assure its end-users and clients that they have continuous visibility of security issues and faster remediation times.

“We now have a source of truth that we’ve been getting from Qualys.”