Infosec Teams Race To Secure DevOps

With DevOps adoption spreading, infosec teams are scrambling to address the new security challenges stemming from DevOps’ accelerated code development and app deployment. But while IT organizations have made notable progress adapting security to their DevOps processes, work remains to be done.

That’s a key finding from SANS Institute’s “Secure DevOps: Fact or Fiction” report, which was discussed recently in a two-day webcast (Part 1 & Part 2) co-sponsored by Qualys. A revealing statistic: Under 50% of respondent organizations have fully “shifted left” to embed security throughout their DevOps pipelines, a figure that should be higher.

“Security is still being built in at the end, whereas risk reduction should start earlier in the software development lifecycle,” said Barbara Filkins, a SANS analyst. With security in the early stages of application design, “we can eliminate many issues that we’d see at the back end,” she said.

Threading security throughout DevOps also preserves the benefits of continuous and quick software delivery, like improved customer support and employee productivity. 

“As a DevOps engineer, you’re looking to automate security at the speed of what business needs,” said Qualys Product Management Director Hari Srinivasan.

“The goal is enabling a transition from DevOps to secure DevOps that is factual, not fiction,” Filkins said.

Read on to learn about DevSecOps challenges, best practices and case studies.

Real world DevSecOps

Srinivasan described how several Qualys customers have successfully implemented DevSecOps by automating and integrating security and compliance checks.

At a large U.S. bank, in order to certify the security of its Amazon Machine Images (AMIs), the DevOps and security teams emailed scan reports and fix requests in a back-and-forth loop.

“It took about two weeks for each AMI to get certified,” Srinivasan said. With Qualys’ help, the bank revamped and automated the process.

The DevOps team was given API access to the security team’s Qualys vulnerability management and policy compliance products. This allowed developers to run scans themselves, get reports, remediate and re-scan as needed, without involving the security team. This shortened the process to under 24 hours.

The bank also seeds the Qualys Cloud Agent on every AMI deployed to production, so it’s alerted immediately about newly-discovered security and compliance issues on live instances. The agent has boosted accuracy of detection of vulnerabilities and mis-configurations, slashing false positives, and quickening scan data availability.

“This is an example of how security can be transparently embedded within DevOps processes,” he said.

Srinivasan also detailed how a large retailer takes advantage of Qualys’ integration with Microsoft’s Azure Security Center (ACS) console to secure VMs in its DevOps pipelines.

The integration allows ACS to detect Azure VMs and deploy lightweight Qualys Cloud Agents in bulk to them. The agents gather vulnerability data and send it to the Qualys Cloud Platform, which in turn, provides vulnerability and system health monitoring data back to the ACS administrator.

“This shows how a vendor can transparently orchestrate security into the cloud provider space, removing the friction and overhead the IT ops team would have rolling out security tools,” Srinivasan said.

Srinivasan also highlighted:

• How Qualys’ Web Application Scanning (WAS) product can be integrated via API with the popular Jenkins CI/CD tool so that DevOps teams can run security checks on applications at the staging, test/QA and development environments.

• Qualys’ ability to secure the complete container pipeline — build, ship and runtime — with its Container Security (CS) product. Qualys can also help block vulnerable container images from entering Docker repositories.

Infosec faces old and new challenges

A clear picture emerges from SANS’ survey of almost 300 organizations: Security teams are striving to keep up with DevOps teams’ constant use of emerging technologies, while also protecting legacy software.

In practice, this means infosec teams are learning how to secure serverless apps, containers, IoT systems and cloud workloads, as they also defend mature web, mobile and off-the-shelf apps.

“Legacy apps remain a priority but does it divert the attention from the new platforms and risks that are rapidly becoming mainstream?” Filkins said.

Thus, infosec teams must strike the right balance, so that they don’t fall behind with either set of apps. That’s easier said than done, of course.

“Almost everyone we surveyed is dealing with serious technical and security debt issues with their legacy systems,” said SANS analyst Jim Bird.

DevOps: Full speed ahead

Meanwhile, DevOps teams aren’t slowing down. Their frequency of system changes deployed to production apps increased notably from last year.

“Security teams must keep up — or get left behind,” Filkins said.

They’re trying. Respondents testing business-critical apps twice or more per month rose from 13% to 24% this year. Organizations testing daily and continuously almost doubled.

Unfortunately, the percentage of vulnerabilities repaired promptly and satisfactorily increased only marginally. “Time to patch shows no improvement, or at least not enough,” Filkins said.

The reason may be surprising. “It’s not because they can’t,” Bird said. Management either doesn’t allow or doesn’t encourage them to.

Organizational issues affect DevSecOps success in general, taking the form of skills and personnel shortages, inadequate budgets, and communication silos.

Best practices

Asked to name useful DevSecOps tools and practices, respondents ranked security training first, followed by continuous vulnerability scanning, threat modeling, and continuous IOC monitoring.

Also fundamental: “Shifting left,” so that security is integrated and automated throughout the software development and delivery cycle.

However, the survey reveals that more than half of respondents aren’t meshing security until the development stage or later ones, like QA and implementation.

“That’s a little too late to start thinking about security because it becomes a bolt-on, instead of a holistic approach in the application design,” Filkins said.

In fact, when respondents were asked about DevSecOps success factors, the top one was the integration of automated security testing into build / delivery tools and workflows.

Other DevSecOps success tips from SANS include:

• Making security transparent and adding it into engineering backlogs
• Addressing organizational issues
• Making engineers responsible for building secure code, and providing them with the necessary training and tools
• Creating security champions throughout the organization
• Improving communication and collaboration between DevSecOps and management, and building cross-functional teams

Measuring improvement

Once organizations have a DevSecOps program, they should evaluate its effectiveness. SANS recommends tracking metrics like:

• Time to fix vulnerabilities
• Security issues discovered post-deployment
• Builds delayed due to security issues
• Human hours spent resolving security issues

The common denominator here — and ultimate goal — is speed of delivery.

“As the velocity of delivery increases, the security program has to enable velocity, not slow it down,” Bird said.

---

We invite you to view the webcasts and download the report, where you’ll find many more details about DevSecOps challenges and best practices.