Add Ergonomic Security to Your CI/CD Pipeline
Last updated on: December 19, 2022
Wikipedia defines ergonomics as “the application of psychological and physiological principles to the engineering and design of products, processes, and systems. The goal […] is to reduce human error, increase productivity, and enhance safety and comfort with a specific focus on the interaction between the human and the thing of interest.”
I think this is the perfect definition of why agile DevOps became so crucial to business and so effective in delivering the velocity that digital transformation demands to modern organizations.
The need for agility led DevOps teams to start using multiple Continuous Integration / Continuous Deployment pipelines.
For this article, we’ll focus on three:
- Web applications,
- The containers their applications run in,
- Cloud resources instantiated in their environments
This wave of innovation too often ignores security or treats it as an afterthought in a way that is risky and dangerous. How do DevOps teams build security into their CI/CD pipelines in a way that it becomes an ergonomic enabler and not perceived as a tedious obstacle to overcome?
The answer is right in the definition of ergonomics: reduce the human error and increase productivity of the developers in their working environment.
To manage and effectively run their CI/CD pipelines, DevOps teams adopt tools such as Jenkins, Bamboo, TeamCity, CircleCI and many others; this is software designed to progress code builds from the initial versions to the various phases of the CI/CD pipeline: build, test, QA, preproduction, production.
When it comes to security, DevOps Teams have recently started demanding integration of security controls into their tooling, as early in the development phase as possible. The ultimate goal is to prevent vulnerable code, containers, machines, and cloud resources from getting into production-deployed applications and exposing a potentially devastating attack surface.
To achieve this important goal, DevOps teams leverage the plugin architecture of these DevOps tools to perform dynamic security testing during the post-build phase.
The integration should also expose clear and consumable security testing results within the DevOps tool, to enable the team to plan remediation accordingly without needing to interact with security team.
Securing the Application
Let’s take an example from the web application pipeline. We’ll assume an application that creates a web form running as an httpd instance in a container using a Dockerfile.
The application build – the “CI part of the pipeline” – is configured in Jenkins with a screen similar to the one below:
Once the build completes successfully, the software will execute several post-build actions, for example removing existing running instances of the containers and running the latest build of the applications – executing the “CD part of the pipeline” as shown below:
To add application security testing with Qualys, it is enough to install the appropriate Qualys Plugin for your DevOps tool of choice (plugins for Jenkins, Bamboo, and Team City are available):
In our example, selecting the Qualys Web Application Scanning (WAS) will make a new action appearing among the Post Build actions: Scan for vulnerabilities with Qualys WAS.
You will then be able to specify the credentials for the API-based connection to Qualys WAS, the parameters for referencing the app you want to scan, and the profile you would like to use. Most importantly, you will be able to define the severity thresholds and criteria that will cause the build to fail if exceeded.
Here’s an example:
This option ensures that next time the build occurs, the security testing you configured with Qualys WAS will be executed automatically. Security becomes an automated part of the CI/CD pipeline.
And once you’re verified the proof of concept, that’s when the Qualys epiphany happens: you will realize you have many CI/CD pipelines, each of them needing security.
And Qualys has developed specific integrations to support them!
Securing the Container
The application container can be considered a specialized slice of a system, inclusive of the operating system portion and the service portion and the application portion that is needed to operate. Web apps are often built as application containers for better functional isolation and operational risk fragmentation, but a container can also be a service or the graphical interface of an operating system. In this case, developers are not necessarily writing the functional code of an application like in web app development; they assemble layers to form the application container image.
The pipeline logic changes slightly as follows:
- Developers plan and define the layers to build the container image.
- After the build, the container image is pushed into a registry or repository.
- The Ops team takes it from there and instantiates the containers, e.g. in QA for testing or in production.
- The Ops Team monitors the containers and provides feedback to developers to update the image and improve it.
- The loops restarts.
The need for security in containers is even more pressing, since the shift-left approach should start at the build stage but should also move down the pipeline to the other stages of the container life cycle.
The illustration describes the four main phases of a CI/CD pipeline for containers, showing how Qualys helps each phase in a transparent and ergonomic way.
Build: During the Build the dynamic security testing for vulnerabilities is performed at each iteration as a post-build task, ensuring developers’ awareness of the vulnerable surface and remediating steps without abandoning their environment.
Ship: When the container is ready for the Ship, a Qualys sensor will continuously assess the registry to spot new vulnerabilities over time, ensuring that no vulnerable image is used in production.
Run: At Run phase, Qualys uses two simultaneous approaches to integrate security in the pipeline: a sensor-based approach ensures continuous monitoring of the repository where images are stored and containers run, while the instrumentation layer approach of Qualys Container Runtime Security ensures the needed enforcement to prevent any malicious usage of a running container.
Host: Finally, thanks to the Qualys Cloud Agent running at the Host level and using external Qualys Scanners, the whole host can be assessed to understand the vulnerable and exposed attack surface.
Securing the Host/OS
A third example of where Qualys can integrate security into the CI/CD pipeline is the build of cloud resources, the functional components of an application that are combined to provide the desired functionality.
For example, think of an application that allows you to store files in a cloud environment.
This application will require:
- An AWS S3 bucket or some type of cloud storage to store the files,
- A relational database to keep track of file operations, rights on a file, usernames, etc.,
- An access control list (ACL) to define who can access what and under which conditions,
- A network segment to make the storage reachable,
- And identity and access management (IAM) instance to provide user identification, etc.
The pipeline logic can be imagined as follows:
- Developers plan and define the configuration files to define the cloud resources.
- After the build, tools like Jenkins propagate the configuration files to the right place for uploading to cloud environments.
- The Ops team takes it from there and proceeds with resource creation within the cloud account, e.g. in QA for testing or in production.
- The Ops team monitors the instances including their relationships and security, and provides feedback to developers to update the configuration files and improve them.
- The loops restarts.
Starting from the build phase, it is possible to configure your favorite DevOps tool to trigger an infrastructure scan using Qualys Vulnerability Management (VM) for the machine’s vulnerable surface (OS and installed software), and fail the build by defining multiple criteria. For example, the number of vulnerabilities by severity, by QID or by CVE found, by CVSS base score or if the vulnerability found involves PCI.
This ensures that the whole machine is controlled with an integrated approach, preventing infrastructure vulnerabilities from exposing an attack vector.
Beside this testing at development (Dev) phase, the continuous collection of vulnerability information and compliance posture can be moved along the deployment (Ops) pipeline by integrating a Cloud Agent in the golden image and/or by scanning the network surface with external scanners.
This approach ensures that issues can be highlighted both at DevOps and Security teams level automatically, enabling proper and timely resolution planning and execution.
Finally, Qualys does not pretend to cover the vast entirety of the DevOps tools of choice. While our coverage is in constant progression – the most recent Qualys WAS integration covers Team City – we decided to publish the Vulnerability Management plugin as opensource in our GitHub repository, together with many other scripts and tools to empower security to be built-in and not bolted on.
- Qualys Web Application Security
Integrations: Jenkins, Bamboo, TeamCity
- Qualys Container Security
- Qualys Vulnerability Management
- Qualys Cloud Agent
- Integration Library (GitHub)
- Links to connector marketplaces
Transparently! Effectively! Ergonomically!