Qualys API Best Practices: Web Application Scanning API

This API Best Practices Series is designed for Qualys customer programmers or stakeholders with a general knowledge of programming who want to implement best practices for improving the development, design, and performance of their programs that use the Qualys API. For non-customers, the Qualys API demonstrates our commitment to interoperability with the enterprise IT security stack.

Qualys Web Application Scanning (WAS) enables organizations to assess, track, and drive remediation for web application vulnerabilities, helping them secure their web applications and keep the attackers at bay.

If you are a programmer, your enterprise can derive superlative benefits from the WAS data from the step-by-step guide provided in this blog post.  

Part 5 of this series aims to obtain WAS Web Applications, Web Application Details, Web Application Findings, and the WAS Catalog in both compressed JavaScript Object Notation (JSON) form and into the latest timestamped, point-in-time SQLite database.

But, before we get to it, let’s quickly look at the most sought-after capabilities of the Qualys Web Application Scanning (WAS):

• A robust, centrally managed, cloud-based application security product that continuously discovers, detects, and catalogs their web applications and APIs (Application Programming Interface).
• Comprehensive and accurate scans to uncover runtime vulnerabilities, misconfigurations, Personal Identifying Information (PII) exposures, and web malware across modern web applications or APIs.
• Integration Capabilities into CI/CD Pipelines such as Jenkins, AzureDevOps, Bamboo, and Team City.
• An optimized posture of risk reduction and mitigation by leveraging ITSM ticketing automation after complete scanning that can reduce Mean Time to Resolve (MTTR).

We are also providing a blueprint of the example code called QualysETL, which is an opensource code under Apache 2 License for your organization to develop with:

We will provide more details along with live examples to help you effectively extract, transform, load, and distribute Qualys WAS data to the cloud database.

About the Qualys API

The Qualys API is a key component in our API-first model. Since the founding of Qualys in 1999, a rich set of Qualys APIs has been available and continually optimized. As a result, programmers at Qualys customer organizations have automated processing Qualys in new ways, increasing their return on investment (ROI) and improving overall mean-time-to-remediate (MTTR).

The diagram below depicts QualysETL as a workflow, resulting in compressed JSON Data and an SQLite Database. The JSON and SQLite data can be used on your desktop for analysis or as a continuous live data feed to update your corporate data store.

Even with all these advances in our API, some enterprise customers continue to experience suboptimal performance in automation. What are the inherent automation challenges to Extract, Transform, and Load (ETL) Qualys data?

And what do we mean by ETL?

1. Extract refers to extracting WAS data using the Qualys WAS API. In this step, you execute API calls to extract web applications, findings, and catalog data from your subscription.
2. Transform means reading the resulting extracted data from Qualys and transforming it into other forms or formats your organization decides will be helpful (e.g., SQL, DDL, or CSV). This step lets you prepare and enhance the WAS data. For example, injecting security intelligence specific to your environment will help drive remediation.
   Examples of this include data classification or application owner to prioritize risk.
3. Load refers to loading the data into its final form on disk for independent analysis (e.g., SQLite) and/or distributing Qualys data to its destination in the cloud for using it in metrics or visualization tools such as PowerBI or Tableau.

Automation Challenges

With any API, there are inherent automation challenges. Some of these are:

• What are the best practice programming methods to extract WAS from the Qualys API reliably and efficiently?
• How can some or all the WAS Applications, Findings, and Catalogs be obtained?
• How to integrate Qualys data into an SQL database for use in automation?

In the Example JSON Output image below, we have noted some key fields, including:

1. The hasMoreRecords field is on line 728; when set to “true”, the lastId field on line 730 is helpful to get the next batch of Findings.
2. The findingType field on line 135 is set to QUALYS, and as a result, the Qualys Knowledgebase QID 150112 on line 134 is used to find additional information about the Qualys Detection.
3. The type on line 71 is VULNERABILITY and is not a potential vulnerability, as noted on line 72.
4. The severity of vulnerability is “2” on line 73.
5. The lastDetectedDate on line 720 indicates when the QID detection was last executed.

You will want to transform the JSON data and prepare the data for ingestion into a database for future correlations with other corporate data sources. The QualysETL blueprint of example code can help you with that objective.

For more information about our JSON Fields in Qualys WAS, please refer to the WAS API Appendix.

About QualysETL

QualysETL is a blueprint that can be used as a starting point to develop your ETL automation. The image below shows QualysETL workflow, which includes the processes to:

1. Extract: API Calls used to obtain Qualys data
2. Transform: Methods used to enhance data, prepare/cleanse data, and catch errors in data
3. Load: Methods to produce various forms of data useful to your organizations, such as SQLite
4. Distribution: Methods you develop to distribute Qualys data to your organization

In the diagram, we show the base schema for the following tables:

1. Q_Was_WebApp – Web Application Data
2. Q_Was_Findings – Vulnerability Data
3. Q_Was_Catalog – staging area for potential web applications.
4. Q_KnowledgeBase_In_Q_Was_Finding – Knowledgebase information found in Q_Was_Findings data

These tables are automatically created through QualysETL of WAS and will grow over time as Qualys keeps adding new capabilities. See the WAS API Guide for a complete list of available fields.

With WAS data prepared for use, you would want to distribute it for usage by your corporation. For example, you can distribute a timestamped version of the SQLite Database into an Amazon Web Services Relational Database Service or an AWS (Amazon Web Services) S3 Bucket.

Further, you create the SQLite database available locally for analysts so they can process and report on vulnerabilities in your organization using their desktop tool of choice.

Data usage flexibility is achieved at this point.

Contact your Qualys Technical Account Manager and arrange a meeting with David Gregory, Principal Solutions Architect of API, if you are interested in automation to inject data directly into your downstream database from QualysETL.

WAS Activity Diagram

The WAS Activity Diagram below depicts QualysETL pagination to obtain Qualys WAS data and simultaneously load WAS data into an SQL Database. The activities include:

1. KnowledgeBase, Web Apps, and Web Catalog are extracted through Multiprocessing.
2. The WebApp ID List drives extracting Web App Details and Findings for each application.
3. Transform and SQL Load occur as each WebApp Details/Findings batch is available.
4. While QualysETL runs, you can immediately begin distributing your data to downstream systems for Metrics, Visualization, and Analysis to drive remediation.

API Call Recommendations:

• Use the “hasMoreRecords” Flag set to “true” and “lastId” to paginate through your API calls.
• Distribute snapshots of your ETL data for desktop analysis or as a continuous update pipeline in your organization’s data store.

Postman Examples

In the following four examples, we will:

1. Get the web application count.
2. Get web applications using limits and filter criteria.
3. Get web application details.
4. Get web application findings (vulnerabilities)

In the first example below, we use Postman to count web applications using the “/qps/rest/3.0/count/was/webapp” endpoint. “https://{{base_url}}/” is your subscription API Server URL from Platform Identification. 

In the request’s body is the ServiceRequest payload, which includes the last scan date, limiting the count to Web Apps scanned after January 1st, 2023. The resulting ServiceResponse includes a count of sixteen web applications.

If you are new to Postman, check the Qualys API Fundamentals Training to learn more about Postman and how it can be developed with Qualys API.

In the second example, the “/qps/rest/3.0/search/was/webapp” endpoint is used to download web applications, filtered using lastScan.date to limit download of Web Apps to those scanned after January 1st, 2023. Notice that the hasMoreRecords field is true, and the lastId is set to a number. Use the lastId in your next API call to get the next batch of hosts by updating the filter criteria “id” to the value of lastId.

In the third example, The /qps/rest/3.0/get/was/webapp/[id] endpoint is used to download web application details using the unique id for each Web App. Extended Web App configuration details are included with this API call.

In the fourth example, the “/qps/rest/3.0/get/search/was/finding” endpoint is used to download web application findings using the unique ID for each Web App. Details of vulnerability findings for the application are included in the data. See the end of this presentation for links to details describing each field.

You can develop your own integration with the WAS API or leverage the QualysETL Blueprint of open-source Python code to download all your WAS Data with a single command!

QualysETL Installation

To install QualysETL, we recommend you provision a secure, patched, up-to-date virtual machine instance of Ubuntu 22.04 with internet connectivity. When this step is completed, you can log into your Ubuntu instance and follow along through this video to install the application and run your first ETL.

The instructions are located on pypi.org at: https://pypi.org/project/qualysetl/

Next, you can run your own SQL queries to analyze the data and tune the application to meet your needs.

QualysETL Installation Summary

Three Key Takeaways

1. Full automation is possible, as with one command, you can ETL Qualys WAS into an SQLite Database, ready for analysis or distribution.
2. Data is flexible as it can be sent downstream to other SQL Databases for applications such as PowerBI or Tableau to visualize your data.
3. QualysETL is a blueprint of example code you can extend or use as you need because it is open source and distributed under the Apache 2 license.

At this point, you should be able to query your WAS SQLite database. Below is an example SQL query to find data of severity type 4 and finding Type Qualys.

Put QualysETL into Practice

Leverage QualysETL as a blueprint of example code to produce a current WAS SQLite Database, ready for analysis or distribution.

For questions, existing Qualys customers can schedule time through their Technical Account Manager to meet with our solutions architects for help. Non-customers can request access to the Qualys API or QualysETL as a free trial of Qualys WAS to learn more about their full capabilities.

Learn More

• API User Guide: WAS API User Guide
• Blog: Building an AppSec Program with Qualys WAS
• Training Courses: “Web Application Scanning“,”Qualys API Fundamentals“

About This Series

This blog series lets you create a unified view of Qualys data across our cloud services at the Qualys customer organizations. It includes the Qualys VMDR (Vulnerability Management, Detection and Response), CSAM (Cyber Security Asset Management), WAS (Web Application Scanning), and PCRS (Policy Compliance Reporting Service).

The API Best Practices Series will continue to expand over the coming months to cover other key aspects of the Qualys API, with each presentation building on the previous one and, in the aggregate, providing an overall best practice view of the Qualys API.