Building an AppSec Program with Qualys WAS – Additional Configurations and Review & Confirm

Part 4 – Configuring a Web Application or API: Additional Configurations

Now that we have completed the basic information, crawl settings, and default scan configurations, we can shift our attention to additional configurations designed to optimize scanning and provide granular control over how applications and APIs are tested.

Authentication Records

Using authentication to scan applications and APIs allows users to fully assess their assets for vulnerabilities without limiting scans to unauthenticated content. Additionally, in the case of APIs, authentication may be necessary to even interact with the application. Once you toggle the “Authentication Records” option to enable them, you can select any preexisting record to assign to the configuration. Additionally, you can create a new authentication record to use with the web application or API. This process will be covered in greater detail in a future article on authentication, but WAS supports server, form, OAUTH2, and client side certificate based authentication. WAS can also handle complex authentication processes such as Single Sign-On (SSO) and Time-Based One-Time Passwords (TOTP) through the use of Selenium scripts recorded through the Qualys Browser Recorder (QBR), available as a free plugin for both Chrome and Edge browsers. A link to the plugin can be found here: Qualys Browser Recorder.

A common question we receive is if WAS can support any Two Factor Authentication (2FA) or Multi-Factor Authentication (MFA). With the exception of TOTP, we cannot automate 2FA or MFA. If a mechanism existed to automate authentication through 2FA or MFA, that security feature would be obsolete as it’s entire purpose is mitigate the risk of password-based authentication (stolen passwords, password reuse, weak passwords, etc…). TOTP is different in that the token or passcode generated only requires a seed value and a hashing algorithm. Using these with the server time that a request is made allows real-time computation of the correct token or passcode to authenticate to an application. Examples of TOTP include RSA Secure ID, Microsoft Authenticator, and Google Authenticator. Details of how to use TOTP with QBR will be covered in a future article deep-diving into authentication. In the meantime, the WAS user-guide offers a detailed explanation on how to set this up: Qualys Browser Recorder User Guide.

In the event a web application or API uses 2FA/MFA besides TOTP, customers can use Header Injection (see below), scan pre-production environments where 2FA/MFA is not implemented, or work with their identity management teams to use service accounts that do not require 2FA/MFA.

Best Practice Tip! When adding authentication records, be sure to click on the “Set as default” selector for the primary record so you do not need to specify it at scan time.

Header Injection

Header injection allows users to configure a request header that will be sent with every request WAS makes during a web application or API scan. There are two primary use cases for doing this, although other unique situations may leverage header injection as well.

1. WAF bypass: In addition to adding the appropriate Qualys scanner IP address to any firewall or web application firewall (WAF) to allow unflitered scan traffic to web applications and APIs (see WAS Scanner IPs), header injection can be used to provide an additional layer of attribution and security. For example, many contemporary WAFs can block or allow traffic through based on a unique request header or presence of a specific Cookie. If your WAF is configured to look for a unique request header to allow scanning traffic through as an additional layer of protection, such as Security_Scan: WAS_Scan_Dec_2023, you can add this to the header injection configuration in WAS.
2. 2FA/MFA Authentication Bypass: For applications using 2FA/MFA for authentication, users can manually log into the protected application and extract the authentication mechanism from their session for use in on-demand WAS scans. Most client browsers will allow you to view your session Cookies directly. Instructions can be found here for Chrome and Firefox. Additionally, most proxy intercept tools like Portswigger Burp and ZAP will easily allow you to inspect and copy Cookies or other session tokens captured while interacting with web applications or APIs. Remember that 2FA/MFA header injection is for on-demand scans while that session is current. Any time a new scan is launched, a new session token or Cookie will need to be used in the header injection configuration. If using TOTP, however, please see the notes under authentication above as to how this can be automated with the Qualys Browswer Recorder.

API Endpoint Definition

As APIs are just endpoints without HTML or links, it is necessary to define all endpoints, operational methods, and parameters necessary to interact and fully test them. This can be accomplished through Postman Collections, OpenAPI Specification (OAS) / Swagger Files, or Portswigger Burp Proxy Captures. This configuration allows users to upload these files for use in scanning APIs. If accessing a OAS or Swagger file directly by URL, customers do not need to upload any files under API Endpoint Definition (see Part 1 of this series). If using Postman Environment or Global Variables, those files can be uploaded for use as well. Testing APIs are limited to how complete the Postman or OAS/Swagger files are – if endpoints are missing, they will not be tested. Care must be taken when using proxy captures for upload to fully interact with the APIs across all endpoints using all operational methods.

Setup Exclusion Lists

While this category is called exclusion lists, it includes both allow and deny rules for controlling how an application is crawled and tested. Users can select global exclusions, local exclusions, or both.

1. Allow List. The scanner will only crawl the links explicitly provided as URLs or those that match the regular expressions you provide. This is not the same as providing explicit URLs to crawl under the application details section. The scanner will not just “jump” to the allowed URLs to crawl – it must follow links to reach them. If any linkage in that chain is not included in the allowed rules, the crawl may not reach all the links you expect and sections of the application will not be reached. This can be corrected by providing explicit URLs to crawl under the application details section. Please also note that setting anything under allow list will essentially exclude all other links, provided you have not added them explicitly as mentioned previously.
2. Exclude List. WAS will not crawl or test the links explicitly provided as URLs or those that match the regular expressions provided. Please also note that setting anything under exclude list will essentially allow all other links. For this reason, it is important not to set up rules under both exclude and allow list as they may create a conflict with each other and result in erratic behavior.
3. POST Data Exclude List. A POST data exclude list will prevent WAS from submitting a POST request to a specific form to prevent unintended consequences, such as mass-emailing or interacting with a database in a way that is not wanted. By specifying a regular expression to match the POST parameters submitted to a specific form, WAS can identify and exclude those submissions during testing. Just be aware that this impacts the full coverage of a scan, and these forms should be manually analyzed and tested for complete security coverage.
4. Logout Regular Expressions. By design, WAS will identify logout links and not follow them during the course of an authenticated scan. However, web applications or APIs that have links or endpoints that result in automatic session de-authentication can make use of this feature to prevent WAS from unintentionally logging out. Any discovered links that match the logout regular expression will not be followed, maintaining session authentication.
5. Parameter Exclusions. Users can add exclusions for very specific items to omit from the web application scans. The most common and potentially useful use case here is to exclude all non-session related Cookies.  This can result in significant scan efficiency improvement by removing all Cookies not used by the web application – such as Google analytic Cookies or similar tracking Cookies. These are irrelevant to the security of the application (i.e. the application itself does not use a Google analytics Cookie) and otherwise WAS will test all request headers (including these irrelevant Cookies) on each URL in testing scope.

Default DNS Override

The Domain Name Service (DNS) is the process by which a domain name (e.g. Qualys.com) is turned into a IP address (e.g. 64.39.96.133) used by both a client browser and WAS for reaching web applications or APIs on the world wide web or inside private networks. By default, WAS will use the DNS for the web application URL to identify the IP address for crawling and testing the web application or API. If you select a DNS override record, WAS will use the mappings in your record instead.

There a few reasons a customer might want to do this. For example, a web application may not have a DNS entry if it’s in a non-production environment. Or the web application may have a different IP address in a non-production environment (e.g. development or QA) than in production.

Redundant Links

Redundant links are the most efficient method to optimize web application scanning and reduce overall scan times. WAS is designed to test custom source code for web application and API vulnerabilities. When you consider a catalog sites with hundreds, thousands, or even millions of products, no scanning tool will ever completely crawl every single page. And really there is no need to. Many pages in a catalog application use the same identical underlying source code, they just load different content from a database – such as pricing, product descriptions, reviews, pictures, etc. Catalog sites are not the only web applications reusing identical source code – it could be as simple as one or two press releases each week going back 5 years. That’s between 260 and 520 URLs using identical source code that do not need to be individually tested. Redundant links can be used to omit testing of identical source code on pages through regular expression matching. A great resource to test out regular expressions can be found at Regex101.

A few useful things to keep in mind: if you are using complete or partial URLs including the “http://”” prefix, you can include both http and https by using http(s*) – This allows the engine to match on http or https. In many instances this can be avoided by setting up a regular expression to match on something unique in the URLs themselves. For instance, if there is a collection of press releases and you have identified them as being good candidates for redundant links, you can use wildcards instead of specifying complete URLs.

As an example, consider https://www.qualys.com/company/newsroom/news-releases/usa/2022/…  To include both the http and https version of ALL press releases regardless of year of release, a valid regular expression could be: .*\/newsroom\/news-releases.* (note the escape character ‘\’ in front of the backslashes ‘/’).

Sometimes customers will state that even after adding a redundant link, WAS is still crawling and testing URLs that should be omitted. This usually happens as a result of running a vulnerability scan before setting up valid redundant link regular expressions. If a vulnerability scan identifies a vulnerability on any URL, the scanner will ALWAYS attempt to retest the vulnerability regardless of any redundant link rules. The only way to remove this prior vulnerability information is to purge the application, and then recreate it with the correct redundant link rules in place. Therefore, it is important to run discovery scans and ensure you are crawling only the links you desire before launching a vulnerability scan.

Path Fuzzing Rules

Path fuzzing rules, much like redundant links, can help improve the efficiency of WAS scans by identifying and reducing duplicate source code testing. An excellent article on path fuzzing can be found here. In short, many web applications will take a URL with parameters, such as:

https://www.example.com/articles/display.php?issue=17&section=sports&article=28

and “prettify” it to:

http://www.example.com/articles/issue/17/section/sports/article/28

One can imagine there might be dozens, hundreds, or thousands of these pages where the source code is simply display.php and it uses the URL parameters to specify what content to load. Once again, it is not necessary to test every single one of these pages, and with path fuzzing rules, customers can specify a URI template for WAS to use. For example, the above URI could be rewritten as:

http://www.example.com/articles/issue/{issue}/section/{section}/article/{article}

During a WAS scan, WAS will identify URIs matching the path fuzzing rule and only test a few of them, greatly reducing the overall scan time. WAS will identify the alterable parameters (issue, section, article) and fuzz those values for vulnerabilities like SQL or command injection using the rules specified instead of testing every single path component of every URI.

Form Training

Form training allows users to provide input for specific workflows during a discovery or vulnerability scan. These inputs may be necessary to allow the application to “move on” to the next step. For example, if testing a web application for car insurance, one of the steps may require a legitimate US Driver’s License for the application to assess whether or not the license is still valid and the owner can even obtain insurance. WAS users can specify the URL and form values to submit so that the workflow can be completed and the application can move to the next link for crawling and testing.

Best Practice Tip! Selenium scripts can be used to perform the same function as form training, and are often faster and easier to setup. For more details see the “Crawl Links” section of Part 2 of this series.

Malware Monitoring

Malware monitoring is an included service with each WAS subscription. While Qualys WAS identifies vulnerabilities in the source code of web applications and APIs, Qualys Web Malware Detection (MD) identifies malware present on web applications. You can configure a malware scan against any external facing web application and it will run as a separate scan from your normal WAS scanning. While you could have both WAS and MD scans run concurrently, customers can also choose to have them run on their own independent schedules. MD scans simply crawl the application – there are no vulnerability test payloads with any requests. As a result, a MD scan takes significantly less time to complete than a WAS vulnerability scan.

During the crawl, each URL of the web application is inspected for the presence of malware through four methods:

1. Signature – Just like a commercial antivirus solution, MD will identify malware based on known signatures and alert to these findings.
2. Reputation – MD will search for the web application in lists of sites identified by 3rd parties for hosting malware, such as Google Safe Browsing Site Status. Additionally, any links on the web application to 3rd party sites will also be checked for malware reputation and reported if it is flagged.
3. Heuristics – While most live malware is either obfuscated so it’s not easily readable, or carefully inserted into existing code to surreptitiously blend in with the application, certain commands and functions will imply malicious intent, either alone or in combination with each other. MD will identify malware using obfuscation techniques despite it’s attempts to evade detection.
4. Behavioral Analysis – The MD scanner is sandboxed so that not only can it identify known malware through the 3 processes outlined above, but the sandbox itself is monitored to identify previously unknown novel malware in the wild. For example, as MD crawls a site, if the sandbox is observed to attempt unusual read or write accesses, or CPU usage or memory starts to spike, that may imply the presence of malware and the behavior will be reported.

To configure malware monitoring, you can specify the cadence for which the scans take place. Options include a single occurrence, monthly, weekly, daily, or even hourly. You can set the time that the malware scan will take place, and opt to send an email notification to alert that a MD scan is about to take place.

After a scan is complete, an email notification of any findings will be sent to the Qualys WAS web application owner. Additionally, all MD scan reports can be reviewed directly through the MD module included with every WAS subscription.

Now that we have completed these additional configurations, it is time to click Next and go to the final step in adding a new web application or API – Review & Confirm.

Part 5 – Configuring a Web Application or API: Review & Confirm

The final step when adding a new web application or API is to review the settings to make sure it is configured correctly. You can easily navigate back to any of the prior steps to edit the settings and update them. You can also choose to cancel the web application or API configuration and return to the main Web Application tab in WAS if so desired. Otherwise, at this point you can click on the button to either “Save the Web Application” or “Save the Web Application and Scan”.

Selecting “Save the Web Application” will save the configuration and return to the main Web Application tab in WAS. Selecting “Save the Web Application and Scan” will save the configuration but also give you the option to launch a Discovery or Vulnerability Scan on-demand to immediately start testing the web application.

Best Practice Tip! Run a discovery scan before a vulnerability scan to make sure all of your configurations and optimization rules are working properly.

Please join us for our next article where we will cover WAS Option Profiles in detail and provide additional best practices for getting the most out of Qualys WAS.