Qualys TotalAI: The Journey from LLM Scanner to Comprehensive AI Security Solution

Embarking on the AI/ML Journey

The launch of Qualys TotalAI marks a significant milestone in our journey with AI/ML. It all began in March 2024 when we ventured into the rapidly evolving AI/ML landscape and the emerging LLM ecosystem. Recognizing the potential of these technologies to revolutionize cybersecurity, we set out to explore their applications. At the time, tools like LangChain, ChainLit, Ollama, OpenLLM, etc., were gaining traction and we were eager to harness their capabilities.

Our initial focus was straightforward: summarization and Q&A. Early successes inspired us to push further, exploring inferencing capabilities where models uncovered insights beyond the explicit data provided. This deep dive into AI/ML expanded our understanding and revealed broader applications.

As our exploration continued, we delved into advanced concepts like retrieval-augmented generation (RAG) and related tools, building a robust foundation in AI/ML systems. A key turning point came with insights from OWASP, which reshaped our approach. Like many in cybersecurity, we faced challenges assessing the security of AI/ML and LLMs—challenges shared by our customers. This is where an LLM Scanner proves invaluable, empowering teams to tackle complex AI/ML security needs with confidence.

 In February 2024, OWASP released their “LLM AI Cybersecurity & Governance Checklist,” which highlighted the need for LLM-specific security scanners and heightened cybersecurity awareness for these new technologies. Although the paper was released before our journey began, we only encountered it after starting our efforts. This pivotal moment steered us away from simpler use cases toward a more ambitious goal: creating a comprehensive LLM scanner that could help safeguard Gen-AI systems.

The Need for a Proof of Concept

To develop an effective LLM scanner, we set clear objectives for our Proof of Concept (POC). The primary goal was to identify and assess vulnerabilities in LLMs. Success criteria included detecting known vulnerabilities, generating accurate risk scores that reflect severity and impact, and ensuring consistent evaluations. The POC also needed to demonstrate scalability and seamless integration into existing workflows, while keeping the solution lightweight and efficient.

From Vulnerability Scanning to AI Security

At Qualys, our history lies in vulnerability management. We have since grown to offer a robust platform of solutions, including an industry-leading web application and host-level vulnerability scanning solutions. With OWASP’s guidance in hand, we decided to adapt our security expertise to the emerging world of LLMs. We aimed to develop an AI/ML scanner that could help secure the growing ecosystem of LLM applications. This shift required us to rethink our approach to vulnerability management, adapting traditional security concepts to fit the unique challenges posed by AI/ML models. As we explored different possibilities, we faced multiple challenges. Initially, our approach was to add LLM detections to API Security. However, we soon realized the value of creating a dedicated module to consolidate all functionalities at different levels. One significant challenge was how to correlate VM risk with AI risk for in-house use cases, which required a deeper understanding of both infrastructure vulnerabilities and the risks inherent in AI/ML models to provide a unified risk assessment.

We began with the OWASP Top 10 for LLM Applications, focusing on vulnerabilities like prompt injection, insecure output handling, model theft, and more. Initially, we generated basic prompts and questions to evaluate target LLMs like phi2 and Vicuna. The results were promising, but we quickly realized that a more sophisticated approach would be necessary to address the nuanced risks involved. This realization led us to use a Judge LLM, a model capable of evaluating responses and assessing security risks in a more structured and consistent manner.

Using well-known foundational models, we tweaked them with prompt engineering to have them act like a Judge LLM for evaluating the responses from our target LLMs and assessing the security risks. However, the evaluation process was anything but straightforward. We faced challenges, such as ensuring the Judge LLM did not consider the prompt itself when determining the severity of vulnerabilities and maintaining consistent scoring akin to static vulnerability scores like CVSS. These challenges required us to refine our prompt engineering skills, experimenting with different phrasing and techniques to achieve the desired evaluation consistency.

The Proof of Concept: Defining Goals and Success Criteria

Our proof of concept (POC) culminated in a simple application capable of prompting any LLM, evaluating the responses, and scoring vulnerabilities. We compiled a dataset of roughly 1,000 questions spanning 16+ risk categories and subcategories.

The dataset was created by us based on the attack categories listed in OWASP, with questions generated by LLMs, handcrafted, and also sourced from open-source datasets on Hugging Face. The generation of the initial 1,000 questions underwent multiple iterations after thorough scrutiny. We used Hugging Face’s text-generation inference platform to prompt the target LLM and then used our Judge LLM for evaluation. This process allowed us to gain valuable insights into the strengths and weaknesses of various LLMs, as well as the types of vulnerabilities that were most prevalent.

We expanded our approach by incorporating 20+ jailbreak attacks, leveraging open-source research to understand different types of jailbreaks and attack methods. Jailbreak attacks involve bypassing the built-in restrictions of LLMs to make them behave in unintended or potentially harmful ways. Jailbreaking, in particular, opened up new avenues for testing the robustness of LLMs against malicious inputs. Jailbreaking closed-source models is particularly challenging. It requires advanced techniques to bypass their inherent safeguards, such as exploiting weaknesses in model fine-tuning or utilizing complex prompt engineering to evade detection. These techniques include identifying weak points in training data, using subtle prompt manipulations, or leveraging vulnerabilities in model architecture. Moreover, attacks are not limited to text—support for multimodal attacks, including image, audio, and video-based vulnerabilities, is essential to fully assess the security of LLMs.

For simplicity and practicality, we ultimately restricted ourselves to jailbreaks that could be carried out without needing an attacker LLM—keeping our solution lightweight and easier to deploy. This decision allowed us to focus on making the scanner as accessible as possible, ensuring that customers could easily integrate it into their existing workflows without requiring significant additional resources.

Integration into the Qualys Ecosystem

The next step was to materialize our POC into an enterprise solution and integrate it with the existing Qualys ecosystem. We decided against having the LLM scanner operate as a standalone tool or a pip module. Instead, we integrated it into our Web Application Scanner (WAS) infrastructure. This allowed us to leverage our existing vulnerability management (VM) capabilities and deliver a seamless experience for our customers, providing both traditional and AI-specific vulnerability scanning from a single platform. This solution makes Qualys the only company to provide a holistic and full stack solution that gives discovery, VM, and Model Scan, unlike competitors who lack such an integrated approach. This sets Qualys apart by offering a comprehensive end-to-end security solution for AI/ML environments. This integration was key to ensuring that our solution was not only effective but also scalable and easy to manage.

To bolster the AI/ML security efforts, we leveraged the existing service of our Cybersecurity Asset Management (CSAM) team, which already performs fingerprinting and tagging of software running on assets, to fingerprint and tag well-known AI/ML software and list them in our asset inventory. This step was crucial in helping our customers understand their AI/ML footprint and the associated risks. Additionally, our vulnerability researchers worked to identify CVEs related to AI/ML software stacks, adding nearly 700+ QIDs (unique Qualys ID number assigned to a vulnerability) for known vulnerabilities, combined with Software Composition Analysis (SwCA) coverage. This extensive catalog of vulnerabilities helped ensure that our scanner was equipped to detect a wide range of potential threats, providing comprehensive coverage for AI/ML environments.

Mapping to MITRE ATLAS and OWASP

An important aspect of our solution was aligning our scanner’s capabilities with established security standards. We mapped our detections to MITRE ATLAS, a knowledge base of adversary tactics and techniques against AI-enabled systems, as well as to OWASP LLM Top 10 categories. This required a meticulous review of our initial dataset, resulting in a reduction from 1,000 to around 600 questions—ensuring that each question fit into a recognized category and provided value. The mapping process was challenging, as it required us to carefully evaluate each question’s relevance and ensure that it aligned with established tactics and techniques.

By combining our AI/ML scanner with VM detections and OWASP-related findings, we aimed to deliver a comprehensive view of the AI/ML security posture for our customers, ultimately feeding into our TruRisk initiative—our internally developed method for assessing risk scores for assets and infrastructure, which is a proprietary approach unique to Qualys. Qualys Enterprise TruRisk Management empowers organizations to identify, assess, and mitigate risks effectively, including those related to AI/ML data. Feeding AI/ML scanner data into TruRisk complements the risk assessment process, providing a valuable and holistic risk perspective. With comprehensive risk visibility across environments, AI/ML insights, automated workflows, and integrated threat intelligence, it enhances decision-making and aligns security strategies with business objectives, ensuring a proactive approach to managing enterprise-wide risks.

Moving Toward Product

With all the groundwork laid, we set our sights on the first version of Qualys TotalAI that we could be proud to release to the world. This included:

• AI Fingerprinting with CSAM: Identifying AI/ML workloads and providing visibility into the AI assets within an organization.
• VM for AI Workloads: Focusing on infrastructure security to ensure that the underlying systems supporting AI models are secure.
• TotalAI UI Module: A dedicated interface for managing AI/ML security, providing users with an intuitive way to interact with the scanner and view results.
• Model Scan (External Only): Scanning endpoints hosted on platforms like Hugging Face, AWS Bedrock, Azure AI, Google Vertex, and others, ensuring broad compatibility with popular AI services.
• OWASP LLM Top 10 Coverage: Limited but focused coverage of the most critical vulnerabilities affecting LLMs, providing a solid foundation for future expansion.
• TruRisk Report: Providing a comprehensive risk score that takes into account both traditional and AI-specific vulnerabilities, helping organizations prioritize their remediation efforts.

Over the next three months, our team refined the LLM scanner, expanded its compatibility with different inferencing endpoints, and worked on UI mockups to ensure a smooth user experience. Our CSAM, Threat Research, and VM teams also played vital roles, with successful fingerprinting of AI/ML software and additional vulnerability detections. These efforts were instrumental in ensuring that our solution was both effective and user-friendly, providing a seamless experience for our customers.

Competitors and Differentiation

By the time we reached this stage, several LLM scanner solutions had emerged. These solutions typically operated as pip modules, scanning LLMs through prompts—similar to how we started. However, our approach went beyond prompt-based scanning. By integrating AI fingerprinting, vulnerability scanning, OWASP attacks, and MITRE ATLAS-mapped mitigations (MITRE ATLAS is a knowledge base of adversary tactics and techniques against AI-enabled systems), we offered a more holistic security solution with a unique risk-scoring mechanism through TruRisk (TruRisk is our proprietary method for assessing risk scores across assets and infrastructure). This differentiation was key to positioning Qualys TotalAI as a comprehensive solution that provided more than just basic vulnerability detection.

Our approach also emphasized the importance of integrating AI/ML security into existing security workflows. Rather than treating AI/ML security as a separate concern, we aimed to make it an integral part of an organization’s overall security strategy. This holistic approach allowed us to provide a more complete picture of an organization’s security posture, helping them address both traditional and AI-specific risks in a cohesive manner.

Challenges and Future Directions

Despite our progress, new challenges were emerging. As our solution matured, we recognized that the evolving AI/ML landscape presented new complexities to address. The rapid evolution of the AI/ML landscape meant we needed to extend our scanner to support multimodal attacks, including image, audio, and video-based vulnerabilities. These new types of attacks required us to rethink our approach to vulnerability detection, developing new techniques and datasets to address the unique challenges posed by multimodal models. Additionally, the rise of retrieval-augmented generation (RAG) systems, agentic workflows, and Public Cloud integration, which almost all our customers use as PaaS or SaaS, and evolving guardrail technologies were key areas of focus for future research and development.

We also explored the use of guardrails, such as NVIDIA NeMo and Guardrails AI, to prevent malicious or jailbreak prompts. These guardrails act as a protective layer by analyzing inputs and outputs for potentially harmful patterns and restricting unsafe actions. They fit into our overall security strategy by providing an additional, proactive mitigation measure against malicious activities, enhancing the robustness of our LLM scanner. These guardrails, however, often rely on pattern matching and can be circumvented—indicating that vulnerabilities are far from eradicated. This opens an opportunity for Qualys TotalAI to integrate guardrails as a mitigation layer, calibrating them based on scan findings to offer customers an adaptable security solution. By incorporating guardrails into our scanner, we aimed to provide an additional layer of defense that could help mitigate the risks posed by malicious prompts and other attacks.

Wrapping Up

Our journey from a simple proof of concept to a product offering has been both challenging and rewarding. We’ve learned a great deal about AI/ML security, prompt engineering, and integrating new technologies into our existing security ecosystem. Throughout this process, we have faced numerous challenges, from developing effective evaluation techniques to integrating our solution into the broader Qualys platform. Each challenge has provided us with valuable insights and helped us refine our approach.

We’re confident that Qualys TotalAI will play a significant role in helping organizations secure their AI/ML infrastructure as the technology continues to evolve. By integrating AI fingerprinting, vulnerability scanning, OWASP attacks, and a unique risk-scoring mechanism through TruRisk, TotalAI offers a comprehensive solution that stands apart from other products in the market.

By providing a comprehensive, integrated solution that addresses both traditional and AI-specific vulnerabilities, we believe we can help our customers navigate the complexities of AI/ML security. And as we continue our work, there will always be new advancements and vulnerabilities to tackle—keeping the journey exciting and pushing us to innovate.

Looking Ahead

Moving forward, we plan to expand our solution to support additional attack types and modalities, ensuring that TotalAI remains at the forefront of AI/ML security. We are also working on making the scanner work internally for models that are not exposed publicly, using our internal scanner. Additionally, we intend to continue refining our risk-scoring methodologies, providing our customers with the insights they need to make informed decisions about their security posture. The AI/ML landscape is evolving rapidly, and we are committed to staying ahead of the curve, and helping our customers secure their AI assets in an ever-changing environment.

Contributors

• Girish Aher, Senior Manager, Data Platform, Qualys
• Kedar Phadnis, Senior Manager, Program Management, Qualys
• Ramesh Mani, Principal Architect, Data Platform, Qualys
• Sheela Sarva, Director, Web Application Security, Qualys