Part I: Implementing Effective Cyber Security Metrics That Reduce Risk Realistically

As a CISO or business leader, some burning questions that often come to your mind are: 

• How vulnerable is our cybersecurity posture? 
• Are we better protected than we were three months or a year ago? 
• Have our investments improved the cybersecurity posture and yielded any tangible benefits? Are my vendors delivering on their commitments to protect my business? 
• If so, to what extent? If not, where should we invest our time and budget?

Although such questions are easy to raise, finding straightforward answers is equally tricky. The crux of the issue lies in identifying the right metrics to measure and a way to report them over time consistently. After all, navigating the dynamic and complex cybersecurity landscape to determine the most influential metrics is no small feat.

So, how can organizations tackle this challenge? Is it by developing a standardized, universally acceptable, and applicable approach to their security metrics? 

Let’s find out in this three-part blog series as we delve into a step-by-step guide on how to build a cyber risk metrics program from the ground up. 

We’ll also examine how to holistically leverage these data points to act as a catalyst that drives organizational change and enables your business to become more cyber-resilient. 

For any metric to be insightful, it must offer supportive data points for strategic and tactical audiences. In this blog series, we will pave the way to start your security metric journey with a framework that will enable you to: 

• Kick-start your own successful cyber risk metrics program.
• Maintain an ongoing series of “incremental metric wins.” 
• Provide actionable tidbits that take your cyber risk metric program to the next level regarding process improvement. 

Let’s understand the three primary areas critical to successfully launching a cyber risk metrics program:

1. Understanding how to start your Cyber Security Metrics journey using proper outcome definition and baselining.
2. “Bread & Butter” security metrics for articulating the risk posture for CISOs and other similar executives and tactical teams.
3. Closing out, creative ways to influence the business to reduce your organization’s attack surface using these tangible risk-centric numbers. 

Getting Started with Security Metrics

Many organizations struggle with the most fundamental questions regarding metrics: Where to start?

As a result, you tend to take a “boil-the-ocean” approach, which quickly becomes overwhelming for everyone involved.

A poor definition of outcomes can lead organizations to choose an easy way forward. And, then it simply becomes a ‘tick-the-box’ exercise causing the program to fizzle out even before it gets off the ground. Organizations that try to digest too many numbers and charts at once often need a clearer picture of their security posture. Moreover, the initiative steals precious work cycles from security teams without providing any value to them or the C-Suite.

In the beginning, it’s helpful to look ahead and pose the following questions: 

1. What outcomes do you want to illustrate with the security metrics you implemented after launching the program? 
   
2. What narrative do you need these metrics to tell, and who will consume this data? (For example, is this for your cyber defense team, C-suite, ultimately the Board of Directors, the business, GRC teams, or others)?
   
3. What do good metrics look like after you’ve completed the first step? 
   
4. What process improvements can be done in the future?
   
5. How can I enhance measuring the cyber risk in the near and long term? (This question will be helpful for you to document what you like to do now during the launch phase but aren’t able because of things out of your control – like external forces – like project timing, budget, and work politics) 

---

Pro Tip: The key here is to document your current limitations, the leading cause of these limitations, and the estimated time to complete this list; when the timing is right, you are ready to execute. 

---

It also allows you to “pre-answer” questions from your management, such as, “Why didn’t you do this to start with when this program journey began?” Since this is a fair question to ask, having these answers already in hand demonstrates your passion and commitment to ensuring your cyber risk metric program will be a success for your organization.

Getting your team to ask these questions will help you define the outcomes you expect from the program and not steer away from the right path as it matures. 

A successful approach to metrics demonstrates your cyber risk posture and means to generate actionable insights for its intended target audience. This clean and simple approach will get your metrics program off the ground and set you up for future success. 

For better understanding, let’s divide this approach into two groups based on the intended audience: Above the Line (ATL) – leadership and Below the Line (BTL) – tactical and operations. 

I used this two-group approach for metric outcome generation in all my previous work environments, where it was well-received and meaningfully produced the desired outcomes for my cyber risk metrics programs. 

Baselining

The next step is a solid understanding of your organization’s cyber risk posture. A baselining exercise will help you get there quickly. It entails creating a baseline risk posture of your perimeter and inside computing environments.

It is a fundamental step as it will be the measuring stick of your cyber risk posture that helps you explicitly to note which of your tools protect all your assets and computing environments. With these defensive concepts in mind, frame this approach in terms of what preventative, detective, and corrective controls deliver results across your enterprise. The delivery of some solutions may not be 100% under the security team’s control, but they should still be in scope for establishing the baseline.

Let’s explain this with an example: Your configuration database management (CMDB)/asset management repository or web application firewall (WAF) should be considered for inclusion. Typically, within an enterprise, these solutions are owned by asset management for CMDB, not the security operations team. WAF solution is under the purview of the network/network security team as part of a typical NGFW (Next Gen Firewall) suite.

Having at least six months of historical data to start the benchmarking phase will be helpful. Review the historical data and conduct a trend analysis to establish the baseline for all these identified solutions. This action will set your program up for success. 

---

Pro Tip: This is also an excellent time to execute a self-audit of your data retention standards to ensure that your organization meets these standards. 

---

Data retention standards traditionally state minimum and maximum retention times for specified data types. If you are struggling, consult your Data Classification standard to answer the duration, what kind of data is in scope here, and for how long – i.e., the data retention periods.

The “Bread & Butter” Security Metrics

Now that we have successfully established baselining standards, next is defining the “Bread and Butter” security metrics. This step will enable you to understand your organization’s risk posture. Let’s divide this step into milestones based on targeted audiences and desired outcomes directly influencing business resilience to help you define the metrics.

I divided this approach into milestones based on the targeted audience and desired outcomes by focusing on “Bread & Butter” cyber risk metrics and metrics that directly influence business resiliency. 

The “Bread & Butter” cyber risk metrics are called ‘Cyber Risk Hygiene’ or ‘Security Risk Posture.’ As mentioned, this category of metrics must be tailored for either operation teams (Below the Line) or an organization’s security and business leaders (Above the Line).

BTL are metrics for operations teams that are tactical/operational-focused. Their target audience includes managers and technical staff for vulnerability management, infrastructure, and IT operations teams. This type of audience requires metrics that show a granular level of details helpful for daily operational decisions that ultimately contribute to strengthening their organization’s security posture. 

ATL metrics (leadership centric) are strategically focused numbers that demonstrate the overall security health of your computing environment for both on-premises and cloud environments using summarized data from BTL metrics. 

The Security Perimeter Health Score is comprised of alert types & frequency of the Next-Generation Firewall (NGFW) rules based on these locations (if applicable to your organization):

1. Cloud environments (derived from Cloud Security Posture Management [CSPM], Cloud-Native Application Protection Platform [CNAPP]
2. Data Center on-premises
3. Regional/Remote offices
4. Remote/Mobile (users of virtual private network [VPN] services)

NGFW-native capabilities that enrich this perimeter health score metric calculation include

1. URL content filtering
2. Intrusion Detection Service/Intrusion Prevention Service (IDS/IPS)
3. Vendor data lake analysis reporting against industry peers.

An external Risk Detector (also known as External Attack Surface Management) is key to completing this perimeter enrichment process.

Metrics for security agent coverage during the past 30 days can be listed by OS and applications installed on that OS and their respective agents. These metrics reveal which vulnerability, endpoint detection and response, and anti-virus agents are installed and working correctly. Other metrics to consider for measuring the inside cyber risk posture: the effectiveness of user Security Awareness Training via simulated phishing attempts and internally focused SIEM alerts.

As a result, this overall cyber hygiene health score comprises a summarized security perimeter health score and security agent endpoint data. The target audience for these metrics includes the board members, C-suite, and other peer IT leaders.

Depending on the deviation from its established baseline, each metric gets assigned a score. (This demonstrates the importance of why baselining is highly integral to the success of your cyber risk metric program, as previously discussed.)

Finally, these scores are summarized and displayed in a single “RAG” color, indicating the Overall Security Baseline Status and an “At-A-Glance Monthly Cyber Risk Health Score”; this display can drill down to the BTL numbers, if desired, by the leadership.

The RAG Definition:
R=Red/Take Notice Immediately | A=Amber(Yellow)/Caution status | G=Green/a Good status 

The formula for calculating Security Baseline RAG Status:
OnPrem NGFW Score + Security Agent Coverage + Security Tools Alerts = Security Baseline RAG Status [Red/Amber/Green]

Concluding Thoughts

It’s always better to focus on the essential outcomes as you develop your security metrics program. Be aware of telling the right story to specific stakeholders: technical metrics for IT and security professionals and business-related metrics for executives.

In the next blog, from this three-part series, let’s take a closer look at how to elevate your cyber risk profile to drive support of your organization’s attack surface reduction strategy. We’ll review several real-world use cases that will be practical to accomplish your mission.

We’ll also check how implementing a data-driven approach and using the Qualys platform can help your team confidently fine-tune your organization’s cyber defense approach, reduce your risk profile, and strengthen your security posture.