6 Security Metrics That Matter – And 4 That Don’t

One of the most challenging executive tasks for CISOs is quantifying the success and the value of the cybersecurity function.

Indeed, security leaders and their organizations have used a myriad of metrics over the years. Yet, many executives and board members have complained that those measures failed to provide them with adequate insight or understanding of how well the security department is performing, how it’s improving, and where it’s falling short.

“Too much technical jargon is being presented to the chief executive and the board. CISOs are still telling the board about critical vulnerabilities and the number of patches, but the board doesn’t understand that because there’s not any proper context provided,” says Jarrett Kolthoff, president and CEO of security firm SpearTip.

He adds: “Those numbers might be great for the CISO, but the CISO needs to work [on developing metrics] that offer context so the board understands risk and how much investment in security is needed.”

Cybersecurity experts, including Kolthoff, said there’s no one metric that can work for all CISOs to demonstrate how well their security efforts are working and whether they’re improving over time. But there are some metrics, or the right combination of measures and narrative, that are more useful than others.

Security metrics that matter to the business

Curtis Simpson, CISO of the tech firm Armis and former CISO of Sysco Foods, believes metrics are more important than ever, considering the increasingly high stakes of getting security right and the growing board oversight in this space.

Like others, though, Simpson says it’s about having the right metrics. “My favorite metrics are the ones the business actually cares about,” he says. As such, he seeks out measurements that narrate how security helps the business achieve its objectives.

As an example, he points to the metrics he used at Sysco, which had a stated goal of serving its global customers on a 24-hour basis. “I had to tell a story that explained how high risk would challenge the outcome of that objective,” he explains.

Instead of reporting on the number of attacks that the company was experiencing, he used that to measure the impact those attacks had on areas such as productivity and operations and showed show how improvements could be made, at what cost, how those would reduce risk and ultimately improve the metrics on the business impact – all to achieve that 24/7 customer support.

“That resonates every time, because you’re talking about the things that business wants to achieve,” Simpson says.

Simpson acknowledges that particular metric might not work for other CISOs. He advises them to find the metrics that can help them measure security-related business impact, risk to key objectives and mitigation success over time.

Experts agree, saying what’s important isn’t just about the numbers used but how those measures highlight that business story and illustrate what CISOs are doing to solve problems and further business goals.

Richard Stiennon, chief research analyst at IT-Harvest and author of Security Yearbook 2020, says he worked with a company in the defense industry that tracked threats and categorized them from low level to weaponized and reported on that.

In effect, he says, this company turned what is often a meaningless number (number of threats) and gave it context that other executives and board members understand and can use to make meaningful decisions around investing in security improvements.

“The lesson here is [to] look beyond the numbers to terms that matter to everyone,” Stiennon adds.

Others offer similar advice.

“Aligning your metrics to key business functions within the organization is what’s really important to the board,” says Shawn P. Murray, president and CEO of Murray Security Services and chief operating officer of the Information Systems Security Association (ISSA). “The whole idea is for the CISO to work with the business units to understand what critical processes need to be maintained in order for the business to succeed. We do that by measuring the right things.”

He advises CISOs to establish key risk indicators, based on information classification aligned to assets and objectives.

So, if an organization’s security goal is to minimize disruptions, that is an objective that can be measured and tracked. Or if an organization wants to see improved security alignment to technology deployments, the CISO could create and track measurements that show how, when, and where the security team engages with tech-related procurements and how those improve over time.

User satisfaction is another metric to consider, according Bil Harmer, a 30-year IT and cybersecurity leader who is now CISO at SecureAuth. “The thing about security always has and always will be about a balance between usability and security,” he says, noting that usability issues often precipitate workarounds that negate the intended security benefits.

Harmer and others, though, say whether it’s usability or other metrics, it’s important for the CISO to find areas that actually yield quantifiable information, that they can actually obtain data to generate those measures and can do so consistently, and that it measures security’s effectiveness as it relates to business objectives.

“All of the functions under the CISO’s portfolio should align with the needs of the business and the CISO should understand the alignment,” Murray adds. “Once the alignment is understood, the CISO can then establish good metrics that measure the performance of his or her objectives to ensure the overall organizational strategies are properly aligned and are providing the expected level of success for the business.”

6 conventional metrics that remain valuable

Although metrics that evaluate how security is doing vis-à-vis business goals are rising in use, veteran CISOs and security management advisors say they still see value in many metrics historically used by the security team.

However, they also say that CISOs should consider putting additional context around these metrics as well. The board doesn’t care about things like how many phishing emails you get, says Derrick A. Butts, the chief information and cybersecurity officer at Truth Initiative, the anti-tobacco nonprofit organization. “What’s more important is measuring the effectiveness of our systems to protect against those and to protect against their impact on the business.”

Here are some of the measures that Butts and other security leaders use that provide that much-needed context.

Results of simulated phishing attacks. Butts uses simulated phishing attacks to help him evaluate how well awareness training is working and set targets for improvement.

Mean time to recover. Harmer measures the percentage of users impacted by an incident, how quickly the security team resolved the issue, and whether that time meets, exceeds or falls short of targeted times based on the organization’s established appetite for risk.

Mean time to detect. Stiennon says he recommends using metrics like mean time to detect – the measure of how long it took from the time of a successful attack to the time of detection – because that, too, indicates how well a security program works and can be tracked to show improvement. And, he says, such metrics help the CISO discuss with the C-suite and board what investments are needed to bring about improvements. Furthermore, such metrics encourage continuous improvement: Get the mean time to detect down to minutes, and the CISO can aim to reduce it to seconds.

Penetration testing. Like simulated phishing attacks, metrics around penetration testing indicate how well an organization can resist such events and can track improvements over time. For Harmer, this is information that he as a CISO as well as other executives and board members understand and value.

Vulnerability management. Murray suggests CISOs develop a metric that they can use to report on the effectiveness of their vulnerability management program. He says this shouldn’t report the number of patches done, but rather measure the security department’s ability to manage vulnerabilities to the greatest effect based on the organization’s security posture. After all, he says, it’s not about getting 100 low-risk patches implemented but rather ensuring that the one that poses the greatest risk is done as fast as possible.

“If it’s not relevant or it’s not critical, then as a CISO I’m not going to report it. I’m only going to report the ones the board needs to know about because it impacts the business. That’s what I should be prepared to measure,” Murray adds.

Enterprise security audits. Butts uses a scorecard developed for his organization from National Institute of Standards and Technology (NIST), Information Technology Infrastructure Library (ITIL) and Center for Internet Security (CIS) frameworks. “It’s a good snapshot to show how things are working,” he says.

4 metrics to abandon

As a new, improved list of metrics for measuring the security function’s effectiveness emerges, experts suggest the following metrics should get minimal use – or be abandoned all together.

Number of attacks.  “Nobody cares if you show you have 100,000 attacks in a month and stop them. That’s the one that makes people say, ‘If you’re at 100% why should I give you another $1 million?’” Simpson says. Besides, it’s not about stopping 100,000 low-level attacks but thwarting the one crippling attack that can put the company out of business.

Patches completed. Vulnerabilities identified. Viruses blocked. While these measures might make sense for CISOs as an internal measurement of work done or be required to confirm an organization is compliant with certain regulations, they have little to no value in and of themselves. “Plus,” says Stiennon, “they might lull you into a false sense of security.”