You Can’t Manage What You Can’t Measure: Tips to Help You Build an Information Security Measurement Program

There’s a mantra in the business world that says “You can’t manage what you can’t measure” and no truer words have ever been spoken in information security. Building an information security metrics program is not glamorous, but it’s an invaluable tool to help measure and visualize KPI’s (Key Performance Indicators) to help you improve security across the organization. By displaying the evidence in an objective manner to your selected key stakeholders you will be able to get your point across regarding risks, areas of improvement and highlight the company’s achievements in protecting the organization.

As an information security practitioner, here are some of my tips to help you define the basics of an information security metrics program.

Who are the right stakeholders:

Metrics by themselves are useless. They need to be directed towards key stakeholders in the organization who will analyze and rip them apart (if necessary) in order to provide valuable input that will help to make a real difference to your security strategy.

So ideally you should identify key CxO stakeholders who are responsible for specific business units. To help identify the right stakeholders, ask yourself:

• Are they important to information security?
• Can they assist you with building your program?
• Will they influence others to bring more security into the enterprise?
• Do you need their approval to be successful?

Where to start:

This is always a tough question to ask when first starting an information security metrics program. Remember not to try and measure everything all at once, it’s a marathon not a sprint. There are many data points you can start measuring right off the bat, but try to prioritize the factors that are important to your stakeholders first. These will be adjusted over time as the stakeholders will want to see tweaks to the current metrics and you’ll most likely want to add additional items yourself. For starters I suggest you focus on the following areas:

• Compliance and regulations: This is a topic that CxO’s normally gravitate towards, so starting with compliance metrics is probably a going to get their attention. This shouldn’t be your driving factor, but it is an important part of your program.)

• Technology Metrics, such as patch updates, incidents, vulnerability trending, etc. This shows tangible improvements in a security programs progression. From my experience CIOs love to review these, plus it shows improvements in a visual way to non-techies.

• Core Project Updates (relative to the each stakeholders department).  These metrics will show process improvements for specific projects and key milestones that have been reached, or those that are coming up.

• Security Risks, across the enterprise and whether the company as a whole is making improvements in protecting against them. Note, some of this information might be very sensitive, so it’s up to the Security Manager to decide if the information can be disseminated to the target audience. 

Creating Metrics:

Now that we know which types of metrics to initially focus your program on, we need to start building them. This is easier said than done, so be careful not to bite off more than you can chew. There’s always the initial overzealous effort when starting something new, but be wise about your metric collection methods and metrics in general. You should be able to collect them in an efficient manner, without too much effort. If the process isn’t easily repeatable and is too laborious there’s an issue. This shouldn’t become a full time job.

A few areas we need to focus on are as follows:

• Collecting Data:
  • Preparing metrics by hand is horrible and prone to errors. While there are times you wont be able to avoid manual data collection, you should use automation as much as possible. There will be times when your systems can’t generate the metrics automatically and you’ll need to do it manually, so keep it simple and make sure your collection methods are accurate. The first time a metric is wrong it will bring your entire program under suspicion.

• Displaying Data
  • There are many tools on the market now that will assist with aggregating data and displaying it in a manner that will be useful for your metrics program. These tools can either assist completely, or supplement your manual collection of data. There are also the reports that can be run from your information security systems that can provide you with outputs for your reports.

• Metrics Template
  • Create a standardized template that you’ll use consistently for your reports. This can either be created by a 3^rd party tools or an Excel spreadsheet. I find that a tool is the best method and provides a more streamlined output than a spreadsheet. But not every tool has ability to collect and display all the metrics you’re attempting to portray. 

Presenting Metrics: 

• K.I.S.S (Keep it simple stupid)
  • Don’t over complicate in the presentation of your data. For one, it will take longer to process, and more importantly it might confuse those trying to review them. Your stakeholders may not be overly technical, so showing them the information in a simple manner is the best way for them to consume the information.

• Get  Your Point Across
  • While displaying your metrics to stakeholders, be sure to get your point across. Show this empirical data as just that and make sure its presented as objective as possible – we’re not trying to blame others, but to show risk in our environments. Metrics can be twisted to get your point across, don’t do this. Show the data as it is and be prepared to answer for them. Also, make sure that those responsible for other areas of the metrics are in attendance while the metrics are being reviewed. We as Information Security practitioners need to be careful not to use FUD (fear, uncertainty and doubt) to get our point across. Focus on the risk and present the data as it is. Also, give credit where credit is due.

• Metric Review Meetings
  • During this review make sure you have all key stakeholders invited, as well as those responsible for other departments who can speak to some of your metrics. (e.g. If there’s a metric that shows Linux vulnerabilities spiking in the past month, make sure you have someone from the Linux team who can explain why this might have occurred). Also, make sure to show trends in the data so that you can show improvements or deterioration in your program. And make sure there’s someone from your team taking notes. This will assist with follow ups and to-do items.

In closing, many people are worried about getting the newest technology in place to help defend against attackers, which is also important, but if you don’t measure your success with the tools you already have in place you’re missing out on a core component of your information security strategy.