Meaningful Baselines for Human Factors: Here's How To Do It.

One of the hardest questions when it comes to change management is how do you measure the results. How do you know that your program actually worked, reduced risk, and that the outcomes you aimed for are in place. Making change happen, and demonstrating the results, can be muddy and difficult - but it does not have to be. 

What is a baseline 

The secret to proving that your transformation program is effective is to measure the state before your program starts, and then measure it again after your program is completed. The first measurement is referred to as a baseline metric. This baseline is the initial state of your program, before or at the time of your program starts. 

A baseline can be simple, with only one or a few metrics, or it can be complex, consisting of any number of different metrics. The important thing to keep in mind is that a baseline should be designed using metrics that will be available before, during and after your program has been implemented. Furthermore, your metrics must be relevant and reliable. It is all too easy to use vanity metrics, metrics that are designed to make you look good, instead of being designed to prove effectiveness. 

Knowing what to measure

It can be difficult to select the right things to measure, especially in a world where there are so many variables and opinions. I suggest you keep it simple, and that you focus on identifying specific metrics that are most likely to be influenced by the change - directly or indirectly. This sounds easy, and it can be. It can also be quite difficult, which is why many security awareness programs lack relevant metrics. 

Here are some examples of metrics that are being used to better understand your organization, and to create relevant baselines:

• Employee engagement: for example surveys, observations, completion rates, content quality scoring;
• Specially designed surveys and assessments, like workplace surveys, culture surveys, skills assessments;
• Behavioral assessments, like phishing assessments;
  
  

These kinds of data points are usually easy to get at, in that your SAT vendor will provide you with some or all of these things. The challenge with these numbers is that they only tell a part of the story, and it can be quite hard to understand which part that is, and to what extent the story they do tell is actually relevant for your baseline. To make matters worse, using the wrong kind of metric is likely to make you look good, but not make the changes you are aiming for.

Other sources of information that we recommend to use include: 

• Actual behavioral data: these can be found in many existing computer systems by examining logs and other traces that are directly or indirectly affected by human behaviors; 
• Leveraging multiple datasets to provide broader and deeper understanding of what is happening;
• Use modern tools that allow for sentiment analysis, communication networking diagramming, and risk modeling based on real-life data
  
  

The best way to create a baseline is to use only a few metrics. Then as your knowledge in the area grows, you can add more data points, allowing you to compare on a more detailed basis, as well as compare information across multiple data sets. This lets you discover which data is more relevant than others, and thus you can start to improve your metrics as you learn more. 

Why do you need a baseline? 

The baseline serves a number of purposes, some of them listed below: 

• Provide a state to compare progress against, which will help you track the effectiveness of your actions during the program itself, and thus allow you to correct course if necessary; 
• Create a (better) understanding of the current situation, which provides more evidence for the actions you plan to take. Sometimes, the evidence may also give your insights that your change is not needed - so you can refocus your efforts and budgets elsewhere; 
• Help inform actions to take during the program, which will help you design a better program with relevant actions and activities, better align your program with the needs of your employees and organization; 
• Demonstrate the effectiveness of your program, by providing a point of comparison, which will help you prove that your program was successful and beneficial to your organization.
  
  

The most important reason for a baseline is to be able to know that what you do is the right thing to do, and that the things you do create the intended results. You can think of your baseline as a tool that allows you to continuously improve your program, while demonstrating its effectiveness at the same time. 

If you want to know how to build your baseline, you can contact us here: 

If you are ready to start building your own baseline, you can try out the Praxis Navigator here: