Growing IT security threats, coupled with regulatory mandates to protect information and ensure privacy, are generating a renewed focus on security awareness training programs, which can heighten security awareness, improve application and infrastructure security, and enhance security incident handling and response. Such programs not only benefit IT security staff and application developers, but they benefit the enterprise workforce at large.
Improved employee understanding of appropriate behaviors and best practices for enhanced information security reduces security risks and helps ensure compliance with regulations such as Sarbanes-Oxley, HIPAA, the Payment Card Industry Data Security Standards (PCI DSS) and others. But merely providing security training is not enough. Organizations need to know if training programs have been successful in changing behavior.
In order to provide an effective security training program, metrics must be set in place from the start. Measurements help establish a baseline of individual and organizational competencies in enterprise security. Additionally, metrics help identify gaps in current training initiatives that should be remedied and improve the methodology and/or content of training programs. Measuring training effectiveness can also be useful in validating the competency of the training entity itself.
Documented metrics can provide proof of an enterprise's considerable commitment to understanding regulatory requirements, implementing technical solutions and instilling behaviors to meet the organization's security policies and achieving certification. Using metrics to evaluate training effectiveness can also establish a positive effect on the business.
By conducting a penetration test before and after performing application security principles training, it is possible to gauge the impact of training. A baseline penetration test may be in the form of a gap-analysis and, if required, may include recommendations for additional actions to address any issues. This best practice provides a measurement against which later training results can be compared.
Perhaps the most employed method to evaluate learning achievement is the Kirpatrick method. First published in a series of training and education journal articles in 1959, Dr. Donald L. Kirkpatrick's four-level learning model (reaction, learning, behavior and results) is the most widely used and accepted method for measuring learning effectiveness today. By employing Kirpatrick's education model and the security industry best practices, organizations can successfully assess the effectiveness of its security awareness training program, measure the results, and further improve the training.
Gauging trainee reaction
A carefully designed training questionnaire provides valuable insight for improving training courses and achieving the greatest learning effectiveness in future sessions. Reaction measurement has evolved beyond the simple question, "Did you like the training?" A truly useful measure of trainees' reactions includes questions about topics such as relevance of the content, the training environment, and whether the training seemed to be an efficient use of time. Sample questions include:
- How well did the course material cover the topic?
- What are your impressions of the instructor's methods?
- Comment on the schedule, the audio visuals, and the printed materials.
- How would you rate the material's relevance to your job?
- Were you surprised or disappointed at anything not covered in the course?
- What are two strengths/weaknesses of the course?
When trainees are asked the right questions, and asked in the right way, evaluators gather data that can be used to improve all aspects of the course. This level of evaluation requires the least effort and lowest cost to conduct. Survey sheets or an online questionnaire are routinely used in the majority of training scenarios to gauge trainees' reactions to their learning experience.
Although surveys offer valuable insight into whether the training interests the trainees or offers perceived value in the workplace, surveys alone will not deliver a complete picture of training effectiveness and behavioral application. Further levels of evaluation are needed to achieve this goal.
Learning the material
The second level of the Kirpatrick method involves learning demonstrations that measures whether or not trainees actually learned the skills or content the program was designed to teach. For many security awareness topics, a test remains the best way to evaluate learning, whether using a practical test, a written quiz, or another form of testing to demonstrate learning levels.
By conducting both pre-testing and post-course testing, organizations can better evaluate the amount of increased knowledge or skills acquired that can be attributed directly to the training. This level of evaluation is routinely conducted on easy-to-quantify skills learned during the training event, and is most often performed at the end of a training program, as either a test or demonstration of skill.
To measure more complex changes in behaviors or attitudes (e.g., those relevant to an individual's responsibilities), the potential increased expense and subjective nature of the evaluation may prove prohibitive for many enterprises. This is especially true if the training program has been poorly designed or executed. Hence, alternative methods of evaluating training effectiveness are needed.
Changing behavior
While testing trainees at the conclusion of the course can measure the value of the educational materials and methodology, an unannounced test conducted in the work environment provides improved insights into training retention and application effectiveness. Evaluating trainees after a period of time has passed since the end of the training – perhaps after two weeks, or in other cases, two months – provides a convincing measure of the training's effectiveness.
Because this level of evaluation requires more resources and can incur higher costs, companies use it less frequently in the majority of training situations. However for meaningful, cost-effective training in critical areas such as information security, this level of evaluation is often warranted.
For example, to determine the effectiveness of training in the area of strengthening passwords, trainee administrators can use their existing policy compliance tools to compare the length, syntax, and composition of passwords both prior to the training and after the training.
Achieving results
The fourth step of this methodology measures the effects on the business environment. These effects can be determined by comparing the costs and benefits of undertaking the training in different ways. For example, the costs of purchasing web-based training can be compared to developing a similar training program internally. These costs are evaluated in light of the measured results of the training effectiveness for overall training results.
The option of hiring security experts to quickly develop and administer the training also has its benefits. Leveraging the established expertise from organizations like Symantec enables more rapid realization of downstream benefits such as security risk mitigation, avoiding costs of security breach cleanup, improving customer retention and rapid regulatory compliance.
Investing in a professional, proven training provider can not only ensure that course materials are understood and retained, but that the course receives positive "reviews" in the workplace. When potential trainees hear that the training is fun, interesting, or relevant, trainees are much more likely to enter the classroom with a receptive attitude.
No matter the topic of the course offering, enterprises that invest in security training benefit from a company-wide heightened awareness and understanding of the critical importance of information security. Security training and education offers employees at all levels in the workforce the appropriate behaviors and best security practices they need to help their organizations reduce IT risks.
In the past, many organizations neglected the measurement of training effectiveness because of the time or expense involved. Many managers perceived that resources were better allocated to the development of new training programs rather than examination and evaluation of current ones. As corporate stakeholders have increasingly pressed executives for accountability in all endeavors, including security training, the requirement for measurable results grows.
Given the environment of government oversight and regulation, skyrocketing amounts of collected personal data, and the adoption of new information technology, enterprise training to improve information security is likely to remain a high priority.
- Luis Navarro is senior consultant for security awareness practice at Symantec
