How well does certification demonstrate ability? As an information security instructor, I am often asked this question, particularly when it is raised in the media.
How well does certification demonstrate ability? As an information security instructor, I am often asked this question, particularly when it is raised in the media. A recent magazine article argued that current certifications do not effectively match knowledge gained through experience, nor do they equal the depth of education gained through a graduate program. I agree.
However, I think the comparison is inappropriate. Certifications are not designed as a substitute for education, nor as a surrogate for experience. They are intended to provide a measure of the expertise gained through education and experience.
The CISSP examination, singled out in the article, is meant to provide validation of a baseline level of knowledge with a predetermined level of assurance. This statement has two implications.
First, the test is designed to measure whether or not an individual has achieved a minimum standard, not to rank degrees of knowledge between qualified professionals. This is not unprecedented. In fact, the Stanford-Binet IQ test was originally designed to ensure that military applicants met a minimum intellectual standard qualifying them as fit for military service. It was not designed to provide a quantitative measurement of human intelligence or intellectual potential, as is often thought. In fact, successful CISSP candidates are not told their passing score for precisely that reason – passing the examination is intended to verify a minimum level of expertise, not to establish a ‘pecking order’ among professionals.
“Predetermined level of assurance” carries a second implication. It quantifies the degree of uncertainty, or chance, associated with measurement. If successfully passing the exam indicates that a subject possesses “adequate knowledge” of the Common Body of Knowledge (CBK), (ISC)²’s living compendium of industry best practices, with a 95 percent degree of certainty, this recognizes that a certain percentage of people may pass the exam without that requisite knowledge – in this case 5 percent or less, depending upon the candidate’s actual knowledge level. (It should be noted that these numbers have been selected as examples, and are not intended to be representative of the assurance levels selected by the authors of the CISSP certification examination.)
This measured degree of uncertainty is common to any statistical sampling, whether of an individual’s knowledge of a specified subject like information security, or a report on the TV viewing habits of a population, based on sampling a subset of that population. This is also common to all methods of standardized testing, including SAT, ACT, GRE, LSAT, MCAT and others. While some may question the validity of these tests for different populations, they have provided effective indicators of future performance within their areas of interest. In fact, our reliance on such tests is evident in the use of these and similar tests as admissions criteria for institutes of higher learning.
Most people agree that past performance is the best indicator of future performance. In spite of this, we don’t always have the luxury of sufficient time and resources to conduct an in-depth background examination of all business associates we encounter. Frequently, we rely on personal references, e.g., “my friend speaks highly of you.” Where we don’t have personal knowledge or references, we judge people by other criteria. These criteria include employers, the letters after their name, the way they dress and the way they present themselves.
Individually, we may weigh the criteria differently. In the business world, appropriate dress may be weighted more heavily than in an academic environment. And expected business attire may vary when traveling from Los Angeles to London.
We each have standards we apply, consciously and unconsciously, based on the culture in which we work and on our personal experience. However, certain criteria have been accepted by the information security industry, including prominent certifications in the field. For instance, when meeting a CISSP, or CISA, or CCIE, we have expectations derived from our knowledge of the certification and our past experience with others holding the certification.
Sometimes our expectations are met with disappointment. It is commonly accepted that some doctors are less knowledgeable about their field than others. Some lawyers are ineffectual in court, and some CISSPs are less proficient or less comprehensive in their knowledge than others. We recognize that sometimes performance on a standardized test, or accreditation, or a personal reference, does not guarantee competence in the field any more than performance on an SAT or ACT guarantees credible academic performance in school.
But this does not mean that the certification program is not valuable, nor that the examination is not effective, no more than the existence of less competent doctors or lawyers reflects on the validity of the appropriate professional certification and accreditation procedures. Instead, it means that the certification is performing within expected parameters. To provide additional assurance would require more in-depth sampling, which means additional testing. Rather than taking a six-hour test and asking over 200 questions (as in the CISSP examination), it would be necessary to answer 500 questions over a twelve-hour period, or to take a series of exams over a longer period of time, perhaps over a two or four-year period.
But that would be a master’s or graduate degree program. And not only do many professionals not have time to attend a four-year program to validate their hard-earned expertise, but frankly, businesses that require information security expertise don’t have the time to wait.
Clearly, there is a balance between the depth and breadth of an examination and the level of assurance required. Businesses need to determine what level of assurance they need and how much it is worth. As a society, we need to decide how much security is enough to meet our requirements. And isn’t that really what information security is all about?
Lee Imrey, CISSP, is a program development manager and lead instructor with (ISC)² (www.isc2.org). He is a member of the ASIS IT Security Council, and current chair of the Information Systems Security Association’s professional ethics committee.