Consultant’s view: Testing with open eyes

You can wander round in the dark says Gunter Ollmann, but full knowledge penetration tests are much better

I regularly meet with would-be clients to discuss their upcoming security consulting requirements, and with increasing regularity I am finding that they have never had their network infrastructure or business-critical systems assessed by an outside agency.

A starting point for many of these discussions begins with, "can you do penetration tests, and can you do it 'blind'?" This is an interesting question, and a clear indicator that the potential client has either watched too many hacking movies or browsed through one of the mainstream how-to-hack books.

My definition of a 'blind' test is one that replicates where a hacker attempts to access an organization's data with no prior knowledge of its systems. For want of a better name, I call such prospective clients HE-men (after the Hacking Exposed line of books). They are proof that a little knowledge in the wrong hands really can do a lot of damage.

Quantifying network threats

For a HE-man (or HE-woman), the impetus for a penetration test is usually driven by an external source, such as regulatory and/or insurance dictates, or by the company's board of directors in response to an external 'compelling event.' Hence, there is a large amount of internal pressure for HE-man to find a company to check the security of his organization's external-facing infrastructure and provide a report stating that everything is secure - all in the quickest possible time.

As far as the organization is concerned, the penetration test is the best way of proving the corporate infrastructure is secure. HE-man interprets this as meaning that a 'blind' penetration test will most effectively quantify the level of threat from an external hacker, and he subsequently searches for the relevant consulting expertise.

The first thing to understand is that a 'blind' penetration test is highly dependent upon the skill of the consultant and the time available to carry out the exercise. Time and skill are important factors in all penetration testing engagements (whether 'blind,' 'black-box' or 'crystal-box'), and are the primary multiplying factors to the cost of a job.

Choosing the right people

Using the same skilled consultant, a 'blind' penetration test will take considerably longer to discover the same number of security flaws. When conducting a full-knowledge (i.e. 'crystal-box') penetration test, it is a simple process to indicate within a report what information was necessary to make the security findings and what level of skill or knowledge an attacker would need to exploit any vulnerabilities.

Thus, a full-knowledge penetration test provides the same, or greater, level of security information for less time and cost. I would question anyone trying to sell a 'blind' penetration test for less than the cost of a full-knowledge penetration test.

The quality of a penetration test and the findings recorded within the final report is also dependent upon the knowledge of the consultant carrying out the work. Unless the organization's infrastructure to be tested is very simple, or the consultant knows the environment from previous experience, it is unlikely that a single person has all the skills necessary to carry out a detailed security assessment consistently across all the infrastructure components.

As an example, I have worked with consultants who could 'root' almost any Microsoft IIS web server in a matter of seconds. Place them in front of a Cisco router or AIX system and the most you would get would be a few vulnerability scanner tool results. Thus, it is extremely important that multi-disciplined consultants work together when carrying out any work.

Certainly, when I am involved in a complex penetration test or security assessment, I wouldn't think of carrying out the work without the support of other consultants who complement the skills and knowledge I bring to the project. Hence I will usually work with one to three other colleagues in order to deliver the most thorough penetration test possible.

Asking the right questions

Where possible, I would strongly recommend that organizations review the skills and experience of the consultants the supplier is proposing to carry out the security work. Don't be afraid to question the supplier or speak directly to their consultants. After all, the penetration test is only going to be as good as the skills of the consultants carrying out the work and the knowledge they have access to.

Gunter Ollmann is manager of X-Force Security Assessment Services EMEA for Internet Security Systems (

Assessing your assessors

Here are some of the questions to ask your new security assessment supplier:

  • How much experience do you have in my type of environment?
  • How many consultants would be on the job and what are their qualifications?
  • How long have you been in business? What reference sites do you have?
  • What methodology is used? What tools are used and why were they chosen?
  • What additional analysis is done on top of results from scanning tools?
  • What steps are taken to preserve confidentiality of your data?
  • How will the network be protected from interruption of service during testing?


Get daily email updates

SC Media's daily must-read of the most current and pressing daily news

By clicking the Subscribe button below, you agree to SC Media Terms and Conditions and Privacy Policy.