Policy, Critical infrastructure

Product testing and accountability

December 1, 2011
As a director of the Anti-Malware Testing Standards Organization, who also has a close business relationship with an anti-virus company, I'm sensitive to the fact that AMTSO is frequently assumed to be a vendor cartel.

So it's good to be able to report a useful article from the other side of the Great Divide. Actually, Simon Edwards represents Dennis Technology Labs in AMTSO, and is an IT journalist with many years of experience in AV testing, so anything he says about the subject is likely to be of interest. But I found his blog article, Anti-malware testing: behind the scenes, particularly interesting because he took the trouble to answer some of the questions that people really do ask. Plus, he addresses an issue that's received comparatively little attention.

Some professional testing organizations earn a significant proportion of their income from comparative tests sponsored by security vendors in the hope that their own products will perform better than competing products. It is, I guess, probable that if the company that supplied one of the products under test is paying for the test, it will expect to come out on top, and will at least expect the test to take a form that will put its product in the best possible light (which isn't quite the same thing as a guarantee of coming first). But that isn't the same as assuming that the tester will fiddle the results to make sure the sponsor does top the list, and in fact, it's not unknown (or comfortable) for a sponsoring company to be pipped at the post by a competitor.

There will always be people who will inevitably assume foul play, though perhaps this reflects distrust in the security industry as much as (or more than) it shows distrust of the testing industry: In general, people seem likelier to accept uncritically a tester's own estimate of its competence than a vendor's... But attempting to answer straightforwardly and publicly some of the questions asked about test methodology strikes me – apart from being well in accord with the rationale behind the foundation of AMTSO – as being an excellent example of a tester acknowledging that he is accountable to his audience for the accuracy of his testing, as well as to the vendor or publication that sponsors it. Or, you might say, rather than... After all, the vendor or publisher might put up the testing fee, but ultimately it comes out of the pockets of the customer or reader.
prestitial ad