Organization creates anti-virus testing best practices
A group of security companies, anti-virus testers and media companies have created standards for evaluating anti-malware products.The Anti-Malware Testing Standards Organization (AMTSO) announced Monday that it has published best practices for reliable testing, a set of nine guidelines that include ensuring that a statistically valid sample is being used and that the evaluation is taking into account all the ways products detect and prevent malware.
Andreas Marx, CEO and managing director of AV-Test.Org, which performs tests for publications such as PC World, said roughly two-thirds of current reviews exclusively rely on signatures because this is an easy and quick way to test.
But these evaluations fail to take into account other ways of stopping malicious code, such as URL scanning or heuristics, he said.
"Today you can win a review if you have enough signatures," Marx said. "However, a product with less signatures but with a lot of more-advanced protection features might be a lot better in a real-world scenario."
Dave Marcus, director of research and communications at McAfee Avert Labs, said that all too often, anti-virus companies are given poor marks for reasons such as the reviewer using outdated test samples.
"When you're looking at the results, you're not given an accurate description of what's being protected against and what's not being protected against," Marcus said. "You're not telling the consumer the whole story."
Jeff Debrosse, senior research analyst at ESET, said customers often base purchasing decisions on faulty information.
"Not all tests are created equal," he said.
The standards, though voluntary, are expected to receive widespread adoption, involved parties say.
"I'm sure the testing will change because documents are prepared and the industry has said, 'Yes, this is a good idea,'" Marx said. "I think the end result should be a more informed reader and a more informed buyer."
Anti-Malware Testing Standards Organization principles
1. Testing must not endanger the public [by creating new malware].
2. Testing must be unbiased.
3. Testing should be reasonably open and transparent.
4. The effectiveness and performance of anti-malware products must be measured in a balanced way.
5. Testers must take reasonable care to validate whether test samples or test cases have been accurately classified as malicious, innocent or invalid.
6. Testing methodology must be consistent with the testing purpose.
7. The conclusions of a test must be based on the test results.
8. Test results should be statistically valid.
9. Vendors, testers and publishers must have an active contact point for testing-related correspondence.
2. Testing must be unbiased.
3. Testing should be reasonably open and transparent.
4. The effectiveness and performance of anti-malware products must be measured in a balanced way.
5. Testers must take reasonable care to validate whether test samples or test cases have been accurately classified as malicious, innocent or invalid.
6. Testing methodology must be consistent with the testing purpose.
7. The conclusions of a test must be based on the test results.
8. Test results should be statistically valid.
9. Vendors, testers and publishers must have an active contact point for testing-related correspondence.
