How we test
Products are rated in six categories:
• Ease of use,
• Support, and
• Value for money.
Each product is then given an Overall rating, which is determined from a combination of these categories.
Ratings are marked from one to five stars, which is referred to as the star rating:
* * * * * Outstanding in all respects - an "A" on the product's report card
* * * * Exceeds basic expectations and requirements - a "B" on the product's report card
* * * Meets the expectations of the review - a "C" on the product's report card
* * Failed to achieve some basic requirements - a "D" on the product's report card
* Seriously deficient - an "F" on the product's report card.
We test against two sets of criteria. The first is a set of generalized points that are organized according to the six categories.
These are the criteria appearing in our generic (baseline) test matrix database. Under each category there is a list of generic criteria that the tester checks "yes" or "no" (in most cases), indicating the presence or absence of the criterion in the product under test.
For example, the first criterion under ease of use is "management interface intuitiveness." The tester would check "yes" or "no" to indicate whether or not the management interface displayed appropriately intuitive characteristics. Additionally, the tester has space in the database to include comments about the specific criteria.
Finally, each product group has individual tests designed specifically for the group, and each group has a unique test bed configuration designed to facilitate testing of the product types within that group.
The criteria for each major category are listed below as they appear in our generic test matrix.
Ease of use
• Management interface intuitiveness
• Task efficient
• Learning curve
• Integratable with other products
• Easy to work with
• Ease of fixing problems
• Defined set up procedure
• Intuitive installation of
• Errors encountered
• User friendly
• Installation verification
• Stable installation
• Bundled hardware and software
• Installation log file
• Intuitive user interface
• User tutorials or wizards
• Predefined reports
• Graphical reports
• Expected features?
• Missing features?
• Standout features?
• Features of little use?
• Function of advertised features?
• Integrated database
• Scripting language
(Criteria in this category are graded as "strong" or "weak")
• Typical condition performance
• Performance under attack
• Command response
• Event information
• Understandable documentation
• Effective documentation
• Online manuals
• Installation documentation
• User oriented manuals
• Supplemental documentation
• Telephone support?
• Web-based downloads?
• Online forum?
Value for money
• Purchase for your own company?
• Appropriately priced?
• Extra fees
• Updates included
The Overall rating
This page of our database automatically calculates the average rating from each of the scores awarded by the tester in the individual categories.
The reviewer also has the opportunity to change individual scores if he/she thinks it's appropriate. This causes a new calculation for reviewer rating and both the reviewer rating and the tester ratings now appear here in the database for future reference.
The tester provides a paragraph or more on key strengths and a paragraph or more on key weaknesses as observed during the testing the sleep. The reviewer then has the opportunity to write a paragraph or more with additional comments.
Back to Products Review Labs
Sign up to our newsletters
SC Magazine Articles
- Study: Open Source Software use increasing in enterprises but without vulnerability monitoring
- RSA Conference 2015: Prepare for the IoT before it's too late, Sorebo warns
- 'Aaron's Law' returns to Congress
- RSA 2015: Tension continues to grow between govt, cryptographers
- CozyDuke APT group believed to have targeted White House and State Department
- Study: Conficker declared top threat of 2014, but N. America targeted mainly by AnglerEK
- RSA 2015: Straight talk about encryption, bulk surveillance and IoT
- RSA 2015: In the healthcare industry, security must innovate with business
- RSA 2015: Unintended use of aircraft systems next challenge for counterterrorism community
- RSA 2015: Bug hunting and responsible vulnerability disclosure