Readers now see star ratings, testing processes and Best Buy and Recommended designations more clearly defined. In addition, a new recognition has been added to these well-established honors, while previously known stand-alone reviews have been replaced with First Looks written by our newly appointed Technology Editor Peter Stephenson.
First, we've continued the tradition of group product reviews but with a few twists. We've expanded the number of categories and refined the granularity of products we test. That means if the group is IPS, all you will see are products that do nothing but IPS. Each group has a set of minimum requirements that products must meet in order to be included in the tests.
Consumers want to know exactly what to expect when they buy a product. In that vein, we try to duplicate, as closely as we can, the buying experience for a product. If vendors insist upon installing their product for their customers, for example, we invite them to install it in our labs exactly as they would do at the customer site. We grade them in this case on their level of support.
Group reviews aren't "shootouts" or "bake-offs." Each product is tested against a set of SC Lab standards and graded on its own merits. The standards consist of about 50 general points in the six evaluation categories (the "Star Ratings") developed by the Lab in cooperation with the Center for Regional and National Security at Eastern Michigan University, and sets of product-specific test requirements that are drawn from the Common Criteria (ISO 15408).
And because we have only 350 words to review a product in six general categories, we will be sticking to Common Criteria Evaluation Assurance Level (EAL) 1 (functionally tested) for our guidance. Though not an exhaustive Common Criteria evaluation, the approach adds rigor and consistency.
All group reviews pass through the labs, even if we take advantage of outside experts in a particular product area. The format will be consistent so you'll always know what to expect from a group review. The features matrix is evolving to contain more meaningful and useful information as well.
Additionally, our test engineers spend ample time talking to vendor engineers, understanding the products and ensuring a fair, balanced and complete review. In short, you can expect transparency in the reviews process.
While the Best Buy and Recommended recognitions remain, another honor has been added, however, it is not be given in every Group Test as is traditionally done with the others. The Approved for SC Labs designation is awarded to those tools that our reviews team feels are extraordinary stand-outs that will fit into the SC Lab environment, and used in future testing of other products when appropriate. Use by SC Magazine lasts for a period of one year and each time such a product is used in reviewing other products. Finally, it will be noted in the "How we tested" portion of Group Tests.
Each month, a new First Look column written by Stephenson is included. This individual review welcomes late beta products, newer products to the market and tools comprising niches of the market only just gaining footing. For these, Stephenson, again, adheres to and explains testing procedures that reveal his final call on whether the tool will succeed...or not.
As mentioned before, other staff members have been brought on to help Stephenson test and manage Group Tests and First Looks. These pros include Mike Stephenson, lab manager, who maintains the test facility, as well as makes sure the integrity of SC's improved test methodologies are upheld; John Aitken, director of lab operations, handles product shipments, scheduling, etc.; and Judy Traub, editorial assistant, who focuses on managing relationships and lines of communications with vendors and services providers whose offerings will be tested over the course of the 2007.
Although, previously, companies would submit products for reviews through the web site, Stephenson and his team contact companies directly to see if they are interested in participating. Of course, organizations desiring to be considered for inclusion in Group Tests should contact any of our team to get moving - look no further to the publication's masthead or website to gain the correct contacts and their information.
SC Magazine's product tests are well on their way to undergoing a renaissance. We hope you enjoy the revival as much as we already are.