The following are questions we often are asked about the reviews process. If you have queries regarding reviews, please check this first to see if we have answered your question already. We will update this document regularly.
1. Before the review
1.1 Can I send you a product to review?
Yes! But first we need some information about the product. Please request and fill out the form from our Editorial Assistant Judy Traub. After the appropriate information is obtained and agreed, then we’ll move on from there – deciding whether or not the product actually is appropriate for the Group Test in question [see 1.3 for further clarification]. But, please note, we will not review any product without a form to go with it.
1.2 Will you review a beta version of our product?
Yes and no. We do “unrated” reviews of betas in our First Looks column to give readers an impression of what to expect from the product. Beta products are not included in Group Tests.
1.3 I’ve filled out the form – what’s next?
You will receive an email from our editorial assistant when the submission is logged into our system, and further updates when we are ready to conduct the review. Do not send us the product until we request it.
Please note that we require pricing information for every product, as value for money is one of our ratings. Without a price, the form is likely to be rejected.
Also, the questions on the form that relate to product functionality must be answered to ensure that your product actually fits into the group for which you are submitting it. Of course, this will have a bearing on whether or not we test for the Group Test in question.
1.4 We put “Price on application” on the form. Is that OK?
No. Without specific pricing information we are not able to conduct the review fully and fairly to you. Value for money is one of the review ratings, and without a price we would score this as “N/A,” and this can lower the overall rating by up to one star. This reduced score probably would result in the product losing a SC Magazine Best Buy or SC Magazine Recommended recognition. Therefore, it’s in your best interest to give us pricing information. If your pricing scheme is complicated, talk to us. We’ll help you convey your pricing in the manner most consistent with the rest of the review.
1.5 When do you need products for Group Tests?
Deadlines for submission are around three months before publication. However, we will tell you when and where to ship, so wait until you get an email from us. We stage products in and out of the lab so coordinated shipping is necessary to ensure that we get all of the products for a particular review and, when they need to be returned, that we get them back to you.
1.6 Why so far in advance?
Group Tests require a great deal of administration and coordination. Products must be evaluated to make sure they fit the test framework, the lab needs to construct a test environment and ensure that products can be tested, etc. The long lead time reflects the diligence and thorough nature of SC group reviews.
1.7 I’ve missed the deadline! Can I still send in a product?
Maybe. Late submissions may be included, at the discretion of the VP of editorial and/or the technology editor.
1.8 When do you need stand-alone products?
Stand-alone reviews, historically one-page reviews conducted as one-offs and published in the magazine monthly, have now been replaced by a new First Looks column. For this new section, products reviewed represent new, innovative, unusual or late beta products and are reviewed at the invitation of the technology editor.
If you have a product that you think might fit into First Looks, contact the technology editor for a discussion of your product.
1.9 Can we discuss the way you’re testing the product?
Yes, to a certain degree. Vendors have an obvious vested interest in knowing the testing environment beforehand to “tune” products. The review, when published, will describe how the product was reviewed. However, there are some things we’re happy to discuss. In fact, there are some general review points elsewhere in this document. Generally, though, we do not go into detail about specific test methodologies or test beds unless it is necessary to conduct a fair review.
1.10 What other products are in the group test?
We do not disclose other participants prior to publication under any circumstance.
1.11 If I buy an ad, will you review our product?
Buying an ad (or not) has no influence at all on:
· whether SC reviews a product
· when a review is conducted
· the methodology used to review a product
· the contents of the review
· the rating assigned to the product
· whether a product is included in a Group Test
· anything else whatsoever to do with editorial content.
1.12 Will you sign an evaluation agreement?
Probably not. We are asked frequently to sign evaluation forms or NDAs, and we nearly always refuse. This is because the agreements are usually intended for potential customers not magazines, and so include specific problems including:
· Confidentiality. If the agreement requires us not to publish results, we can’t very well publish a review at all.
· Agreement on methodology. If the agreement requires us to discuss or agree testing methodology in a way that is incompatible with our policies (as discussed in this FAQ), we can not sign it.
· Disclosure of results. If the agreement requires us to divulge testing information which we consider to be confidential, we can not sign it.
Bottom line: we are very unlikely to sign your evaluation agreement. Given that we are reviewing your product for publication in a magazine with over 100,000 reader circulation, NDAs and confidentiality agreements really make no sense. We won’t ask you about anything that is proprietary and we really don’t want to know the details of your “secret sauce.” In that context the review is based upon publicly available information, product documentation and the facts that present themselves during the course of testing.
2 Sending products to SC Magazine
2.1 Can you test/download a trial version of our software?
Not if there are any limitations in the download version. We want to review the product as used by customers, and trial versions are frequently limited in functionality, lack the full documentation, and have a tendency to expire halfway through testing. Full versions only, please.
2.2 But can you download the product at all?
Only if your customers do. Many of today’s products are provided through a web portal and activated with a license key. The important thing here is that we are getting the product exactly as your customer would. It must have all of the proper documentation, installation files, etc. If, on the other hand, we are getting a “special” version, we may have to discuss this with you to ensure that the copy we are testing is identical to what you provide to paying customers.
2.3 Where and how should I send the product?
To the address in the email requesting delivery. If you haven’t received that email, please don’t ship it. Mark the product with the name of the product, the vendor and the test for which it is intended: we receive many anonymous brown boxes, often several from the same vendor for multiple tests.
2.4 What do I need to send?
The short answer to this is “everything you would send a purchasing customer.” If the product needs a license key to activate it, for example, this must be sent with the product or with the submission form. If you use a separate email, please be sure to tell us which product (by serial number and model number) the key work with. We also need high resolution (300dpi, about 3” in width) images of the product. If a high resolution image is not available, a screen-grab may be acceptable, but a corporate logo will not be used.
2.5 Who pays for shipping?
All shipping costs, including customs charges and any other costs, must be paid by the vendor both in sending the product to us, and returning it. Similarly, all arrangements for shipping must be made by the vendor. International shipping may require special arrangements: please ensure you know what will be involved beforehand, since unexpected delays can result in products arriving too late for inclusion in a group test.
2.6 When do I get the product back?
As soon as the testing is finished, our operations director will coordinate return delivery with you. If you do not collect the product 30 days after we notify you that the product is ready for collection, we may dispose of it. When sending a courier to collect a product, please ensure that the courier knows what it is they are picking up. We will not release a product unless the courier has this information – there is just too much risk of handing over the wrong product.
3 The reviews process
3.1 Can we send an engineer to install the product?
Yes, but only if you would routinely send one to a purchasing customer. If the reviewer requires technical assistance installing the product, or if the product absolutely requires a representative to be present, this should be discussed with the reviews editor before sending the product.
3.2 How exactly will you be testing our product?
Some details are available and, in fact, are included at the front of this document. Most, however, are confidential. A typical exception to this is for products requiring pre-configuration, with details specific to the test environment.
3.3 Can we set up a conversation between the reviewer and our technical team before or during the review?
Yes. However, please work through our editorial assistant in advance to set up the call, WebEx or whatever you wish to discuss. We do not allow our test engineers to conduct discussions with marketing, sale or public relations personnel. The engineers simply are on deadlines that are too tight to take time out to hear a sales pitch. However, they usually will make time to ensure that they have the technical insights that will help them do a better review more efficiently.
3.4 What about after the review? Can we set up an interview then?
Sometimes, depending upon the topic and our engineers’ schedules. Often vendors want to get a “post mortem” of the testing so that they can apply what they learned to improve the product. As unbiased “honest brokers” we cannot offer consulting advice to vendors. However, time permitting, we can give some insights into why the product may have failed particular tests and received the scores t it did.
3.5 I sent in a product. When are you going to review it?
We try very hard to stage products in and out of the lab in as efficient a manner as we can. The short answer to this, then, is, “as soon as possible.” We want to get the product in, test it and get it out. You can help us by working with our operations director to keep that process smooth. The smoother the process is, the faster we can complete the tests and get your product back to you.
3.6 I sent you a product, but since you received it we’ve released a new version! Can I send it instead?
Certainly, though as always, we can’t guarantee to review it. If the product is in for a group test, we need to know well in advance if a new version is due. The reviews editor may be able to adjust testing schedules to accommodate new releases, but this cannot be guaranteed. Talk to our operations director to ensure that we can fit the switch into the test schedule.
3.7 Who is actually doing the review?
Some reviews are conducted in-house by SC Magazine staff, others by contracted freelance reviewers who are information security practitioners or specialists in particular fields. We rarely reveal the details of a specific reviewer while a product is being tested, to avoid vendors contacting them directly and attempting to influence the test.
3.8 I’ve changed my mind! Can I withdraw the product from testing?
It’s your product – you can demand it back any time you like. However, we may have to indicate in the magazine that the product was withdrawn, particularly in a group test. The exception is if the review is finished and the issue it is appearing in has “gone to bed.” In that case there usually is nothing we can do but publish. Unfortunately, they only “Stop the Presses!!!!” in the movies.
4.1 I’ll send you a product, but I want it reviewed in the magazine, not just online.
Placement of the reviews is entirely at the discretion of the editorial team. Some reviews are published in print and online, some just online. “In print” may mean in both print editions of the magazine (U.S. and U.K.), or just one. And, it’s not “just” online! The SC website receives many hundreds of thousands of hits a month.
4.2 Well, I’ll only send you a product if you review it for the January (etc.) issue!
We can promise that your product will be in the test plan for the product group for which it was submitted if we accepted the product for that group. What that means is that, procedurally, we identify your product as a likely fit for a particular group test. We send you an invitation with a short questionnaire that helps us ensure that it is planned for the appropriate group review. When you respond to the invitation with the applicable forms filled out we will let you know if your product has been accepted for the group test. The only likely reason it would not be accepted is that it does not fit in the group. In that case we will recommend another group that is a better fit and the process repeats. Once you re accepted, barring anything unusual, your product will be in the group review.
4.3 Can we see the review before it is published?
No. We no longer offer the review copy to the vendor in advance of the review. The reasons are that, first, we work on compressed schedules and hard deadlines and it usually is not possible to reverse something in the review before it goes to press. The second reason is that we have a rigorous quality process and we usually do not make errors of fact. When it does happen, on rare occasions, we have mechanisms available to make it right for your product after the fact.
4.4 Will you change the review if I don’t like it?
Not simply because you don’t like it. We will make it right if we’ve made a mistake.
4.5 But what if you’ve made a mistake?
If we’ve made a factual error in the review, we will amend it as necessary according to our standard procedures. However, we have a structured test process that includes fact checking of specific issues such as price, support offerings, etc. When there is any doubt on our part as to what our test results mean, we will discuss with you to ensure that we get it right the first time. If an error does creep in, will work with you to make it right.
4.6 Can I withdraw the review if I don’t like the draft?
This will depend on the circumstances: if it is brought to our attention that we have reviewed a product no longer on the market, for example, we may pull the review. But once written the review is ours to publish. See also Q 3.8 (“I’ve changed my mind!”). However, remember that we no longer provide drafts of reviews, so these types of things must be caught during our post-test fact checking process.
4.7 I sent through some changes, can I see a second draft?
Again, we do not provide drafts so we depend upon our discussions with you during the fact checking to ensure accuracy.
4.8 But it’s really important that you make all the changes I sent!
It is to us too. Our current process includes you directly in the accuracy process so we do not expect that there will be changes that need to be made.
4.9 You didn’t make all my changes! Why not?
Changes, in the rare cases that there are any under the current fact checking process, depend upon several things. Our errors of accuracy always are open for correction. Errors of interpretation may or may not be. Our test engineers interpret test results as trained professionals, but occasionally results are ambiguous. You usually can expect a call from the engineer in that case. Changes that relate to opinion usually will not be made.
4.10 You didn’t mention XYZ cool feature!
We test to a particular structure and we are limited to 350 words per product. We have a structured formula for how we write reviews and there simply is no room to cover every bell and whistle. We try to hit the high points and, from a technical (NOT marketing) perspective we are happy to hear about your differentiators.
Bluntly, it is not the purpose of the review to sell your product for you. That said, our readers want to know what makes your product our Best Buy and we want to tell them. The credibility of that story depends upon us sticking to what we do (testing and reviewing) and you sticking to what you do (building and marketing your product).
4.11 I got a great review, thanks! Can I put it on my web site? In a brochure? Print it as a flyer?
Please contact the team at PARS for more information on licensing your review.