Product Review Policies (Page 2)

3. Should we send review articles to the subjects of those articles in advance of publishing?

This is a sticky question with valid points on both sides of the debate. For reference, most (but not all) print publications do send reviews to subjects ahead of publication.

In favor: Journalists are people, and people make mistakes. Making a mistake in a product review can be embarrassing and can actually cost someone money–the money could belong to product makers or readers. Here I don’t mean an honestly bad review of a product costing the manufacturer; in those cases, journalists fairly evaluate the product and accurately point out its flaws–flaws that are the manufacturer’s responsibility. What I mean is, if a reviewer says a product does something it really doesn’t, readers might choose to buy the product specifically for that reason. Such readers will be disappointed when they get the product home, and, especially with software, there may not be a return/refund path available. Equally damaging to the product maker is if a reviewer specifically says a product doesn’t do something that it, in fact, does. If the error relates to an important feature, readers who might have benefited by purchasing the product may pass it by. Thus, again, both readers and the product manufacturer lose out through no fault of their own. This has happened (not at Designorati, thankfully).

Another point in favor of allowing review subjects to peruse review editorial in advance of press is misdirected technical errors–especially when the reviewed product is a plug-in or add-on for another product. Numerous are the occasion in which Photoshop, for instance, was the culprit behind malfunctions in a third-party plug-in. Often times a plug-in will fail or crash, making it appear to a reviewer that the product is defective. In my experience, about half of those times are not the fault of the plug-in itself–the real problem is later revealed to be within the host application, operating system, hardware, or some other culprit outside the plug-in’s direct control. Deep troubleshooting–or even calling the maker of the host application, operating system, or hardware–is rare for product reviewers; it just doesn’t happen. Even when a call is placed, host application, operating system, and hardware makers often have no idea about the problem; they can only stay aware of so many issues related to integration of third-party products and their own. In most of these cases, the product under review is dinged for the crash or failure, and readers get the impression that the product is buggy.

However, plug-in and add-on makers usually know when and how host applications et al malfunction relative to their own products. If a review noting a bug is passed through the product maker, those issues that are not really the fault of the product can be explained as such. While Photoshop being at fault for Plug-In X crashing still may not make everyone rush out to buy Plug-In X, including in the review the steps to alleviate the problem and make Plug-In X run is a better service to readers than implying that the product is broken and isn’t worth their time.

Getting a price wrong is another compelling reason in favor. Omitting the “1″ in a price of “$199″ will make readers turned intended purchasers very unhappy, and they will take it out on the product maker.

Running a pre-publication review past a representative of the company who makes the product being reviewed can ensure that human errors and misdirected technical issues are addressed in the article accurately–and without accuracy, there’s no point to a review.

Against: Equally strong are the arguments against letting subjects see reviews ahead of printing. Product reviews are a delicate subject. Manufacturers depend on media to get the word out about their products. Naturally, every manufacturer wants his product to be reviewed favorably and score 5-stars. Some manufacturers want that so badly that they will exert pressure on journalists to be a little more favorable than objective. Giving a manufacturer the opportunity to see a review including marks against their product before the public sees it can, to some manufacturers, be perceived as an opportunity to make the review more favorable through coercion. It’s at this time that gifts and promises tend to be given to reviewers, requests for media kits (advertising rates) are made, and accusations of journalists “having it in for” manufacturers tend to develop. Pressure is often exerted to garner a more favorable rewrite of the review. Pressures can be subtle–”we’ll fix that in a dot release next week, so can we just omit that from your review?” or “you didn’t read the documentation, here, this is how you do it right”–or overt–”what can we do for you to get this and this taken out?” or “take out some of the unflattering items in this review, or you’ll never get another review product from us again.”

And, often, such pressures are not deliberate. The job of manufacturers (and the public relations agencies the larger ones use as liaisons with journalists) is to help sell their products. Many of them don’t know the ins and outs of the news business, and often they nudge a little without seeing the harm in it. Sometimes, though, even the most wary of journalists can be subtly influenced by careful manipulation.

Another point on the against side is the forewarned is forearmed theory. Paid-for editorial is not as uncommon as I would like to think. Public relations and copywriting people can write, and many of them begin their careers as article writers. When a manufacturer gets wind of an upcoming unflattering review, they sometimes arrange for publication of a more flattering review, written by an agent of the manufacturer or a journalist with a certain measure of flexibility in his ethics (and a corresponding inflexibility in his pending mortgage payment).

Running a review past a manufacturer in advance of publication provides opportunity for both fair correction of editorial errors or inaccuracies, as well as unfair influence over, or rebuttal to, an objective and accurate review. This is a sticky question to answer. Care to help us? What is your opinion?

These are the questions Designorati is discussing internally right now. We would be very interested in feedback from you, the reader, and even from the manufacturers whose products we do, have, or might review. What is your opinion?