The subject of certain policies and guidelines relevant to product reviews has been a hot topic around the Designorati conference table over the past week. We would like your input on some of it.

On the discussion table are a few questions specific to product and book reviews. In (reversed) order of complexity, these questions are:

1. Do readers really care about stars or other ratings systems?
2. What elements are most useful to readers in a review?
3. Should we send review articles to the subjects of those articles in advance of publishing?

1. Do readers really care about stars or other ratings systems?

Well, do you? We’re all familiar with star rating systems for restaurants and hotels—a 5-star restaurant is the best, while most people avoid 1-star establishments like the plague. Is a 5-star rating system truly that interesting and valuable to you? If you see that Designorati has reviewed Gizmo 3.0 at 1-stars, are you going to avoid buying Gizmo 3.0? Would you rush out and buy the product if we gave it 5-stars?

My opinion—please comment if you disagree—is that ratings systems are an initial guideline. After a time of reading one writer’s articles, you may find that you have a tendency to agree with her on most topics. Then, if that writer gives a product a low rating, you may find yourself inclined to make a judgement on the product yourself, even without reading the full review. Conversely, a 3.5-4.5 star rating from the same writer may prompt you to become interested in the product under review, but also to read the article to find out why it didn’t score higher. In my experience, this is pretty common: The more one reads the work of a given writer, the more one forms an opinion about trusting (or distrusting) that writer’s evaluations of software, hardware, books, and so on.

On the other hand, some people don’t care a bit for reviews. I, for example, ignore movie ratings entirely. Why? Because most professional movie critics are morons who pander to Hollywood politics and peddle star-ratings like maps to celebrities’ homes. (”Independence Day” earned 5-stars from some of the same people that condemned “A Beautiful Mind.” ‘Nuff said.)

So, what do you think? Are 5-star scale ratings useful to you in product reviews? Should we keep or dump them?

2. What elements are most useful to readers in a review?

Here’s where we could really use your feedback. What sections or types of information do you typically want in a product review?

Of course, our software and hardware reviews include real-world testing of the product, and of course we include screen shots or photographs as appropriate. I’m talking about the other bits of information. Which of the following do you find useful in a review, and which tend to clutter it up? And, where would you like to see them—at the very top of the article, at the bottom, or just somewhere in the article?

• System requirements
• Buying advice (the reviewer’s opinion about who should and should not buy the product in question)
• Link(s) to locations where you can buy the product under review
• Platforms compatible with the product (Personally, I like to see these at the top somehow; if I’m looking for a Mac-compatible gizmo, I won’t bother reading a review of a Windows-only gizmo.)
• Price: Suggested retail price from the OEM, or the best price we can find from any source?
• A succinct summary of pros and cons

What else would you like to see that isn’t listed above?

Next: Should we send review articles to the subjects of those articles in advance of publishing?

3. Should we send review articles to the subjects of those articles in advance of publishing?

This is a sticky question with valid points on both sides of the debate. For reference, most (but not all) print publications do send reviews to subjects ahead of publication.

In favor: Journalists are people, and people make mistakes. Making a mistake in a product review can be embarrassing and can actually cost someone money—the money could belong to product makers or readers. Here I don’t mean an honestly bad review of a product costing the manufacturer; in those cases, journalists fairly evaluate the product and accurately point out its flaws—flaws that are the manufacturer’s responsibility. What I mean is, if a reviewer says a product does something it really doesn’t, readers might choose to buy the product specifically for that reason. Such readers will be disappointed when they get the product home, and, especially with software, there may not be a return/refund path available. Equally damaging to the product maker is if a reviewer specifically says a product doesn’t do something that it, in fact, does. If the error relates to an important feature, readers who might have benefited by purchasing the product may pass it by. Thus, again, both readers and the product manufacturer lose out through no fault of their own. This has happened (not at Designorati, thankfully).

Another point in favor of allowing review subjects to peruse review editorial in advance of press is misdirected technical errors—especially when the reviewed product is a plug-in or add-on for another product. Numerous are the occasion in which Photoshop, for instance, was the culprit behind malfunctions in a third-party plug-in. Often times a plug-in will fail or crash, making it appear to a reviewer that the product is defective. In my experience, about half of those times are not the fault of the plug-in itself—the real problem is later revealed to be within the host application, operating system, hardware, or some other culprit outside the plug-in’s direct control. Deep troubleshooting—or even calling the maker of the host application, operating system, or hardware—is rare for product reviewers; it just doesn’t happen. Even when a call is placed, host application, operating system, and hardware makers often have no idea about the problem; they can only stay aware of so many issues related to integration of third-party products and their own. In most of these cases, the product under review is dinged for the crash or failure, and readers get the impression that the product is buggy.

However, plug-in and add-on makers usually know when and how host applications et al malfunction relative to their own products. If a review noting a bug is passed through the product maker, those issues that are not really the fault of the product can be explained as such. While Photoshop being at fault for Plug-In X crashing still may not make everyone rush out to buy Plug-In X, including in the review the steps to alleviate the problem and make Plug-In X run is a better service to readers than implying that the product is broken and isn’t worth their time.

Getting a price wrong is another compelling reason in favor. Omitting the “1″ in a price of “$199″ will make readers turned intended purchasers very unhappy, and they will take it out on the product maker.

Running a pre-publication review past a representative of the company who makes the product being reviewed can ensure that human errors and misdirected technical issues are addressed in the article accurately—and without accuracy, there’s no point to a review.

Against: Equally strong are the arguments against letting subjects see reviews ahead of printing. Product reviews are a delicate subject. Manufacturers depend on media to get the word out about their products. Naturally, every manufacturer wants his product to be reviewed favorably and score 5-stars. Some manufacturers want that so badly that they will exert pressure on journalists to be a little more favorable than objective. Giving a manufacturer the opportunity to see a review including marks against their product before the public sees it can, to some manufacturers, be perceived as an opportunity to make the review more favorable through coercion. It’s at this time that gifts and promises tend to be given to reviewers, requests for media kits (advertising rates) are made, and accusations of journalists “having it in for” manufacturers tend to develop. Pressure is often exerted to garner a more favorable rewrite of the review. Pressures can be subtle—”we’ll fix that in a dot release next week, so can we just omit that from your review?” or “you didn’t read the documentation, here, this is how you do it right”—or overt—”what can we do for you to get this and this taken out?” or “take out some of the unflattering items in this review, or you’ll never get another review product from us again.”

And, often, such pressures are not deliberate. The job of manufacturers (and the public relations agencies the larger ones use as liaisons with journalists) is to help sell their products. Many of them don’t know the ins and outs of the news business, and often they nudge a little without seeing the harm in it. Sometimes, though, even the most wary of journalists can be subtly influenced by careful manipulation.

Another point on the against side is the forewarned is forearmed theory. Paid-for editorial is not as uncommon as I would like to think. Public relations and copywriting people can write, and many of them begin their careers as article writers. When a manufacturer gets wind of an upcoming unflattering review, they sometimes arrange for publication of a more flattering review, written by an agent of the manufacturer or a journalist with a certain measure of flexibility in his ethics (and a corresponding inflexibility in his pending mortgage payment).

Running a review past a manufacturer in advance of publication provides opportunity for both fair correction of editorial errors or inaccuracies, as well as unfair influence over, or rebuttal to, an objective and accurate review. This is a sticky question to answer. Care to help us? What is your opinion?

These are the questions Designorati is discussing internally right now. We would be very interested in feedback from you, the reader, and even from the manufacturers whose products we do, have, or might review. What is your opinion?