![[decorative element]](http://designorati.com/wp-content/themes/d3/images/nav1_topics_OFF.png){kind=small}
Product Review Policies
The subject of certain policies and guidelines relevant to product reviews has been a hot topic around the Designorati conference table over the past week. We would like your input on some of it…
The subject of certain policies and guidelines relevant to product reviews has been a hot topic around the Designorati conference table over the past week. We would like your input on some of it.
On the discussion table are a few questions specific to product and book reviews. In (reversed) order of complexity, these questions are:
- Do readers really care about stars or other ratings systems?
- What elements are most useful to readers in a review?
- Should we send review articles to the subjects of those articles in advance of publishing?
1. Do readers really care about stars or other ratings systems?
Well, do you? We’re all familiar with star rating systems for restaurants and hotels—a 5-star restaurant is the best, while most people avoid 1-star establishments like the plague. Is a 5-star rating system truly that interesting and valuable to you? If you see that Designorati has reviewed Gizmo 3.0 at 1-stars, are you going to avoid buying Gizmo 3.0? Would you rush out and buy the product if we gave it 5-stars?
My opinion—please comment if you disagree—is that ratings systems are an initial guideline. After a time of reading one writer’s articles, you may find that you have a tendency to agree with her on most topics. Then, if that writer gives a product a low rating, you may find yourself inclined to make a judgement on the product yourself, even without reading the full review. Conversely, a 3.5-4.5 star rating from the same writer may prompt you to become interested in the product under review, but also to read the article to find out why it didn’t score higher. In my experience, this is pretty common: The more one reads the work of a given writer, the more one forms an opinion about trusting (or distrusting) that writer’s evaluations of software, hardware, books, and so on.
On the other hand, some people don’t care a bit for reviews. I, for example, ignore movie ratings entirely. Why? Because most professional movie critics are morons who pander to Hollywood politics and peddle star-ratings like maps to celebrities’ homes. (”Independence Day” earned 5-stars from some of the same people that condemned “A Beautiful Mind.” ‘Nuff said.)
So, what do you think? Are 5-star scale ratings useful to you in product reviews? Should we keep or dump them?
2. What elements are most useful to readers in a review?
Here’s where we could really use your feedback. What sections or types of information do you typically want in a product review?
Of course, our software and hardware reviews include real-world testing of the product, and of course we include screen shots or photographs as appropriate. I’m talking about the other bits of information. Which of the following do you find useful in a review, and which tend to clutter it up? And, where would you like to see them—at the very top of the article, at the bottom, or just somewhere in the article?
- System requirements
- Buying advice (the reviewer’s opinion about who should and should not buy the product in question)
- Link(s) to locations where you can buy the product under review
- Platforms compatible with the product (Personally, I like to see these at the top somehow; if I’m looking for a Mac-compatible gizmo, I won’t bother reading a review of a Windows-only gizmo.)
- Price: Suggested retail price from the OEM, or the best price we can find from any source?
- A succinct summary of pros and cons
What else would you like to see that isn’t listed above?
Next: Should we send review articles to the subjects of those articles in advance of publishing?
- Acrobat 8 Professional New Features Create A Worthwhile Upgrade
- Photoshop Lightroom Review At Macworld.com
- Contact Designorati
- Aperture 2.0 Now Available
- Why Lightroom Is Not Included With Any CS3 Suite
- QuarkXPress Reviews
- Fireworks CS3 Is Looking For Its Niche
- Book InDesign Type
- Acrobat 8 News: INTERVIEW With Lonn Lorenz, Acrobat 8 Product Manager
- Contribute CS3 Makes Easy CMS Even Easier
my opinion, is that reviews are part evaluation, part condensing. A reviewer can let a reader know if the product is good, but also in a way that isn’t limited marketing jargon. When a reviewer says Integrated Workflow, you might think they are only regurgitating from a company website. But when they highlight an advantage that they think their readers will care about, or find a feature that is especially cool, but hidden, they are doing a great job I think. Star ratings do little for me unless the rating was bad and the star ratings were given by about 5 different people summarized.
An example I would give is a review of a Corel Product. Corel has limited information on their site in terms of specific product features and product comparisons. I couldn’t tell you the difference in buying Corel X3 Standard or Professional, from Corel other than price. But a reviewer could tell me so. And based on the reviewers comment such as “Professional has these features but I wouldn’t use them much” then I would evaluate that with my own needs. On the other hand, Corel and any other company site will always list every benefit and feature as “you need this.” even when they couldn’t possibly know.
I like the idea of using a (or crosshair?) rating system, as long as it is from a trusted reviewer as well as accompanied by a review that discussses why the product received that rating. I like reviews that feature evolution of a technique, or the introduction of “the new squiggly line tool” that I might not have experienced yet.
As for allowing the subjects to preview the article, I would say I approve. If I trust the writer of the article, it shouldn’t matter what kind of swag they have received. Right? One could always edit in notes that the issues are “currently under review” by the manufacturer, or something to that effect.
Plus, you stand a good chance of getting cool stuff as well as like you’d stated, corrections to any misunderstandings.
Thanks for the input, GraphicsMan.
As the discussion continued internally, we have leaned toward a policy of sending reviews to manufacturers ahead of publication. We’re in a position where we talk to a lot of manufacturers and other members of the press. The more we explore this subject, the more we’ve heard two things we already knew: Online media typically doesn’t send reviews by the their subjects ahead of time, and that published reviews more and more frequently contain errors in material facts.
A reviewer’s job is to provide opinion but also fact; the reviewer must tell you what a product does, what it doesn’t do, and then provide his opinion of both. At the same time, other hard facts must be delivered, facts such as: whether the product (if software) runs on Windows, Mac, or both; the manufacturer’s stated system requirements; any other prerequisites; availability dates (if relevant), and; price(s) of the product. Opinions are opinions, but facts must be checked and verified.
If, for example, a review states that a product retails for $199 when it is actually $99, then the reviewer is doing the reader and the manufacturer a disservice. That extra hundred dollars can, and often does, determine whether a reader decides to investigate the product. If the reader chooses not to, thinking the price is too high, that reader won’t even have the opportunity to see the correct price.
We’ve seen too many factual errors in online reviews lately, including—and especially—reviews published by Big Name creative industry media. Consequently, Designorati has made it a policy to send all reviews to their subjects. When to do it is open. If the subject has requested to see the review in advance, or if the reviewer feels it’s warranted, reviews will be sent ahead of publication. Otherwise, they’re sent to the subject immediately upon publication, which enables rapid correction of factual errors should they occur (none so far, knock on wood).
With this, I must disagree. Journalists—particularly those who write reviews—should never receive swag. No matter how honorable the journalist may truly be, the perception of influence by the readership is what really matters.
For example let’s say I wrote a glowing review of Acme’s Rocket Cycle. Then, it came out that, while writing the review, Acme had sent me a Roadrunner Catching Kit to keep—not for review, just as a gift, for my personal use. While I know I would continue to be impartial in the review, that I know my integrity is not for sale, how would you know that? Even if my prior writing gave you reason to trust me, not knowing me personally or professionally, you would (and should) have a lingering doubt as to whether my Rocket Cycle review truly was influenced by the gift.
It’s happened plenty of times in the past—not with me, of course—but in other media. The radio business, for example, was rife with payola and swag influence in the 1980s.
Journalists should never accept gifts from advertisers, prospective advertisers, review subjects, or prospective review subjects. That’s the only way to maintain both actual impartiality as well as the perception of impartiality.
DESIGNORATI