I wrote something with Josh on the Undercurrent.com blog today, and wanted to share: Trust, Complexity, And The 5-Star Rating.
It was inspired by a meandering conversation two weeks ago – this post on five-star ratings and Yelp had been sent around the office, and the following morning Sam, Joe, Mark, Josh and I met up over coffee to talk about it. The basic gist of our post is that it's difficult to compare rating sites (Yelp and Netflix, for example) to each other like apples to apples, because a lot of things affect people's relationships with them – who the intended audience is, what the thing-in-question being rated is, how sophisticated the recommendation algorithm is, what the user base is like, etc. The two hypotheses we single out are:
1. The primary user benefit of a rating system affects our motivations to rate and review, our rating behavior, and our perception of the aggregate ratings (and the collective who contributed them).
2. The level of trust people place in an aggregate rating system is determined by the cohesiveness of the community and the sophistication of the recommendation engine.
We had a third hypothesis down, but it for various reasons got cut from the final post. But I still love and believe it! So here it is – a "deleted scene," so to speak.
3. The higher the opportunity cost associated with an experience, the more complex the rating system should be.
The more involved the action is, and the greater the cost of the experience, the more complex the rating system for it should be built. For example: Stopping a bad movie and choosing another from Netflix’s Instant Watch catalogue can be done from the comfort of your couch. All this costs is some time (and maybe a little bit of annoyance). The costs associated with going out to eat are much higher – transit (time and financial), wait times, crowds, weather, interacting with staff, and paying for the food. Even more so if the experience sucks and you have to head to another restaurant. Even in a city with a multitude of bars, the opportunity cost of looking for a dance party in Manhattan on a Saturday night is much higher than that of renting a video from Netflix (either by disc or streaming). So a recommendation site's algorithm should not just be as sophisticated like one like Netflix's is – it should be better! Take more things into account, weigh by different variables, triangulate information from a variety of sources, etc. (This is actually related to another post I've had swimming around in my mind about the best hybrid of all going-out-to-eat planning sites; maybe I'll write it up this week).
If you want to read more about all of this, read Trust, Complexity, And The 5-Star Rating and let us know what you think! Any other things you can think of that make no-two-rating-systems-created-equal?
If you're curious:
Rating Systems and Personal Rules by Noah a few years ago
Horsehead Nebula on Wikipedia (from our post image)
