5 Stars: Fixing the Standard Rating System

Months ago, my parents took me and my sister to a fancy shmancy ice cream shop where they mix the ingredients on a slab. Because of the premium prices, the horrible experience was doubly painful. From what I remember, we waited 40 minutes while the manager served people that had gotten there after us. The worker who had initially been serving us left in the middle of preparing our desserts because she told him to go do something after humiliating him in front of the few customers in the shop. After he went missing for 30 minutes, we assumed he quit from how rude she was.

Curious, after leaving I checked the reviews for the place on Google. The creamery had 3.3 stars and over 150 reviews. Google pointed out that frequently mentioned words in reviews included employee and rude. When looking at the reviews, those two words were used together to describe the person who had been rude to us. In fact, almost all of the recent reviews were negative and complained about this employee. Customers explained how they had gone out of their way to call and use other channels to report them but fell on deaf ears (or nonexistent ones). All the negative reviews received a reply from the company asking to email their customer service team which people were claiming had no effect. Clearly, nothing was being done and this shop was able to coast on good reviews from years ago. Holding their 3.3-star rating when reviews from the past year alone would put them at almost 1 star.

This made me think of a rating system different to combat this problem. Something that was different from the rest. Something that accounts for change over time. Something that wouldn’t allow companies to degrade and continue to capitalize on their former glory. Something that assured customers of the present state of the establishment. Something that… was already created by Steam.

Steam’s Review System

Steam doesn’t get the credit it deserves for its review system. It’s still limited in some manners but it’s lightyears ahead of other review systems in many areas. Cyberpunk 2077 is the perfect example to showcase its strengths.

Opening up the game’s store page, the user can instantly see the review sentiment. There are two different review scores- Recent and All. The former is the average of recent reviews, and the latter is the average of all reviews over time. If a game got a bad update, lost support, or was embroiled in any kind of controversy the user is immediately signaled to this. This solves the rude employee problem at the ice cream parlor problem.

Going down into the trenches- past the game info, community updates, and system requirements- sit the actual reviews. Above the written reviews is a graph that shows positive and negative reviews over time. Some reviews were omitted because they were found to be unrelated to the game. If the user wants to see these reviews, they only have to click on the asterisk to see what happened at that time.

Instead of relying on the “most relevant” sorting system where you see old reviews that were marked as positive, you are told a story of how something was received over time and is currently being reviewed. Cyberpunk 2077 had an abysmal launch on all platforms. This is highlighted by the high number of negative reviews at the beginning of the graph. What isn’t clear is if what is shown is all the negative reviews given at the time or if those are the negative reviews that weren’t updated to positive as many did eventually change their opinion over time as fixes were made and CD Projekt RED made amends.

Scrolling down to the user reviews, the first thing that is shown is that my friend gave a review of the game where they recommended it. Because they are my friend I have greater insights into what their preferences are and if we have the same taste. I can see, for all reviews, their like/dislike, hours in the game, hours in the game when they gave a review when there is a dramatic difference, their written review, and how other people have reacted to that review.  The star symbol on the top right corner of all games lets me know if the person paid for the game or got it via a key or gift as that might alter their experience.

The reviews taking up the most space are from the last 30 days and were determined to be the most helpful. The absolute most recent reviews are to the right so as soon as attitudes begin to change people can take notice.

This isn’t the be all end all of review systems though. There are notable caveats to adapting it to other channels and even problems with its current implementation.

Steam’s Limitations

The biggest and most obvious limitation of Steams review system is only being able to like or dislike a game. You are unable to give a meh, super like, or super dislike. (Even though I have two thumbs!!!) This makes it difficult to choose what makes a game worth recommending. If the gameplay is super good but it crashed dozens of times during a playthrough, is it still worth recommending it to others when that is a deal breaker for most people. This is a problem with all review sites where there is no uniformity in what a 1-star product/service is compared to a 5-star product/service. People may also not be evaluating just the product. I have seen negative reviews of products online because the UPS person was rude to them! Giving people only 2 options make it hard for people to draw the line and forces the finger on one side of the scale.

The reason I don’t use the Steam review system is that it requires a written response that is linked to your profile. Not only will my friends see it if they go to the game’s store page, but complete strangers will see it and be one click away from my profile if they do not like what I have to say. I do have the option of a very short review like “game good” or “game bad” but that makes the system harder to use for other people since they do not know specifically what I found good or bad or what threshold needed to be met for those elements.

For assessing reviews, there is a limited background on the reviewers. At a creamery, most walk in wearing traditional clothes, order off a small menu, and pay with a normal form of currency. On Steam, people are all using different hardware, have different values/expectations, and pay different prices. The negative reviews citing poor optimization could be made by somebody with a current day high end system or a decade old then-budget build. Is this somebody who bought a game on a whim or was passionately following the release for years and gave input on several betas? There are far more gaming snobs than ice cream aficionados. When it comes down to payment on Steam, price and cost vary drastically. The price people pay for a game is always changing. Steam sales are constant and a lot of users, like me, Wishlist products they are interested in and wait until a sale to purchase them. A positive review on a $15 game may have been negative if the player paid $45. Steam shows whether someone paid for the game, but it does not show how much they paid. Some reviews do specify if a person got the game on sale or suggest that those considering purchasing a game should wait for a sale. The cost of playing also has a large breadth. Gamers range from dads escaping for an hour while their kid is at soccer practice or teenagers finding something to occupy themselves with for a small part of their summer. The value of time for those two is going to vary dramatically and a satisfactory time killer for one of those people is a waste of time for the other. Steam somewhat combats this problem by showing the amount of time spent by reviewers and how many products they own. Those with more products likely have more time to spend playing games. Steam is not the only game provider, however, which limits this method’s effectiveness.

Outside of the platform, going back to the creamery, this system requires a wealth of reviewers for it to be as effective as it is. Most businesses don’t have that. Steam is used by millions of opinionated people but in the real world, people are less vocal about the quality of service at Arby’s. The effort required to have someone review every location they visit or every product they buy, potentially multiple times for the same thing, is unrealistic. The whole adoption of Steam’s process is a bad idea but aspects like a timeline are not.

Let’s imagine ways to build on Steam’s review system and what a review system of the future might look like.

Fixes and Potential Improvements

While writing this article, I came across an engadget article about upcoming fixes to Steam’s review system. As I mentioned, Steam’s system requires a text response and gives suggested helpful reviews based on reactions. A lot of people took advantage of this situation by using their review slot to post ASCII art and memes which were then reacted to, hurting the platform’s ability to show users high quality reviews. Steam is in the process of implementing AI algorithms to better select suggested reviews but is still set on requiring responses.

Because reviewers who had negative experiences are more likely to complain than people who had an okay or good experience are to publicly compliment an experience, I would like to perform a test where some users are allowed to like or dislike a game without a text entry. I am curious if the number of reviews coming through noticeable increase and whether that increase would have an effect on the overall review score. I am also curious as to what impact that would have on the number of reviews with written responses that are serious and unhelpful responses.

Another improvement, for Steam, would be giving incite to what cause a serious of negative reviews or why reviews got redacted. Hovering over an area might reveal the message, “Game was on sale for $1,” or “Game received game breaking patch that took a week to fix.” This would give a potential buyer greater insight. After all, those that are looking into reviews to this extent might like to see an explanation of any abnormalities.

Including statistics like average playtime across users, % of players who left a review, or any other statistic that could shed light onto the quality of the game or reviews would be helpful.

The Future of Reviews?

I take reviews with a grain of salt because I know that not everyone leaves a review for everything they buy or do. I rarely leave reviews even for things I like. My couch is extremely comfortable, but I don’t have the motivation to find it online and type out a paragraph. My shag carpet has an edge full of loose carpet that I had to stick under my couch which I thought was annoying, but I don’t want to review that either.

When I imagined a review system of the future, it wasn’t managed by people submitting formalized reviews or controlled by the party being reviewed. It was a third-party review system that took data from all over the internet and in person and created a report for a product or place.

Jeremy walks into the gas station and buys a pack of cigarettes. Security cameras there stream the video onto a local server owned by a third party whose algorithms do a sentiment analysis of his verbal and nonverbal communication. That analysis contributes to a larger report made up of other customers attitudes and behaviors at the site and mentions of the location across the internet. Written reviews can be submitted to the third party directly but would only be a small portion of the pie. The service can identify when customers come back for unsatisfied returns, when they do not like dealing with certain employees, and how they feel about everything else in the store. Companies would be obligated to monitor their experiences constantly so that they do not begin to slip as everything contributes to their rating and not just a small percentage that chooses to be vocal.

My idea is to have customers information to be trashed as soon as the data is pulled from the video. Nothing should be stored long term, and customer profiles will not be created nor sold. Because nothing personal is ever collected or stored, this should be no different than having security cameras. This service would work for both online and brick and mortar locations as it uses the internet to collect publicly given information. If something similar to this idea comes out, I hope it follows my privacy expectations as the only reasons I see it failing are companies unwillingness to pay for unbiased review monitoring and privacy concerns from the public.