TL;DR: Amazon has implemented a new strategy of giving old reviews a significantly lower weight than the newer ones. And should anybody post a single bad review as of recent, which would be normal for outdated tech books, that review will pretty much define the quality of the book (== terrible).
Gaming reviews has been a big issue for any marketplace, and in particular on Amazon. Amazon does a pretty so-so job at weeding out fake reviews out as it is gamed freely by some Chinese vendors in an inventive way that appears legit. But it has recently deployed a new strategy of how it weighs each given review. Reviews with no ‘Verified Purchase’ tag get a lesser weight than those with the tag, and older reviews get progressively less weight in the game.
(note: I have no direct source to confirm that this is so, and I derived this by looking at multiple listing, asking for help from others and doing the math, which after several days of investigation lead to what appears to be a good understanding of what is probably going on. So please don’t take this as a definitive answer.)
Let’s Destroy my Book’s Reputation
My first book was published in 2003. Now long forgotten by most, the Apache/mod_perl technology became outdated around 2010, and moreover, this particular book was written about the first generation of this technology, and which got superseded by the second generation in 2004. As you can tell this book is not relevant for today’s tech world, unless you happen to maintain a 15-year old system that refuses to quit.
Until recently, that book had an almost excellent total 4.4 rating with 3 5-star reviews, and 4 4-star reviews. Which I think reflects well the reality. It’s an excellent work of 900 pages that took me 3.5 years of hard work to create. Once I finished the writing, I discovered that writing technology books is a foolishness, because they get outdated before they hit the presses. It’s better now, when books are digital, as compared to dead trees. But even with digital books, for an author to constantly update the book to match the ever-changing at a furious speed APIs and validating that the code still works is an insanely time-wasteful effort.
Now, I know all those reviews on my book were honest and I have never solicited those. Only one of those reviews has a Verified Purchase tag, but many of my books were purchased at the O’Reilly Open Source Convention – I know that, because for 7 years I taught workshops at that conference and signed many of those books that were purchased there. Powell Books was usually the book vendor there.
Someone recently rated my book at 1-star (no review). I can’t imagine, who in their right mind would buy a 17-year old book on a long dead technology, and expect something good out of it? No wonder they must have been enraged when they started reading it and as a result left a I-hate-it 1-star rating. It could have been done out of pure malice too. Who knows.
You might say, what’s the big deal. Indeed, it shouldn’t be a big deal, since no single review should define a book’s quality – but, unfortunately, it is a big deal. Since after that rating was added, the total rating of the book went from 4.4 to 2.9.
Now, you might say, no way. But, unfortunately, numbers don’t lie:
So we have 7 positive and 1 negative ratings and you can see the system gave a whooping 48% weight to a single negative rating! Don’t you find it just odd?
Of course, the book has no chance to balance this machinated wrong-doing, because other than a straightforward manipulation of me asking someone to buy the book, rating it at 5-stars and refunding them the money, nobody in their sane mind is going to do it on their own will, because the book is 17 years old and is no longer relevant.
You might say that, isn’t this a good thing that no longer relevant books get bad rating? It’s a difficult question. It depends on who is looking:
- To a potential buyer of this specific old book it’s a good rating – signaling don’t buy.
- To an author who worked hard to create the book this is a really bad rating as it doesn’t reflect the quality of their work.
- To someone who might be looking at author’s page before buying his/her other books this is a bad rating, since who wants to take the risque of buying new books if their other books have a bad rating, and this is clearly a misrepresentation of the truth.
- To a potential publisher who might be considering giving this author a contract for a new book this is a bad rating.
Moreover, depending on whether you’re logged in or not, Amazon is likely to show you a different rating. For example, my book shows 3.1/5 when I’m logged out.
Note, how the 1-star rating went from 48% in weight to 41%. Why do I get a different rating whether I’m logged in or not?
Another thing you may notice is that Amazon started incorporating International reviews and ratings into the total ratings. This too is very confusing for some books, since you may get a totally different rating depending on whether you’re logged in or not. For example, this book shows a rating of 5/5 if you aren’t logged in (and sometimes even if you are logged in):
But when I log in and I go to the book’s page, it drops from 5/5 to 3/5:
Here, despite 9 5-star reviews, the book is ranked at 3/5. When I log in, a single 2-star International review is being displayed, with a crazy disproportionate weight of 68%, which brings a 9 5-star reviews book, to 3.0. Can you make sense of it? Do you read the fantastic reviews and buy the book, or do you run like hell because Amazon thinks it’s a terrible book – based on a single recent Verified purchase international review?
Have I mentioned that none of those 9 5-star reviews are ‘Verified Purchase’? Bad signal for sure – we don’t trust reviews from those who didn’t pay us money, Amazon communicates.
Note, that this example is not of a technical book, so something else must be going on.
Customer Service Blames Everything on Machine Learning
I contacted Amazon customer service, asking for help to understand what’s going on and got a reply:
The overall star rating for a product is determined by a machine-learned model that considers factors such as the age of the review, helpful votes by customers, and whether the reviews are from verified purchasers. Similar machine-learned factors help determine a review’s ranking in the list of reviews. The system continues to learn which reviews are most helpful to customers and improves the experience over time. Any changes that customers may currently experience in the review ranking or star ratings is expected as we continue to fine-tune our algorithms.
This doesn’t help me, as a customer to understand what’s going on.
Please Remove my Books from Amazon
One thing for sure, as an author, I don’t want my book to be misrepresented and I’m going to ask O’Reilly Media, Inc, the publisher, to pull my book off Amazon and I’m going to try to find a way for Amazon to remove all books published by me and never sell any of my books on Amazon. Amazon banned me from self-publishing a few years ago, because of some internal conflict with a 3rd party vendor, which had nothing to do with me, and without giving me any reason of why suddenly I wasn’t a kosher writer, while continuing selling my other books. Now this situation should hopefully remove any income coming from my works to Amazon. Surely, I will sell less books without Amazon, but I prefer that, to Amazon transmitting falsified information about my works.
Recommendations for Tech Book Authors on Amazon
To conclude, here is what you need to know as an author to prevent Amazon from burying your works over time:
- Your older reviews get less and less valid with time – you need to find a way to get recent reviews (hint, hint)
- When you (hint, hint) get them, make sure they are posted by those who purchased the book on Amazon. Should they buy it elsewhere, too bad – they will have to buy another copy on Amazon. Basically, Amazon forces you to game its system, while making money for Amazon.
- If your book is aging, which in particular impacts technology authors, but I’m sure it affects other categories as well, its reputation is going to get destroyed by recent negative reviews, which you can’t avoid. Because the relevancy of your work is diminishing, the 1-star reviews will prevail. The only recourse here at the moment to keep your reputation unblemished, I think, is to find a way to remove your book from Amazon. I haven’t investigated other marketplaces to tell whether it’s the same situation over there, but I’m sure all markets will follow Amazon’s lead. So, it’s best to remove the aged book altogether and not allow automated systems to misrepresent the child of your hard work.
Recommendations for Marketplace Designers
I doubt Amazon would care for my recommendations, but you never know, someone on their dev team might read them and at least consider them.
After contemplating the current situation here are some recommendations that I came up with:
- When you introduce weights for different signals, visible to users, make sure the outcome makes sense. e.g. the second book presented in this article has a rating of 3.0 and it displays 9 5-star reviews and nothing else – weird, no?
- Weigh ratings w/o reviews at a lower weight than ratings with reviews. It’s too easy to submit a number on the rating scale without giving it much thought, usually resulting in less than reality-reflecting ratings (in both good and bad directions). It takes time and thinking to write a review, so usually ratings bundled with reviews are of a higher fidelity.
- Weigh higher reviews with real customer names as compared to Anonymous reviews. It’s easy to write a negative review when one intends malice (competitor/hater/etc.). If you read listings you’d find few positive reviews by Anonymous. Surely, some people don’t feel safe to write a totally valid negative review with their name on it for fear of persecution. So this recommendation is highly debatable.
- Apply all the weighing smarts for products with dozens and hundreds of reviews, the truth may prevail there (unless it was gamed), but be careful with niche products/books with under 10-15 reviews. Any automated weight re-balancing of products with very few reviews can lead to a disastrous outcome for the creator. A carefully thought out different algorithm or a ML model should be applied there.
- Give authors of technology books (and whatever similar domains fit here) a chance to indicate when a book is outdated, to prevent disappointments from users and almost guaranteed ensuing negative reviews/ratings. The book should probably still be available for some years to support those who maintain old systems. But it should be made loud and clear to the customers, that they are buying an outdated manual before they hit the ‘Pay’ button. That is if the author was given a way to signal this situation.
- Separate second hand market ratings from the normal books. Very often I see a 1-star post from a second-hand book buyer: “I received a damaged copy”. What does this rating have to do with the quality of the contents of the book. This is very damaging to the total book rating. Why do you do nothing about this? Perhaps, the rating form can be adjusted to indicate whether the user provides feedback on the physical quality of the book as compared to the content.
I honestly don’t know what to do about the ‘Verified purchase’ reviews outweighing the reviews w/o that tag. From one point, it’s easier to add fake reviews w/o purchasing the product. On the other hand it’s just as easy to make people buy a book/product, write a review and then refund them the cost via paypal (a common current way to game the system). And as shared earlier in this article, this forces more sales for Amazon, and discourages competition. I trust Amazon has a good insight on this one.
This situation took several days to sort out and I’m glad the picture is finally clear.
The automated future is looking is great for the masses. Niche creators watch out!
Tags: authors, machine learning, manipulations, ratings, reviews
great post. how is china related to this?
Thank you, Andy, for the kind words.
I was referring to purchased reviews.
What I’m trying to communicate is that Amazon punishes innocent authors, while Chinese vendors are at large ensuring that they get Verified Purchase reviews, gaming the system by trading reviews for freebies. I’m sure Chinese vendors aren’t the only ones who do that. This is just that I had the experience with 2 Chinese vendors in the last few months that offered me to be a receiver of free products in exchange for reviews (after I bought them from Amazon, so they get the Verified Purchase review, and only after a review they would refund me via paypal).