Half and Half - Oct 12, 2022

[ebirnholz]

@HoldThatThought:

I've honestly tried to stop relying on star ratings for any commodity -- movies, restaurants, books, you name it. I'd rather not let my judgment be influenced by ratings before I experience the product myself. I'm not always perfect about that, especially with more expensive purchases like a hotel room, but even then I take a closer look at actual customer comments rather than star ratings in case there are major red flags to watch out for.

And at least with those other sites like Yelp or IMDB or Amazon or Google, you will usually have thousands of reviews at your disposal that can give you information on the product you're getting. That's not the case with crosswords. There aren't many sites that review non-NYT puzzles, and with Fiend, a handful of cranks who drop 1-star ratings for no actual reason have an outsized effect on other solvers, the ones who use them to determine which puzzles they solve.

In some cases I'm also opposed to star ratings on principle since some businesses like Uber actively use them to evaluate their employees. The thought of a company punishing its workers because some customers didn't give them 5-star ratings is something I find really disturbing. The stakes with the star ratings at Crossword Fiend aren't as high as that, obviously, but it would still be nice if something as complex as a crossword -- its theme, its answers, its clues, how it all ties together, and ultimately something that a constructor pours a huge amount of time into creating for others' enjoyment and often not much money -- weren't cheapened with the same kind of algorithm that we use to evaluate every other product in our lives. It's an artform, not something that should be reduced to a star rating.

You said earlier that no experienced constructor gets heartbroken over a handful of 1-star ratings. Maybe not, but I know several veteran constructors (Matt Gaffney, Matt Jones, Pete Muller, among them) who have publicly expressed their own frustration with people dropping 1-star ratings without bothering to provide any feedback. For my part, I don't even look at the star ratings. Literally. I scroll past them immediately on opening up the page. Better to ignore them if they're not going away.

And yes, I agree that it'd be nice if everyone who visits Fiend could take the ratings with a grain of salt, but unfortunately that's not what happens in practice. Again, I've already heard from many commenters who have said they use them to decide which puzzles they solve. You can say "well it's those people's fault for relying on an algorithm," but I blame the system for encouraging that thinking rather than individual solvers. These are most likely people who aren't super-fast solvers, don't have as much time in the day to solve every puzzle out there, probably don't spend any time thinking about how the star ratings are flawed and unreliable. They're still potential folks who would become regular solvers of any puzzle reviewed on Fiend, but aren't because the star ratings have convinced them otherwise. The star rating system is the cause of that.

I guess my question is this: What meaningful benefit is there to the star ratings, and what would be the downside to abandoning the star rating system altogether? It a) isn't a reliable measure of a puzzle's quality, b) causes unnecessary stress for at least some constructors, and most importantly, c) prevents anyone who relies on it from becoming regular solvers of puzzles that don't meet some arbitrary numerical threshold. All of these are problems that could be solved if the ratings just weren't there. I have a very hard time imagining how there's some actual benefit to them that outweighs those issues.

[HoldThatThought]

Evan: Thank you, again, for your thoughtful, and thought provoking response to my comments. I think we're both very much on the same page.

I sense that we entirely agree that no one should be much-swayed in their puzzle-selection choice by the reviews of anonymous star raters; especially, as you say, when there's no commentary, and therefore, context on the reason(s) for the rating.

We might differ, though, in how we're thinking about the entire mindset - not just of interpreting reviews (and relying upon them), but, also, of how reviews are made, and what they represent.

Most reviews are inherently subjective, and simply answer the question "how did this make me feel?". Diners are rarely reviewing a chef's knife skills, movie viewers are rarely reviewing a director's scene blocking or set dressing. Most reviews - even the most thoughtful and genuine - are nothing more than an answer to the question "How did I enjoy the totality of the experience?" Puzzle reviewers are probably not commenting on the technical merits of the puzzle (cluing voice, grid design, lazy reliance on crosswordese, sloppy editing, meta or theme originality) any more than a diner is commenting on the chef's nuanced use of locally sourced herbs. A rating comes down to an utterly subjective opinion about "how did this make me feel?"

I have a personal history with ratings that has certainly influenced the way I think about them. My first career was in academia; as a young and inexperienced university professor, I was extremely interested in student reviews, man, I could not wait to see what my students had to say about my teaching skills. On the one hand, I was hoping to read praise, but I was at least equally anxious to learn how I could improve. Even the simple stuff: did I talk too fast, did I speak clearly, was my tone and language appropriate to my audience? Obviously, I also wanted to know "was I getting through? Was I making the material understandable? Was I effectively teaching, and was I inspiring an interest in my subject matter?" In this environment, at least at the school that I was teaching at, the reviews had real consequence - lazy deans would evaluate a new lecturer's performance on the basis of student reviews.

I got that first packet of reviews, and tore that envelope open; eager to be praised, eager to improve. And the reviews? All over the place. Dozens making me smile, and giving me confidence with comments like "Dr HoldThatThought has a skill in breaking down even the most difficult material in a way that I could understand it." "The professor makes the class fun with lots of humor and funny anecdotes that help the otherwise dry material come alive", "I like the way that he so carefully follows the syllabus, and lays out his expectations for student performance, and how our grade will be determined", "The instructor is generous with his office hours, and always seems happy to answer my questions, before and after class", and my all-time favorite comment of all, "Dr HoldThatThought does not wear socks." I'm not making that one up.

And what else do you think was in the packet? "Poor syllabus, which he never follows anyway" "Tells boring and inappropriate stories that he thinks are funny" "Lectures have nothing to do with the exams." "Never available outside of class, and if you ask him a question, he'll make you feel stupid for asking it" "Worst class I have ever taken. Will tell everyone I know to avoid this professor."

So, were the reviews useful? Was their anything to learn from them? I think so. Was I being evaluated objectively, on the technical merits of my knowledge and teaching skills? Some of the time, maybe. But, mostly, these students were telling me if they "enjoyed" the class or not. Clearly, some hated it, me, the experience they were having at the university, and whether their boyfriend was being a jerk when they had lunch together, just before that class. This is what a review is, of dinner, of teaching, of movies, and of crossword puzzles. There is informational value in reviewing reviews, but a creator should appreciate them for what they are, and people who are deciding whether to bother with a crossword puzzle should, too.

And yeah, the aforementioned lazy ass dean did want to "chat" about a few of the reviews that weren't as encouraging as the 20x as many that were.

Thanks, again, for this collegial discussion.

[Joe Ross]

Thanks to everyone for this discussion.

[ebirnholz]

Thanks, @HoldThatThought. Before my full-time life in crosswords I also spent time in academia, with a semester as a teaching assistant for a Gender & World History class. The reviews I got from students were just like yours: all over the map. So I appreciate you sharing them.

To clarify, I differentiate between the star ratings and comments left on Fiend and (especially) the actual blog posts. I get the most informational value from bloggers' reviews since they actually take the time to analyze the puzzle in detail after they've solved it. I get some value from reading the comments; I know many constructors operate under the mantra of "never read the comment section" but I still do it since then I can get a sense of what these solvers are thinking, what makes them tick, etc. I don't always agree with either the Fiend blog posts or the comments -- and there are occasional trolls who can be a pain -- but there's usually information there that I can at least consider.

But many years ago I realized I was getting zero useful information from the star ratings. How am I supposed to know what caused someone to rate the puzzle 1 star? Was the theme too difficult? Did the clues have pop culture references the solver didn't know? Do they just not like seeing the answer OREO? I have no way of knowing any of that from the star ratings, and frankly it just doesn't matter. I feel confident after all these years that I don't write 1-star-quality puzzles, so if someone would rate my work like that I'd just dismiss it out of hand. If it's all just an exercise for the solver to say "I enjoyed this puzzle" or "I didn't enjoy this puzzle" (without actually specifying what they did or didn't enjoy about it), well then that's true of literally every puzzle -- some will like it, others won't. They can still express the same sentiments and get the same level of enjoyment out of a puzzle if the star ratings were abandoned.

Again, I'm with you that it'd be great if everyone could just ignore the star ratings, or at least not treat them as though they say anything about a puzzle's quality. But since there are people who have admitted they use them to choose which puzzles they solve, that's reason enough for me to try to convince others that the star ratings need to go.

(p.s. Sorry to everyone else that this may be way, way more discussion about the Crossword Fiend star ratings than you cared to read!)

[HunterX]

Interesting discussion about star ratings! I can't add much to the discussion, nor do I want to. You two are very articulate and have covered the topic thoughtfully and thoroughly. You also have more that you agree on than you disagree.

I do have a question, though. Isn't the star rating's purpose just to have something quantifiable that can be averaged to give others a general sense of the overall sentiment towards whatever is being rated? I think we all have become accustomed to using it as a very general, and not very accurate, guide, knowing that we have to look at specific commentary to get anything truly meaningful or helpful. And, or course, even then, you have to take what you get with more grains of salt. The devil is in the details. And even then, sometimes the devil just had a bad day.

Plus, I wonder whether people who don't even try low rated puzzles wouldn't just find another criteria for rejecting some if the rating wasn't there, such as how quickly the leader board fills up. Some of us only have so much time and need to use some sort of criteria to pick and choose what we spent time and attention on.

FWIW, I don't use ratings to determine whether I do a puzzle. I use the length of my to-do list. Speaking of which, I have to get back to it.

But thanks for the thought provoking discussion! I give it 4.9 stars.