Are You Biased? Let's Find Out

Dunky said:
Nice analysis. What do you make of Ashita no Joe 2's graph? That one seems to have the highest negative bias I'm aware of. Guessing it got hit particularly hard by fake votes because it doesn't have the popularity to offset them.

Deago said:


Welcome to another episode of "anime statistics"

Today, I'd like to share a new advanced method to discover biased ratings within members of a certain title. This is a novel observational study, and if you dislike math don't worry, it only requires observation.

I hope most of you are familiar with histograms.

An example of normal distribution histogram

you can see how smooth it is around the mean "the average" (7/10), this means the ratings are mostly health, and free from bias.


However, the example above was taken from "relatively unpopular and mediocre quality anime", and when I say "quality", I am referring here to the "relative critical claim" or "the popular opinion".





Now, let's see (High Popularity+ Low Quality):



The general critical opinion about SAO is negative, yet it still has a large fan-base, you may notice how 10 scores are higher than 9 score, BUT 8 score are also higher than 9 scores. In statistics, this term is called "systematic bias", but for the sake of simplicity, we will call this "Positive Bias".

Here are examples of "Negative Bias", you can notice how 1 score are higher than 2 score:





Postive Bias (notice how it increases as popularity decreases, this is also known as "Sequal Effect", but doesn't always applies; e.g. Gintama, Sword Art Online.)



We can see a correlation between "Fanboyism" and "Positve Bias" this is prominent in anime that the majority consider "critically bad". These would typically have "unfair" positive ratings.




Same goes for "critically good" anime, some people call this "haters", "haters" only target anime that is popular around let's say the forum, LOGH for example is associated with "elitism" which has a bad connotation, hence it received more negative biased scors than, let's say Monster.




of course, I am aware of bots, but these are also created by humans, whether it is haters or fanboys:











Tip:

Dunky said:
Nice analysis. What do you make of Ashita no Joe 2's graph? That one seems to have the highest negative bias I'm aware of. Guessing it got hit particularly hard by fake votes because it doesn't have the popularity to offset them.

Turnch said:
This is a really cool exercise in understanding the community's reactions and expectations towards different kinds of anime. Good stuff!

Also, even if it's not exactly the topic, I think we're all biased, and there's nothing we can do to get away from that. We use our personal experiences and the media/criticism we've been exposed to when informing our own opinions/judgements. Even in attempting to review something "objectively" by dividing criticisms into numerical categories or only considering inarguable factors such as budget or community reception, one still leans towards criteria they personally believe hold more merit than others. Entertainment isn't a science or legal proceeding. There's no harm in accepting our feelings and personal opinions in how we experience and rate it.

NubFix said:
this is a superb thread!
can the mods do anything about it?

Deago said:


Bot are easily spotted by opening the "stats" tab of an anime, you will find users of whose score is 1/10 has no real genuine scores, you'll notice how most, if not all their scores are only 10s or 1s.
But even so, they are too many to be easily handled by mods I assume.

TheBlackob said:
While I don't think that you didn't think this through I got the feeling you missed some aspects.
If you score anything it is mostly a gut feeling (unless you are are someone who does reviews with passion, but the amount of that is so few, that it is neglectable). And gut feeling is often based on comparison.

SAO with its popularity is not only watched by people who watched many anime. It is also watched by people just joining the anime world. People who can't compare this to 500 other anime.
SAO was one of my first 20 anime, long befor I even joined MAL and begann rating anime. Non the less internally I scored it high. It was just a very very good anime. One of the ones who told me, that anime might be a hobby for me. For the longest time it was a 10 for me after joining MAL. Only once I hit 400 anime, and updated a lot of anime scored years ago, it went down to a 9. It isn't one of my favorites anymore because I watched more, better stuff but I still rate it high, because I remember it being fun to watch and I still think it is one of the perfect anime for beginner.

AOT on the other side is just bad. It isn't bad enough to be a 1, (I have yet to find one bad enough to get a 1) but it has a bad artstyle, bad animation, unlikeable characters and so on (mind, that this is my personal opinion) each and every aspect isn't to my liking.


I don't disagree that there definetly are people with bias, but there also seem biased by your observation while in reality, they are not.

Deago said:
Why not try checking your own histogram?
https://anime.plus/
You can check "Ratings" and see if your personal histogram is normally distributed around your "mean score".

TheBlackob said:
While I don't think that you didn't think this through I got the feeling you missed some aspects.
If you score anything it is mostly a gut feeling (unless you are are someone who does reviews with passion, but the amount of that is so few, that it is neglectable). And gut feeling is often based on comparison.

SAO with its popularity is not only watched by people who watched many anime. It is also watched by people just joining the anime world. People who can't compare this to 500 other anime.
SAO was one of my first 20 anime, long befor I even joined MAL and begann rating anime. Non the less internally I scored it high. It was just a very very good anime. One of the ones who told me, that anime might be a hobby for me. For the longest time it was a 10 for me after joining MAL. Only once I hit 400 anime, and updated a lot of anime scored years ago, it went down to a 9. It isn't one of my favorites anymore because I watched more, better stuff but I still rate it high, because I remember it being fun to watch and I still think it is one of the perfect anime for beginner.

AOT on the other side is just bad. It isn't bad enough to be a 1, (I have yet to find one bad enough to get a 1) but it has a bad artstyle, bad animation, unlikeable characters and so on (mind, that this is my personal opinion) each and every aspect isn't to my liking.


I don't disagree that there definetly are people with bias, but there also seem biased by your observation while in reality, they are not.

Fario-P said:
Holy shit, how long did it take for you to make this thread?
If I am allowed to be completely honest, this thread is admittedly way too academically high for me to understand or figure out how to give a more meaningful response. But I will at least congratulate you on being one of the few users I've seen on here to make a very high quality thread during this... frankly oversaturated time on the MAL forums. Well done.



I can only hope to make a thread as amazing and well-put-together as yours.



I've just got on MALgraph to check if my ratings are "normally distributed" as you've said, and I dunno, I think the histograph I was given makes sense with my mean score for both my anime and manga lists. I didn't realize I gave so many 7s until now though lol

alshu said:
I love how the so called "statisticians" regularly pull nonsensical therms like "health".
Also I admire how they use only the upper part of the scale 7-10.

Even for layman like me this math sounds broken.
And yes, I believe in the data itself, I just don't buy into asspull interpretations.

Deago said:

Sorry, I meant "healthy"

Deago said:
which means the frequency falls down regularly away from the mean (average).
In science, healthy histograms may refer to cohort selections that are free from selection bias and considered reliable to conduct the study.

alshu said:

Which is equally BS therm.



People will interpret "healthy" and "bias" in a very different way...and you guys know that perfectly well.
Popularity", "Low Quality+Low Popularity" ect.

Same goes for "High Quality+High Popularity", "High Quality+Low Popularity", "Low Quality+High

blxlb said:
The main bias you see is unavoidable: it's volunteer bias. Nobody is required to watch, review, or rate anything. Generally people will try to watch things that they think they'll like (Typically based on a combination of rating, popularity, and genre)

And the sequel effect is attrition bias. Most sequels are of similar quality to the original, but only people who hate themselves are going to watch a sequel if they didn't like the first season.

Deago said:

Bias existence in a histogram is not subjective.

raiderublaze said:
As an MAL user who is interested and has also taken on part of some MAL statistics, I am impressed on your thread. It really is informative and hope that many more people understand this angle of bias on MAL.

alshu said:

But people will interpret it as "prejudice for or against"...and for example not liking Fairy Tail is not prejudice if you ask me.

alshu said:
I love how the so called "statisticians" regularly pull nonsensical therms like "health".
Also I admire how they use only the upper part of the scale 7-10.

Even for layman like me this math sounds broken.
And yes, I believe in the data itself, I just don't buy into asspull interpretations.

Adampk said:

I don't think the post meant u should have 7-10 score to be unbiased. Even a person with low mean score eg 5 can have a non baised histogram. I think the post meant histogram should be more like a triangle/pyramid to be more non biased. It doest matter even if the peak is lower no like 5 or 3. Atleast that's what I think the post meant

Deago said:

I'll give you an example,
if you see some gives Ashita no Joe a 1/10, then there's a huge chance this user is biased against the anime, and low chance he/she has a reasonable justification for the low score, they definitely think it's "overrated".

Same for anime with two spikes 10 and 8 such as DBZ,, the huge noticible 10/10 spike, means the person has a high chance of being biased for that particular anime, and probably thinks it's "underrated".

Adampk said:

I don't think the post meant u should have 7-10 score to be unbiased.

alshu said:

Actually this sounds like your bias towards those shows (Ashita no Joe not deserving 1/10 and DBZ not deserving 10/10) with which I coincidently agree...but for battle shounen fans DBZ is at least 9/10 so 10/10 is not bias. And if I am not mistaken there are tons of batlle shounen fans.
The only reason DBZ is not higher in the ranks is another biased group, the one I call "Everything older than 2010 is obsolete". Also those same guys score only with 7-10.
Like every solid group in the fandom has some sort of bias. It's impossible to account for them with statistical bias.



I don't think that too, my point was that the OP takes as sample people who vote exclusively with 7-10...and those are mainly newer fans.

jal90 said:
Why do you seek for normal distributions in a rating scale of ten numbers that, in principle, have an equal chance of being picked? Fixing the tendency towards one number of the scale and neglecting the others is another form of bias.

katsucats said:
It's called the central limit theorem. Look it up.

Slawadia said:
iirc fmab only has that many 1/10's because users wanting ishuzoku reviewers to reach the top in MAL ratings proceeded to mass low-score the competition earlier this year.

katsucats said:
You have no idea what you're talking about. In statistics, we say a sample is biased if the mean deviates from 0. An individual cannot be biased according to statistics. The "bias" in an individual comes from subjectivity, which has nothing to do with the statistical definition of bias. Since we're all part of the overall population, we're all random variables of that distribution. Distributions are, by definition, random, meaning there's a range of values, so it makes no sense to call an RV of a certain distribution "biased" with respect to that distribution.

Secondly, it's highly questionable to consider popular opinion "the critical claim", as if popularity makes quality. I don't think I even have to refute that.