Danny
Asked 8 years ago
|
|
The "Shift" Effect: Re-Scoring Movies
|
[archived]
|
Hi all,
I have a little theory/worry/thought. It goes something like this...
When I first joined MovieLens, I started the fun process of rating films - very soon I was nicely around 150 or so.
It was around the 150 point that my scoring system started to alter, and I found that the first few films that I'd rated needed re-scoring - I suppose I would put this down to "getting used to the system".
BUT around the 400 ratings mark, I started to think that ALL my scores should shift - and I think I still want them to. I want to drop all my scores down 0.5, leaving about only 12 films with a 5.0 rating (as opposed to my current amount of 26).
Another solution would be to mark the 12 TRUE 5.0 films with a 5.0* (indicating even better than 5.0), but that negates the 10 point scale and isn't possible!
At the moment, 50% of my films have a 4.0 score or higher, which I feel can't be right. (31% of my films have a 4.0 rating, 18% have a 4.5, and 5% have a 5.0.)
Has anyone else experienced a similar "Shift" effect?
|
|
|
Answers
PolarisDiB
Answered 8 years ago
|
|
It's easier for me. I break it down like this:
Script: 1 Star.
Acting: 1 Star.
Directing: 1 Star.
Editing and other technicalities: 1 Star.
Entertainment Value: 1 Star.
If something is "okay" and not particularly interesting, not technically bad but who cares, it gets a half a star. If it's bad, it doesn't get a star, and if it's good, it gets a star.
Sometimes a movie, despite lacking in something, say acting for instance, is just so incredibly good that I feel it deserves a bit extra, then I'll go ahead and give it full ratings, but that hardly happens because I often think that a movie is so good because everything in it is done well.
With this system, my most common ratings are 3.5 for movies I don't really care about, 4.5 for movies I like, and 1.5 for movies I hate. Very rarely does a movie get 5.0 mostly because even the best done movies don't seem that entertaining and vice versa, and I can then understand having to at least put a .5 star on bad movies because anything that bad provides entertainment in the form of loving to hate it.
--PolarisDiB
|
|
|
movsom
Answered 8 years ago
|
|
Hi all,
I have a little theory/worry/thought. It goes something like this...
When I first joined MovieLens, I started the fun process of rating films - very soon I was nicely around 150 or so.
It was around the 150 point that my scoring system started to alter, and I found that the first few films that I'd rated needed re-scoring - I suppose I would put this down to "getting used to the system".
BUT around the 400 ratings mark, I started to think that ALL my scores should shift - and I think I still want them to. I want to drop all my scores down 0.5, leaving about only 12 films with a 5.0 rating (as opposed to my current amount of 26).
Another solution would be to mark the 12 TRUE 5.0 films with a 5.0* (indicating even better than 5.0), but that negates the 10 point scale and isn't possible!
At the moment, 50% of my films have a 4.0 score or higher, which I feel can't be right. (31% of my films have a 4.0 rating, 18% have a 4.5, and 5% have a 5.0.)
Has anyone else experienced a similar "Shift" effect?
First of all, you only rate movies you have actually SEEN and if this is the case here, then the rating you have given these movies should reflect what you actually think about them. If you think a movie deserves a 5, then it deserves a 5, no matter how you have rated the other movies. If you think all the movies deserves a 5, then let it be so. Your rating history doesn't have to obey some sort of rating law.
|
|
|
abintra
Answered 8 years ago
|
|
The predictions I get I find not to be very good and have attributed it to the way I ranked my films rather than based on the suggested rankings listed in the upper right hand corner of 1 awful, 2 fairly bad, 3 ok, 4 will enjoy, 5 must see.
For example, the system only gives me 2 4 stars titles, 2 pages of 3 1/2 stars, and then it drops down to 3 stars. That seems incredibly low and off for me so thought it must be because of the way I rated rather than strictly adherring to the above suggested system.
My personal system was based on the following..
5=masterpiece
4.5=terrific
4=great
3.5=good
3=slightly above typical/average
2.5=average
2=below average
1.5=poor
1=really bad
0.5=without value
|
|
|
Danny
Answered 8 years ago
|
|
So, if everyone seems to have different methods of ranking, then how is the system so good at making predictions?! Very cool, I think!
|
|
|
xflies
Answered 8 years ago
|
|
(shouldn't this thread be in the Movielens forum?)
So, if everyone seems to have different methods of ranking, then how is the system so good at making predictions?! Very cool, I think!
While I don't find the system to make good predictions, I can give you a insight into how it can account for people rating things differently.
How does a system like this handle people rating things differently? Even though the rating system says "5 stars = Must see" some people may be careful about labeling something a "Must See" while otheres (like me) happily labels something "Must See" if it was a fun experience to watch the movie. Movielens, assuming it is still based on a resnick predictor variation (as vaugely described in the grouplens paper), I'm sure it's been modified a lot since (any movielenser want to update me?), could roughly be seen to account for different rating behaviours as follows:
(1) Movielens calculates the average rating YOU give to movies. Probably this value is 3.0 or near. Suppose that you rate a lot of movies with 4 stars and 5 stars, then your average rating may be slightly higher than 3.0. If you are a careful about giving out high grades (few 5 stars) your average may be lower than 3.0.
(2) People that have watched and rated same movies as YOU. If they deviate from THEIR average rating on a movie YOU haven't seen, then so should YOU do.
Example:
You rate
Move A: 1
Move B: 4
Move C: 4
Your average rating is (1+4+4)/3 = 3
A friend rates:
Move A: 2
Move B: 5
Move C: 5
Move D: 4
Friends average (2+5+5+4)/4 = 4
You haven't seen D, but you and your friend have seen some same movies and have given similar grades to them, hence you will correlate with your friend (you'll correlate better with someone who rates the same as you ofcourse, but the issue here was that of making predictions). When predicting a grade for YOU on movie D the predictor starts out with your average grade, 3 stars in this case, and depending on what your friends thinks of it this grade will be adjusted.
So next we look at your friend, he has given D a 4 and his average rating is 4, your friend with similar movie taste doesn't deviate from his average rating, so neither should the prediction on D for you do. So the final prediction on D is 3 stars even though your friend gave it a 4 simply because you are more careful about high grades.
Note that this is just theory and a dumb example, wheter it works or not can only be tested practically.
The predictions I get I find not to be very good and have attributed it to the way I ranked my films rather than based on the suggested rankings listed in the upper right hand corner of 1 awful, 2 fairly bad, 3 ok, 4 will enjoy, 5 must see.
Fascinating, so if I were to construct a random recommender system that basically just predicts random ratings for movies. You'd still attribute the wrong predictions to yourself doing something wrong? It doesn't occur to you that the system simply might not be as precise as you want it to be? There is not call for assuming the system is perfect, it's experimental afaik.
|
|
|
PolarisDiB
Answered 8 years ago
|
|
All of this and more, I believe, is published on this site if you look for it. It's just lengthy and didactic.
I believe a focus of the way the predictions work is based on similar-minded people, not necessarily averages. If you rate a few movies very closely to how another person rates those same movies, then a movie he's rated but you haven't will be predicted to you to be close to his rating.
--PolarisDiB
|
|
|
movsom
Answered 8 years ago
|
|
Hmm, this is how i have understood how the algorithm work that MovieLens is build on (according to their papers on GroupLens), so if any MovieLens member out there think otherwise, please let me know. Basically you can divide it into two separate steps, similarity calculations and the prediction process:
1. For every user, calculate a similarity score between them based on how similar they have rated common movies.
2. For every user, predict ratings for movies he has not rated.
Some important things to observe is this:
-In step 1, when calculating similarity between user A and user B, only movies they both have in common are used, it doesn't matter if they have rated thousands of movies each, if they only have ten movies in common, then those ten is used for calculating the similarity score.
As a sidenote, think about this, how similar taste do you really have to another users if you only have 10 out of thousand movies in common?
-In step 2, when predicting a value for a movie to a user, the algorithm uses those users that have rated that movie and depending on how much similarity they have to the user they are going to predict for, the more they will have to say about the final prediction score. And it is here that the average thing plays an important role and that deserves some extra explanation.
Just look at two people that have posted in this thred; PolarisDiB and abintra.
For PolarisDiB a 3.5 means "I don't really care" and for abnitra 3.5 means "it's good".
You all see the problem here? Two different interpretation of the same grading score! If abintra would say to PolarisDiB that a movie was 3.5, then PolarisDiB would not go see it but if abnitra said "it's good", then PolarisDiB would consider to go and see it. This makes it impossible for users that have different rating scales to recommend movies to each other.
The solution is to look at each users average rating score, what's above it is what the user thinks is better then what's below it, no matter what the average rating score is.
So, lets look at xflies example where user B is to make a prediction for user A. First MovieLens calculates user A's average score to see where it lies and then it calculates user B's average score to see where it lies. And since the average score differ the algorithm makes some adjustments, called normalization of the rating scale, before it finally predict a value for movie D to user A. Since user B has rated movie D with an 4, which is what he thinks is average, the predicted value to user A should also be average, namely a 3, which is what user A's average is.
Suppose user B had rated movie D with 3.5, then the predicted value to A should have been 2.5.
Suppose user B had rated movie D with 4.5, then the predicted value to A should have been 3.5.
And so on...
You all see the pattern here? By using A's average and from that adding or subtracting whats deviate from user B's average you got an prediction that is adjusted according to user A's rating scale!
You can also say that the prediction is personal, it's based on your similarity score to the user and YOUR rating scale and thats why it is wrong to say what PolarisDiB says "a movie he's rated but you haven't will be predicted to you to be close to his rating".
And that answer Dannys question "if everyone seems to have different methods of ranking, then how is the system so good at making predictions?! " the system takes users different ratingscales under consideration when it makes it predictions.
It does that by normalizing the rating scale and it does so by using users average rating score.
|
|
|
Danny
Answered 8 years ago
|
|
Really insightful ideas and explanations - thanks! And, yes, I should have posted this in the movielens forum - oops. Oh well.
|
|
|
Sumytra1
Answered 8 years ago
|
|
I rate with my gut reaction and after 1455 films Movielens is a pretty accurate predictor. It hasn't predicted a 5 star for me yet but there are a few 4.5 stars. Most of the predictions fall in the 3.5 and 3.0 range which is accurate. I think most films are rather middle of the road.
|
|
|
bossywalker
Answered 7 years ago
|
|
I think that people should use the stick to the movielens rating system. It in the top right corner.
1 = Awful
2 = Fairly Bad
3 = It's OK
4 = Will Enjoy
5 = Must See
Personally 5 star movies are things such as Donnie Darko, the original Star Wars films. Yeah give them 5 if you absolutely love them but also if you think they are something that people must see. Like Power of One or Schindlers List
|
|
|
This question is closed to new answers.
|
