APBRmetrics Forum Index APBRmetrics
The statistical revolution will not be televised.
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 

A new idea on testing the accuracy of a player rating system

 
Post new topic   Reply to topic    APBRmetrics Forum Index -> General discussion
View previous topic :: View next topic  
Author Message
Statman



Joined: 20 Feb 2005
Posts: 94

PostPosted: Sun Jan 13, 2008 8:19 am    Post subject: A new idea on testing the accuracy of a player rating system Reply with quote

This just occurred to me yesterday when I was trying to figure out how close my college player rating system would have predicted an outcome AFTER the fact - after we already know the exact minutes played of every player in a game on each team. I did a one game test of my system (Arizona vs Houston) - and actually ended up with almost the exact ratio of Arizona's rating to Houston's to the actual ratio of Arizona's points to Houston's points in the game (pure luck it happened the first game I try). Here's the thread where I mention it - it's the 9th post down where I do this "test":

http://pointguardu.com/cats/showthread.php?t=22937

Anyway - I realized - all this debate about the accuracy of different systems (Usually PER vs. Berri's Winscore or whatever it's called) in evaluating players - couldn't one retroactively test these rating systems on a game to game basis on an entire season, taking the standard deviation of the difference in ratios of predicted outcomes to actual outcomes? The system that has the smallest standard deviation from game to game would in essence be the most "accurate".

For example - for one game, Team A beats team B 110 to 100, so the TRUE ratio of points scored for the game ends up 1.10. Now - if you take the PER (final PER for the season mind you) of every player that played and multiplied it by the minutes the player played, and summed each team - the goal in a perfect world (if PER were 100% accurate) would be that if you divided the sum of Team A by the sum of team B, you'd get 1.10. If it actually ends up, say, 1.25, then the prediction deviated by 0.15.

Does that make sense? Has anyone tried this on a large scale? Wouldn't this maybe be the best way to "test" these systems? The closer a system gets to mimicking actual results when going BACK over the season - then the better that system reflects true individual player performance.

Is there a better way to grade a rating system than trying to retroactively test final season results with past game outcomes?
_________________
Statman

http://wildcatsportsreport.com/forums/blog.php?u=44
Back to top
View user's profile Send private message Send e-mail
Ryoga Hibiki



Joined: 06 Oct 2007
Posts: 13

PostPosted: Sun Jan 13, 2008 11:36 am    Post subject: Re: A new idea on testing the accuracy of a player rating sy Reply with quote

Statman wrote:

For example - for one game, Team A beats team B 110 to 100, so the TRUE ratio of points scored for the game ends up 1.10. Now - if you take the PER (final PER for the season mind you) of every player that played and multiplied it by the minutes the player played, and summed each team - the goal in a perfect world (if PER were 100% accurate) would be that if you divided the sum of Team A by the sum of team B, you'd get 1.10. If it actually ends up, say, 1.25, then the prediction deviated by 0.15.

I'm not sure you're measuring how accurate those rating systems are.
Once you sum up all together it's very likely going to work because the tough part is to split the credit for every change in the scoreboard within the team => most of the errors are going to be corrected once putting everything together again!
I really don't see how to test them at the player level in a quantitative way, actually.
Back to top
View user's profile Send private message
Harold Almonte



Joined: 04 Aug 2006
Posts: 373

PostPosted: Sun Jan 13, 2008 3:43 pm    Post subject: Reply with quote

I'm not sure if I understood. A player's performance against some team or some games not necesarily is correlated with his average season performance. If such prediction at the individual game level could be possible, Las Vegas would know it and betting would dissapear. The most closed correlation that I think can be done is with wins (no margins).

I have not the level to revise Rosembaum's tests, but my common sense says me that if you compare the ratings with 4 (eight really) factors and the W% of each action (scoring, ballhandling, rebounding and FTs/foulings), the PER's lack of points allowed will produce problems in the def. eFG% prediction, but the win% relationship eFG% + R% supossedly should produce more problems in WP, but is not the case. I think, and I'm not sure either, is because altough rebounds are overrated, the importance of rebounding rating (20%) is reduced; scoring is underrated but the importance of efficiency (40%) is rised, and probably is why WP remains relatively predictive. Somebody please correct it if that's possible.
Back to top
View user's profile Send private message
Statman



Joined: 20 Feb 2005
Posts: 94

PostPosted: Sun Jan 13, 2008 5:30 pm    Post subject: Re: A new idea on testing the accuracy of a player rating sy Reply with quote

Ryoga Hibiki wrote:
Statman wrote:

For example - for one game, Team A beats team B 110 to 100, so the TRUE ratio of points scored for the game ends up 1.10. Now - if you take the PER (final PER for the season mind you) of every player that played and multiplied it by the minutes the player played, and summed each team - the goal in a perfect world (if PER were 100% accurate) would be that if you divided the sum of Team A by the sum of team B, you'd get 1.10. If it actually ends up, say, 1.25, then the prediction deviated by 0.15.

I'm not sure you're measuring how accurate those rating systems are.
Once you sum up all together it's very likely going to work because the tough part is to split the credit for every change in the scoreboard within the team => most of the errors are going to be corrected once putting everything together again!
I really don't see how to test them at the player level in a quantitative way, actually.


Not if you sum the absolute values of the deviations of each game.

But yes - overall, a system should sum close fairly close to zero (if you are just summing the differences, some games negative, some positive) - or it's probably a bit off.
_________________
Statman

http://wildcatsportsreport.com/forums/blog.php?u=44
Back to top
View user's profile Send private message Send e-mail
Statman



Joined: 20 Feb 2005
Posts: 94

PostPosted: Sun Jan 13, 2008 5:57 pm    Post subject: Reply with quote

Harold Almonte wrote:
I'm not sure if I understood. A player's performance against some team or some games not necesarily is correlated with his average season performance. If such prediction at the individual game level could be possible, Las Vegas would know it and betting would dissapear. The most closed correlation that I think can be done is with wins (no margins).

I have not the level to revise Rosembaum's tests, but my common sense says me that if you compare the ratings with 4 (eight really) factors and the W% of each action (scoring, ballhandling, rebounding and FTs/foulings), the PER's lack of points allowed will produce problems in the def. eFG% prediction, but the win% relationship eFG% + R% supossedly should produce more problems in WP, but is not the case. I think, and I'm not sure either, is because altough rebounds are overrated, the importance of rebounding rating (20%) is reduced; scoring is underrated but the importance of efficiency (40%) is rised, and probably is why WP remains relatively predictive. Somebody please correct it if that's possible.


As for the first part - yes, this is true. However, we aren't looking at the players as individuals, but summing the players every game in proportion to their minutes. Of course every game will have variance (many games large variance) - even if the rating was quite "accurate" - but over the course of a whole season, the absolute value of the sums of the deviations (I forgot to mention before absolute value) still would be lower for the "more accurate" systems I would think.

I think some may be confused a little by my one game example - one game being "accurate" obviously doesn't tell us that much (although I was happy the one game I tested in mine wasn't WAY off) - but over thousands of games - I would think it could tell us alot when comparing one system to another.

If I were a programmer - I could probably have all the data pulled from every game box score (all I need is final score and individual player minues). From here one could test PER, Berri's, maybe Mike G's, etc. against each other (IF I had all their final player ratings) - see whose system seems to best on average to retroactively "predict" the past results.

As for your second point - I'm sure what you are getting at. I agree PERs biggest problem would probably lie most in not measuring points allowed, and Berris problem would lie probably elsewhere (overvaluing rebounding, undervaluing usage?) - I'm just curious which would be more "off" from real life results.

The system I use does use opposing points as part of a factor, which is why it's a ratio (100 being average): players summed results (scaled to actual team points) divided by opponent's points. I also take into account playing time in the final result as kinda a "Bruce Bowen" adjustment (low rated guys who play alot on good teams probably are doing more things not reflected in the box score, and vice versa). For college it's trickier - because of the inclusion of SoS, which could be skipped in the NBA without too much "error" (though there would be some, ie the East being much weaker than the West in a given season).
_________________
Statman

http://wildcatsportsreport.com/forums/blog.php?u=44
Back to top
View user's profile Send private message Send e-mail
Mike G



Joined: 14 Jan 2005
Posts: 1712
Location: Delphi, Indiana

PostPosted: Tue Jan 15, 2008 10:32 am    Post subject: Reply with quote

I would expect a system that (like mine) matches point-differential to player rates to yield closer predictions than one which doesn't. However, credit to individual players is still not adequate. If Bowen doesn't play, do the Spurs suffer a little, or do they suffer a lot?

Last year, team expected wins (based on my eWins-producing stats) averaged an 'error' of 2.5 wins from pythagorean-expected. But actual wins 'erred' by an average of 2.7 from pyth. Can you beat that?
_________________
40% of all statistics are wrong.
Back to top
View user's profile Send private message Send e-mail
gabefarkas



Joined: 31 Dec 2004
Posts: 958
Location: Durham, NC

PostPosted: Tue Jan 15, 2008 6:25 pm    Post subject: Re: A new idea on testing the accuracy of a player rating sy Reply with quote

Statman wrote:
For example - for one game, Team A beats team B 110 to 100, so the TRUE ratio of points scored for the game ends up 1.10. Now - if you take the PER (final PER for the season mind you) of every player that played and multiplied it by the minutes the player played, and summed each team - the goal in a perfect world (if PER were 100% accurate) would be that if you divided the sum of Team A by the sum of team B, you'd get 1.10. If it actually ends up, say, 1.25, then the prediction deviated by 0.15.

Does that make sense? Has anyone tried this on a large scale? Wouldn't this maybe be the best way to "test" these systems? The closer a system gets to mimicking actual results when going BACK over the season - then the better that system reflects true individual player performance.

I've tossed around the idea of something like that too. It seems incredibly monumental though.

One question: are you referring to the PERs over the entire season, or the PERs calculated based on stats only up to that game?
Back to top
View user's profile Send private message Send e-mail AIM Address
Statman



Joined: 20 Feb 2005
Posts: 94

PostPosted: Tue Jan 15, 2008 7:05 pm    Post subject: Re: A new idea on testing the accuracy of a player rating sy Reply with quote

[quote="gabefarkas"]
Statman wrote:

One question: are you referring to the PERs over the entire season, or the PERs calculated based on stats only up to that game?


I was thinking whole season PER.

I do think Mike G's would probably be more "accurate" in this type of testing than John H's or Berri's. Mine (which I haven't worked on recently since I've been doing alot of college stuff) would probably be also, since Mike G & I have a number of similarities in our approach (I make sure the linear weights totals exactly match team points totals, and I also use opposition scoring as a factor).
_________________
Statman

http://wildcatsportsreport.com/forums/blog.php?u=44
Back to top
View user's profile Send private message Send e-mail
BadgerCane



Joined: 05 Sep 2007
Posts: 18
Location: University of Miami-Florida

PostPosted: Tue Jan 15, 2008 11:28 pm    Post subject: Reply with quote

This idea is interesting, but does not work for a few reasons. For instance, if our only measure of player productivity is points, then if we add up all the players with that measure then the team whose players add up to a higher point total will win. Similarly, if we jut use Offensive Rating as a measure, and add all of that up, the team with the higher offensive rating will win. If we "believe what we say we believe," that shot creation has value, that efficiency goes down as usage goes up, then such a system does not work. The difficulty in basketball analysis is not figuring out what leads to wins. The difficulty is in figuring out how to apportion credit for these stats among players.
Back to top
View user's profile Send private message Send e-mail AIM Address
Statman



Joined: 20 Feb 2005
Posts: 94

PostPosted: Tue Jan 15, 2008 11:49 pm    Post subject: Reply with quote

BadgerCane wrote:
This idea is interesting, but does not work for a few reasons. For instance, if our only measure of player productivity is points, then if we add up all the players with that measure then the team whose players add up to a higher point total will win. Similarly, if we jut use Offensive Rating as a measure, and add all of that up, the team with the higher offensive rating will win. If we "believe what we say we believe," that shot creation has value, that efficiency goes down as usage goes up, then such a system does not work. The difficulty in basketball analysis is not figuring out what leads to wins. The difficulty is in figuring out how to apportion credit for these stats among players.


I'm really not sure what you are saying here.

IF you are saying that there isn't a metric that can accurately reflect true player impact on a team - that may be somewhat true.

However, I do think it is possible to have a metric that does a fairly solid job of measuring player performance. While not perfect by any means, or necessarily fair to every type of player (ie Bruce Bowen), I would think it is possible to put together something that has some substance.

I was wondering HOW to figure out if a metric seems to hold water - and this idea was the best I've come up with. IF we are going to appoint some type of number to a player's performance for the season (say PER), then I would think if we were to go back and test the results - we shouldn't have a huge deviation from the ACTUAL results on average. What would be considered too big a deviation - I really don't know, unless we tested many metrics and saw what kind of results we got. I don't think it would take long to figure out which metrics are obviously WAY off, and which aren't too bad.

But, there will obviously NEVER be a perfect metric to accurately measure true player performance.
_________________
Statman

http://wildcatsportsreport.com/forums/blog.php?u=44
Back to top
View user's profile Send private message Send e-mail
BadgerCane



Joined: 05 Sep 2007
Posts: 18
Location: University of Miami-Florida

PostPosted: Wed Jan 16, 2008 12:07 am    Post subject: Reply with quote

Well, I think that Dave Berri accomplished this with Wages of WIns. He took the end results of every team in every year since they kept all the stats (I think '76?) and found out what team totals had the highest correlation to winning. (You can download pretty much every result ever from www.basketballdatabase.com.) So, this would seem to answer the "whose stats add up to most wins" question. This is where the big debates between Wages and BOP come in. In Wages, the rebounder pretty much gets credit for the forced miss and the rebound. Also, Wages gives no value to shot creation. This isn't to say that the "what stats add up to wins" question is not worth asking. I do not want to seem like I am dismissing the idea. Nor do I want to say that I do not find value in a lot of what Wages has to say about basketball. I'm just trying to question the soundness of using a system of measurement, "what adds up to the most wins," that actually provides one of the main forks in the road to agreement between different schools of basketball thought.
Back to top
View user's profile Send private message Send e-mail AIM Address
Statman



Joined: 20 Feb 2005
Posts: 94

PostPosted: Wed Jan 16, 2008 12:22 am    Post subject: Reply with quote

BadgerCane wrote:
Well, I think that Dave Berri accomplished this with Wages of WIns. He took the end results of every team in every year since they kept all the stats (I think '76?) and found out what team totals had the highest correlation to winning. (You can download pretty much every result ever from www.basketballdatabase.com.) So, this would seem to answer the "whose stats add up to most wins" question. This is where the big debates between Wages and BOP come in. In Wages, the rebounder pretty much gets credit for the forced miss and the rebound. Also, Wages gives no value to shot creation. This isn't to say that the "what stats add up to wins" question is not worth asking. I do not want to seem like I am dismissing the idea. Nor do I want to say that I do not find value in a lot of what Wages has to say about basketball. I'm just trying to question the soundness of using a system of measurement, "what adds up to the most wins," that actually provides one of the main forks in the road to agreement between different schools of basketball thought.


He did this at the team level - that is a different ball of wax from what I am saying.

Did he later, AFTER he created his ratings and got his results for a given season, go back and test these INDIVIDUAL results against the actual results? If he did, what was his standard deviation in an average NBA game (of the summed individual results) from actual results? Did he test this deviation against against individual factors on their own (like, say, the standard deviation if one used only scoring rate, or rebound rate, etc)? Did he make sure his metric had a lower deviation than any others he could test?

I'm guessing he didn't.
_________________
Statman

http://wildcatsportsreport.com/forums/blog.php?u=44
Back to top
View user's profile Send private message Send e-mail
Harold Almonte



Joined: 04 Aug 2006
Posts: 373

PostPosted: Wed Jan 16, 2008 8:14 am    Post subject: Reply with quote

They have the WScore metric at the individual game. I think its standard deviation would be similar to any other one without team adjusts, except for the out of scale in rebounding (probably neither of them is right at the scoring efficiency).
The basketball win laws were already discovered in 4 (eight) factors, and every metric is obligued to fit with these proportions and the main of all (the zero sum approach at every action of the game). What rest in metrics is the weight of skills (shot creation, ballhandling, rebounding skills, etc.), a secondary thing. And the usage worth (linked with the weight of skills): why players are above and under average in attempts (defensive attempts too) at every factor, and what is the worth of that. Another thing would be the quality of play time and how that change the weight of stats. And finally decission makings (linked with usage).
Back to top
View user's profile Send private message
Display posts from previous:   
Post new topic   Reply to topic    APBRmetrics Forum Index -> General discussion All times are GMT - 5 Hours
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum


Powered by phpBB © 2001, 2005 phpBB Group