
APBRmetrics The statistical revolution will not be televised.

View previous topic :: View next topic 
Author 
Message 
jsill
Joined: 19 Aug 2009 Posts: 73

Posted: Wed Nov 04, 2009 3:41 pm Post subject: 


Schtevie:
Thanks very much for your feedback.
It's hard to say for sure how big the APM estimates should be for the best players, but let's suppose for the sake of the argument that you're right. Suppose that some allknowing being told us that the average APM for the top 10 players in the league ought to be 9 instead of what I have (which I think averages to about 5.5 for my 3year top 10 from RAPM, although I'm just eyeballing it).
Suppose you have the results of another APM one that doesn't use regularization which has an average magnitude for the top 10 that's "correct", i.e., around 9. That doesn't necessarily mean that this second APM analysis is more accurate than the one with the values which are "too small". The second analysis could have the right magnitude but still be off on an individual playerbyplayer basis to such an extent that it's not as accurate as RAPM.
A related point is the distinction between the true parameter magnitudes and the magnitudes which can be reliably estimated given a limited amount of highly noisy data and significant evolution over time of player abilities (what statisticians sometimes call nonstationarity). If we had 50,000 games each year, it might well be that the models would converge on larger magnitude APM values. We don't, though, and my experiments suggests that given the limited, noisy data that we do have, penalizing large APM values improves accuracy.
Regarding the plausibility of the magnitudes for the top players, it's worth remembering that the number is computed relative to a minutesweighted average APM player. When you hear "average NBA player" casually, you might tend to think of a simple average over everyone on an NBA roster, which would mean "average NBA player" might correspond roughly to a sixth or seventh man on an average team. However, since it's minutesweighted, that means the "average NBA player" is a good bit better than that. I haven't looked at the numbers carefully, but it probably roughly corresponds to the third or fourth best player on an average team in other words, a fairly decent player. That's why the majority of players have scores below zero.
So if the model says Chauncey Billups gets you 3 extra points in margin of victory (given that he plays 35 mpg) relative to an average NBA player, that's relative to a fairly decent player. That's about a 3 points per game boost relative to Andre Miller or Mike Bibby (just to take a couple of players for whom I get an RAPM close to 0).
Also, let's remember that 3 points a game in average margin of victory over an entire season is quite a lot. 3 points a game is the difference between Denver (+ 3.4 ppg) and Philadelphia (+ 0.1 ppg) last year. I realize that there's a Pythagorean model for translating margin of victory into expected wins, but from eyeballing it, it looks like a rough rule of thumb is that 1 ppg in margin of victory over a whole season maps to about 3 extra wins on average. So that's 9 extra wins if you replace a minutesweighted average NBA player which again, is not the second guy off the bench but a decent starter with Billups.
Finally, can you clarify what you mean by this?
Quote:  the straight APM gives a dramatically different result (that is consistent whether it uses one or more seasons 
Are you saying that the magnitude of the bestplayer APM estimates from standard regression seem to be the same whether it uses one or more seasons? I admit I haven't looked at that carefully, but my impression is that you tend to get more extreme values with 1 season of data than with multiple seasons. 

Back to top 


jsill
Joined: 19 Aug 2009 Posts: 73

Posted: Wed Nov 04, 2009 3:46 pm Post subject: 


Aaron:
Quote:  For our reference, have you compared your predictions to predictions using other approaches (e.g. PER, Win Score,...)? 
I have not looked at this yet. I need study some of these other systems in more detail. Does something like PER yield testable predictions on unseen data regarding the outcome of games? It's not obvious to me that it does,but maybe there's a way to extract that from PER.
I'll try to look into this some more. 

Back to top 


jsill
Joined: 19 Aug 2009 Posts: 73

Posted: Wed Nov 04, 2009 4:11 pm Post subject: 


deepak_e:
Quote: 
If you have the numbers readily available, could you publish the leaders in fast break points per game (teamwise, or even playerwise) over the last several years? I can't find that information elsewhere. 
I don't have the numbers readily available, but it might not be too hard for me to generate them. As you might have seen, on my site I have some results regarding how likely fast break points are depending on who grabbed the defensive rebound, so I might be able to tweak that code in order to get the info you wanted. I'll try to get to it, but ping me again in a week or two if I haven't gotten to it yet. 

Back to top 


deepak
Joined: 26 Apr 2006 Posts: 664

Posted: Wed Nov 04, 2009 4:12 pm Post subject: 


Thanks for this work.
I'd be very interested to see the APMs split into offensive and defensive impact. Also it would be useful to have 1year APMs in each of the last 3 seasons. If there is good year to year consistency (for vets, in particular) compared to other APM methods, it would be good to show. 

Back to top 


gabefarkas
Joined: 31 Dec 2004 Posts: 1311 Location: Durham, NC

Posted: Wed Nov 04, 2009 4:54 pm Post subject: 


jsill wrote:  DSMok1: I do yet not have the standard errors for each player. Because I'm using regularization, this becomes more complicated than getting standard errors in a classic regression. In theory, we should be able to get an "a posteriori" distribution for the parameters which is a consequence of combining the a prior distribution from which the regularization term stems with the data. I need to do some research on how to do this, though. 
What software package are you using to perform the Ridge Regression? 

Back to top 


schtevie
Joined: 18 Apr 2005 Posts: 400

Posted: Wed Nov 04, 2009 5:11 pm Post subject: 


jsill wrote:  It's hard to say for sure how big the APM estimates should be for the best players, but let's suppose for the sake of the argument that you're right. Suppose that some allknowing being told us that the average APM for the top 10 players in the league ought to be 9 instead of what I have (which I think averages to about 5.5 for my 3year top 10 from RAPM, although I'm just eyeballing it).
Suppose you have the results of another APM one that doesn't use regularization which has an average magnitude for the top 10 that's "correct", i.e., around 9. That doesn't necessarily mean that this second APM analysis is more accurate than the one with the values which are "too small". The second analysis could have the right magnitude but still be off on an individual playerbyplayer basis to such an extent that it's not as accurate as RAPM. 
Though I agree it is hard to say for sure exactly how big the APM estimates should be for particular players, I don't think it is true that one cannot get a sense of the general value of the best players. Let's stipulate for current purposes that the 1/3 offense, 2/3 defense (that shown in Stephen Ilardi's recent "stabilized" result) is the appropriate credit breakdown for KG. By your numbers, this says that a KG inspired defense is only 3 points better on defense than were he replaced by an average player. This simply doesn't sit right.
And given this, I am not sure what concept of accuracy we should be using here. Why should we care about going from an Rsquared of nine to sixteen, if as a result we lose what I think is a fundamental reality of the league: that there are star players who drive outcomes on the court.
Let me expand on the concept with additional, supporting evidence. If, in fact, talent is relatively "compressed", I would expect that the record of the NBA would show a lot more instances of teams that cobbled together lesser talent to win championships. This is almost an empty set. Additionally, I have the impression that despite the imperfections in the labor market, that NBA salaries would be much less unequal.
jsill wrote: 
Finally, can you clarify what you mean by this?
Quote:  the straight APM gives a dramatically different result (that is consistent whether it uses one or more seasons 
Are you saying that the magnitude of the bestplayer APM estimates from standard regression seem to be the same whether it uses one or more seasons? I admit I haven't looked at that carefully, but my impression is that you tend to get more extreme values with 1 season of data than with multiple seasons. 
I wasn't trying to say that oneyear APM delivers the same results as multipleyear APM. Rather, I was specifically saying that they consistently show greater range and "spread" than yours.
With this is mind, could you please comment on how one should interpret the distinct differences between multiyear, standard APM and your results?
Thanks. 

Back to top 


DSMok1
Joined: 05 Aug 2009 Posts: 560 Location: Where the wind comes sweeping down the plains

Posted: Wed Nov 04, 2009 5:31 pm Post subject: 


schtevieyou're saying the same things as I was... basically, it appears that the reason everything is compressed is that all of the players are being "regressed" toward 0, which is why this is a better projection of the future/out of sample numbers. It should be far better, I think, if all players were not being regressed toward 0, but some better Bayesian prior used. 

Back to top 


gabefarkas
Joined: 31 Dec 2004 Posts: 1311 Location: Durham, NC

Posted: Thu Nov 05, 2009 1:36 pm Post subject: 


schtevie wrote:  jsill wrote: 
Finally, can you clarify what you mean by this?
Quote:  the straight APM gives a dramatically different result (that is consistent whether it uses one or more seasons 
Are you saying that the magnitude of the bestplayer APM estimates from standard regression seem to be the same whether it uses one or more seasons? I admit I haven't looked at that carefully, but my impression is that you tend to get more extreme values with 1 season of data than with multiple seasons. 
I wasn't trying to say that oneyear APM delivers the same results as multipleyear APM. Rather, I was specifically saying that they consistently show greater range and "spread" than yours.  Is the coefficient of variation what you're looking for here? 

Back to top 


gabefarkas
Joined: 31 Dec 2004 Posts: 1311 Location: Durham, NC

Posted: Thu Nov 05, 2009 1:37 pm Post subject: 


DSMok1 wrote:  schtevieyou're saying the same things as I was... basically, it appears that the reason everything is compressed is that all of the players are being "regressed" toward 0, which is why this is a better projection of the future/out of sample numbers. It should be far better, I think, if all players were not being regressed toward 0, but some better Bayesian prior used.  What would be a better prior for APM than 0? For every point scored for, there's a point scored against. 

Back to top 


Crow
Joined: 20 Jan 2009 Posts: 773

Posted: Thu Nov 05, 2009 2:10 pm Post subject: 


jsill wrote: 
Regarding the plausibility of the magnitudes for the top players, it's worth remembering that the number is computed relative to a minutesweighted average APM player. 
Yes and the magnitude is also an average and would assume that they are at that level every singe game. Relax this assumption and for example Kevin Garnett could be +5 points on defense 7 out of 10 times on the court and a bit off 3 times with say a 0, 2 and 3 and get that average of +3 on defense.
On defensive efficiency last season the Celtics were about 6 points better than league average. Garnett could be half that, in his minutes, most nights. I.e. playing at +5 on most of his good nights prorated to 6075% of the minutes would get you to a 3 point impact or so. That doesn't sound outlandish.
Still these ratings are per 48 minutes maybe the magnitude should be in between the previous Adjusted ratings and the first regularized version. Maybe variance from the mean should not be reduced as much when the Adjusted +/ consistent with a player's SPM or team point differential so stars are still estimated as having higher perhaps truer impact.
But on defense preventing 1 less offensive rebound or getting a full one more steal per game relative to a minutesweighted average APM player is pretty rare. Even preventing one less basket on average every game is substantial. A +3 guy probably has to do several of these things on average every game or even more most of the time to make up for the occasional off night.
Last edited by Crow on Thu Nov 05, 2009 3:19 pm; edited 4 times in total 

Back to top 


DLew
Joined: 13 Nov 2006 Posts: 222

Posted: Thu Nov 05, 2009 2:37 pm Post subject: 


Given that a point per game in differential is worth roughly 2.5 wins, then a player who is a +8 per 100 possesions is worth about +6 per 40 minutes which is what most stars play in a game. So that's 15 wins better than average and roughly 25 wins better than replacement. To me that does not seem unreasonably low for the best player in the league and that seems to be what the numbers suggest.
One possible check might be, and I could be wrong on my theory here but it sounds right, to look at his home court advantage coefficient and see how it compares to coefficient found in other places. If the numbers are too strongly biased towards zero then perhaps his home court advantage coefficient will show this. If it has a similar magnitude as reported else where then this would suggest that little or no signal has been lost, just noise. 

Back to top 


DSMok1
Joined: 05 Aug 2009 Posts: 560 Location: Where the wind comes sweeping down the plains

Posted: Thu Nov 05, 2009 2:42 pm Post subject: 


gabefarkas wrote:  DSMok1 wrote:  schtevieyou're saying the same things as I was... basically, it appears that the reason everything is compressed is that all of the players are being "regressed" toward 0, which is why this is a better projection of the future/out of sample numbers. It should be far better, I think, if all players were not being regressed toward 0, but some better Bayesian prior used.  What would be a better prior for APM than 0? For every point scored for, there's a point scored against. 
Unfortunately, the math involved here is beyond me. It may not be possible in the framework jsill is using to apply any prior other than 0.
That said, the choice of 0 as the prior is not particularly accurate, and influences the results stronglyfor example, all players with few minutes have a RAPM of about 0.
Recall that NBA players are drawn from the tail end of a normal distribution (of the general population's basketball skill/talent). What this means is that player skill is not normally distributedrather, there are ~an infinite number of very bad players and very few excellent ones. This is why regression to the mean is wrong.... the only reason it works is because IF a player is playing, there is a reasonusually their true talent has not been fully exposed yet (if they are on the bad end). To reflect a more correct distribution, I would propose using some sort of prior based on minutes per game. This would avoid assuming all players with few minutes per game are close to 0 APM....when actually, they are most likely worse than the players with more minutes per game (otherwise they would be playing).
Whether this is possible for jsill's framework I don't know. 

Back to top 


Crow
Joined: 20 Jan 2009 Posts: 773

Posted: Thu Nov 05, 2009 3:47 pm Post subject: 


Minutes alone is useful but isn't enough to guide that well on player quality for big minute players on bad teams or a lot of bench players on good teams; hence I suggested minutes somewhat adjusted by team win performance. 

Back to top 


mtamada
Joined: 28 Jan 2005 Posts: 375

Posted: Fri Nov 06, 2009 3:07 am Post subject: 


Excellent stuff. The ridge regression seems to improve things substantially. And if I understand your crossvalidation correctly, you're basically using the Retrodiction technique (meaning, using the actual player minutes for the OutOfSample data)  but rather than trying to retrodict the entire 200809 season, you're looking at the last few months' of the 200809 season?
Looks like really good work. Three ideas occur to me as possible next steps.
It's too bad the the optimal cutoff was M = 1200 minutes. Maybe the Generalized Ridge Regression technique that tpryan mentioned could be useful here ... larger lambdas for players with small minutes played.
So the optimal value of D was 0.25, i.e. pretty small, and no wonder the oneyear results don't differ that much from the multiyear results. Several people have mentioned the notion of adding career curves to the model, i.e. players at the peak (or more accurately plateau) of their career would have relatively large values of D, whereas young players or clearly declining old players would have small values of D  or might even have their nextseason's parameters extrapolated, rather than directly based on current and recent seasons with no attempt to predict possible changes for next season.
I also like RyanP's random effects hierachical regression ... he seemed to get an ever larger reduction in RMSE. I don't know if someone has worked out the estimator (or if it can even be done), but how about combining the two: a regularized or ridge regression of a random effects hierarchical model?
Oh a fourth thought: multicollinearity reduces our ability to get good estimates of the effects of any one predictor variable (or player), due to their high correlation. But even in a collinear situation, one can get good estimates of the impact of a combination of variables.
An intuitive example: if Stockton and Malone always started and subbed out at the same time, and never played without each other, obviously we'd never be able to tell what Stockton's APM was, distinct from Malone's, and viceversa. But what we could do is estimate the APM of the StocktonMalone duo.
Which fits in nicely with the notion of looking at lineups and lineup effects that has been floating around. Maybe we can't estimate whether Stockton was a better PG than Isiah Thomas, because Stockton's value is inextricably tied up with Malone's  but if Jerry Sloan never plays Stockton without Malone anyway, that's not such a big issue, from the perspective of team evaluation, predicting wonloss records, etc.
That fourth thought was where my thoughts had been drifting the last several weeks, but RyanP's random effects model and JoeS's regularized regression results make me think there could still be new and useful stuff to extract from the good ol' Adjust PlusMinus model, suitably upgraded with more sophisticated estimation techniques. 

Back to top 


Crow
Joined: 20 Jan 2009 Posts: 773

Posted: Fri Nov 06, 2009 4:44 am Post subject: 


Another idea could be to run "1 on 1 games" between Mr. Laker Lineup Factors playing against the counterparts of Mr. Celtic Lineup Factors and all the other matchups. That would get Adjusted Lineup Factors. Maybe that would help predict the future even better?
You could go with the straight 4 Factors on offense and defense but I might divide it up as 3 pt shooting, non3 point shooting, free throw shooting, turnovers and rebounding to get 5 to match the 5 when you use players for the heck of it and to add a bit more detailed breakout of scoring activity. Run the regression just looking at the play by play for a single of these factors at a time. Turnovers and offensive rebound findings would be converted into points based on regression findings or value of possession.
This would help illuminate which Factor is influencing a lineup most toward winning or losing their matchups. And give you better measurements / more obvious clues about what you might try to fix about them than the raw data itself since the quality of lineup matchup and quality of Factor matchup among lineups will probably vary a lot and make surface comparisons often misleading.
Last edited by Crow on Fri Nov 06, 2009 2:01 pm; edited 1 time in total 

Back to top 




You cannot post new topics in this forum You cannot reply to topics in this forum You cannot edit your posts in this forum You cannot delete your posts in this forum You cannot vote in polls in this forum

Powered by phpBB © 2001, 2005 phpBB Group
