And yes, WINVAL and my adjusted plus/minus ratings do differ (I think Aaron would tell you that his metric is not yet a full adjusted plus/minus measure), but that is only because we are describing different things, i.e. we differ in how much we value clutch/non-clutch time and/or we differ in how we want to handle changes over time in player value. .
Wow, this is a nice long, detailed thread. I'm having trouble keeping up with the flurry of posts. Dan, you definitely characterized correctly that I'm still working on having a coherent, fully adjusted plus/minus rating.
I'm hoping to have the same data I have available for 2005-2006 up for 2006-2007 ASAP, then I'll start to release some of the building blocks of a "back-to-basics" adjusted +/- that doesn't use regression. I think there is not enough data generated in a single season to make the adjustments as I had originally planned, but I still think there are adjustments that can be made.
As far as some of the philosophy in this thread, I agree with what I think is the general theme that even if adjusted +/- correctly valued everyone's contribution on a unit in one year(say, 2005-2006), it would still be unlikely to predict with 100% accuracy what would happen if you randomly chose 5 players to make up a unit the next year(say, this season). There's still intangibles of chemistry that can't be predicted.
Joined: 03 Jan 2005 Posts: 413 Location: Greensboro, North Carolina
Posted: Fri Nov 17, 2006 2:35 pm Post subject:
Dave Berri has a very nice discussion of PER, using a "model" to construct player ratings, and the use (or lack thereof) of statistical analysis in NBA decision-making.
Looking at the specific weights Hollinger chooses we see another problem. In discussing the NBA Efficiency metric – which the NBA presents at its website – I argued that this measure fails to penalize inefficient shooting. The regression of wins on offensive and defensive efficiency reveals that shooting efficiency impacts outcomes in basketball. The ball does indeed have to go through the hoop for a team to be successful.
The same critique offered for NBA Efficiency also applies to Hollinger’s PERs, except the problem is even worse. Hollinger argues that each two point field goal made is worth about 1.65 points. A three point field goal made is worth 2.65 points. A missed field goal, though, costs a team 0.72 points.
Given these values, with a bit of math we can show that a player will break even on his two point field goal attempts if he hits on 30.4% of these shots. On three pointers the break-even point is 21.4%. If a player exceeds these thresholds, and virtually every NBA played does so with respect to two-point shots, the more he shoots the higher his value in PERs. So a player can be an inefficient scorer and simply inflate his value by taking a large number of shots.
Quote:
Less than 15% of wins in the NBA are explained by payroll. Regressions are nice, but not always understood by everyone. So to further illustrate the lack of association between pay and wins I took another approach. Specifically I ranked the teams in the NBA last year in terms of payroll and then divided this ranking into five equal segments. The results revealed that the teams in the top 20% spent an average of about $78 million on players and won –on average – 35.7 games. The next 20% spent $61 million and won 42.5 games. In the middle we see teams that spent only $54 million and won 39.7 games. When we look at the last two groupings – the teams that spent the least – we see clearly the very weak link between pay and wins in basketball. The 20% of teams ranked just below the middle in payroll won 47.7 games while spending $47 million on players. And the teams at the very bottom of the payroll rankings spent less than $38 million on its players and won 39.5 games. Yes, the teams at the bottom spent less than half what the teams spent at the top and actually won more games.
Okay, pay and wins do not have a strong link. What does this tell us about player evaluation? In football payroll explains less than 5% of wins. But in football we also see very little consistency in player performance. So decision-makers cannot easily know how to spend money to ensure success in the future. A similar problem – though to a lesser extent – exists in baseball. In basketball, though, players are much more consistent across time. The correlation between a player’s per-minute Win Score this season and last season is 0.84. As we detail in The Wages of Wins, the consistency we observe in basketball exceeds what we observe in either baseball or football. Despite this consistency, though, payroll is still not strongly linked to wins. In sum, decision-makers have a greater ability to predict the future in the NBA, yet the payroll-wins relationship still remains very weak.
When we look at what determines salary we see the problem. The primary player characteristic that dictates wages in the NBA is scoring. Shooting efficiency, rebounds, turnovers, and steals – factors that all impact outcomes – are not strongly linked to player pay. Given this evidence, we think players are evaluated incorrectly in the NBA. Too much emphasis is placed on scoring, and not enough on all the other factors that impact outcomes.
I would like to reiteratre that despite my many criticisms of Wages of Wins, there are a lot of good ideas in the book and at the blog.
Joined: 03 Jan 2005 Posts: 460 Location: Washington, DC
Posted: Fri Nov 17, 2006 3:12 pm Post subject:
Berri & company have raised an interesting point, and I'm kinda torn on it. Yes, shooting efficiency is extremely important -- there's no question. But, someone has to take the shots. Does treating all shot attempts as negative (which I think their metric does) reflect reality, though?
Like I say, I'm torn on the subject. I know that shooting efficiency is valuable, but at the same time, does it really hurt a team when Player A misses and his team gets the offensive rebound?
Dan, didn't you have a number showing a very low shooting percentage as a sort of benchmark for positive adjusted +/-? _________________ If you can't explain it simply, you don't understand it well enough.
Joined: 03 Jan 2005 Posts: 413 Location: Greensboro, North Carolina
Posted: Fri Nov 17, 2006 3:37 pm Post subject:
The problem is that Wins Produced doesn't really use regression to answer this question. It assumes that the average points per possession throughout the league tells us all we need to know about the breakeven shooting percentage for the marginal shot.
It assumes players convert shots on the margin at the same rate as their average shot. And so if we remove those players who shoot a lot (and draw a lot of the focus of the defense), the shooting percentages of remaining players will not change. Wins Produced is very similar to Bob Chaikin's simulator along those lines. Coming from a sports economics literature dominated by baseball, it is very natural to make this assumption, since this issue does not come up in a sport where every player gets his turn at bat.
When I look at this issue empirically using adjusted plus/minus statistics, I tend to find that the breakeven shooting percentage is somewhere between what PER/NBA Efficiency uses and what Wins Produced uses. Wins Produced rightly addresses a problem with PER/NBA Efficiency, but in my opinion it so overcorrects that the resulting metric does a poor job explaining how players impact winning.
Last edited by Dan Rosenbaum on Fri Nov 17, 2006 5:09 pm; edited 2 times in total
Joined: 13 Oct 2005 Posts: 183 Location: Atlanta, GA
Posted: Fri Nov 17, 2006 4:06 pm Post subject:
I found this to be one of Berri's most well-reasoned posts at the WoW Journal. I've said it before -- PER doesn't have the "power of language", because it lacks units. You can say that it's kind of like Points Produced per 40 minutes, but it really isn't, because it mixes apples and oranges (defensive and offensive stats) right and left, the weights are sometimes totally subjective (Why is an assist worth .67 "points"? And frankly, saying that, "on an assist, the passer does one thing and the scorer does two things, so the passer must get 1/3 credit," just doesn't cut it), the league average is forced to be 15, etc. So, yeah, Berri has some legit gripes about NBAEff and PER, because Wins Produced is at least derived pseudo-scientifically.
If he's lurking, I wonder how John would respond to Berri's post (probably wishful thinking on my part, but still)...
Joined: 13 Jan 2005 Posts: 168 Location: Iowa City
Posted: Fri Nov 17, 2006 5:06 pm Post subject:
Dan Rosenbaum wrote:
The problem is that Wins Produced doesn't really use regression to answer this question. It assumes that the average points per possession throughout the league tells us all we need to know about the breakeven shooting percentage for the marginal shot.
It assumes that a player converts shots on the margin at the same rate as their average shot. And so if we remove those players who shoot a lot (and draw a lot of the focus of the defense), the shooting percentages of remaining players will not change. Wins Produced is very similar to Bob Chaikin's simulator along those lines. Coming from a sports economics literature dominated by baseball, it is very natural to make this assumption, since this issue does not come up in a sport where every player gets his turn at bat.
I'm not an economist, but I've studied economics a little bit and it seems to me that they would be the most likely to see this distinction (between marginal and average shot). In fact, I'm surprised that an economist could miss it.
Joined: 27 Jan 2005 Posts: 423 Location: cleveland, ohio
Posted: Fri Nov 17, 2006 6:21 pm Post subject:
It assumes players convert shots on the margin at the same rate as their average shot. And so if we remove those players who shoot a lot (and draw a lot of the focus of the defense), the shooting percentages of remaining players will not change. Wins Produced is very similar to Bob Chaikin's simulator along those lines......it is very natural to make this assumption...
might you care to expound on what this assumption is that the simulator uses?...
(Why is an assist worth .67 "points"? And frankly, saying that, "on an assist, the passer does one thing and the scorer does two things, so the passer must get 1/3 credit," just doesn't cut it)
What is the appropriate way to weight assists? It's never satisfying to just pick a number, but what other choice do you have? How do other rating systems solve it?
Another thing that bothered me about crediting assists:
Does it really make sense for the passer and shooter to share credit on an assisted score, but for the shooter to be exclusively penalized on the miss?
Joined: 13 Oct 2005 Posts: 183 Location: Atlanta, GA
Posted: Sat Nov 18, 2006 8:36 pm Post subject:
Basketball is troublesome in the way that it fails to lend itself to regression. In baseball, if I wanted to find the various weights (in runs) for all of the offensive events that lead to run scoring, I would simply regress team singles, doubles, walks, home runs, etc. against team runs scored, and come out with something like this. If I regressed the relevant offensive stats in basketball against points, however, I would always come out with this formula: Pts = (2*FG) + 3FG + FT. In other words, assigning partial "points" credit to assists and offensive rebounds is simply not intuitive, and pretty much any method that does this is going to have to fudge on the "value" of an assist.
But if you regress on wins, you don't have the problem of points being totally dependent on a few of the variables that you want to regress. I presume that this is why Berri based his research on "wins produced" and not "points produced" -- linear regression is feasible.
asimpkins wrote:
davis21wylie2121 wrote:
(Why is an assist worth .67 "points"? And frankly, saying that, "on an assist, the passer does one thing and the scorer does two things, so the passer must get 1/3 credit," just doesn't cut it)
What is the appropriate way to weight assists? It's never satisfying to just pick a number, but what other choice do you have? How do other rating systems solve it?
That's just it -- they don't do it any better than JH. Whether it be Bellotti's "Points Created", Tendex, NBA Efficiency, etc., they all arbitrarily pick a weight for assists and go with it. At least Hollinger subtracts out the assistant's credit from the assistee's field goals -- I would venture to say that most of these linear weights methods do not.
However, while I agree with many of its results (and I used to be a huge PER devotee), I have grown wary of a lot of the logic behind PER. The formula is as arbitrary as that used by the much-maligned IBM Award, but Hollinger justifies his assertion that the PER is superior by saying that it confirms his pre-existing conceptions of how players should rank -- after all, those other methods must be hogwash, because they don't have Shaq as the number one center, or they have David Robinson ranking ahead of Michael Jordan (I've got news for you, John, PER has been known to dothesame thing). It's called confirmation bias, and Berri's right -- that's not how science works.
I believe DeanO's Points Produced uses probability theory, which is a huge step in the right direction. Similarly, Wins Produced, while flawed, is at least an empirical approach (until they employ their fudge factor at the end of the calculation, that is). We all laughed when Berri came out with Rodman as the 1998 NBA MVP, but the important thing is that he's trying to derive results scientifically. I don't like the way he's cut himself off from this group, or the way he shuns all criticism, or the way he treats Wins Produced as the be-all and end-all of stats. That attitude can only be bad for APBRmetrics. But I welcome the attempt to do better than simply "picking a number".
Joined: 03 Jan 2005 Posts: 413 Location: Greensboro, North Carolina
Posted: Sat Nov 18, 2006 9:53 pm Post subject:
First point, Wins Produced is designed to predict net team efficiency (offensive minus defensive efficiency) and if only one year was used in the prediction, it would perfectly predict net team efficiency. Their results tell us nothing more than that net team efficiency does a pretty good job predicting wins.
Second, Berri often argues that he uses regression to come up with the relative value of points, rebounds, field goal attempts, etc., but in general that is not true.
For points, field goal and free throw attempts, rebounds, steals, and turnovers, Berri simply uses the leaguewide average of points per possession to arrive at the relative value of the linear weights. I guess that is emprical, but since it is so far removed from anything at the individual level, I would argue that it is only marginally more emprical than PER or other linear weights methods. Berri runs a regression, but he did not need to, so I think it is misleading to argue that this is really regression-based.
For blocks and personal fouls, he does use empirical work, but not regression, to arrive at the appropriate weights. Only for assists does he really use regression analysis.
At the end of the day, what we really want to know is how much better does a team, on average, do with a guy who is more of an assister or a scorer or a rebounder. And it only the value of the assister that Berri really uses emprical work to ask the appropriate question.
So I guess in sum I would argue that Wins Produced is much closer to just "picking a number" than it really appears.
Maybe this is ultimately a trivial point, but I strongly disagree with Berri's disdain for "the laugh test". Confirmation bias is selectively looking for evidence to fit the test hypothesis, but conventional wisdom is not the test hypothesis, it is (or at least should be, in my mind) the null hypothesis. The null hypothesis is supposed to get the benefit of the doubt.
Surprising results shouldn't cause us to reject the method behind them, but they should cause us to scrutinize it and understand why they occur. In the case of Berri's previous effort, the rating for Dennis Rodman revealed some serious flaws - those flaws, in this case, being tied to the effort to determine the value of individual statistics through regression at the team level. If we literally just laughed, that would not have been useful, but neither would have been accepting the results because they were "scientific."
There's also a pragmatic argument. For the most part, we aren't dealing with scientists, so the scientific method is not always appropriate. There is no question that fans and front office personnel are using the laugh test. Should we just ignore that?
Joined: 14 Jan 2005 Posts: 971 Location: Delphi, Indiana
Posted: Sun Nov 19, 2006 9:09 am Post subject:
davis21wylie2121 wrote:
...they all arbitrarily pick a weight for assists and go with it. At least Hollinger subtracts out the assistant's credit from the assistee's field goals -- I would venture to say that most of these linear weights methods do not.
.
Not all points are equally 'assisted'. 82games tracks assisted% of FG for players, though (as far as I know) this info isn't available before season's end. This makes it possible to scale down scoring by use of the unassisted% of players' points. The remainder of credit can then go to the assist man.
This year, I'm experimenting with estimating player unAst%, and also with scaling actual assist rates to some power of (TmPPG/OppPPG). Since I'm correlating player eWins to team pythagorean-expected (pW), I get an exponent that boosts assist-men's eW appropriately when they assist for winning teams.
Thus far, it seems by multiplying the assist rate by (TmPPG/OppPPG)^4 , I get a best fit. This is a pretty large exponent, and it exaggerates the difference between good assists for good teams, and for bad teams. Set the exponent to zero (no good-team credit), and Utah's Deron Williams' per-36 assist rate is 7.5, while Indiana's Jamaal Tinsley is at 7.7.
With the ^4 distinction, the Utah PG soars to an 'effective' rate of 10.3, while Tinsley sags to 6.6 . Of course, this early in the season, PPG-diff is hardly the last word on 'good/bad' teams. So it's an ongoing experiment.
BobC has referred to 'assists that lead to wins' (or not), which he determines thru his sim. I'm proposing there may be other ways of guessing at a 'sliding scale' assist weight. _________________ 40% of all statistics are wrong.
Maybe this is ultimately a trivial point, but I strongly disagree with Berri's disdain for "the laugh test". Confirmation bias is selectively looking for evidence to fit the test hypothesis, but conventional wisdom is not the test hypothesis, it is (or at least should be, in my mind) the null hypothesis. The null hypothesis is supposed to get the benefit of the doubt.
The idea behind the laugh test is that if the results of some analysis are strongly out-of-whack with one's pre-existing notions of what the results should look like, then there must be something wrong with the analysis and it should be restructured until its results ultimately do achieve some harmony with expectations. If that isn't confirmation bias I don't know what is.
Still, I'd agree that the laugh test is to some extent a useful methodological tool, for these reasons.
1: Presumably, the threshold for confirmation bias endorsed by the laugh test is rather high-- the results of an analysis have to be so far from expectation as to induce (or at least suggest the image of) laughter. So we only enact the "bias" when our two sources of judgment-- statistical analyses and direct observation and social discourse-- are heavily dissonant. This ties into
2: We should regard the "common sense" set of judgments, evaluations and expectations of the general basketball-watching community to have some degree of value, even if they are not infallible. This is not in the least part due to the fact that these judgments take into account observations that do not factor into any given statistical analysis (i.e. there is useful information to be obtained from watching games that is opaque to the numbers extracted for a given kind of analysis, and of course vice versa).
Ultimately the laugh test could be seen as enforcing a kind of consistency of judgment across two disparate but useful sources of information and evaluation of NBA games and players. Conventional wisdom about basketball can serve as a useful counterbalance to statistical analysis, and arguably such a counterbalance is needed since analysis of basketball is so difficult and murky of a prospect for all the well-known reasons. To say the laugh test has no value is essentially to say that judgments formed from watching NBA games have no value (as opposed to, say, limited value), which seems a rather bold claim.
Joined: 03 Jan 2005 Posts: 413 Location: Greensboro, North Carolina
Posted: Sun Nov 19, 2006 3:59 pm Post subject:
Berri and co-authors in their previous work came up with a metric that did not include assists, blocks, or fouls. None of those statistics are directly tied to points scored or possessions, so it is possible (with a team adjustment) to come up with a metric that perfectly predicts team efficiency even without using assists, blocks, or fouls.
So why did Wages of Wins move away from that metric? It passed through the peer-review process, and if they had used a team adjustment (I am not sure they did), it would have predicted team wins just as well as the current Wins Produced.
I think they moved from that metric through lots and lots of e-mails from an APBRmetric member (not me) that convinced them that this older metric did not pass the laugh test. That old metric has almost no correlation with adjusted plus/minus statistics (not that they knew this or would have cared). It led to perverse results in several published papers.
So how can the scientific method and peer-review process fail so badly? Models can be wildly off the mark and so if they are only judged on some form of internal consistency, they can produce perverse results. Calibrating a model to perfectly predict subjective evaluations may not be scientific, but completely ignoring (and belittling like they do in the book) all non-statistical analysis is NOT how good empirical work is done in other areas of economics.
Good empirical work is always a combination of theory and evidence that incorporates as much subjective knowledge as is practically possible. We may risk being non-scientific if we overfit the data to predict our subjective evaluations. But we also run the risk of being irrelevant if we completely ignore subjective evaluations.
You cannot post new topics in this forum You cannot reply to topics in this forum You cannot edit your posts in this forum You cannot delete your posts in this forum You cannot vote in polls in this forum
FAQ
Search
Memberlist
Usergroups
Register
Profile
Log in to check your private messages
Log in
