 |
APBRmetrics
The statistical revolution will not be televised.
|
| View previous topic :: View next topic |
| Author |
Message |
mtamada
Joined: 28 Jan 2005
Posts: 375
|
 Posted: Tue Oct 06, 2009 7:08 pm Post subject: |
 |
|
| HoopStudies wrote: | | Good to see some of the people with that more advanced stat knowledge providing the input. I didn't know about ridge regression until this discussion. Relating it to Bayesian stuff is definitely helpful... So many tools and you have to know when to use which ones... |
A side note: ridge regression has been around for decades, but it always sounded like total gibberish to me (why would one make up a number and -- plunk -- add it to the X'X data matrix?), until somewhere on the web I read about the Bayesian interpretation, and weighting the data matrix to bias the estimates toward 0. So a thank you to -- unfortunately I have completely forgotten where I read it, much less who wrote it.
This stuff is probably common knowledge among statisticians (and from MoonBeamLevels' posting, they've evidently progressed well beyond it), but ridge regression was never well explained, nor used for that matter, in the econometrics classes that I took. |
|
| Back to top |
|
 |
tpryan
Joined: 11 Feb 2005
Posts: 96
|
| Posted: Wed Oct 07, 2009 4:47 pm Post subject: |
|
|
| mtamada wrote: | | HoopStudies wrote: | | Good to see some of the people with that more advanced stat knowledge providing the input. I didn't know about ridge regression until this discussion. Relating it to Bayesian stuff is definitely helpful... So many tools and you have to know when to use which ones... |
A side note: ridge regression has been around for decades, but it always sounded like total gibberish to me (why would one make up a number and -- plunk -- add it to the X'X data matrix?), until somewhere on the web I read about the Bayesian interpretation, and weighting the data matrix to bias the estimates toward 0. So a thank you to -- unfortunately I have completely forgotten where I read it, much less who wrote it.
This stuff is probably common knowledge among statisticians (and from MoonBeamLevels' posting, they've evidently progressed well beyond it), but ridge regression was never well explained, nor used for that matter, in the econometrics classes that I took. |
A few more thoughts.
The number that is added to the diagonal elements of X'X must be the realization of a statistic that is intelligently chosen, or one could use a ridge trace approach and pick the constant based on a subjective determination of the point at which the parameter estimates tend to stabilize.
Actually, the shrinkage is not toward zero, at least not for a small ridge constant, nor should it be since each parameter estimate is not degraded to the same extent by multicollinearity, and some estimates might not be degraded at all. It is also possible to directly employ a shrinkage estimator that shrinks in the direction of a desired subspace, as in Oman's 1982 paper in Technometrics, which I refereed.
The extent to which the betahat vector is "too long", relative to the (of course, unknown) length of the beta vector, is a function of the reciprocals of the eigenvalues of X'X, which will blow up when there is extreme multicollinearity. Thus, shrinkage makes sense.
A good - if I may say so - presentation of ridge regression with some recent references is my 22-page chapter on ridge regression (Chapter 12) in the 2nd edition of my regression book, which came out last November. (shameless plug LOL)  |
|
| Back to top |
|
|
mtamada
Joined: 28 Jan 2005
Posts: 375
|
| Posted: Thu Oct 08, 2009 4:11 pm Post subject: |
|
|
| Ryan J. Parker wrote: | | Thanks for that link mtamada. I haven't seen much work in trying to look at the underlying assumptions and testing if they work or not. In fact, I haven't seen any such talk about when you'd have to revert back to a fixed effects model! |
It was a revolutionary test, to be used not just for random effects vs fixed effects, but in general when there's the possibility of the residuals being correlated with the righthand side variables (necessitating the use of less efficient but more robust techniques such as instrumental variables), or for that matter a wide variety of tests of model specification.
Nobel-prize worthy stuff, but Hausman hasn't won one. One possible reason is that he was actually beaten to the punch by a previously published article by Wu (but people didn't pay much attention to Wu's article); sometimes the test is even called a Wu-Hausman test. Actually, now I see that it's sometimes called the Durbin-Wu-Hausman test, nor sure where/why Durbin came in but he's certainly a recognizable name (assuming it's the same Durbin). So maybe Hausman's work had been very much prefigured by Durbin and by Wu. |
|
| Back to top |
|
|
haralabob
Joined: 11 Apr 2007
Posts: 25
|
| Posted: Thu Oct 08, 2009 7:49 pm Post subject: |
|
|
| mtamada wrote: | Correcting something that I wrote:
| Quote: | | So Winston's APM is similar to the other APMs out there, but with some modifications. Do these modifications result in much smaller standard errors? It'd be hard to imagine how they could |
Actually I should have said that his standard errors may indeed be reduced, but it's hard to imagine that he's reduced them by more, or much more, than other standard techniques such as ridge regression manage to do.
The whole point of ridge regression is to reduce the standard errors of the coefficient estimates, by purposely biasing the estimates (ie. doing that weird thing of adding an artificial number to the X'X data matrix). Usually we want to avoid using biased estimators, but sometimes a biased estimator can have such a good (i.e. smaller) standard error that it's worthwhile to do so: most (all?) of the estimators which utilize regression to the mean (or to zero in this case) are examples.
If Winston has come up with a technique which reduces standard errors and does it substantially better than other techniques, that'd be a major advance and he could publish it in an economics or statistics journal and get accolades aplenty. Maybe he has made such an advance, but wants to keep it proprietary so he can keep raking in the 6-figure consulting fees from the Mavs. More likely I think is that he twiddles with the figures somewhat, possibly in a way which makes sense for NBA data, but is not generalizeable to other data sets, which may indeed yield smaller standard errors but my guess is not by a substantial amount. |
Mtmada or any other R junky, do you know which R package would be best suited for forcing the avg adj +/- coefficient to be zero (using a Bayesian prior)?
I have only recently converted to R and can't seem to find a suitable package for this task.
haralabob |
|
| Back to top |
|
|
Ryan J. Parker
Joined: 23 Mar 2007
Posts: 706
Location: Raleigh, NC
|
| Posted: Thu Oct 08, 2009 7:55 pm Post subject: |
|
|
I am unaware of such a package. The only way I've ever gotten a 0 average is with a Bayesian analysis in WinBUGS. I pretty much stick with R, so I don't know how others do it with their software package of choice (or if they just do it "by hand", so to speak).
An R alternative I've been looking into involves creating a custom MCMC algorithm with C++ that uses the framework of the MCMCpack package.
_________________
I am a basketball geek. |
|
| Back to top |
|
|
mtamada
Joined: 28 Jan 2005
Posts: 375
|
| Posted: Mon Oct 12, 2009 7:44 pm Post subject: |
|
|
| Hmm, I replied a few days ago but it seems to have been eaten by gremlins. But it was a negative reply anyway: I don't use R (because I have some nifty and not-so-nifty stat software at work). For doing "customized" estimation such as imposing a constraint that the (weighted presumably) coefficients sum to 0, I'd think that R would be one of the best apps to use (because of its programmability) but I don't have any hints to give. Amongst the ones that I use, I'd look at Stata first for that sort of estimation problem ... Stata's also cheaper than many of the well-known stat apps, but it's still several hundred dollars. R is great because it's (a) powerful and (b) free. |
|
| Back to top |
|
|
tpryan
Joined: 11 Feb 2005
Posts: 96
|
| Posted: Tue Oct 13, 2009 2:31 pm Post subject: |
|
|
| I have Stata on my PC and have used it some. It is good for doing unusual things and that seems to be the niche the company has sought. |
|
| Back to top |
|
|
haralabob
Joined: 11 Apr 2007
Posts: 25
|
| Posted: Wed Oct 14, 2009 6:42 am Post subject: |
|
|
| Thanks for the replies guys, I was previously using Stata but found the stata programming language to be a bit odd and frustrating to deal with and have made the switch to R. |
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
Powered by phpBB © 2001, 2005 phpBB Group
|
FAQ
Search
Memberlist
Usergroups
Register
Profile
Log in to check your private messages
Log in
