APBRmetrics

Tango · Joined: 18 Mar 2005 Posts: 24

Hi all:

I've been putting together the following types of graphs for wins/loss trending analysis like the following...

I'm trying to automate the collection of the data so that it's easier to create these graphs. Currently it's a pretty manual process of copying and pasting standings & team schedules+outcomes to do this.

What sources of daily standings and schedules do you all recommend for doing this?

I did visit Doug Steele's site so i could simplify the process some by taking the daily scores and loading them into a flat file for crunching. I was looking for a little more sophistication to do this.

I would appreciate everyone's thoughts on the matter. Thanks!

Ed **Küpfer** · Joined: 30 Dec 2004 Posts: 785 Location: Toronto

wtbarron · Joined: 10 Feb 2005 Posts: 12

If you're a coder, you could do what I do and parse everything straight out of NBA.com.

Tango · Joined: 18 Mar 2005 Posts: 24

Hi Ed:

Thanks for the suggestion. I was referring to Doug's Game Results page as well. I could certainly do a "save as" of his page into a txt file and then use Excel or Access to clean up & format the data as I need it. That saves a bit of work for sure! However I was looking at doing something with a little more automation- something more along the lines of what wtbarron is suggesting.

How much work are you doing with your spreadsheet to make the data collection (e.g. macros etc.) more painless for you? I'd be curious to know to help me gauge what level of effort I want to put into the automation.

wtbarron:

I'm a novice at all this so you'll have to bear with me Smile

! However I'm tech savvy enough to try the coding to get the parsing to work though. Can you give me some advice as to how I would go about doing that? I'm comfortable coding in PHP, have a tad of experience with JS, and know how to manipulate mysql DB's. What tools and techniques would you suggest for parsing data from websites? Thanks so much!

wtbarron · Joined: 10 Feb 2005 Posts: 12

Tango · Joined: 18 Mar 2005 Posts: 24

wtbarron:

Thanks for the excellent tips!

Here's a tutorial for any interested in coding in PHP to do this taking wtbarron's steps and breaking it out more in a bit more detail.
http://www.jjwdesign.com/data_mining_functions.html

I just tried the snoopy.php.class created by someone to do the fetching (step 1 listed by wtbarron) and it works like a charm. Now working on how to format the output for parsing.

wtbarron · Joined: 10 Feb 2005 Posts: 12

My pleasure, Tango. Great idea, finding that PHP project. Wink

apparition312 · Joined: 16 Feb 2005 Posts: 6 Location: Chicago, IL

I've done this for a fantasy basketball site I'm working on. I use ESPNs box scores. First I set up a MySQL database with several tables: games, standings, playerstats, teamstats and players.

Next, I populated the player info by converting Patricia's rosters into a CSV file, and populated the games info from the daily schedule on NBA.com.

Then I wrote a script that takes each box score and matches it to a game in the games table. Then it goes through each stat line and adds the info to the playerstats table. It takes the totals for each team and adds them to the teamstats table.

To get the standings, I start with 1 entry for each team that includes all zeroes and, based on what happened in the game being parsed, increment the values accordingly.

All I need to do is download each day's HTML boxscores and upload each one into my script. (I could probably make it so I can do all of them at once, but I like seeing the results of each game as I upload them.)

Once all the data is in the database, I can make scripts to do pretty much whatever I want - standings, box scores, player profiles, league leaders, stat querys, etc.

If you want more info about setting up the tables or parsing the stats, let me know.

Tango · Joined: 18 Mar 2005 Posts: 24

apparition312:

Thanks for the tips. I could use help with the code for parsing the data. My current understanding is to take the following approach:

(1) find the start and end "markers" I'm looking for in the html
(2) serialize the text between the start & end into a string
(3) go through the string and break it down into the parts I'm looking for and store them into an array
(4) write the array into the DB

Does this sound about right? I'm now looking for the appropriate PHP functions to use for all this - especially for #1 and #2.

apparition312 · Joined: 16 Feb 2005 Posts: 6 Location: Chicago, IL

Tango, what I do is look through the HTML source of the boxscore to find unique identifiers for the info I'm looking for. I like ESPNs boxscores because their code is pretty easy to parse and they use first and last names in the boxscores which makes matching stat lines to the player database easier.

Basically I parse the HTML for the game date and home and away teams to match to the games database (and get a corresponding game id). Then I go through each stat line and grab the stats. The method is pretty much the same for each.

I start with a pattern - the away team, for example - then use preg_match() to find the pattern within the HTML ($data) and store the value in an array called $away_info.