My first reaction after I’ve finished post about Accuracy of Predictions for Wins in 2010/11 was this: “that’s nice but what about consistency of authors?”. Are writers and bloggers consistent over time or is it basically a semi-random outcome every year? Is it an educated guess or just a guess by educated people?
To answer this question I simply needed more data and because I didn’t want to wait a couple more years I went the other way – into the past. I collected some predictions for NBA standings before 2009-10 season and before 2008-09 season [btw, it's not a closed list so if you want to add any predictions let me know in the comments]. Then I again compared them to actual results by Root Mean Square Deviation [read post about 2010/11 for explanation]. If you want to play with numbers or check any single entry just download NBA Predictions Accuracy 2008-2009-2010.pdf but if you are interested only in the results here they are…
Those lists are unfortunately shorter than I hoped for because:
- sadly APBR forum was hacked so their batch of past predictions vanished forever,
- Wages of Wins Network either didn’t exist or didn’t post any win predictions for 2009/10 or
I can’t quickly find them [they merged and relocated],
- surprisingly few authors even published their win predictions 3 years ago or google just can’t find them easily.
I didn’t get a big sample but it’s still bigger than just one year so
I think I can make a couple of additional points…
- last season had a higher level of accuracy than two previous ones. Was it a classic fluke year? ;-)
- Vegas odds-makers clearly know what are they doing.
So I’m just wondering… as a blogger without any good system shouldn’t you just copy their predictions with some small changes regarding those teams where you feel veeery strongly they are wrong?
- wisdom of the ESPN crowd had two really good years in a row – coincidence or this system works?
Again, I wish there was more data but they started it two years ago.
- John Hollinger had two top seasons and one average so either he is really good at this game or very lucky.
- there’s an interesting path for Bill Simmons. He had a spectacular results for 2008-09, really bad ones for 2009-10 and he didn’t even finish his predictions for 2010-11! So did he… just quit this game?
- WARP, Wins Produced and Win Shares did a surprisingly poor job in the last 3 seasons so even though it doesn’t tell us anything about those ratings I have a theory: you shouldn’t base your predictions on numbers from previous year and just plug them into a system because there are too many unpredictable variables… you really should get a little wild and add some changes not based on numbers. True or false?
- overall, it DOES seem to be at least somewhat consistent… although again I wish it was a bigger sample.
We will get there in a couple of years ;-)