My first reaction after I’ve finished post about Accuracy of Predictions for Wins in 2010/11 was this: “that’s nice but what about consistency of authors?”. Are writers and bloggers consistent over time or is it basically a semi-random outcome every year? Is it an educated guess or just a guess by educated people?
To answer this question I simply needed more data and because I didn’t want to wait a couple more years I went the other way – into the past. I collected some predictions for NBA standings before 2009-10 season and before 2008-09 season [btw, it's not a closed list so if you want to add any predictions let me know in the comments]. Then I again compared them to actual results by Root Mean Square Deviation [read post about 2010/11 for explanation]. If you want to play with numbers or check any single entry just download NBA Predictions Accuracy 2008-2009-2010.pdf but if you are interested only in the results here they are…
Those lists are unfortunately shorter than I hoped for because:
- sadly APBR forum was hacked so their batch of past predictions vanished forever,
- Wages of Wins Network either didn’t exist or didn’t post any win predictions for 2009/10 or
I can’t quickly find them [they merged and relocated],
- surprisingly few authors even published their win predictions 3 years ago or google just can’t find them easily.
I didn’t get a big sample but it’s still bigger than just one year so
I think I can make a couple of additional points…
- last season had a higher level of accuracy than two previous ones. Was it a classic fluke year? ;-)
- Vegas odds-makers clearly know what are they doing.
So I’m just wondering… as a blogger without any good system shouldn’t you just copy their predictions with some small changes regarding those teams where you feel veeery strongly they are wrong?
- wisdom of the ESPN crowd had two really good years in a row – coincidence or this system works?
Again, I wish there was more data but they started it two years ago.
- John Hollinger had two top seasons and one average so either he is really good at this game or very lucky.
- there’s an interesting path for Bill Simmons. He had a spectacular results for 2008-09, really bad ones for 2009-10 and he didn’t even finish his predictions for 2010-11! So did he… just quit this game?
- WARP, Wins Produced and Win Shares did a surprisingly poor job in the last 3 seasons so even though it doesn’t tell us anything about those ratings I have a theory: you shouldn’t base your predictions on numbers from previous year and just plug them into a system because there are too many unpredictable variables… you really should get a little wild and add some changes not based on numbers. True or false?
- overall, it DOES seem to be at least somewhat consistent… although again I wish it was a bigger sample.
We will get there in a couple of years ;-)
Daniel M (@DSMok1)
December 30, 2011 at 18:20
I have some (1100 pages) of the old forum for APBR on my hard drive, and will post it on APBRmetrics when I have time.
This is from 07-08 on APBRmetrics
Predicted Wins Difference (Absolute values)
Pyth+ tm JohnH DW21 MikeG Mtn JohnH DW21 MikeG Mtn
72.3 Bos 51 50.2 54.0 51 21.3 22.1 18.3 21.3
65.0 Det 48 48.7 47.6 53 17.0 16.3 17.4 12.0
46.5 Was 33 39.9 39.5 37 13.5 6.6 7.0 9.5
34.4 Phi 21 29.7 25.7 28 13.4 4.7 8.7 6.4
37.9 Ind 28 32.0 30.1 28 9.9 5.9 7.8 9.9
50.7 Orl 49 45.8 41.0 40 1.7 4.9 9.7 10.7
38.0 Atl 42 33.4 31.9 35 4.0 4.6 6.1 3.0
42.9 Tor 40 40.0 45.5 42 2.9 2.9 2.6 .9
27.6 Mil 35 27.4 29.7 32 7.4 .2 2.1 4.4
34.6 NJ 39 36.9 40.1 42 4.4 2.3 5.5 7.4
27.1 Cha 34 29.7 35.3 30 6.9 2.6 8.2 2.9
23.3 Mia 43 37.6 35.9 40 19.7 14.3 12.6 16.7
20.1 NY 38 32.7 37.3 42 17.9 12.6 17.2 21.9
35.9 Cle 54 57.1 53.5 49 18.1 21.2 17.6 13.1
32.1 Chi 55 57.7 53.2 54 22.9 25.6 21.1 21.9
39.2 east 40.7 39.9 40.0 40.2 12.1 9.8 10.8 10.8
Pyth+ tm JohnH DW21 MikeG Mtn JohnH DW21 MikeG Mtn
46.4 Por 26 21.8 18.6 26 20.4 24.6 27.8 20.4
54.9 LAL 43 40.2 38.2 40 11.9 14.7 16.7 14.9
54.0 NO 39 40.3 41.4 43 15.0 13.7 12.6 11.0
45.8 GS 36 34.4 39.6 39 9.8 11.4 6.2 6.8
49.7 Den 53 44.4 42.6 53 3.3 5.3 7.1 3.3
56.5 Phx 54 56.9 58.6 57 2.5 .4 2.1 .5
31.7 Sac 27 36.0 32.0 35 4.7 4.3 .3 3.3
47.1 Uta 50 45.3 54.1 50 2.9 1.8 7.0 2.9
26.3 LAC 20 33.7 33.6 28 6.3 7.4 7.3 1.7
57.4 SAS 59 65.2 58.8 59 1.6 7.8 1.4 1.6
54.2 Dal 58 62.2 63.5 58 3.8 8.0 9.3 3.8
28.3 Mem 34 35.2 37.5 34 5.7 6.9 9.2 5.7
23.2 Sea 35 34.0 27.9 29 11.8 10.8 4.7 5.8
14.6 Min 22 26.2 26.9 23 7.4 11.6 12.3 8.4
42.8 Hou 61 55.2 56.3 53 18.2 12.4 13.5 10.2
42.2 west 41.1 42.1 42.0 41.8 8.3 9.4 9.2 6.7
10.2 9.6 10.0 8.7
I think that’s about all I’ve got, though.
wiLQ
December 30, 2011 at 18:32
That’s a great batch of new data, thanks! I’ll add it if I can.
By any chance do you have something similar for 2008/09 and/or 2009/10?
wiLQ
January 1, 2012 at 17:03
I’ve calculated RMSE for your numbers and results look pretty bad (JohnH=11.64, DW21=11.29, MikeG=11.14, Mtn 10.51) but without any context it’s hard to tell whenever it was just a difficult year for predictions or it was a learning process or what.
ek
December 31, 2011 at 20:41
This is very good stuff, wiLQ.