Now that nearly every team has played at least 10 games, one might think we have enough data to form an accurate assessment of any team based on what they have done on the court this season. Then why still have the influence of pre-season ratings in the current ratings? Because you actually don’t have enough data to work with. The opinion one had of a team before the games started being played still has some predictive value.
To illustrate this, I looked at the teams that had deviated the most from their preseason rating at this time last season. For instance, shown below are the ten teams that exceeded their preseason rating the most heading into the 2011 Christmas break, listed with their preseason rank and their ranking on December 24.
Pre 12/24 Wyoming 273 88 La Salle 217 69 Middle Tenn. 178 58 Mercer 247 116 W. Illinois 332 216 Wagner 206 80 Indiana 50 6 Virginia 88 24 Wisconsin 10 1 St. Louis 62 15
(To compare ratings differences, I’m using the Z-score of the Pythagorean winning percentage. If this means nothing to you, basically I’m accounting for the fact that a certain difference in Pyth values at the extremes of the ratings is equivalent to a larger difference in the middle of the ratings. Or Indiana’s move in the ratings represented the same improvement as Wagner’s even though the Hoosiers moved up fewer spots.)
If the preseason ratings are weighted properly, then there shouldn’t be a pattern to how these teams will trend from December 24 through the rest of the season. Some teams will see their numbers improve and some will see their ranking get worse. I’ve expanded the outlier list to 20, and added two columns – each team’s final ranking and the difference in that ranking from the December 24 edition.
Pre 12/24 Final Diff Wyoming 273 88 98 -10 La Salle 217 69 64 + 5 Middle Tenn 178 58 60 - 2 Mercer 247 116 91 +25 W Illinois 332 216 186 +30 Wagner 206 80 112 -32 Indiana 50 6 11 - 5 Virginia 88 24 33 - 9 Wisconsin 10 1 5 - 4 St. Louis 62 15 14 + 1 Georgia St. 182 76 71 + 5 Toledo 337 267 208 +59 Cal Poly 192 103 165 -62 Murray St. 110 43 45 - 2 Lamar 214 121 113 + 8 Ohio 111 48 62 -14 Denver 159 75 80 - 5 Illinois St. 181 98 81 +17 Georgetown 48 14 13 + 1 Oregon St. 124 61 85 -24
I suppose if you had some interest in Toledo you might have had a legitimate beef with the preseason influence on December 24. But the other teams didn’t move all that much, except for Cal Poly which moved downward significantly. If you average the ranking differences (I realize this isn’t the most scientific way to do this analysis), you get -0.9 per team. Pretty much unbiased.
For symmetry, let’s take a look at the teams that were outliers in the other direction. These programs underperformed their preseason ratings the most through December 24.
Pre 12/24 Final Diff Utah 140 316 303 +13 Wm & Mary 160 322 285 +37 Mt St Mary's 210 314 294 +20 Maryland 47 166 134 +32 UC Davis 251 330 326 + 4 Nicholls St. 276 338 332 + 6 Grambling 343 345 345 0 Towson 309 343 338 + 5 Monmouth 262 328 277 +51 UAB 72 175 133 +42 Rider 147 248 199 +49 N Illinois 315 340 330 +10 N Arizona 229 301 341 -40 Portland 130 229 278 -49 Jacksonville 151 239 228 +11 Binghamton 324 342 343 - 1 Arizona St. 70 158 223 -65 Kennesaw St. 272 323 313 +10 UC Riverside 224 291 284 + 7 Rhode Island 180 258 202 +56 VMI 200 268 254 +14
There’s more of a trend here. Each team’s ranking improved by 10 spots on average between Christmas and the final ratings. (The average for the top 10 was 21 spots, with every team but Grambling improving. And Grambling’s numbers did improve, but they were so far in last place they couldn’t catch #344.)
For comparison, let’s look at a world where preseason ratings aren’t used. They’re created for fun and discarded once games are played. The next set of tables looks at the same groups of teams, but the 12/24 column depicts a team’s ranking had there been no preseason influence on 12/24. First, the early-season improvers.
Pre 12/24 Final Diff Wyoming 273 43 98 -55 La Salle 217 62 64 - 2 Middle Tenn 178 44 60 -16 Mercer 247 82 91 - 9 W Illinois 332 122 186 -64 Wagner 206 58 112 -54 Indiana 50 6 11 - 5 Virginia 88 15 33 -18 Wisconsin 10 1 5 - 4 St. Louis 62 13 14 - 1 Georgia St. 182 73 71 + 2 Toledo 337 205 208 - 3 Cal Poly 192 72 165 -93 Murray St. 110 30 45 -15 Lamar 214 88 113 -25 Ohio 111 33 62 -29 Denver 159 50 80 -30 Illinois St. 181 75 81 - 6 Georgetown 48 11 13 - 2 Oregon St. 124 46 85 -39
Where the average ranking decline in the world influenced by preseason ratings was about one spot, this group drops by an average of 23 spots. Clearly, the lack of preseason influence would cause a bias. The opposite effect is observed with the decliners…
Pre 12/24 Final Diff Utah 140 330 303 +27 Wm & Mary 160 340 285 +55 Mt St Mary's 210 331 294 +37 Maryland 47 236 134 +102 UC Davis 251 335 326 + 9 Nicholls St. 276 338 332 + 6 Grambling 343 345 345 0 Towson 309 343 338 + 5 Monmouth 262 336 277 +59 UAB 72 201 133 +68 Rider 147 276 199 +77 N Illinois 315 341 330 +11 N Arizona 229 321 341 -20 Portland 130 252 278 -26 Jacksonville 151 261 228 +33 Binghamton 324 344 343 + 1 Arizona St. 70 188 223 -35 Kennesaw St. 272 324 313 +11 UC Riverside 224 313 284 +29 Rhode Island 180 279 202 +77 VMI 200 292 254 +38
The average improvement under the preseason-weighting scheme was 10 spots, but in a no-preseason scheme it’s 27 spots. Without preseason influence at this time of year, you can be nearly certain that a team that has overachieved relative to the initial ratings would be overrated. Likewise, a team that has dramatically underachieved would be almost certain to see its rating improve. That is to say, the ratings would be biased without preseason influence.
And this is because a dozen games are not enough to get an accurate picture on a lot of teams, especially when most of those games involve large amounts of garbage time. That’s not to say there isn’t a lot of value in the games that have been played. The fact that the preseason ratings are only given 2-3 games worth of weight at this point is an indication of that. Teams that have deviated substantially from their preseason ratings are almost surely not going to revert to that preseason prediction. But what’s nearly as certain is that a team’s true level of play is closer to their preseason prediction than their performance-to-date suggests.
If you’ve made it this far, you’ve earned some bonus visuals. So let’s take a look at how the ratings changed last season in the entire D-I population, comparing change from the beginning of the season to Christmas and change from Christmas to the end of the season (using Z-score).
The plot on the top is without preseason ratings and the plot on the bottom is under the existing system. Notice that without preseason ratings, the change between the beginning of the season and Christmas is correlated with the change between Christmas and the end of season. While in the case with pre-season ratings, the two changes are almost uncorrelated, as they should be in an unbiased system.
Another conclusion that can be drawn from these plots is that the system would be more volatile without the influence of preseason ratings. Changes after December 24 are greater in the plot on the top than the one on the bottom. This begs the question: How long should preseason influence last? Based on Nate Silver’s findings, there’s some strong evidence that it would improve predictions to include one or two games worth of preseason expectation through the end of the season instead of having them expire in late January. The plot on the bottom suggests there’s still enough rebound that the Christmastime ratings should include more preseason juice. But it appears the mix is close enough to being right – certainly much closer than not including preseason ratings at all – to not lose any sleep over.