A while back, I expressed my support for the pre-season AP poll. A refresher: During the season I have no use for the polls because nobody really knows what they represent. But before games are played – and voters minimize the importance of scoring margin, game location, and schedule strength – everyone is trying to rank the best 25 teams in the country.

As I prepare to release my own preseason ratings (in about a week), built strictly from a formula, I again wondered about the pre-season AP poll and how useful it is as a forecast tool. Not just for the team ranked at the top as was examined in the previous piece, but for teams ranked at each spot in the poll.

So I went back through the 64-team era and recorded the eventual tournament seed for each team ranked in the preseason. That data is summarized below.

                         P r e a s e a s o n   R a n k
        1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25
   1   19 13 11  9 12  3  7  2  3  2  6  3  1  0  0  1  3  1  2  0  1  2  0  2  0
   2    5  8  7  4  6 10  4  9  5  5  1  3  6  1  3  1  0  2  2  1  2  1  0  1  5
S  3    3  3  4  5  5  3  3  1  2  4  3  3  2  4  2  2  1  1  1  4  0  4  3  3  1
e >3    1  3  6  9  3 11 13 15 15 14 18 19 19 23 23 23 24 24 23 23 20 15 20 17 17
e NT    0  1  0  1  2  1  1  1  3  3  2  5  5  7  6  3  8  5  6  7  7  2  6  8  4
d %1   68 46 39 32 43 11 25  7 11  7 21 11  4  0  0  4 11  4  7  0  4  7  0  7  0
  %1-3 96 89 79 67 88 59 52 44 40 44 36 32 32 18 18 15 14 14 18 18 13 32 13 26 26
  %NT   0  4  0  4  8  4  4  4 12 12  7 18 18 25 21 11 29 18 21 25 30  9 26 35 17

The top ranked team became a one-seed 19 times, or 68% of the time. It has received at least a three-seed 27 of 28 times (sorry, 2000 UConn), or 96% of the time, and has never missed the tournament (NT). Even after 28 seasons, the data is still rather noisy on a ranking-position basis. For instance, I wouldn’t take seriously the idea that the team ranked 17th in this season’s poll has a better chance at a one-seed than the team ranked 8th based on past performance.

One can overcome some of the noise issues by grouping similarly-ranked teams. The following table takes all teams ranked within two spots of the ranking shown. So the column labeled “3” includes all teams ranked one through five. This offers a more coherent picture of what a pre-season ranking means.

                       5 - s p o t   a g g r e g a t e                                                
      1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25
%1         47 35 31 24 20 13 15 12 11  9  7  4  4  4  5  5  5  4  4  4  4    
%1-3       84 76 69 62 57 48 43 39 37 32 27 23 19 16 16 16 16 19 19 20 22    
%NT         3  4  4  4  6  7  8 11 13 16 18 19 21 21 20 21 24 21 23 25 24    

The chances of being a one-seed get really slim once you get past the top 12 or 13, while the chances of missing the tournament altogether are very real for the teams in the bottom half of the poll.

It’s striking that there isn’t much difference between the forecast of team ranked 15 or 25. So if you are feeling strongly that a team ranked 25 should be 15 or vice versa, you’re not actually talking about a significant difference. Even at the end of the season, with 30 games of data to rely on, it’s a challenge to distinguish between the 15th and 25th best teams. In the preseason, without that information, there are so many uncertainties involved that the difference is practically impossible to identify.

This exercise also appears to give some quantification to the meaning of the regular season. In most seasons, there isn’t much suspense to the regular season for the top team or two. Exhibit A – the Kentucky/North Carolina game last season. There was already strong reason to believe that both would be one-seeds and that the result of the contest would have no impact on their future. (For those that fear the 96-team era – and I don’t look forward to it – that game was also an illustration that a game with no “meaning” in a championship sense can still generate a lot of interest anyway.)

However, a team ranked as high as eighth has less than a 50% chance of being a top three-seed, necessary for a reasonable chance at a title. Teams ranked in the mid-teens have a one-in-five chance of completely missing the tournament. The uncertainty in the non-elite-team’s future provides the incentive to take advantage of each practice and game to get better. I’m not sure if an expanded tournament field would reduce that incentive, but I kind of like the idea that a team universally thought to be really good in November could find itself watching the tourney in March if things don’t go right.

Finally, it’s worth discussing whether the pre-season poll is getting better. I’ve attacked this a few different ways and haven’t found a trend. (But if you’d like to take a crack, I’ve included the raw data at the end of this post.) Frankly, I would expect voters to be improving considering the increased information available in the internet age. There are two reasons I can think of that are preventing this.

The simple explanation is that it’s more difficult to pick the top teams than it was nearly thirty years ago. Picking the top 20 of 250 teams is marginally easier than picking the top 20 of 350 teams.

It’s also possible that the increased connectivity in the internet age is having a negative effect. The pre-season poll should represent the wisdom of the crowd, but that’s only true if everyone’s opinion is independent of the other voters. Independence was easier to achieve when people had to battle 1200-baud modems to connect to CompuServe and QuantumLink than it is now when everyone sees Andy Katz’s top 25 the day after the title game.

Of course, not everyone is cheating off the answer key. That’s why I’ll always have Scott Mansch’s back, no matter how wacky his ballot is.

Tournament seed for each top 25 team by year

                              P r e a s e a s o n   R a n k
        1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25
  2012  1  1  2  9  1  2  5  7  4 XX  8  3  2 10  4 XX XX  4  9 XX  6  3  7 12  2
  2011  1 10  5  1  1  9  1  2  2  3  4 11  9  3 11 XX  8  7 12  6 XX  7  9  3  2
  2010  1  5  8  1  2 XX  4  2  1  6  5 XX  8 11 XX  2 XX XX  9  3 XX 10 XX  7 11
  2009  1  1  1  6  1  2  7  2 XX  4  5  2  2  9  6  6 XX 10 XX XX  4 XX  3  3 12
  2008  1  1  1  1  2  3  2  5  8  4  6  9  2  7  2  9 10  6  9 11 XX  4  3 XX 11
  2007  1  1  1  3 XX  2  1  2  2  8 XX  6  3  2  7  8 XX XX 10 XX  4  8 10  7  5
T 2006  1  2  1  6  1  6 XX  3  8  8  4  1 XX  6 10  5  4 XX  2  3  8  5  6 XX XX
o 2005  3  2  5  1  1  4  2  2  2  3  1  9  5  4 XX  8  9  5 10 XX  6  1  4 XX  3
u 2004  2  1  7  9 XX  4  5  5  6  2  1  3  5 XX  6 10  1  4  1  4 XX  3 XX  3  2
r 2003  1  2  1  1  2  3  2 10  7  3  8  5  6 XX  5  *  1  3  6 13  7  9  8 XX 13
n 2002  1  1  4  4  8  5  1 12 XX XX XX XX XX  8 XX 10 11  7 XX  4 XX  5  6  2  2
a 2001  2  1  1  1  3  2  4  1  8 XX  3  2 XX XX  7  6  4  5  6  7 XX  4  6  5  2
m 2000  5  2  1  7  3  8  2  5  1  1  8  6  1  5  8  4  4  2  4  9  5  3  6 10  6
e 1999  1  1  2  3  1  2  6  6  4  2  3  5  9  7  3 10 XX  4  4  9  8  6 12 XX XX
n 1998  1  1  1  1  6  6  3  2  2  6  4  2 XX  3 XX  3  7  8 XX 10  8 XX  4  7 XX
t 1997  3  1  1  3  2  2  4  1 XX  2  6 XX XX XX 11 10  3  6  4  4  5  1  8 XX  7
  1996  1  2  3  4  2  1  1  6  5  4  2  6  5 XX  7 12  7  9 XX  6  2  9  6  1 12
Y 1995  2  2  2  1  5  1  3 XX  9 10  1  7  7  4  6  9 XX  5  2  3  4  3 XX  1 11
e 1994  1  3  1  2  3  5  3  4  4  6  4  5  5 XX  9  7  8  2  8  4  1  2 XX 10 XX
a 1993  1  2  3  1  1  2  1 10  3  2  4 XX  4  4 XX XX 11 XX  8 XX  2  *  3  9 10
r 1992  1  2  3  2  3  7  1  4  4  7  1  1  2 XX  9  6  5  5  4  6  9  9  7 XX  8
  1991  1  1  2  5  1  2  4  2  8  1  4  6  2  6 XX  8 11  7 10 XX 11  5 XX 11  4
  1990  1  5  2  3  3  2  8  5  4  3  3  4  7  8 11  1 10 XX XX  6 XX  4 XX XX  6
  1989  2  1  3  4  1  2  4  2  1  4  1 XX  6  3  7  4 XX  5 XX  3          
  1988  3  1  2  2  2  4  6  6  3  7  5  1  5  6  2  8  1  5  1  5          
  1987  1 XX  1  3  1  7  6  5  8  2  8  8  2  3  2  3 11  1 10 XX          
  1986  2  3  2  2  1  1  4  4  2  8  1  3  4 11  3  6  6  4  5  7          
  1985  1  3 10 XX  1  3  1  2  5  5  4  7  3  4  9  9 XX 12  3  2          

*Teams ranked here were ineligible for the tournament