{"id":2392,"date":"2020-09-22T15:09:32","date_gmt":"2020-09-22T21:09:32","guid":{"rendered":"http:\/\/kenpom.com\/blog\/?p=2392"},"modified":"2021-01-11T01:42:07","modified_gmt":"2021-01-11T07:42:07","slug":"the-eye-test-makes-you-blind","status":"publish","type":"post","link":"https:\/\/kenpom.com\/blog\/the-eye-test-makes-you-blind\/","title":{"rendered":"The eye test makes you blind to the important stuff"},"content":{"rendered":"

After months of uncertainty, it\u2019s looking like a college basketball season will exist in some form. The number of games played will be limited<\/a> and some conferences may not play a non-conference slate.\u00a0<\/span><\/p>\n

However, there is a misconception promoted by the media and the highest levels of the NCAA, that without non-conference games one cannot use objective data to select and seed teams for a postseason tournament. In reality, almost every other ratings system besides the NET will produce ratings this season, even for teams that do not play non-conference games, because it\u2019s not a very difficult obstacle to overcome.\u00a0<\/span><\/p>\n

College basketball has always been an outlier in the sports world in using opinions to influence who makes its post-season tournament. <\/span>If subjective opinions are used exclusively, this season will be even worse. In the event of no or very few non-conference games, how does one compare a 10-10 Big East team to an 11-7 Pac-12 team to a 16-0 MAC team using the eye test? It is a bizarre exercise to consider. <\/span><\/p>\n

At least in the past there\u2019s been a historical standard that is loosely understood by the participants even if nobody can precisely explain it. This season there may be no standard. Just ten sets of eyes to make a guess as to which team is most deserving. Do wins and losses matter? Do style points matter? Nobody knows and that’s not fair to participants in the game, and thus the game itself.<\/span><\/p>\n

Pandemic or no, the NCAA should be moving away from subjectivity when it comes to selecting teams. Making decisions purely with one\u2019s gut is a risky exercise. And unfortunately, people who express the most confidence in their intuition often seem to be the worst at using it. We have seen that principle play out in a role fundamental to college basketball administrators – the colleagues of people on the selection committee – and that’s the hiring of head coaches.\u00a0<\/span><\/p>\n

Per the nifty<\/span> NCAA Demographics Database<\/span><\/a>, 28% of D-I head coaches in 2019 were Black. At non-HBCU’s, this figure is 24%. That 24% of non-HBCU head coaches are Black contrasts sharply with the fact that Blacks make up 53% of players and 46% of assistant coaches. In 2019, 6 of 27 (22%) non-HBCU coaching hires without prior D-I head coaching experience were Black.<\/span><\/p>\n

Whether this is the result of overt racism or subconscious bias is beyond the scope of this piece. But you have to tie yourself into a serious logical knot to believe that African-Americans are qualified to be assistant coaches, but not head coaches. However, people will tie themselves into those knots.<\/span><\/p>\n

Sadly, it appears coaching hires are often made by the eye test. For head coaches with previous experience there is an actual track record with wins and losses to consider. But for aspiring assistants, there clearly isn\u2019t as much data.\u00a0<\/span><\/p>\n

Well, there is <\/span>some<\/span><\/i> data. The most important thing for a college basketball program is getting players. Whether it\u2019s the five-star high school player or the undervalued gem that is the perfect fit for a particular system, having great players supersedes strategy. The best strategic coach is not going to win without talent. But the evidence indicates that athletic directors tend to ignore skill in recruiting in favor of\u00a0 focusing on a coach’s strategic knowledge.<\/span><\/p>\n

This is not to say that White coaches are better with strategy than Black coaches. But there\u2019s no doubt that basketball people, whether media or fellow coaches, think so. <\/span>This 2013 piece<\/span><\/a> from espn.com polled D-I head coaches to name the top 25 X’s and O’s coaches. And 24 of them were White. (Actually two other coaches received enough votes to make the list – Stevens and Ben Howland – but were excluded because they weren’t active college coaches. So make it 26 of 27.) <\/span>This 2015 ranking<\/span><\/a> from coachstat.com seems to be well-thought and thoroughly researched but lists 25 White guys among its top 25 X’s and O’s coaches.<\/span><\/p>\n

In all there are 44 unique names on these lists and 43 are White. (Congrats, John Thompson III.) Now, there aren’t a lot of Black head coaches to begin with, so you wouldn’t expect a lot of diversity. Still, with 24% of non-HBCU head coaches being Black, and these lists having just 2% of Black head coaches, we might want to re-examine our biases in this area.<\/span><\/p>\n

Now, wisely, nobody appears to have made such a list in the last five years. Because as you can see, people (other coaches, no less) either think Black coaches aren’t qualified for such consideration or they have a blind spot. I would invite anyone to try to come up with an objective way to measure in-game coaching performance that produces a list where the top 25 coaches are White. (I\u2019ve tried and <\/span>found no racial trend<\/span><\/a>.)<\/span><\/p>\n

But administrators, tasked with finding the person to run their multi-million dollar basketball program, have a clear bias in selecting coaches. And that\u2019s just one way the eye test fails in a profound way. We could name others – and have on this blog from <\/span>time<\/span><\/a> to <\/span>time<\/span><\/a> – but let\u2019s get to the point: <\/span>Playing basketball in a pandemic shouldn\u2019t require teams to have to choose between scheduling the maximum number of games and risking the spread of a highly-contagious virus without a vaccine just for the sake of helping out ten people who want more games to watch.<\/p>\n

And it doesn’t have to be this way. In the event no non-conference games were played, the simplest fix to make a ratings system work would be to give teams from each conference their own ratings baseline. I’d use my pre-season ratings to establish this baseline but the NET doesn\u2019t have pre-season rating. However, one could also do this with the average of previous seasons of a conference\u2019s rating. (The trick is you have to use the <\/span>current<\/span><\/i> conference membership to compute those ratings.)\u00a0<\/span><\/p>\n

One might say that it\u2019s unfair to assume a certain conference will be better than another. Under normal circumstances, perhaps. But unusual circumstances require creative solutions. And this works well in practice. The ACC has never been worse than the WCC on average, despite the fact the WCC usually has a team that would be capable of competing for an ACC title. We know that the Big Ten will be better than the AAC, which will be better than the Mountain West, and all of those leagues are always among the top ten conferences.<\/span><\/p>\n

To illustrate how well this works, let\u2019s first look at how my pre-season ratings have fared for both teams and conferences over the past decade.<\/span><\/p>\n

\"\"\"\"<\/span><\/p>\n

They are significantly more accurate predicting the end-of-season rating for conferences, which shouldn\u2019t be surprising. The errors for each team are random, and if you group a bunch of teams together, those random errors tend to cancel out.<\/span><\/p>\n

It\u2019s more interesting to look at past seasons to see what the ratings would look like in a situation where no non-conference games were played and this method was used.\u00a0<\/span><\/p>\n

Here\u2019s a summary from 2019:<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0Reg. Season\u00a0 \u00a0 \u00a0 Conf-only<\/b>\r\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0Team\u00a0 \u00a0 \u00a0 Rk\u00a0 AdjEM\u00a0 \u00a0 Rank\u00a0 AdjEM<\/b>\r\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0Virginia \u00a0 \u00a0 \u00a0 1\u00a0 35.65 \u00a0 \u00a0 \u00a0 1\u00a0 36.58<\/span>\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0Gonzaga \u00a0 \u00a0 \u00a0 2\u00a0 32.86 \u00a0 \u00a0 \u00a0 2\u00a0 35.69<\/span>\r\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0Duke \u00a0 \u00a0 \u00a0 3\u00a0 31.96 \u00a0 \u00a0 \u00a0 7\u00a0 29.46<\/span>\r\n \u00a0\u00a0Michigan St. \u00a0 \u00a0 \u00a0 4\u00a0 31.11 \u00a0 \u00a0 \u00a0 3\u00a0 32.48<\/span>\r\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0Michigan \u00a0 \u00a0 \u00a0 5\u00a0 29.67 \u00a0 \u00a0 \u00a0 5\u00a0 29.85<\/span>\r\n North Carolina \u00a0 \u00a0 \u00a0 6\u00a0 29.19 \u00a0 \u00a0 \u00a0 4\u00a0 30.26<\/span>\r\n \u00a0\u00a0\u00a0\u00a0\u00a0Tennessee \u00a0 \u00a0 \u00a0 7\u00a0 28.33 \u00a0 \u00a0 \u00a0 8\u00a0 27.76<\/span>\r\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0Kentucky \u00a0 \u00a0 \u00a0 8\u00a0 27.48 \u00a0 \u00a0 \u00a0 6\u00a0 29.51<\/span>\r\n \u00a0\u00a0\u00a0\u00a0Texas Tech \u00a0 \u00a0 \u00a0 9\u00a0 26.77\u00a0 \u00a0 \u00a0 10\u00a0 26.94<\/span>\r\n \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0Purdue\u00a0 \u00a0 \u00a0 10\u00a0 26.04 \u00a0 \u00a0 \u00a0 9\u00a0 27.49<\/span><\/pre>\n
The data on the left are the actual ratings prior to the field selection and the numbers on the right include just conference games using the baseline method (and conference tournaments). It\u2019s neat that each of the top ten teams is still in the top ten even using limited data with no cross-conference information. We have less data and yet, for practical purposes, the top teams end up where they would have been with a full season.\u00a0<\/span><\/p>\n
That\u2019s just the 2019 season and I have done preseason ratings since the 2011 season. But I lost the ratings for that season so we\u2019ll have to settle for data since the 2012 season. Over those eight years, an average of 8.6 teams from the actual top ten ended up in the conference-only top ten.\u00a0<\/span><\/p>\n
Going deeper, there were 14 teams that finished in the top 25 of either the actual ratings or the conference-only scenario and had a ranking difference of at least ten spots between the two. So only about two per season:<\/span><\/p>\n
 Season Seed\u00a0 \u00a0 \u00a0 \u00a0 \u00a0     Team Actual ConfOnly     diff<\/b>\r\n \u00a0\u00a02012\u00a0 \u00a0 8\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 Iowa St. \u00a0 \u00a0 31 \u00a0 \u00a0 \u00a0 17 \u00a0 \u00a0 \u00a0+14<\/span>\r\n \u00a0\u00a02012\u00a0 \u00a0 5\u00a0 \u00a0 \u00a0 \u00a0 Vanderbilt \u00a0 \u00a0 18\u00a0 \u00a0 \u00a0 \u00a0 8 \u00a0 \u00a0 \u00a0+10<\/span>\r\n \u00a0\u00a02013\u00a0 \u00a0 6 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 Arizona \u00a0 \u00a0 17 \u00a0 \u00a0 \u00a0 28\u00a0 \u00a0 \u00a0 -11<\/span>\r\n \u00a0\u00a02013\u00a0 \u00a0 6 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 Memphis \u00a0 \u00a0 34 \u00a0 \u00a0 \u00a0 23 \u00a0 \u00a0 \u00a0+11<\/span>\r\n \u00a0\u00a02013 \u00a0 11 \u00a0 \u00a0 \u00a0 \u00a0 Minnesota \u00a0 \u00a0 23 \u00a0 \u00a0 \u00a0 39\u00a0 \u00a0 \u00a0 -16<\/span>\r\n \u00a0\u00a02013\u00a0 \u00a0 5 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 VCU \u00a0 \u00a0 20 \u00a0 \u00a0 \u00a0 33\u00a0 \u00a0 \u00a0 -13<\/span>\r\n \u00a0\u00a02014\u00a0 \u00a0 5\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 Oklahoma \u00a0 \u00a0 29 \u00a0 \u00a0 \u00a0 17 \u00a0 \u00a0 \u00a0+12<\/span>\r\n \u00a0\u00a02016\u00a0 \u00a0 4\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 Duke \u00a0 \u00a0 14 \u00a0 \u00a0 \u00a0 28\u00a0 \u00a0 \u00a0 -14<\/span>\r\n \u00a0\u00a02016\u00a0 \u00a0 -\u00a0 \u00a0 \u00a0 \u00a0 Louisville \u00a0 \u00a0 10 \u00a0 \u00a0 \u00a0 20\u00a0 \u00a0 \u00a0 -10<\/span>\r\n \u00a0\u00a02016 \u00a0 14 Stephen F. Austin \u00a0 \u00a0 44 \u00a0 \u00a0 \u00a0 17 \u00a0 \u00a0 \u00a0+27<\/span>\r\n \u00a0\u00a02017\u00a0 \u00a0 6 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 SMU \u00a0 \u00a0 14\u00a0 \u00a0 \u00a0 \u00a0 3 \u00a0 \u00a0 \u00a0+11<\/span>\r\n \u00a0\u00a02018\u00a0 \u00a0 7\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 Nevada \u00a0 \u00a0 24 \u00a0 \u00a0 \u00a0 39\u00a0 \u00a0 \u00a0 -15<\/span>\r\n \u00a0\u00a02018\u00a0 \u00a0 5\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 Ohio St. \u00a0 \u00a0 15\u00a0 \u00a0 \u00a0 \u00a0 5 \u00a0 \u00a0 \u00a0+10<\/span>\r\n \u00a0\u00a02019\u00a0 \u00a0 4\u00a0 \u00a0 \u00a0 \u00a0 Kansas St. \u00a0 \u00a0 23 \u00a0 \u00a0 \u00a0 11 \u00a0 \u00a0 \u00a0+12<\/span><\/pre>\n
And there were 17 teams that finished in the top 50 of either the actual ratings or the conference-only scenario and had a ranking difference of at least 20 spots between the two.\u00a0<\/span><\/p>\n
 Season Seed\u00a0 \u00a0 \u00a0 \u00a0 \u00a0     Team Actual ConfOnly     diff<\/b>\r\n \u00a0\u00a02012 \u00a0 14 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 BYU \u00a0 \u00a0 49 \u00a0 \u00a0 \u00a0 69\u00a0 \u00a0 \u00a0 -20<\/span>\r\n \u00a0\u00a02012 \u00a0 \u00a0-\u00a0 Middle Tennessee \u00a0 \u00a0 47 \u00a0 \u00a0 \u00a0 82\u00a0 \u00a0 \u00a0 -35<\/span>\r\n \u00a0\u00a02012\u00a0 \u00a0 6\u00a0 \u00a0 \u00a0 \u00a0 Murray St. \u00a0 \u00a0 38 \u00a0 \u00a0 \u00a0 61\u00a0 \u00a0 \u00a0 -23<\/span>\r\n \u00a0\u00a02012 \u00a0 \u00a0-\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 Oregon \u00a0 \u00a0 71 \u00a0 \u00a0 \u00a0 49 \u00a0 \u00a0 \u00a0+22<\/span>\r\n \u00a0\u00a02012 \u00a0 12 \u00a0 \u00a0 South Florida \u00a0 \u00a0 59 \u00a0 \u00a0 \u00a0 39 \u00a0 \u00a0 \u00a0+20<\/span>\r\n \u00a0\u00a02012 \u00a0 14 \u00a0 St. Bonaventure \u00a0 \u00a0 66 \u00a0 \u00a0 \u00a0 46 \u00a0 \u00a0 \u00a0+20<\/span>\r\n \u00a0\u00a02012\u00a0 \u00a0 - \u00a0 \u00a0 \u00a0 \u00a0 Tennessee \u00a0 \u00a0 56 \u00a0 \u00a0 \u00a0 34 \u00a0 \u00a0 \u00a0+22<\/span>\r\n \u00a0\u00a02012 \u00a0 \u00a0-\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 UCLA \u00a0 \u00a0 64 \u00a0 \u00a0 \u00a0 44 \u00a0 \u00a0 \u00a0+20<\/span>\r\n \u00a0\u00a02013\u00a0 \u00a0 9 \u00a0 \u00a0 \u00a0 \u00a0 Villanova \u00a0 \u00a0 54 \u00a0 \u00a0 \u00a0 31 \u00a0 \u00a0 \u00a0+23<\/span>\r\n \u00a0\u00a02014 \u00a0 13 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 Tulsa \u00a0 \u00a0 79 \u00a0 \u00a0 \u00a0 47 \u00a0 \u00a0 \u00a0+32<\/span>\r\n \u00a0\u00a02014\u00a0 \u00a0 - \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 Vermont \u00a0 \u00a0 74 \u00a0 \u00a0 \u00a0 42 \u00a0 \u00a0 \u00a0+32<\/span>\r\n \u00a0\u00a02016 \u00a0 14 Stephen F. Austin \u00a0 \u00a0 44 \u00a0 \u00a0 \u00a0 17 \u00a0 \u00a0 \u00a0+27<\/span>\r\n \u00a0\u00a02018 \u00a0 11 \u00a0 \u00a0 \u00a0 Arizona St. \u00a0 \u00a0 45 \u00a0 \u00a0 \u00a0 86\u00a0 \u00a0 \u00a0 -41<\/span>\r\n \u00a0\u00a02018 \u00a0 \u00a0-\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 Nebraska \u00a0 \u00a0 57 \u00a0 \u00a0 \u00a0 30 \u00a0 \u00a0 \u00a0+27<\/span>\r\n \u00a0\u00a02018\u00a0 \u00a0 7 \u00a0 \u00a0 \u00a0 \u00a0 Texas A&M \u00a0 \u00a0 30 \u00a0 \u00a0 \u00a0 59\u00a0 \u00a0 \u00a0 -29<\/span>\r\n \u00a0\u00a02019\u00a0 \u00a0 9\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 Oklahoma \u00a0 \u00a0 38 \u00a0 \u00a0 \u00a0 59\u00a0 \u00a0 \u00a0 -21<\/span>\r\n \u00a0\u00a02019 \u00a0 \u00a0-\u00a0 \u00a0 South Carolina \u00a0 \u00a0 73 \u00a0 \u00a0 \u00a0 48 \u00a0 \u00a0 \u00a0+25<\/span><\/pre>\n
There just aren’t a bunch of huge changes. (Somehow 8 of the 17 cases on this list occurred in 2012 alone, which could be the result of my preseason ratings being not as good back then. But also seven of those cases involved a ranking change of between 20 and 23 spots.)<\/p>\n
Now you might say, Stephen F. Austin was not the 17th best team in the country in 2016, Ken. Would you give them a five-seed with this approach? Well, first I\u2019d say that it\u2019s awfully rude of you to judge my work on the worst example among many good ones. Like, maybe I should go to your job and watch you do the thing that you do like 300 times and then find the worst one of those 300 and tweet to your boss about it.\u00a0<\/span><\/p>\n
Armed with the knowledge of how difficult every win is for every game played by every team, you can use any <\/span>number of approaches<\/span><\/a> to figure how impressive a team\u2019s record is against its schedule. We can compare teams that play 13 games to teams that play 25 games. We can compare teams that play no non-conference games to teams that do. No program has to feel pressure to schedule a ton of games in these challenging times in order to impress the selection committee. <\/span><\/p>\n
Nor do we have to worry about top 50 and top 100 wins. And through no fault of their own, some very good teams may not get the chance to play any games against that kind of competition this season.<\/span><\/p>\n
My final plot is a comparison of the ratings from the COVID scenario to the actual pre-tourney ratings. <\/span><\/p>\n
\"\"<\/p>\n
Obviously there are differences. But considering the limitations placed on the system – no conference connections and about 40% less data – it\u2019s acceptable.<\/span><\/p>\n
Could ten humans do better? They\u2019d be fine getting the top teams mostly correct. Though I don\u2019t think they\u2019d be exclusively using the eye test for that. We know which conferences are best year-in and year-out and we can look at the top of the conference standings to get an idea which teams are best. <\/span>But figuring out the last at-large teams, which comes down to distinguishing the difference between the 44th and 45th most-accomplished teams, is an impossible human task even in a normal season.\u00a0<\/span><\/p>\n
No doubt a few non-conference games will help both the subjective and objective approaches. (Although, just because a few non-conference games will be played doesn\u2019t mean the principles discussed here wouldn\u2019t need to be implemented.) If these games can be played safely, great. But they don\u2019t need to happen simply to appease the eye-testers.<\/span><\/p>\n
Whether it\u2019s hiring coaches or determining the whether a college basketball program gets a chance to earn a massive payout and publicity by making the tournament, there\u2019s always been a lack of humility regarding how fallible human decision-making can be when ignoring available objective data. Eventually this will change. We are closer than we were before. But the attitude heading into this season suggests we are still not very close at all.<\/span><\/p>\n","protected":false},"excerpt":{"rendered":"
After months of uncertainty, it\u2019s looking like a college basketball season will exist in some form. The number of games played will be limited and some conferences may not play a non-conference slate.\u00a0 However, there is a misconception promoted by the media and the highest levels of the NCAA, that without non-conference games one cannot […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":[],"categories":[1],"tags":[],"_links":{"self":[{"href":"https:\/\/kenpom.com\/blog\/wp-json\/wp\/v2\/posts\/2392"}],"collection":[{"href":"https:\/\/kenpom.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/kenpom.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/kenpom.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/kenpom.com\/blog\/wp-json\/wp\/v2\/comments?post=2392"}],"version-history":[{"count":24,"href":"https:\/\/kenpom.com\/blog\/wp-json\/wp\/v2\/posts\/2392\/revisions"}],"predecessor-version":[{"id":2436,"href":"https:\/\/kenpom.com\/blog\/wp-json\/wp\/v2\/posts\/2392\/revisions\/2436"}],"wp:attachment":[{"href":"https:\/\/kenpom.com\/blog\/wp-json\/wp\/v2\/media?parent=2392"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/kenpom.com\/blog\/wp-json\/wp\/v2\/categories?post=2392"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/kenpom.com\/blog\/wp-json\/wp\/v2\/tags?post=2392"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}
But also, my ratings are not designed to pick teams for the tournament. It should be used as a very good estimate for how difficult it is to beat a particular team. (This is something that humans seem to be not very good at judging.) Beating SFA would have been viewed as beating the 17th-best team. Nobody beat them in conference play that season anyway, so <\/span>the question is moot<\/span><\/a>.<\/span><\/p>\n