As people who have followed my twitter feed have figured out, I’m more than a little uncomfortable with the use of plus/minus data in college basketball analysis.

I once produced something called an HD Box score which contained +/- data. It was a thrill to make the breakthrough to compute such data because after all, plus-minus captures everything. All of those hustle plays that lead to points are rewarded. The guy who’s able to force his man into repeatedly missing shots gets rewarded for this defense. It’s perfect. And I basically said some dumb things along those lines back then.

Only, my experience since then has led me to believe that it’s far from perfect. This well-intentioned analysis (which is accompanied with plenty of caution) indicates that Kyrie Irving was a huge drain on Duke at the beginning of the season. Would you go to Coach K and tell him with a straight face to bench Kyrie Irving (if he didn’t have that whole toe problem already)? With Irving’s offensive stats, he would literally have to be shooting for the other team to be a net drain on the Blue Devils. That’s just my conjecture, though. Legitimate conjecture that makes basketball sense, I would say, but conjecture nonetheless.

I decided to do a test to determine the value of plus-minus-related stats. I created a player who has absolutely no impact on the game. For both his team and his opponent, there’s a 3% chance of scoring one point on a possession, a 30% chance of scoring two, and a 15% chance of scoring three. These probabilities are unchanged whether the player is in the game or on the bench.

I am assuming that every game is 70 possessions long and that my player will be on the floor for 35 possessions and on the bench for 35 possessions. With minimal programming experience you can code up such a scenario, run it, and track Mr. Irrelevant’s plus-minus. I played 20 such games and tallied said player’s plus-minus for those possessions for which he was on and off the court. For a limited time you can go to this page and refresh it to do as many such tests as you want.

My first experiment resulted in the following output…

      On Off MOV
 1.   5   3   8
 2.   6  -3   3
 3.  -1   7   6
 4.  -4  10   6
 5.  11 -11   0
 6.  -1   4   3
 7.  13   7  20
 8.  -8   7  -1
 9. -11   7  -4
10.  -9  23  14
11. -14   1 -13
12.   6  -4   2
13. -12  -7 -19
14. -13  -6 -19
15.   4  -7  -3
16. -13 -12 -25
17.   9   6  15
18.  -6  -2  -8
19.  -1   9   8
20.  -3 -17 -20
Tot -42  15 -27

If plus-minus were truly just, there would be nothing but zeroes on the list. Sure, there’s going to be some noise in any statistic, but this illustrates to me that the amount of noise in plus-minus is unacceptable. Heck, the amount of noise in final score is pretty amazing. My team is playing a schedule against teams it’s evenly-matched with. Yet in game 16 we were outscored by 25 points! You can imagine the game-stories written about that one. How my team was out-coached, or how the opponent is just a bad matchup for my team, or how my team quit, showed no heart, etc.

But I can assure you, none of that occurred! It was just a bunch of bits and bytes that produced the outcome. Think about that the next time somebody feeds you an explanation of why a lopsided result occurred in what was expected to be an even game. (For reasons too boring to explain here, this methodology overstates the variance in final margin by about 20%. Still, you get the point.)

In this case, my team was -42 with my player on the court and +15 with him off. Do the math and his value works out to -5.7 points per 40 minutes. The conclusion is that I need to bench this guy. But since this is an experiment, I know he actually has no impact on my team whether he’s in the game or not. It’s simply randomness that is creating these numbers. And keep in mind this is a more controlled environment than the real world. The quality of my player’s teammates remains constant. The quality of the players on the other team is constant. In the real world, there is variation in both of these things, which only adds more noise. In addition, the real world would contain games that are big-time mismatches, with plenty of garbage time. That would clutter things up as well.

Just for consistency, I decided to run my 20-game experiment 50 times. The results should give a more accurate assessment of the error associated with plus-minus. It wasn’t difficult to get a crazier result than the one described above. The third trial in this new run gave my player a -43 on the floor and a +52 off the floor. A few trials later, my guy was +48 on and -73 off. My dud had become a stud. I hope he doesn’t get hurt! For the 50 sets of 20 games, the average error in measuring my worthless player was 4.8 points per 40 minutes. There are few players in the country who, if they got hurt, would move the Vegas line for their team by more than a couple points.

One can imagine if we had an eight-man rotation of equal players that after 20 games we would see enough variation among players to convince some people that the best player was much better than the worst.

So where does this leave us? I will state my beliefs here:

1) In-game plus minus is useless. As you can see from the first trial above, there were games when my player looked very good and games when he looked bad. Sure there are things that a player does to help or hurt his team that are not reflected in the box score, but plus-minus for a single game is often not reflective of this. If a player has a significant effect on his team by doing things not accounted for in the box score, I suggest you actually see those things occur before giving the player credit or blame.

2) Multi-game plus-minus isn’t much better. I suppose there are cases where there’s enough of a difference over a few games to draw a meaningful conclusion about a player. But I think that pertains to the kinds of players for whom you already know the answer (like Greg Oden, although I don’t necessarily stand by everything I wrote in that piece).

3) Season-long adjusted plus-minus might have limited use. I haven’t talked about adjusted plus-minus, but in the simplest terms possible, it accounts for who a player is playing with and against. It’s useful in the NBA, but the NBA has many more games, more minutes per game, and its star players see a fair amount of time on the bench during non-garbage time. Even so, there are always some curious results with adjusted plus-minus and the most accurate version uses two years worth of data.

4) Respect randomness. This should really be the objective analyst’s credo because it’s where we have an advantage over people that fall in love with their own eyes. It’s true plus-minus captures everything that’s happening, but that includes a whole lot of random things that lead to a hoop or a stop. Things that have nothing to do with the ability of the player you want to analyze. In basketball analysis, we should be filtering out randomness, not embracing it.

In summary, plus-minus, while neat to look at, is a poor tool in college basketball analysis.