Follow me on twitter

The good stuff

At other venues...
  • ($)
  • Deadspin
  • Slate

  • Strategy
  • Whether to foul up 3 late
  • The value of 2-for-1’s

  • Philosophy
  • Brady Heslip’s non-slump
  • The magic of negative motivation
  • A treatise on plus-minus
  • The preseason AP poll is great
  • The magic of negative motivation
  • The lack of information in close-game performance
  • Why I don’t believe in clutchness*

  • Fun stuff
  • The missing 1-point games
  • Which two teams last lost longest ago?
  • How many first-round picks will Kentucky have?
  • Prepare for the Kobe invasion
  • Predicting John Henson's free throw percentage
  • Can Derrick Williams set the three-point accuracy record?
  • Play-by-play Theater: earliest disqualification
  • Monthly Archives

  • July 2014
  • May 2014
  • April 2014
  • March 2014
  • February 2014
  • January 2014
  • December 2013
  • November 2013
  • October 2013
  • September 2013
  • August 2013
  • July 2013
  • June 2013
  • May 2013
  • April 2013
  • March 2013
  • February 2013
  • January 2013
  • December 2012
  • November 2012
  • October 2012
  • September 2012
  • August 2012
  • July 2012
  • June 2012
  • May 2012
  • April 2012
  • March 2012
  • February 2012
  • January 2012
  • December 2011
  • November 2011
  • October 2011
  • September 2011
  • August 2011
  • July 2011
  • June 2011
  • April 2011
  • March 2011
  • February 2011
  • January 2011
  • December 2010
  • November 2010
  • October 2010
  • August 2010
  • July 2010
  • June 2010
  • May 2010
  • April 2010
  • March 2010
  • February 2010
  • January 2010
  • December 2009
  • November 2009
  • October 2009
  • July 2009
  • February 2009
  • January 2009
  • December 2008
  • November 2008
  • October 2007
  • September 2007
  • July 2007
  • June 2007
  • May 2007
  • April 2007
  • March 2007
  • February 2007
  • January 2007
  • December 2006
  • November 2006
  • October 2006
  • September 2006
  • August 2006
  • July 2006
  • June 2006
  • May 2006
  • April 2006
  • March 2006
  • February 2006
  • January 2006
  • December 2005
  • November 2005
  • October 2005
  • September 2005
  • August 2005
  • July 2005
  • June 2005
  • May 2005
  • April 2005
  • March 2005
  • February 2005
  • January 2005
  • December 2004
  • November 2004
  • October 2004
  • September 2004
  • August 2004
  • July 2004
  • June 2004
  • May 2004
  • April 2004
  • March 2004
  • February 2004
  • January 2004
  • December 2003
  • November 2003

  • RSS feed

    A treatise on plus/minus

    by Ken Pomeroy on Monday, January 10, 2011

    As people who have followed my twitter feed have figured out, I’m more than a little uncomfortable with the use of plus/minus data in college basketball analysis.

    I once produced something called an HD Box score which contained +/- data. It was a thrill to make the breakthrough to compute such data because after all, plus-minus captures everything. All of those hustle plays that lead to points are rewarded. The guy who’s able to force his man into repeatedly missing shots gets rewarded for this defense. It’s perfect. And I basically said some dumb things along those lines back then.

    Only, my experience since then has led me to believe that it’s far from perfect. This well-intentioned analysis (which is accompanied with plenty of caution) indicates that Kyrie Irving was a huge drain on Duke at the beginning of the season. Would you go to Coach K and tell him with a straight face to bench Kyrie Irving (if he didn’t have that whole toe problem already)? With Irving’s offensive stats, he would literally have to be shooting for the other team to be a net drain on the Blue Devils. That’s just my conjecture, though. Legitimate conjecture that makes basketball sense, I would say, but conjecture nonetheless.

    I decided to do a test to determine the value of plus-minus-related stats. I created a player who has absolutely no impact on the game. For both his team and his opponent, there’s a 3% chance of scoring one point on a possession, a 30% chance of scoring two, and a 15% chance of scoring three. These probabilities are unchanged whether the player is in the game or on the bench.

    I am assuming that every game is 70 possessions long and that my player will be on the floor for 35 possessions and on the bench for 35 possessions. With minimal programming experience you can code up such a scenario, run it, and track Mr. Irrelevant’s plus-minus. I played 20 such games and tallied said player’s plus-minus for those possessions for which he was on and off the court. For a limited time you can go to this page and refresh it to do as many such tests as you want.

    My first experiment resulted in the following output…

          On Off MOV
     1.   5   3   8
     2.   6  -3   3
     3.  -1   7   6
     4.  -4  10   6
     5.  11 -11   0
     6.  -1   4   3
     7.  13   7  20
     8.  -8   7  -1
     9. -11   7  -4
    10.  -9  23  14
    11. -14   1 -13
    12.   6  -4   2
    13. -12  -7 -19
    14. -13  -6 -19
    15.   4  -7  -3
    16. -13 -12 -25
    17.   9   6  15
    18.  -6  -2  -8
    19.  -1   9   8
    20.  -3 -17 -20
    Tot -42  15 -27

    If plus-minus were truly just, there would be nothing but zeroes on the list. Sure, there’s going to be some noise in any statistic, but this illustrates to me that the amount of noise in plus-minus is unacceptable. Heck, the amount of noise in final score is pretty amazing. My team is playing a schedule against teams it’s evenly-matched with. Yet in game 16 we were outscored by 25 points! You can imagine the game-stories written about that one. How my team was out-coached, or how the opponent is just a bad matchup for my team, or how my team quit, showed no heart, etc.

    But I can assure you, none of that occurred! It was just a bunch of bits and bytes that produced the outcome. Think about that the next time somebody feeds you an explanation of why a lopsided result occurred in what was expected to be an even game. (For reasons too boring to explain here, this methodology overstates the variance in final margin by about 20%. Still, you get the point.)

    In this case, my team was -42 with my player on the court and +15 with him off. Do the math and his value works out to -5.7 points per 40 minutes. The conclusion is that I need to bench this guy. But since this is an experiment, I know he actually has no impact on my team whether he’s in the game or not. It’s simply randomness that is creating these numbers. And keep in mind this is a more controlled environment than the real world. The quality of my player’s teammates remains constant. The quality of the players on the other team is constant. In the real world, there is variation in both of these things, which only adds more noise. In addition, the real world would contain games that are big-time mismatches, with plenty of garbage time. That would clutter things up as well.

    Just for consistency, I decided to run my 20-game experiment 50 times. The results should give a more accurate assessment of the error associated with plus-minus. It wasn’t difficult to get a crazier result than the one described above. The third trial in this new run gave my player a -43 on the floor and a +52 off the floor. A few trials later, my guy was +48 on and -73 off. My dud had become a stud. I hope he doesn’t get hurt! For the 50 sets of 20 games, the average error in measuring my worthless player was 4.8 points per 40 minutes. There are few players in the country who, if they got hurt, would move the Vegas line for their team by more than a couple points.

    One can imagine if we had an eight-man rotation of equal players that after 20 games we would see enough variation among players to convince some people that the best player was much better than the worst.

    So where does this leave us? I will state my beliefs here:

    1) In-game plus minus is useless. As you can see from the first trial above, there were games when my player looked very good and games when he looked bad. Sure there are things that a player does to help or hurt his team that are not reflected in the box score, but plus-minus for a single game is often not reflective of this. If a player has a significant effect on his team by doing things not accounted for in the box score, I suggest you actually see those things occur before giving the player credit or blame.

    2) Multi-game plus-minus isn’t much better. I suppose there are cases where there’s enough of a difference over a few games to draw a meaningful conclusion about a player. But I think that pertains to the kinds of players for whom you already know the answer (like Greg Oden, although I don’t necessarily stand by everything I wrote in that piece).

    3) Season-long adjusted plus-minus might have limited use. I haven’t talked about adjusted plus-minus, but in the simplest terms possible, it accounts for who a player is playing with and against. It’s useful in the NBA, but the NBA has many more games, more minutes per game, and its star players see a fair amount of time on the bench during non-garbage time. Even so, there are always some curious results with adjusted plus-minus and the most accurate version uses two years worth of data.

    4) Respect randomness. This should really be the objective analyst’s credo because it’s where we have an advantage over people that fall in love with their own eyes. It’s true plus-minus captures everything that’s happening, but that includes a whole lot of random things that lead to a hoop or a stop. Things that have nothing to do with the ability of the player you want to analyze. In basketball analysis, we should be filtering out randomness, not embracing it.

    In summary, plus-minus, while neat to look at, is a poor tool in college basketball analysis.