Rating Systems Challenge: Historical Performance

This post is part of a series about the 2016 NCAA Tournament. Follow along here.

Photo credit: Me (Some Rights Reserved)

Over the years, some rating systems have performed better than others in an NCAA Tournament bracket. A minority of systems successfully outperform chalk on a consistent basis. The table below compares each system based on how each performed in the ESPN Tournament Challenge scoring system over the last five years.

*PAC = Points above chalk. For systems that I have tracked for fewer than five years, I regressed them to even performance against a chalk bracket for years with no data.

*In the future, I'll probably retroactively calculate Massey's performance, as that data is easily retrievable.

It's stunning how few systems reliably outperform a strategy that merely involves picking the better seed (Pure Chalk). If you are looking to use a ratings system to pick a bracket in the future, I would suggest picking the ESPN National Bracket's hive mind, as well as Jeff Sagarin's, Nate Silver's and Christopher Long's. The ESPN Insider's game simulator, USA Today Preseason Coaches Poll (with chalk as a tiebreaker), and Sports Reference's adjusted net ratings may also be worth a look.

Oh, and RPI? Please go away.