Rating System Challenge: Ken Pomeroy, Nate Silver and Chalk Looking Strong

Through the first two full rounds, the Pomeroy system is leading the pack. Those of you who picked a Pomeroy-based bracket currently lead 97.9% of all ESPN Tournament Challenge entrants. Nate Silver's FiveThirtyEight bracket (which is partially based on Pomeroy) is in second, trailing by 10 points and 2 percentiles.

Then again, those of you who picked pure chalk look just as smart as Nate Silver at the moment, so let's not read too much into this too quickly.

It's early yet, and it's beginning to look like March 2011 is particularly idiosyncratic. Pomeroy's bracket is only outperforming pure chalk by the equivalent of one win in the field of sixty-four. FiveThirtyEight has performed exactly as well as pure chalk. Every other system is worse, and in some cases much worse.

If we can't say that Pomeroy or any other system has significantly outperformed simple seeding, then we can't say that any of these systems are particularly useful. At least not yet.

RNK	ENTRY	R64	R32	S16	E8	FF	NCG	CHAMPION	PPR	TOTAL	PCT
1	Pomeroy	260	200	0	0	0	0	Ohio St	1080	460	98.0
2	FiveThirtyEight	250	200	0	0	0	0	Ohio St	1120	450	96.0
2	Pure Chalk	250	200	0	0	0	0	Ohio St	1120	450	96.0
4	Nolan Power Index	230	200	0	0	0	0	Kansas	1200	430	85.4
4	Jeff Sagarin	230	200	0	0	0	0	Ohio St	1080	430	85.4
4	Sonny Moore	230	200	0	0	0	0	Ohio St	1080	430	85.4
7	ESPN National Bracket	230	200	0	0	0	0	Ohio St	1120	420	78.0
8	Vegas	210	200	0	0	0	0	Ohio St	1120	410	68.9
8	LRMC Bayesian	250	160	0	0	0	0	Ohio St	1080	410	68.9
10	NCAA Official RPI	220	180	0	0	0	0	Kansas	1200	400	58.4
10	Lunardi RPI	220	180	0	0	0	0	Kansas	1040	400	58.4
12	ESPN Decision Tree	230	160	0	0	0	0	Kansas	1160	390	47.8
13	Preseason AP Poll	240	140	0	0	0	0	Duke	880	380	37.7

More analysis after the jump.

Market-based systems, such as the ESPN National Bracket and the Vegas picks are performing quite poorly so far. The least chalky systems—LRMC Bayesian, Lunardi RPI and the Preseason AP Poll—are bringing up the rear. And woe is anyone who relied on ESPN's Decision Tree to pick their bracket so far.

Why is chalk doing so well in a year with remarkable upsets, including Virginia Commonwealth's shocking advance to the Sweet Sixteen? Simple: the upsets that the systems predicted didn't occur, and the upsets that occurred the systems didn't predict.

Image credit: Wikipedia