2013 Projection Systems Review: RA9 and WHIP

Thursday I compared five projection systems with their projections for weighted on base average. Today I’m looking at two different pitching categories, runs allowed per 9 innings (RA9) and WHIP, walks plus hits per innings pitched.

Tom Tango kindly highlighted my post yesterday, but suggested that one of my charts was useless, because it did not compare the systems on the same players. So today I’ll just have two charts per stat, one using only those players projected by all systems, and another filling in a “missing” value for any player a system did not project.

The systems are the same five:

And, as before, I’m calculating RMSE and MAE, and sorting by the former. The error is bias-adjusted, so I’m first comparing each player’s stat to the average of the system, and then I compare those deltas with the actual delta. I’m using actual innings pitched to weight the averages. First up, RA9.

These are the 373 pitchers projected by all the systems:

Source Num Avg RA9 MAE RMSE
Actual 373 4.1006 0.0000 0.0000
Steamer/Razzball 373 4.2557 0.8416 1.1176
Consensus 373 4.2288 0.8465 1.1293
Marcel 373 4.1553 0.8836 1.1620
MORPS 373 4.1651 0.8810 1.1735
RotoValue 373 4.2009 0.8842 1.1809
CAIRO 373 4.3407 0.9230 1.2250
y2012 373 4.0932 1.1731 1.5998

Steamer again is the best, followed by the Consensus. There’s a much higher level of errors here, at 25% or more of the statistic, compared to about 10-12% when projecting wOBA. Also, the systems vary more in how well they do. Pitching is indeed harder to predict.

Next I’ve assumed an RA9 of 0.50 above the league average for any players not projected by a system:

Source Num RA9 MLB RA9 StdDev MAE RMSE Missing
Actual 664 4.1765 664 4.1752 1.4593 0.0000 0.0000 0
Steamer/Razzball 515 4.2975 664 4.3477 0.5198 0.9656 1.3825 198
Consensus 872 4.3286 664 4.3403 0.5480 0.9836 1.4049 113
MORPS 499 4.2385 664 4.2846 0.6549 1.0162 1.4381 226
Marcel 841 4.3405 664 4.2872 0.5407 1.0230 1.4390 136
CAIRO 489 4.3972 664 4.4681 0.6919 1.0400 1.4644 234
RotoValue 836 4.3470 664 4.3361 0.7283 1.0521 1.4916 133
y2012 630 4.3382 664 4.4439 1.9624 1.4250 2.3074 168

This comparison shakes up the order quite a bit compared to using the subset of all projected pitchers. Steamer still had the lowest errors, but MORPS and CAIRO now do much better, while Marcel, the Consensus average, and my RotoValue projection did relatively worse in this test.Steamer and Consensus are still the top two, while MORPS moves ahead of Marcel, and CAIRO passes RotoValue. Including more, and worse, pitchers raises the errors of all the systems.

Now on to WHIP, for the 373 pitchers projected by all:

Source Num Avg WHIP MAE RMSE
Actual 373 1.2816 0.0000 0.0000
Consensus 373 1.3032 0.1359 0.1838
RotoValue 373 1.2848 0.1360 0.1844
Steamer/Razzball 373 1.3045 0.1348 0.1845
Marcel 373 1.2716 0.1396 0.1870
MORPS 373 1.2913 0.1405 0.1925
CAIRO 373 1.3582 0.1735 0.2292
y2012 373 1.2591 0.1798 0.2448

The RMSEs here are a little tighter than for RA9, but till much wider than for wOBA. My model edged out Steamer/Razzball in RMSE here, but was a bit behind it in MAE, while the Consensus did even better than mine.

Now here’s adding league-average WHIP plus 0.100 for any players not projected:

Source Num WHIP MLB WHIP StdDev MAE RMSE Missing
Actual 664 1.2995 664 1.2993 0.2354 0.0000 0.0000 0
Steamer/Razzball 515 1.3139 664 1.3250 0.0895 0.1522 0.2176 198
Marcel 841 1.3055 664 1.2952 0.0911 0.1589 0.2232 136
Consensus 872 1.3231 664 1.3280 0.1101 0.1579 0.2233 113
RotoValue 836 1.3136 664 1.3097 0.1100 0.1592 0.2246 133
MORPS 499 1.3129 664 1.3226 0.1335 0.1626 0.2307 226
CAIRO 489 1.3720 664 1.3874 0.1733 0.1864 0.2529 234
y2012 630 1.3049 664 1.3219 0.3179 0.2175 0.3630 168

Steamer comes out on top in this test, while my model and the Consensus do drop back. Pitching statistics take longer to stabilize, so it’s not so surprising to see the relatively higher errors here. Very observant readers might have noticed a slight difference in the actual WHIP and RA9 in the two columns of the second tables. That’s an artifact of a small number of pitchers who did not record an out, and thus had 0 IP. When I’m computing errors, these pitchers aren’t included (since I weight by IP, they have a weight of 0), but their runs, hits, and walks allowed do slightly increase the aggregate actual statistics.

7 February 2014: I found and  fixed a bug in my code that generated the tables using a worse than league average value for missing players, so the affected tables in this post were changed to reflect that.

About Geoff

Dad, hacker, fantasy sports entrepreneur.
This entry was posted in Major League Baseball, Projections, Sabermetrics. Bookmark the permalink.