Playing-time Neutral Projection Comparison

In response to Jared Cross’s suggestion, I’ve done one more set of RotoValue comparisons of projection systems. This time, I’m taking players’ actual 2012 AB or IP, and scaling the projections from each system to match that level of playing time. Also, since commenters Rudy Gamble and mcbrown were asking for ZiPS data, I’ve included that system as well. Since ZiPS didn’t project saves at all, I’m following mcbrown’s suggestion of using data from another source: I set ZiPS save projections to be the average of the save projections from the other four tested systems.
So in this run, the systems are:

Jared is one of the developers of the Steamer system, which has been the strongest system in my previous 2012 testing. This time, however, Steamer no longer dominates all the formats, sometimes finishing mid-pack, and in one configuration it had the highest errors of any projection system. As I expected, when adjusting for playing time my old RotoValue projections do worse, as does actual 2011 data. Much like my previous examinations of rate stats, 2011 was last in all the rankings, with RotoValue usually penultimate (which sounds more impressive than second-to-last; hey, this is in part a promotional blog).
I suspect that 2011 is worse here because, somewhat counter-intuitively, using 2011 playing time may be better than some of the playing time forecasts of other systems, so the gap narrows. But when the projected stats are all prorated to fit actual playing time, 2011 is by far the worst. Similarly, it seems that my old RotoValue model did markedly better at playing time projections than some of these systems, and so in the unadjusted comparison it was relatively stronger than it is here. Similarly much of Steamer’s advantage in my previous testing is probably due to playing time (although it did very well in pitching percentages, too).
Of course, a fantasy player cares about both a player’s skill level and his playing time. I reiterate that all of this is just one year of data; if Steamer continues to do well in both it may indeed be a better one-stop solution for fantasy players. I do note that this year it is offering two sets of forecasts, one using Razzball‘s playing time projections, and another using fan projections at fangraphs.com. I find it interesting that the Steamer folks are now outsourcing playing time projections, given how strong they seemed to be in 2012. But in 2012 I only had access to freely available projections which usually did not particularly worry about playing time.
As for my RotoValue MLB projections, I’m improving my rate statistics, and also taking another look at playing time. For 2012, I certainly adjusted for known injuries, and I applied a “normalizing” algorithm to ensure that team totals (and even totals within a likely position on a team) for playing time seemed sane. That likely helped overcome relatively weak rate statistics projections for my model.
RotoValue.com’s fantasy pages will show whatever projection sources I can readily load in. And I may also generate a consensus average from the projection sources I have, as previous research has found consensus projections seem more consistently strong than even the best individual forecast. Or, rather crowd-sourcing works as well or better than anything else in baseball projections.

As in my last post, RMSE0 and MAE0 are the root mean square error and mean absolute error when I assume any player not projected by a system has a price of $0.00.
First up, the 4×4 AL League – $260 Cap:

Source Num Avg Price MAE RMSE
2012 240 10.832 0.000 0.000
SteamerAdj 240 10.833 5.634 7.242
MarcelAdj 240 10.833 5.692 7.796
CAIROAdj 240 10.833 5.751 7.841
ZiPSAdj 240 10.833 5.984 7.932
RotoValueAdj 240 10.833 6.513 8.361
2011Adj 240 10.833 8.627 10.724

Top 240 actual players players in 2012

Source Num Avg Price MAE RMSE MAE0 RMSE0
2012 240 10.832 0.000 0.000 0.000 0.000
MarcelAdj 222 10.421 5.235 7.034 5.360 7.236
ZiPSAdj 228 9.884 5.557 7.213 5.746 7.653
SteamerAdj 231 10.069 5.599 7.441 5.541 7.385
CAIROAdj 235 9.600 5.593 7.584 5.530 7.519
RotoValueAdj 211 10.450 6.059 7.866 6.200 8.422
2011Adj 218 8.822 8.534 10.964 8.252 10.709

This time Steamer was lowest among the top 240 projections from each system, but it wound up behind Marcel when looking at the top 240 actual players, and also behind ZiPS when only averaging projected players.
Now the 4×4 NL League – $260 Cap:

Source Num Avg Price MAE RMSE
2012 230 11.304 0.000 0.000
CAIROAdj 230 11.304 6.147 8.051
ZiPSAdj 230 11.304 6.408 8.076
SteamerAdj 230 11.304 6.636 8.340
MarcelAdj 230 11.304 6.809 8.703
RotoValueAdj 230 11.304 6.993 8.916
2011Adj 230 11.304 9.630 12.141

Top 230 actual players players in 2012

Source Num Avg Price MAE RMSE MAE0 RMSE0
2012 230 11.304 0.000 0.000 0.000 0.000
ZiPSAdj 221 10.118 6.284 7.984 6.288 7.978
CAIROAdj 229 9.746 6.220 8.098 6.197 8.081
MarcelAdj 222 10.108 6.453 8.256 6.450 8.302
RotoValueAdj 209 10.494 6.851 8.915 6.689 8.766
SteamerAdj 229 9.531 7.182 9.209 7.155 9.189
2011Adj 215 8.275 9.517 12.003 9.270 11.782

Here CAIRO and ZiPS did quite well, while Steamer was mid-pack among systems’ best 230, and ahead of only 2011 data among 2012’s top 230 players.
In the 5×5 AL League $260 Cap:

Source Num Avg Price MAE RMSE
2012 240 10.833 0.000 0.000
SteamerAdj 240 10.833 4.668 6.005
ZiPSAdj 240 10.833 4.791 6.308
MarcelAdj 240 10.833 4.668 6.325
CAIROAdj 240 10.833 4.733 6.427
RotoValueAdj 240 10.833 5.361 6.816
2011Adj 240 10.833 8.173 10.068

Top 240 actual players players in 2012

Source Num Avg Price MAE RMSE MAE0 RMSE0
2012 240 10.833 0.000 0.000 0.000 0.000
MarcelAdj 222 10.882 4.306 5.808 4.495 6.190
ZiPSAdj 228 10.375 4.629 6.036 4.831 6.548
SteamerAdj 230 10.451 4.635 6.070 4.573 6.009
CAIROAdj 234 10.103 4.653 6.258 4.579 6.188
RotoValueAdj 213 10.869 4.967 6.357 5.213 7.175
2011Adj 219 8.860 8.200 10.520 7.959 10.319

This ordering is very much like the 4×4 AL, and Steamer was stronger here, but not dominant.
The 5×5 NL League – $260 Cap:

Source Num Avg Price MAE RMSE
2012 230 11.304 0.000 0.000
CAIROAdj 230 11.304 5.016 6.558
ZiPSAdj 230 11.304 5.238 6.600
SteamerAdj 230 11.304 5.283 6.698
MarcelAdj 230 11.304 5.497 6.944
RotoValueAdj 230 11.304 5.617 7.105
2011Adj 230 11.304 9.071 11.392

Top 230 actual players players in 2012

Source Num Avg Price MAE RMSE MAE0 RMSE0
2012 230 11.304 0.000 0.000 0.000 0.000
ZiPSAdj 223 10.748 5.111 6.617 5.193 6.703
CAIROAdj 229 10.473 5.132 6.767 5.113 6.752
MarcelAdj 224 10.697 5.339 6.838 5.408 7.002
SteamerAdj 229 10.451 5.577 7.237 5.556 7.221
RotoValueAdj 213 10.969 5.569 7.374 5.562 7.419
2011Adj 218 8.428 8.863 11.215 8.755 11.134

Again it’s ZiPS and CAIRO that were the best NL projection systems. Adding the extra two categories made little difference.
Finally the shallow 5×5 Mixed League – $260 cap:

Source Num Avg Price MAE RMSE
2012 230 11.304 0.000 0.000
SteamerAdj 230 11.304 7.564 9.583
ZiPSAdj 230 11.304 7.464 9.635
CAIROAdj 230 11.304 7.350 9.672
MarcelAdj 230 11.304 7.741 10.052
RotoValueAdj 230 11.304 8.238 10.512
2011Adj 230 11.304 12.369 15.394

Top 230 actual players players in 2012

Source Num Avg Price MAE RMSE MAE0 RMSE0
2012 230 11.304 0.000 0.000 0.000 0.000
MarcelAdj 223 9.565 6.970 8.993 7.048 9.062
ZiPSAdj 225 9.387 7.354 9.454 7.403 9.509
SteamerAdj 229 9.097 7.585 9.737 7.574 9.722
CAIROAdj 230 9.097 7.429 9.741 7.429 9.741
RotoValueAdj 221 9.101 7.781 10.058 7.897 10.371
2011Adj 223 5.096 11.786 14.839 11.680 14.703

In this format, Steamer was best when judged by its own top players, while Marcel wins when judged by 2012’s best.
One other comment – while I’ve not published the tables, I did rerun comparisons including ZiPS with the average save totals of other systems, but no other changes. When doing that, ZiPS was not among the best systems in any of the configurations I ran.

1 comment

Leave a comment

Your email address will not be published. Required fields are marked *