In response to Jared Cross’s suggestion, I’ve done one more set of RotoValue comparisons of projection systems. This time, I’m taking players’ actual 2012 AB or IP, and scaling the projections from each system to match that level of playing time. Also, since commenters Rudy Gamble and mcbrown were asking for ZiPS data, I’ve included that system as well. Since ZiPS didn’t project saves at all, I’m following mcbrown’s suggestion of using data from another source: I set ZiPS save projections to be the average of the save projections from the other four tested systems.
So in this run, the systems are:
- CAIRO – from S B of the Replacement Level Yankees Weblog.
- Marcel – the basic projections from Tom Tango, coauthor of The Book.
- Steamer – developed by Jared Cross, Dash Davidson, and Peter Rosenbloom.
- ZiPS – projections from Dan Szymborski of Baseball Think Factory and ESPN.
- RotoValue – my own old projection algorithm.
- 2011Adj – 2011 statistics prorated to 2012 playing time.
Jared is one of the developers of the Steamer system, which has been the strongest system in my previous 2012 testing. This time, however, Steamer no longer dominates all the formats, sometimes finishing mid-pack, and in one configuration it had the highest errors of any projection system. As I expected, when adjusting for playing time my old RotoValue projections do worse, as does actual 2011 data. Much like my previous examinations of rate stats, 2011 was last in all the rankings, with RotoValue usually penultimate (which sounds more impressive than second-to-last; hey, this is in part a promotional blog).
I suspect that 2011 is worse here because, somewhat counter-intuitively, using 2011 playing time may be better than some of the playing time forecasts of other systems, so the gap narrows. But when the projected stats are all prorated to fit actual playing time, 2011 is by far the worst. Similarly, it seems that my old RotoValue model did markedly better at playing time projections than some of these systems, and so in the unadjusted comparison it was relatively stronger than it is here. Similarly much of Steamer’s advantage in my previous testing is probably due to playing time (although it did very well in pitching percentages, too).
Of course, a fantasy player cares about both a player’s skill level and his playing time. I reiterate that all of this is just one year of data; if Steamer continues to do well in both it may indeed be a better one-stop solution for fantasy players. I do note that this year it is offering two sets of forecasts, one using Razzball‘s playing time projections, and another using fan projections at fangraphs.com. I find it interesting that the Steamer folks are now outsourcing playing time projections, given how strong they seemed to be in 2012. But in 2012 I only had access to freely available projections which usually did not particularly worry about playing time.
As for my RotoValue MLB projections, I’m improving my rate statistics, and also taking another look at playing time. For 2012, I certainly adjusted for known injuries, and I applied a “normalizing” algorithm to ensure that team totals (and even totals within a likely position on a team) for playing time seemed sane. That likely helped overcome relatively weak rate statistics projections for my model.
RotoValue.com’s fantasy pages will show whatever projection sources I can readily load in. And I may also generate a consensus average from the projection sources I have, as previous research has found consensus projections seem more consistently strong than even the best individual forecast. Or, rather crowd-sourcing works as well or better than anything else in baseball projections.
As in my last post, RMSE0 and MAE0 are the root mean square error and mean absolute error when I assume any player not projected by a system has a price of $0.00.
First up, the 4×4 AL League – $260 Cap:
Source | Num | Avg Price | MAE | RMSE |
---|---|---|---|---|
2012 | 240 | 10.832 | 0.000 | 0.000 |
SteamerAdj | 240 | 10.833 | 5.634 | 7.242 |
MarcelAdj | 240 | 10.833 | 5.692 | 7.796 |
CAIROAdj | 240 | 10.833 | 5.751 | 7.841 |
ZiPSAdj | 240 | 10.833 | 5.984 | 7.932 |
RotoValueAdj | 240 | 10.833 | 6.513 | 8.361 |
2011Adj | 240 | 10.833 | 8.627 | 10.724 |
Top 240 actual players players in 2012
Source | Num | Avg Price | MAE | RMSE | MAE0 | RMSE0 |
---|---|---|---|---|---|---|
2012 | 240 | 10.832 | 0.000 | 0.000 | 0.000 | 0.000 |
MarcelAdj | 222 | 10.421 | 5.235 | 7.034 | 5.360 | 7.236 |
ZiPSAdj | 228 | 9.884 | 5.557 | 7.213 | 5.746 | 7.653 |
SteamerAdj | 231 | 10.069 | 5.599 | 7.441 | 5.541 | 7.385 |
CAIROAdj | 235 | 9.600 | 5.593 | 7.584 | 5.530 | 7.519 |
RotoValueAdj | 211 | 10.450 | 6.059 | 7.866 | 6.200 | 8.422 |
2011Adj | 218 | 8.822 | 8.534 | 10.964 | 8.252 | 10.709 |
This time Steamer was lowest among the top 240 projections from each system, but it wound up behind Marcel when looking at the top 240 actual players, and also behind ZiPS when only averaging projected players.
Now the 4×4 NL League – $260 Cap:
Source | Num | Avg Price | MAE | RMSE |
---|---|---|---|---|
2012 | 230 | 11.304 | 0.000 | 0.000 |
CAIROAdj | 230 | 11.304 | 6.147 | 8.051 |
ZiPSAdj | 230 | 11.304 | 6.408 | 8.076 |
SteamerAdj | 230 | 11.304 | 6.636 | 8.340 |
MarcelAdj | 230 | 11.304 | 6.809 | 8.703 |
RotoValueAdj | 230 | 11.304 | 6.993 | 8.916 |
2011Adj | 230 | 11.304 | 9.630 | 12.141 |
Top 230 actual players players in 2012
Source | Num | Avg Price | MAE | RMSE | MAE0 | RMSE0 |
---|---|---|---|---|---|---|
2012 | 230 | 11.304 | 0.000 | 0.000 | 0.000 | 0.000 |
ZiPSAdj | 221 | 10.118 | 6.284 | 7.984 | 6.288 | 7.978 |
CAIROAdj | 229 | 9.746 | 6.220 | 8.098 | 6.197 | 8.081 |
MarcelAdj | 222 | 10.108 | 6.453 | 8.256 | 6.450 | 8.302 |
RotoValueAdj | 209 | 10.494 | 6.851 | 8.915 | 6.689 | 8.766 |
SteamerAdj | 229 | 9.531 | 7.182 | 9.209 | 7.155 | 9.189 |
2011Adj | 215 | 8.275 | 9.517 | 12.003 | 9.270 | 11.782 |
Here CAIRO and ZiPS did quite well, while Steamer was mid-pack among systems’ best 230, and ahead of only 2011 data among 2012’s top 230 players.
In the 5×5 AL League $260 Cap:
Source | Num | Avg Price | MAE | RMSE |
---|---|---|---|---|
2012 | 240 | 10.833 | 0.000 | 0.000 |
SteamerAdj | 240 | 10.833 | 4.668 | 6.005 |
ZiPSAdj | 240 | 10.833 | 4.791 | 6.308 |
MarcelAdj | 240 | 10.833 | 4.668 | 6.325 |
CAIROAdj | 240 | 10.833 | 4.733 | 6.427 |
RotoValueAdj | 240 | 10.833 | 5.361 | 6.816 |
2011Adj | 240 | 10.833 | 8.173 | 10.068 |
Top 240 actual players players in 2012
Source | Num | Avg Price | MAE | RMSE | MAE0 | RMSE0 |
---|---|---|---|---|---|---|
2012 | 240 | 10.833 | 0.000 | 0.000 | 0.000 | 0.000 |
MarcelAdj | 222 | 10.882 | 4.306 | 5.808 | 4.495 | 6.190 |
ZiPSAdj | 228 | 10.375 | 4.629 | 6.036 | 4.831 | 6.548 |
SteamerAdj | 230 | 10.451 | 4.635 | 6.070 | 4.573 | 6.009 |
CAIROAdj | 234 | 10.103 | 4.653 | 6.258 | 4.579 | 6.188 |
RotoValueAdj | 213 | 10.869 | 4.967 | 6.357 | 5.213 | 7.175 |
2011Adj | 219 | 8.860 | 8.200 | 10.520 | 7.959 | 10.319 |
This ordering is very much like the 4×4 AL, and Steamer was stronger here, but not dominant.
The 5×5 NL League – $260 Cap:
Source | Num | Avg Price | MAE | RMSE |
---|---|---|---|---|
2012 | 230 | 11.304 | 0.000 | 0.000 |
CAIROAdj | 230 | 11.304 | 5.016 | 6.558 |
ZiPSAdj | 230 | 11.304 | 5.238 | 6.600 |
SteamerAdj | 230 | 11.304 | 5.283 | 6.698 |
MarcelAdj | 230 | 11.304 | 5.497 | 6.944 |
RotoValueAdj | 230 | 11.304 | 5.617 | 7.105 |
2011Adj | 230 | 11.304 | 9.071 | 11.392 |
Top 230 actual players players in 2012
Source | Num | Avg Price | MAE | RMSE | MAE0 | RMSE0 |
---|---|---|---|---|---|---|
2012 | 230 | 11.304 | 0.000 | 0.000 | 0.000 | 0.000 |
ZiPSAdj | 223 | 10.748 | 5.111 | 6.617 | 5.193 | 6.703 |
CAIROAdj | 229 | 10.473 | 5.132 | 6.767 | 5.113 | 6.752 |
MarcelAdj | 224 | 10.697 | 5.339 | 6.838 | 5.408 | 7.002 |
SteamerAdj | 229 | 10.451 | 5.577 | 7.237 | 5.556 | 7.221 |
RotoValueAdj | 213 | 10.969 | 5.569 | 7.374 | 5.562 | 7.419 |
2011Adj | 218 | 8.428 | 8.863 | 11.215 | 8.755 | 11.134 |
Again it’s ZiPS and CAIRO that were the best NL projection systems. Adding the extra two categories made little difference.
Finally the shallow 5×5 Mixed League – $260 cap:
Source | Num | Avg Price | MAE | RMSE |
---|---|---|---|---|
2012 | 230 | 11.304 | 0.000 | 0.000 |
SteamerAdj | 230 | 11.304 | 7.564 | 9.583 |
ZiPSAdj | 230 | 11.304 | 7.464 | 9.635 |
CAIROAdj | 230 | 11.304 | 7.350 | 9.672 |
MarcelAdj | 230 | 11.304 | 7.741 | 10.052 |
RotoValueAdj | 230 | 11.304 | 8.238 | 10.512 |
2011Adj | 230 | 11.304 | 12.369 | 15.394 |
Top 230 actual players players in 2012
Source | Num | Avg Price | MAE | RMSE | MAE0 | RMSE0 |
---|---|---|---|---|---|---|
2012 | 230 | 11.304 | 0.000 | 0.000 | 0.000 | 0.000 |
MarcelAdj | 223 | 9.565 | 6.970 | 8.993 | 7.048 | 9.062 |
ZiPSAdj | 225 | 9.387 | 7.354 | 9.454 | 7.403 | 9.509 |
SteamerAdj | 229 | 9.097 | 7.585 | 9.737 | 7.574 | 9.722 |
CAIROAdj | 230 | 9.097 | 7.429 | 9.741 | 7.429 | 9.741 |
RotoValueAdj | 221 | 9.101 | 7.781 | 10.058 | 7.897 | 10.371 |
2011Adj | 223 | 5.096 | 11.786 | 14.839 | 11.680 | 14.703 |
In this format, Steamer was best when judged by its own top players, while Marcel wins when judged by 2012’s best.
One other comment – while I’ve not published the tables, I did rerun comparisons including ZiPS with the average save totals of other systems, but no other changes. When doing that, ZiPS was not among the best systems in any of the configurations I ran.
1 comment