My last post effectively compared projection systems by how well they predicted player skill level: I scaled each projection to match 2012 actual ab/ip, but used the rates each system projected to generate stats.
Tango asked if I could run the reverse, a comparison where I used prorated actual 2012 stats but kept only the playing time projections of various systems. So I’ve run those numbers now also for the same 5 league formats I’ve been doing. Steamer usually has the lowest errors in this format, with my old RotoValue model usually second lowest. This seems reasonable, as the other models don’t make any specific attempt to project playing time.
As in my last post, RMSE0 and MAE0 are the root mean square error and mean absolute error when I assume any player not projected by a system has a price of $0.00.
First up, the 4×4 AL League – $260 Cap:
| Source | Num | Avg Price | MAE | RMSE |
|---|---|---|---|---|
| 2012 | 240 | 10.832 | 0.000 | 0.000 |
| PTSteamer | 240 | 10.833 | 4.419 | 6.251 |
| PTRotoValue | 240 | 10.833 | 5.084 | 6.954 |
| PTMarcel | 240 | 10.833 | 6.100 | 9.042 |
| PT2011 | 240 | 10.833 | 8.627 | 10.724 |
| PTZiPS | 240 | 10.833 | 7.730 | 10.795 |
| PTCAIRO | 240 | 10.833 | 7.901 | 10.851 |
Top 240 actual players players in 2012
| Source | Num | Avg Price | MAE | RMSE | MAE0 | RMSE0 |
|---|---|---|---|---|---|---|
| 2012 | 240 | 10.832 | 0.000 | 0.000 | 0.000 | 0.000 |
| PTSteamer | 231 | 9.829 | 4.308 | 6.153 | 4.298 | 6.138 |
| PTRotoValue | 207 | 11.250 | 4.834 | 6.734 | 5.110 | 7.483 |
| PTMarcel | 222 | 10.061 | 5.385 | 7.561 | 5.498 | 7.712 |
| PTZiPS | 228 | 8.721 | 6.241 | 7.900 | 6.395 | 8.272 |
| PTCAIRO | 235 | 8.393 | 7.007 | 9.289 | 6.914 | 9.203 |
| PT2011 | 218 | 8.822 | 8.534 | 10.964 | 8.252 | 10.709 |
Steamer winds up on top here, followed by RotoValue, and then the other systems further back. This makes intuitive sense, because none of Marcel, ZiPS, or CAIRO make any effort to projecting playing time. My old RotoValue model does account for known preseason injuries at the time I last run the projections, and it also tries to adjust time via an algorithm when team totals, or even position totals within a team, get too high or too low.
Now the 4×4 NL League – $260 Cap:
| Source | Num | Avg Price | MAE | RMSE |
|---|---|---|---|---|
| 2012 | 230 | 11.304 | 0.000 | 0.000 |
| PTSteamer | 230 | 11.304 | 5.605 | 7.534 |
| PTRotoValue | 230 | 11.304 | 5.947 | 8.086 |
| PTMarcel | 230 | 11.304 | 6.438 | 8.477 |
| PTZiPS | 230 | 11.304 | 7.683 | 10.475 |
| PT2011 | 230 | 11.304 | 9.630 | 12.141 |
| PTCAIRO | 230 | 11.304 | 8.983 | 12.515 |
Top 230 actual players players in 2012
| Source | Num | Avg Price | MAE | RMSE | MAE0 | RMSE0 |
|---|---|---|---|---|---|---|
| 2012 | 230 | 11.304 | 0.000 | 0.000 | 0.000 | 0.000 |
| PTMarcel | 222 | 9.913 | 5.851 | 7.480 | 5.869 | 7.559 |
| PTSteamer | 229 | 9.277 | 5.822 | 7.726 | 5.800 | 7.710 |
| PTRotoValue | 206 | 10.969 | 5.951 | 7.903 | 5.921 | 7.937 |
| PTZiPS | 221 | 8.971 | 6.604 | 8.484 | 6.596 | 8.459 |
| PTCAIRO | 229 | 7.957 | 7.522 | 9.498 | 7.493 | 9.478 |
| PT2011 | 215 | 8.275 | 9.517 | 12.003 | 9.270 | 11.782 |
Here Marcel wound up with the lowest errors among the actual top 230, even after accounting for players it did not project (I realize Tango has defined a Marcel projection for anybody not in the file, but that basically gives a fantasy player no predictive information on which non-projected players might do better; also I’d need to hack something to make that adjustment).
Among the system’s projections, Steamer was lowest, followed by RotoValue and then Marcel.
Next the 5×5 AL League $260 Cap:
| Source | Num | Avg Price | MAE | RMSE |
|---|---|---|---|---|
| 2012 | 240 | 10.833 | 0.000 | 0.000 |
| PTSteamer | 240 | 10.833 | 4.707 | 6.485 |
| PTRotoValue | 240 | 10.832 | 5.439 | 7.310 |
| PTMarcel | 240 | 10.832 | 6.275 | 9.047 |
| PT2011 | 240 | 10.833 | 8.173 | 10.068 |
| PTZiPS | 240 | 10.833 | 8.178 | 11.031 |
| PTCAIRO | 240 | 10.833 | 8.560 | 11.277 |
Top 240 actual players players in 2012
| Source | Num | Avg Price | MAE | RMSE | MAE0 | RMSE0 |
|---|---|---|---|---|---|---|
| 2012 | 240 | 10.833 | 0.000 | 0.000 | 0.000 | 0.000 |
| PTSteamer | 230 | 9.609 | 4.483 | 6.175 | 4.428 | 6.111 |
| PTRotoValue | 209 | 10.751 | 4.997 | 6.912 | 5.237 | 7.601 |
| PTMarcel | 222 | 9.587 | 5.333 | 7.503 | 5.446 | 7.694 |
| PTZiPS | 228 | 7.968 | 6.705 | 8.172 | 6.804 | 8.468 |
| PTCAIRO | 234 | 7.793 | 7.465 | 9.604 | 7.321 | 9.489 |
| PT2011 | 219 | 8.860 | 8.200 | 10.520 | 7.959 | 10.319 |
Steamer again had the lowest errors, followed by RotoValue. I do notice that RotoValue had the fewest projected players among actual in many formats, and its error averages rise when counting such players at $0.00, which is appropriate (a system should be penalized for failing to project valuable players).
The 5×5 NL League – $260 Cap:
| Source | Num | Avg Price | MAE | RMSE |
|---|---|---|---|---|
| 2012 | 230 | 11.304 | 0.000 | 0.000 |
| PTSteamer | 230 | 11.304 | 5.500 | 7.251 |
| PTRotoValue | 230 | 11.304 | 6.219 | 8.240 |
| PTMarcel | 230 | 11.304 | 6.552 | 8.490 |
| PTZiPS | 230 | 11.304 | 8.225 | 10.994 |
| PT2011 | 230 | 11.304 | 9.071 | 11.392 |
| PTCAIRO | 230 | 11.304 | 9.791 | 13.392 |
Top 230 actual players players in 2012
| Source | Num | Avg Price | MAE | RMSE | MAE0 | RMSE0 |
|---|---|---|---|---|---|---|
| 2012 | 230 | 11.304 | 0.000 | 0.000 | 0.000 | 0.000 |
| PTSteamer | 229 | 9.208 | 5.616 | 7.420 | 5.595 | 7.404 |
| PTMarcel | 224 | 9.595 | 6.044 | 7.594 | 6.095 | 7.724 |
| PTRotoValue | 210 | 10.354 | 6.098 | 8.059 | 6.106 | 8.121 |
| PTZiPS | 223 | 8.351 | 7.373 | 9.027 | 7.387 | 9.026 |
| PTCAIRO | 229 | 7.220 | 8.077 | 9.811 | 8.045 | 9.789 |
| PT2011 | 218 | 8.428 | 8.863 | 11.215 | 8.755 | 11.134 |
Here Marcel beats RotoValue as runner up when looking at actual best players, while Steamer still has the lowest errors overall.
Finally the shallow 5×5 Mixed League – $260 cap:
| Source | Num | Avg Price | MAE | RMSE |
|---|---|---|---|---|
| 2012 | 230 | 11.304 | 0.000 | 0.000 |
| PTSteamer | 230 | 11.304 | 7.585 | 10.562 |
| PTRotoValue | 230 | 11.304 | 8.221 | 11.345 |
| PTMarcel | 230 | 11.304 | 9.029 | 13.301 |
| PT2011 | 230 | 11.304 | 12.369 | 15.394 |
| PTZiPS | 230 | 11.304 | 12.033 | 17.864 |
| PTCAIRO | 230 | 11.304 | 13.038 | 18.748 |
Top 230 actual players players in 2012
| Source | Num | Avg Price | MAE | RMSE | MAE0 | RMSE0 |
|---|---|---|---|---|---|---|
| 2012 | 230 | 11.304 | 0.000 | 0.000 | 0.000 | 0.000 |
| PTSteamer | 229 | 7.891 | 7.084 | 9.897 | 7.076 | 9.881 |
| PTZiPS | 225 | 5.999 | 8.497 | 10.448 | 8.521 | 10.477 |
| PTMarcel | 223 | 7.145 | 8.016 | 10.791 | 8.061 | 10.799 |
| PTRotoValue | 218 | 7.675 | 7.942 | 10.879 | 8.045 | 11.130 |
| PTCAIRO | 230 | 5.046 | 8.960 | 11.188 | 8.960 | 11.188 |
| PT2011 | 223 | 5.096 | 11.786 | 14.839 | 11.680 | 14.703 |
It’s interesting to see ZiPS performing rather well in this model among 2012 top players, although in a shallower league, the player pool is mostly good players all expected to get a lot of time. So it’s not so surprising that simple methods like some weighted average of recent performance could hold up rather well. Also I think ZiPS projects players based on previous playing time, be it in MLB or the minors, and so it projects a lot of time for several players who will simply not be playing much at the MLB level. Some of those players may make the top 230 prices based on ZiPS projections, and thus its errors in that format would be much larger than simply looking at the actual top players.