As I’ve noted before, my older projection model has not done as well at projecting batters’ weighted on base average (wOBA) as several other publicly available systems, so I’ve been trying a different method.
I’ve now tried doing something very similar to Tom Tango’s Marcel method – take a weighted average of recent performance, regress towards the mean, and make an adjustment for the player’s age. I tried two variations of this new model, and to test it I projected 2012 based on previous years’ data, and then reran my previous projection comparison for wOBA.
The two variations were a raw projection, and another that took the raw projection and adjusted batters hits, assuming that batting average on balls in play, or BABIP, should be the same for all batters.
Here’s the results of averages for players each system projected:
| Source | Num | Avg wOBA | MAE | RMSE |
|---|---|---|---|---|
| Actual | 393 | 0.3277 | 0.0000 | 0.0000 |
| Babip | 393 | 0.3293 | 0.0287 | 0.0358 |
| CAIRO | 393 | 0.3288 | 0.0260 | 0.0330 |
| Marcel | 393 | 0.3311 | 0.0267 | 0.0341 |
| NoBabip | 393 | 0.3321 | 0.0266 | 0.0340 |
| RotoValue | 393 | 0.3229 | 0.0312 | 0.0407 |
| Steamer | 393 | 0.3324 | 0.0256 | 0.0329 |
| ZiPS | 393 | 0.3284 | 0.0258 | 0.0328 |
| y2011 | 393 | 0.3289 | 0.0359 | 0.0492 |
This is the same table I generated earlier this week, just with additional lines for the Babip and NoBabip projection models I’m testing. Interestingly, giving all batters the same BABIP results in larger errors than not doing that, which implies that for batters there is some significant skill in getting hits on balls in play. The NoBabip’s errors were almost identical to Marcel’s, and so this represents a marked improvement in projecting rate stats over my old RotoValue model. I likely will use this as the basis for the 2013 RotoValue model.
I’ve updated my comparison program to be able to compare other rate statistics also, and so here’s a table showing average slugging percentage, as well as mean average error and root mean square error for those. As with the wOBA computations, I compute as error the difference between a player and the league average for the projection minus the difference between the player’s actual data and the league’s actual average:
| Source | Num | Avg Slg | MAE | RMSE |
|---|---|---|---|---|
| Actual | 393 | 0.4194 | 0.0000 | 0.0000 |
| Babip | 393 | 0.4165 | 0.0500 | 0.0620 |
| CAIRO | 393 | 0.4164 | 0.0461 | 0.0578 |
| Marcel | 393 | 0.4210 | 0.0484 | 0.0608 |
| NoBabip | 393 | 0.4200 | 0.0487 | 0.0610 |
| RotoValue | 393 | 0.4101 | 0.0551 | 0.0721 |
| Steamer | 393 | 0.4237 | 0.0455 | 0.0593 |
| ZiPS | 393 | 0.4172 | 0.0463 | 0.0580 |
| y2011 | 393 | 0.4181 | 0.0640 | 0.0867 |
Here CAIRO edges out ZiPS and then Steamer as the best RMSE, while Steamer has the lowest mean absolute error, just ahead of ZiPS and CAIRO. As with wOBA, Marcel and my highly similar NoBabip model had very similar errors, while the model with the Babip adjustment was a little further back. My old RotoValue model was far backm but still far ahead of using unadjusted 2011 data to project 2012.
Next let’s check batting average:
| Source | Num | Avg Avg | MAE | RMSE |
|---|---|---|---|---|
| Actual | 393 | 0.2605 | 0.0000 | 0.0000 |
| Babip | 393 | 0.2610 | 0.0262 | 0.0321 |
| CAIRO | 393 | 0.2613 | 0.0225 | 0.0284 |
| Marcel | 393 | 0.2651 | 0.0229 | 0.0293 |
| NoBabip | 393 | 0.2645 | 0.0229 | 0.0293 |
| RotoValue | 393 | 0.2573 | 0.0271 | 0.0364 |
| Steamer | 393 | 0.2621 | 0.0224 | 0.0281 |
| ZiPS | 393 | 0.2619 | 0.0225 | 0.0284 |
| y2011 | 393 | 0.2625 | 0.0305 | 0.0434 |
This shows basically the same ordering as slugging percentage, with Steamer coming out slightly on top in both RMSE and MAE, just ahead of ZiPS and CAIRO.
Finally on base percentage:
| Source | Num | Avg OBP | MAE | RMSE |
|---|---|---|---|---|
| Actual | 393 | 0.3248 | 0.0000 | 0.0000 |
| Babip | 393 | 0.3295 | 0.0267 | 0.0339 |
| CAIRO | 393 | 0.3284 | 0.0244 | 0.0310 |
| Marcel | 393 | 0.3298 | 0.0249 | 0.0316 |
| NoBabip | 393 | 0.3326 | 0.0242 | 0.0312 |
| RotoValue | 393 | 0.3217 | 0.0286 | 0.0371 |
| Steamer | 393 | 0.3311 | 0.0240 | 0.0307 |
| ZiPS | 393 | 0.3272 | 0.0243 | 0.0307 |
| y2011 | 393 | 0.3276 | 0.0326 | 0.0443 |
This shows the same basic big-picture story, with ZiPS tying Steamer for best RMSE, and CAIRO third lowest. My NoBabip projection did almost as well as the three best, and was a little better than Marcel at projecting OBP than any of the other stats I tested.
To summarize, Steamer, ZiPS, and CAIRO performed the best in these tests, with Marcel not far behind. My basic attempt to match Marcel (the NoBabip model) basically succeeded in doing that, and at times had somewhat lower errors, but if I try to assume that all batters should have the same BABIP, then that makes the projections a little worse. But based on these results I’ll be retiring my older RotoValue projection model, and instead move forward with this. Thanks to Tom Tango for writing open documentation of Marcel – that certainly helped me get a handle on regression towards league mean as an important factor above simply doing a weighted average.
All of these comparisons are of rate statistics, and so they ignore playing time projections – the weightings are done by the actual playing time of the batter. From a fantasy perspective having some idea of playing time is important also, and so I want to do another comparison where a system gets credit for projecting playing time well.
In my next post I’ll take a look at pitching projections from all of these systems. Currently I plan on comparing ERA, WHIP, BB/9, and K/9, but if there are other particular pitching statistics you’d like to see, e-mail me (geoff at rotovalue dot com) and I can probably do that also.