Testing New Projection Models, and More Offensive Stats

As I’ve noted before, my older projection model has not done as well at projecting batters’ weighted on base average (wOBA) as several other publicly available systems, so I’ve been trying a different method.

I’ve now tried doing something very similar to Tom Tango’s Marcel method – take a weighted average of recent performance, regress towards the mean, and make an adjustment for the player’s age. I tried two variations of this new model, and to test it I projected 2012 based on previous years’ data, and then reran my previous projection comparison for wOBA.

The two variations were a raw projection, and another that took the raw projection and adjusted batters hits, assuming that batting average on balls in play, or BABIP, should be the same for all batters.

Here’s the results of averages for players each system projected:

Source Num Avg wOBA MAE RMSE
Actual 393 0.3277 0.0000 0.0000
Babip 393 0.3293 0.0287 0.0358
CAIRO 393 0.3288 0.0260 0.0330
Marcel 393 0.3311 0.0267 0.0341
NoBabip 393 0.3321 0.0266 0.0340
RotoValue 393 0.3229 0.0312 0.0407
Steamer 393 0.3324 0.0256 0.0329
ZiPS 393 0.3284 0.0258 0.0328
y2011 393 0.3289 0.0359 0.0492

This is the same table I generated earlier this week, just with additional lines for the Babip and NoBabip projection models I’m testing. Interestingly, giving all batters the same BABIP results in larger errors than not doing that, which implies that for batters there is some significant skill in getting hits on balls in play. The NoBabip’s errors were almost identical to Marcel’s, and so this represents a marked improvement in projecting rate stats over my old RotoValue model. I likely will use this as the basis for the 2013 RotoValue model.

I’ve updated my comparison program to be able to compare other rate statistics also, and so here’s a table showing average slugging percentage, as well as mean average error and root mean square error for those. As with the wOBA computations, I compute as error the difference between a player and the league average for the projection minus the difference between the player’s actual data and the league’s actual average:

Source Num Avg Slg MAE RMSE
Actual 393 0.4194 0.0000 0.0000
Babip 393 0.4165 0.0500 0.0620
CAIRO 393 0.4164 0.0461 0.0578
Marcel 393 0.4210 0.0484 0.0608
NoBabip 393 0.4200 0.0487 0.0610
RotoValue 393 0.4101 0.0551 0.0721
Steamer 393 0.4237 0.0455 0.0593
ZiPS 393 0.4172 0.0463 0.0580
y2011 393 0.4181 0.0640 0.0867

Here CAIRO edges out ZiPS and then Steamer as the best RMSE, while Steamer has the lowest mean absolute error, just ahead of ZiPS and CAIRO. As with wOBA, Marcel and my highly similar NoBabip model had very similar errors, while the model with the Babip adjustment was a little further back. My old RotoValue model was far backm but still far ahead of using unadjusted 2011 data to project 2012.

Next let’s check batting average:

Source Num Avg Avg MAE RMSE
Actual 393 0.2605 0.0000 0.0000
Babip 393 0.2610 0.0262 0.0321
CAIRO 393 0.2613 0.0225 0.0284
Marcel 393 0.2651 0.0229 0.0293
NoBabip 393 0.2645 0.0229 0.0293
RotoValue 393 0.2573 0.0271 0.0364
Steamer 393 0.2621 0.0224 0.0281
ZiPS 393 0.2619 0.0225 0.0284
y2011 393 0.2625 0.0305 0.0434

This shows basically the same ordering as slugging percentage, with Steamer coming out slightly on top in both RMSE and MAE, just ahead of ZiPS and CAIRO.

Finally on base percentage:

Source Num Avg OBP MAE RMSE
Actual 393 0.3248 0.0000 0.0000
Babip 393 0.3295 0.0267 0.0339
CAIRO 393 0.3284 0.0244 0.0310
Marcel 393 0.3298 0.0249 0.0316
NoBabip 393 0.3326 0.0242 0.0312
RotoValue 393 0.3217 0.0286 0.0371
Steamer 393 0.3311 0.0240 0.0307
ZiPS 393 0.3272 0.0243 0.0307
y2011 393 0.3276 0.0326 0.0443

This shows the same basic big-picture story, with ZiPS tying Steamer for best RMSE, and CAIRO third lowest. My NoBabip projection did almost as well as the three best, and was a little better than Marcel at projecting OBP than any of the other stats I tested.

To summarize, Steamer, ZiPS, and CAIRO performed the best in these tests, with Marcel not far behind. My basic attempt to match Marcel (the NoBabip model) basically succeeded in doing that, and at times had somewhat lower errors, but if I try to assume that all batters should have the same BABIP, then that makes the projections a little worse. But based on these results I’ll be retiring my older RotoValue projection model, and instead move forward with this. Thanks to Tom Tango for writing open documentation of Marcel – that certainly helped me get a handle on regression towards league mean as an important factor above simply doing a weighted average.

All of these comparisons are of rate statistics, and so they ignore playing time projections – the weightings are done by the actual playing time of the batter. From a fantasy perspective having some idea of playing time is important also, and so I want to do another comparison where a system gets credit for projecting playing time well.

In my next post I’ll take a look at pitching projections from all of these systems. Currently I plan on comparing ERA, WHIP, BB/9, and K/9, but if there are other particular pitching statistics you’d like to see, e-mail me (geoff at rotovalue dot com) and I can probably do that also.

About Geoff

Dad, hacker, fantasy sports entrepreneur.
This entry was posted in Major League Baseball, Sabermetrics. Bookmark the permalink.