Reviewing Five 2012 MLB Projection Systems

Last year I ran some comparisons of a few projection systems for major league baseball, comparing several projection systems against 2011 data. So I thought it would be interesting to run a similar comparison of the five sets of projections I made visible for 2012 on RotoValue:

• CAIRO – from S B of the Replacement Level Yankees Weblog.
• Marcel – the basic projections from Tom Tango, coauthor of The Book.
• Steamer – developed by Jared Cross, Dash Davidson, and Peter Rosenbloom.
• ZiPS – projections from Dan Szymborski of Baseball Think Factory and ESPN.
• RotoValue – my own old algorithm.

Just like last year, I’m computing both mean absolute error and root mean square error of weighted on base average (wOBA), a good rate statistic for evaluating offensive performance. Also, I’m adding Actual (2012 real data) and 2011 data to the table.
The above table is only comparing wOBA for the 393 players who were projected by all the systems. Here ZiPS edged out Steamer for the lowest RMSE (0.03287 to 0.03290), with CAIRO quite close behind, and Marcel in the same ballpark. Steamer had the lowest mean average error, slightly ahead of ZiPS and CAIRO. Marcel was further back again, and my older RotoValue model was, like in 2011, much further behind the other systems. It was still better than simply using the previous year’s data, but it was not particularly good.

Here’s a larger table, showing averages of all players each system projected who had at least one plate appearance in 2012:

In this table num refers to the total number of players projected by the system, and the first wOBA column is the cumulative wOBA of all projected statistics. MLB refers to the number of projected players who got at least 1 plate appearance in MLB, and the second wOBA is the average wOBA of those players, weighted by the number of plate appearances.
Here the MAE and RMSE rise, as the players only projected by some systems were harder to project, but the overall ordering is about the same: Steamer had the lowest MAE, and ZiPS had the lowest RMSE and MAE, followed closely by Steamer, with Marcel 3rd and CAIRO, which projected the most total players, now just behind.
The differences between the best four systems in 2012, ZiPS, Steamer, CAIRO, and Marcel, are still quite small, so I’m still reluctant to say any of these are assuredly better than the others. But my RotoValue projections have lagged behind the others for both years that I’ve run these numbers, only bettering the simplest projection of just using last year’s data. So I’m looking into revisiting what I’m doing, to see if I can come up with a better algorithm at projecting future data.
From a fantasy perspective, I think looking at any (or perhaps even an average of all) of the better systems is helpful – a baseball season is long, but it is not long enough for many performance statistics to stabilize tightly around a player’s current skill level. So looking at something which somehow averages a few years of recent performance is much better than just relying on the previous year’s data.
I’ll be reworking my model, and I hope to make better RotoValue baseball projections available on the site for 2013, along with other projection systems as well.
Update 22 Jan 2013: Tom Tango e-mailed asking whether I’d get exactly the same results if I added a fixed amount to each wOBA. In theory I should. In testing for this, I discovered a bug that caused me to omit a player from the StdDev, MAE and RMSE calculations when his wOBA was actually 0. This didn’t affect projection systems much (at least for players all systems projected), but it made a larger difference when using 2011 stats as a naive projection. So my old RotoValue projection’s averages are further ahead of 2011 data, but they still lag the other systems. I’ve updated the tables and adjusted text to reflect this.
If you’ve got projections you’d like to make available on RotoValue, contact me (geoff at rotovalue dot com), and I’d gladly add the data, and include you in a review of 2013 numbers next year!