Comparing Replacement Level for Starters and Relievers

There was a discussion at Tom Tango’s blog which led to an interesting question: what’s the difference in effectiveness between a replacement-level starter and a replacement-level reliever?

To answer this, you’d need to have some way of estimating replacement level for starters and relievers. Since I have a database with boxscore data since 2010, I set out to do exactly that. First, I compiled player statistics by role, either starter or reliever (the same player might appear in both roles), and then I sorted players by some effectiveness statistic (runs per 9 innings, ERA, FIP, whatever). I bucketed players in groups – the top 10%, next 10%, and so on, and took the average effectiveness statistic for each group.

I figured the bottom 10% effectiveness should be a good proxy for replacement level: if you’re performing in the bottom 10%, you’re replaceable. Sounds good. Here’s a table from 2013 data using FIP:

Bucket # Start FIP # Relief FIP Diff IP Start IP Relief
0 30 2.688 53 1.705 0.983 2928.34 919.67
1 30 3.313 52 2.506 0.807 4933.33 1967.33
2 29 3.584 53 2.962 0.622 4331.67 1959.68
3 30 3.890 52 3.251 0.639 3424.66 2416.67
4 29 4.090 53 3.589 0.501 3641.68 2464.00
5 30 4.359 52 3.924 0.435 3153.33 1644.34
6 30 4.629 53 4.280 0.349 2924.67 1473.32
7 29 4.999 52 4.973 0.026 1944.33 1245.66
8 30 5.691 53 6.221 -0.531 1058.33 659.00
9 29 7.635 52 9.597 -1.963 361.00 233.33

Immediately it looks like small sample size here is a big problem: the 52 relievers in the worst bucket average under 5 IP each, and the 29 starters average just about 12 IP. I tinkered for a while with adding arbitrary innings limits, like, say, 100 IP as a starter, and 30 IP as a reliever, ignoring players that don’t reach these totals. That gave more stable looking numbers, but the difference was greatly affected by where I might draw the line, and that was a purely arbitrary choice.

When I model fantasy sports pricing, I use the roster sizes for determining replacement level for a given league, so I thought I’d try that here. Each of the 30 teams typically uses 5 starters and carries a bullpen of at least 6 relievers. So I changed my filter to look at only the top 150 starters and 180 relievers by innings pitched. That led to this table:

Bucket # Start FIP # Relief FIP Diff IP Start IP Relief
0 15 2.714 18 1.985 0.729 2716.67 1092.00
1 15 3.260 18 2.632 0.628 2846.00 1045.00
2 15 3.452 18 2.884 0.568 2830.00 1124.67
3 15 3.596 18 3.087 0.509 2425.00 1121.67
4 15 3.833 18 3.251 0.582 2515.99 1181.00
5 15 4.010 18 3.437 0.573 2454.00 1057.00
6 15 4.136 18 3.612 0.524 2181.01 1025.67
7 15 4.354 18 3.808 0.546 2185.33 1088.67
8 15 4.555 18 4.133 0.422 2194.00 911.66
9 15 5.010 18 4.871 0.139 1954.00 888.66

That looks much better. The cutoff was 79 IP for a starter, and 36.67 IP for a reliever.

Here’s the same algorithm, just computing/bucketing by RA9 instead of FIP:

Bucket # Start RA9 # Relief RA9 Diff IP Start IP Relief
0 15 2.718 18 1.684 1.035 2579.00 1122.34
1 15 3.292 18 2.374 0.918 2706.67 1099.34
2 15 3.530 18 2.732 0.798 2605.34 1060.66
3 15 3.737 18 3.024 0.713 2620.34 1226.34
4 15 3.936 18 3.322 0.614 2641.33 1159.66
5 15 4.187 18 3.625 0.562 2530.00 1003.00
6 15 4.419 18 3.976 0.443 2220.00 1025.33
7 15 4.806 18 4.315 0.491 2236.00 1038.67
8 15 5.296 18 4.640 0.656 2200.67 975.67
9 15 6.103 18 5.509 0.594 1962.66 825.01

In the discussion on Tango’s site I had speculated that the difference between elite relievers and elite starters was likely greater than the overall average, and these tables support that view: the top bucket showed the widest gap in both FIP and RA9.

The top buckets aren’t relevant to the question of replacement level, but it is good to see that in general teams give more innings to pitchers who are performing better. That the best bucket doesn’t have the highest IP is also expected, as being the very best in performance rate requires some good luck as well as skill, and it’s easier to post very good rate stats in smaller slices of playing time.

Next I’ll track just the bottom bucket, but over the past 4 years:

Bucket # Start FIP # Relief FIP Diff IP Start IP Relief
2010 15 5.375 18 5.392 -0.016 1981.67 906.33
2011 15 5.095 18 5.131 -0.036 2113.33 850.67
2012 15 5.411 18 5.031 0.380 1858.67 893.33
2013 15 5.010 18 4.871 0.139 1954.00 888.66

And the same years using RA9:

Year # Start RA9 # Relief RA9 Diff IP Start IP Relief
2010 15 6.195 18 6.279 -0.084 2071.67 926.00
2011 15 5.945 18 5.898 0.047 2036.00 824.00
2012 15 6.320 18 6.075 0.245 1744.33 960.00
2013 15 6.103 18 5.509 0.594 1962.66 825.01

The minimum innings threshold varied from 79 to 85.33 for starters, and from 34.33 to 37 for relievers, as each year I looked at just the top 180 relievers and 150 starters by IP. It’s interesting to see that in both FIP and RA9, there was basically no difference in bottom percentile performance in 2010 and 2011 between starters and relievers. Since then, however, relievers have done a little better than starters, which is what I’d have expected.

This analysis is sensitive to where you draw the lines to exclude players for low playing time. Increasing the numbers in the pool (either by straight numerical count or by lowering an innings threshold) will make that group of pitchers look worse relative to the other group.