Farm system rankings, and competing in the future

There are two disclaimers to this post. Actually, there are lot more than two, as I'm going to be doing ill-advised number crunching on small sample sizes, but there are two I want to get out of the way. First, everything written below pales in comparison to this succinct but head-spinning legacy post by Matt Swartz, via Fangraphs, from a few years ago. That post says a lot more than this one will, in better form. So if you're interested in the topic, I suggest reading that first. I'll wait. Second, this post might read a bit like it's been soured by "mood affiliation" (the tendency of observers or commentators to highlight evidence that agrees with preconceived notions or feelings rather than objectively assessing evidence as presented and perhaps reconsidering said preconceived notions in the face of contrary indications). I can't convince you that it isn't, though I hope that you'll trust me and take my words at face value when I say that I didn't know what to expect when looking into this, and that my results are independent of the fact that I think the Braves would've been better off competing in 2015 rather than rebuilding.

This has been a fairly exciting week in terms of coverage of the Braves' farm system. Our own Top 25 list is being unveiled. Keith Law ranked the Braves' system sixth, which is a huge turnaround from where it started the offseason, likely somewhere in the bottom five of all teams. Kiley McDaniel gave the Braves the same ranking on Fangraphs, reinforcing what's been a quick turnaround from a moribund system with organizational players clogging up the Top 20 prospects list. But while salivating over prospect hit tools, run grades, and the like is very exciting, the key question is, as always, how well these things might translate into wins down the line.

As far as I know, the Braves' brass have never explicitly come out and said that they're targeting 2017 as the campaign where their rebuilding efforts will culminate in a resurgent team, but that's the date bandied around a fair bit. That gives the Braves two rebuilding years, and then a hopefully-successful 2017 campaign that coincides with the opening of Suntrust Park. The downside to any rebuilding effort, of course, is that rebuilding (like everything else in baseball, I guess), is not a sure thing. All sorts of things can happen, including highly-ranked prospects not developing, injuries, not enough prospects developing at once, bad seasons from complementary team pieces that make the prospect contributions not enough, and so on. Combine that with the fact that a bunch of prospects need to be successfully graduated at once to really proceed with a full rebuild, the fact that rebuilding teams don't generally have high payrolls, and your good ol' regular baseball volatility, and you've got a concoction that might make risk-averse folks feel a bit queasy. Trying to yank some success out of this mix, and peg it to a specific year, is easier said than done. Or, well, even easier said than projected.

Out of curiosity, I went back to 2012 and pulled various organizational rankings that were publicly-available at the time, each from the spring of 2012:

• Baseball Prospectus;
• Baseball America; and
• Fangraphs (Marc Hulet minor league system rankings)

• Wins three years later (in 2014);
• Wins over the whole period (2012-2014); and
• Pythagorean wins three years later, as well as total pythagorean wins.

There wasn't really a relationship. Here's an example:

The R-squared, which in oversimplified layman's terms is just the explanation that the x-axis variable has on the y-axis term, is essentially zero (you can read this as, "The variation in Fangraphs 2012 farm system ranks explains 0.17 percent of the variation in 2014 wins"). The coefficient is not statistically significant at any meaningful level. But more than that, you can see that stuff is just all over the place, Let's just agree to ignore the fact that the line slopes up a tiny bit, which one can (but you shouldn't, not on my watch!) interpret as a minor sign that having a worse minor league ranking actually increases your win total three years later.

Here's another one:

The average farm system rank explains about ten percent of the variation in total wins between 2012 and 2014. The coefficient isn't quite significant, but at least there's some tiny semblance of a relationship where a better farm system leads to more wins over the next three seasons. Still, pretty much all of the data and relationships look like this, and there's not much that can be said about a good farm system specifically increasing record three seasons out. Of course, farm system ranks are at least somewhat "sticky," so if farm systems are more predictive of wins in, say, two seasons, and the Braves farm system continues to be highly ranked in the future, that may have a bit more bearing on their future (including 2017) records.

One thing that struck me, of course, is that winning is the product of many things, and farm system wouldn't really be expected to be a really big one. Payroll isn't a huge determinant these days, but given Matt Swartz's linked article above, and the fact that it's really easy to add payroll in, let's see what happens when we test future winning against farm system rankings and payroll. Below is some very rudimentary linear regression output from STATA. If you're interested but have never seen this sort of thing before, I'm happy to discuss in the comments, but I'm going to attempt to economize on post length by not going on at length with a specific explanation of what everything in the below means.

In the above, I'm trying to see whether either the average farm system ranking (avg) or payroll have any impact on wins three seasons in the future (at least as far as 2012 -> 2014 data go). As you can see, the "fit" of the model (the R-squared in the upper right hand portion) is far from great, but somewhat better than in the first scatterplot in this post, largely from including payroll as a variable. However, even payroll itself isn't quite statistically significant at any level I'd be that interested in (though that 1.59 t-statistic is fairly close to the 90% confidence level, so I guess there's that). This is likely because we still have a very small sample of just 30 teams and team wins vary a fair bit for various reasons season-to-season. (As a professional, I would not advise anyone to do a regression with just 30 data points... or at least not take the results too seriously. More like a "cool story bro" rather than "incontrovertible evidence!")

What's more interesting is if we carry this analysis to three-year wins from 2012 to 2014, rather than looking at 2014 in a vacuum. I wasn't initially thinking this way, but the note in Swartz's article about farm systems having a greater impact at +2 years rather than +3 years led me down this path.

This output looks pretty similar to the other one in structure; I just replaced w2014 (2014 wins) with totw (total wins, 2012 through 2014). The significance of the "avg" coefficient (farm system rank) is much higher, not exactly at a great level of statistical significance, but getting there. There's a decent chance that there's a relationship between average farm system rank in 2012, and three-year wins 2012-2014, at a ratio of moving up one ranking increasing wins by about 0.9 wins. That's not much, but it does indicate that all else equal, over this period, the difference between the best-ranked and worst-ranked farm system was 25 wins (29 x 0.87), or about 8 wins a season. That's nothing to sneeze at, if that relationship holds for other years. Of additional interest to me is the payroll variable in this case, and this shouldn't be that surprising: the variable is highly significant and indicates that for each million of additional payroll, you get about 0.3 wins over that three year period, assuming equivalent farm system rankings. Of course, an extra million of additional payroll isn't much; a $24 million gap between two teams (like say, the Braves and Nationals) would result in about 2-3 more wins each season for three seasons for the better-endowed team. (Given that the real gap between the highest and lowest payrolls is an absurd $190M, this indicates that on the basis of payroll alone, you'd expect a 19-wins-per-season swing between the richest and poorest teams, but in reality, we know this is an underestimate. There are a bunch of other effects not captured here, as evidenced by the R-squared indicating only 30% of the variation in three-year wins explained by these two factors.)

So it looks like farm system rankings might matter after all. Here's some additional food for thought, from Swartz's article (seriously, if you've gotten this far and haven't clicked that link, just do it):

If you regress NM WAR in each of the next five years against a team’s Baseball America farm system ranking, you’ll find that the difference in NM WAR between the best and worst teams in the league is about 41 expected wins over five years.... The biggest effect is two years later, where the gap is 10 wins... So we have some numbers: Each team can expect about 1.4 more wins in the next five years than the team ranked below it, with the largest gap occurring two years later with about 0.33 wins difference for each spot in the rankings. That’s definitely important, but it’s hardly a crystal ball.

And hey, consistent with that, check it out:

These results are fairly consistent with the ones quoted above, though way overstated, since the change in rankings affects wins by about two-thirds of a win by moving up one spot in the rankings, rather than the stated one-third. But still, that's pretty cool, right? (Note also the better fit for wins two years out rather than three years out or three-year wins, and also that the effect of payroll is fairly consistent across either three-year wins or wins two years out.)

As far as I can tell from a bunch of other things I'm not pasting here for length purposes, that two-year window is really the sweet spot for win contributions from a high-ranked farm. The effect isn't really there in the first year, and the third year appears to be too far away. So, given the pre-season 2014 farm rankings, we can expect some additional win contributions to the team in 2016. That's a bit before the window. Of course, this doesn't speak to how the farm system might be ranked in 2017. But these data indicate, at least as well as a very rudimentary analysis can, that if the Braves are intending to compete in 2017, it would behoove them to bolster the farm even more before the 2016 season. And as for me? I went into this thinking that the prognosis for 2017 was murky, with not much evidence that rebuilding now would help 2017. I'm still fairly iffy on that point, but at least I feel better about 2016, which I wasn't even thinking about until running these numbers. So I guess we'll see where the farm system rankings are next year, as that should be a better barometer of potential 2017 success than what we know right now.