Alternate Baseball Universes

Jamie found a NYTimes op-ed by a grad student and a professor from Cornell, outlining some research they did into alternate baseball universes. The goal was to find out how unlikely in fact was Joe DiMaggio's 56-game hitting streak, played out in the 1941 season. No one since has even come close to that record. The math guys ran simulations of the entire history of baseball from 1885 on — 10,000 of them. For each simulation they put each player up to the plate for each at-bat in each game in each year, just like it happened; and they rolled the dice on him, based on his actual hitting stats for that season. (Their algorithm sounds far simpler than whatever the Strat-O-Matic guys use.) The result: Joltin' Joe's record is not merely likely, it's basically a sure thing. Every alternate universe produced a streak of 39 games or better; one reached 109 games. Joe DiMaggio was not the likeliest player in the history of the game to accomplish the record, not by a long shot.

Comments

If its so likely, they why hasn't it happened?

[quanticle]

I know the statisticians among you are going to bash me with a cluestick for such a naive question, but I'll ask anyway - if this event is so likely to occur, then why hasn't it happened again?

Re:If its so likely, they why hasn't it happened?

[Anonymous Coward]

Clearly they aren't factoring in the stress and nerves the average ballplayer would be dealing with as they got closer to the mark.

Re:If its so likely, they why hasn't it happened?

[Martin Blank]

It was likely to occur early in the history of baseball, and fell off dramatically after the 1930s. The early years tended to be batting competitions (in some ways like today's) rather than pitching competitions, and a pitcher's repertoire was limited to about a half-dozen pitches, plus whatever grease, oil, jelly, file, sandpaper, thumbtack, or razor blade he could conceal.

Re:If its so likely, they why hasn't it happened?

[hedwards]

The most likely reason is that statistics isn't the appropriate method by which to study this problem.

This sort of a study is really more about curiosity, it doesn't deal with things like changes to the way in which the game is played. For instance early on, and for quite a while later, it was common for a pitcher to pitch 9 innings every game, and in many cases to pitch both games out of a double header. Meaning more opportunity for errors and since batters get time to rest up, there's a bit of an edge under that style of play to the batter which doesn't exist today.

That also doesn't include the variety of pitching which players see today or the fact that a player might get to see 3 different pitchers in a single game.

Even the length of the season has an effect on how players play. None of those things are easily quantified, much less analyzed by statisticians.

Re:If its so likely, they why hasn't it happened?

[Martin Blank]

No, they reran 1871-2005 through the simulator a total of 10,000 times. This is clear not only from the statement that says as much ("Using a comprehensive collection of baseball statistics from 1871 to 2005, we simulated the entire history of baseball 10,000 times in a computer"), but from the mention that the record was set most often in 1894.

Re:If its so likely, they why hasn't it happened?

[ByteSlicer]

Because baseball players aren't dice?

Re:If its so likely, they why hasn't it happened?

[hedwards]

While it might be true that statistics is not an appropriate technique to study baseball, none of the things you mention are evidence against it being useful. All of the factors you mention influence a player's batting average, and the hypothesis they are using is that once you know the batting average you can calculate a set of possible histories of hits for that player, with the right statistical weight. They are assuming that the probability of a batter getting a hit in any game is uncorrelated with his performance in previous games. Actually, they are reasons why statistical analysis is not appropriate in this instance.

Statistical analysis isn't inappropriate in terms of studying baseball, it is just inappropriate to use it in this manner.

What you are suggesting is a good example of the gambler's fallacy. And it breaks down in this case for the reasons that I mentioned, the underlying conditions in which those batting averages were collected has changed in such a way that they no longer accurately reflect the present conditions.

The GP was asking if the occurrence is that common, why hasn't it happened since, and the answer I gave was that there was a fundamental change in the way that the game is played which changed who has the advantage. It's similar to why nobody has had a .400+ season since 1930.

Re:If its so likely, they why hasn't it happened?

[Frequency Domain]

No bashing, it's not a bad question. The answer is because it still qualifies as a "rare event". The thing that's kind of counter-intuitive, but easy to demonstrate, is that having a particular rare event happen is rare, but having some rare event happen is common.

A good illustration of this is the so-called "birthday paradox", which asks what's the probability of having duplicate birthdays in a group of n people (whose birthdays are independent of each other). Think of adding the people to the room one by one. The first person doesn't have any chance of having a duplicate birthday, because there's nobody else in the room. The second person has 1/365 chances of duplicating, 364/365 of missing the first one. Let's follow up on the misses, they're easier to work with. In general, if we've got k people in the room without a duplicate, that means they've used up k of the 365 days in the year, and the next person we introduce to the room has to miss all of those days to avoid a duplication. So the probability of everybody missing everybody else, by the time we get up to n people in the room, is (365/365)*(364/365)*(363/365)*...*((365-n+1)/365), which starts diving towards zero really fast. The probability of having one or more duplicates is 1 - P(no duplicates), which correspondingly climbs to one really fast. If you write a short program to do the exact calculations, you'll find that by the time you have 23 people in the room the probability is greater than 0.5 of having a duplicate, and by the time you get 57 people it's greater than 0.99!

If you pick one particular person and ask what's the probability of duplicating that birthday it remains quite small. That's the difference between having a particular rare event rather than having some rare event. For a large enough group, some pair of people will almost surely share a birthday but the odds of it being you (or any other designated person) remain quite small.

Just to preserve my computing geek cred, this is why you need collision resolution for hashing algorithms. You don't know which entries will share hash values, but collisions are almost certain to happen by the time you've loaded 3 * sqrt(Hash Table Capacity) values, e.g., if your hash table has capacity 10000 you will almost surely see a duplicate within the first 300 entries.

Re:If its so likely, they why hasn't it happened?

[GoodbyeBlueSky1]

I wish my mod points hadn't just expired, because you just summed it up perfectly. Silly study with no basis in reality.

In other news, I've just started a fund of stocks that are held and traded based on historical data. If you invest in it, I guarantee a large return, because complex systems that rely heavily on myriad human variables are of course determined entirely by statistics.

Re:If its so likely, they why hasn't it happened?

[Anonymous Coward]

The early years tended to be batting competitions (in some ways like today's) rather than pitching competitions If by "early years", you mean 1920 and later, yeah.

Otherwise, buddy, you're way off base.

NL year-by-year stats.

Look at those ERAs pre-1920. Before 1920, the ERA on the NL never significantly exceeded 3.00. After 1920, it never dropped below 3.3 or so, with the exception of a 2.99 in 1968, after which MLB made changes to the rules, amongst them lowering the acceptable height of the pitcher's mound.

The time prior to 1920 was marked by pitchers such as Cy Young, Mordecai Brown, Walther Johnson, Ed Walsh, Christy Mathewson. You've probably heard of most of them.

Here are the single-season MLB ERA leaders. Outside of Bob Gibson in the aforementioned 1968, you have to go all the way to Greg Maddux in 1994 at #48 all time to find a season after 1920 on the list. Barely 10 of the 100 lowest single-season ERAs in MLB history occurred after 1920. And that's only because Pedro Martinez in 2000 and Ron Guidry in 1978 tied with 9 others for #100 on the list. So only 8 of the best single-season ERAs happened after 1920.

You need to research "dead ball era", and the response by baseball to "Black Sox". (Hint: just like the response to the 1994 strike, it involves the ball...)

The fact that you got a +5 out of such a demonstrably incorrect post is a major indictment of the baseball knowledge of the Slashdot faithful.

Re:If its so likely, they why hasn't it happened?

[Kamineko]

Like Einstein said: "God does not play baseball!"

I think.

Re:If its so likely, they why hasn't it happened?

[Vellmont]

I wish my mod points hadn't just expired, because you just summed it up perfectly.

Really? For the purposes of this article, why?

It seems perfectly reasonable to me to take a set of data and try to model how likely a particular outcome is. That's a very valid question to ask that a statistical model can answer. The model may be flawed, need improvement, or whatever, but that doesn't mean the question isn't one that can't be answered by science.

If you invest in it, I guarantee a large return, because complex systems that rely heavily on myriad human variables are of course determined entirely by statistics.

This is simply an invalid analogy. The article isn't saying it can predict the future (or even the past!) based on a statistical model. All it's saying is "just how likely was it for DiMagio to get his streak, given past performance".

Re:If its so likely, they why hasn't it happened?

[khallow]

So what does that have to do with the study? Statistics applies to a lot more than dice. No offense, but your observation sounds like one of those cute but irrelevant observations that just add noise.

Re:If its so likely, they why hasn't it happened?

[ByteSlicer]

Well, they modeled the batter using random numbers and their player stats. The problem is that real people don't behave deterministically. They might hit better on their birthday, or when it was a clear sky the night before the game, just because they believe that (baseball players are extremely superstitious). The model doesn't take into account that some player might get psyched out by a certain number (and always screw up on the 13th consecutive hit), or just by the pressure of wanting to break the record. It doesn't take into account the pitcher, weather conditions, and a lot of other things that matter to real people but not to computers.

You might be able to model some long term behavior that way, but never the short term stuff, because the model is too simplified (man versus dice).

Re:If its so likely, they why hasn't it happened?

[khallow]

You might be able to model some long term behavior that way Like the probability of winning streaks over the lifespan of baseball? Pretty much what they are doing.

Re:If its so likely, they why hasn't it happened?

[OldBus]

But the point they were trying to make is that the statistics aren't valid. The guys writing the paper were measuring independent random events. Baseball hits may or may not be random, but the key thing is that they are not independent.

Players suffer from pressure because of streaks. A player who goes several games without hitting is under a huge amount of pressure to hit, and their form may suffer. A player who hits for a few games is likely to have high confidence and keep hitting. However, if they do keep hitting for a few games more, everyone starts talking about it. They get asked questions at interviews etc. Many players crack under that sort of extra pressure. Di Maggio's great streak is great because he overcame that. Sportsmen in all sorts of sports are heard to make comments such as, "I'm just focussing on hitting/scoring" or "I'm taking one game at a time". This is because they're trying to avoid that sort of pressure.

I'm sure the guys who wrote the article know about gamblers' fallacy and would be quick to point out someone's mistake. Eg the gambler who bets extra money on a six being rolled because no six has come up on the last 20 rolls and 'a six must be due'. Of course, a six is no more likely than it was on any other roll because each roll is random and independent of any other roll. In the case of sporting streaks, each game is not independent, so the argument is just and flawed and invalid as the gamblers' fallacy.

Re:If its so likely, they why hasn't it happened?

[Gospodin]

Baseball hits may or may not be random, but the key thing is that they are not independent.

Announcers and actual players to the contrary, they actually are remarkably independent. Studies have been done. The one to start with is: Albright, S. C. (1993), "A statistical analysis of hitting streaks in baseball," Journal of the American Statistical Association , 88, 1175-1183. He shows that there is almost no evidence that hitting streaks are anything other than statistical noise.

Re:If its so likely, they why hasn't it happened?

[Heian-794]

:::The early years tended to be batting competitions (in some ways like today's) rather than pitching competitions

::If by "early years", you mean 1920 and later, yeah.

:Otherwise, buddy, you're way off base.

The only one off base is yourself -- check your own link (baseball-reference.com is an amazing site and I recommend it to anyone) and pay extra attention to the 1890s. In the years immediately after the pitcher's mound was moved back to its current 60 feet 6 inches, offensive totals soared far beyond what we're used to seeing. Given that you're familiar with the lowering of the mound for 1969, I'm surprised that you're not familiar with when it was fixed at its current distance.

The article even mentions that the record was most likely to have been set in 1894, when the National League ERA was well over 5.00, and there were 11.6 hits per team per game, more than 20% more than we see now.

Look at those ERAs pre-1920. Before 1920, the ERA on the NL never significantly exceeded 3.00.

I'm looking at them. The "5.32" for 1894, which is somewhat more than three, is particularly striking.

After 1920, it never dropped below 3.3 or so, with the exception of a 2.99 in 1968, after which MLB made changes to the rules, amongst them lowering the acceptable height of the pitcher's mound.

...

You need to research "dead ball era", and the response by baseball to "Black Sox". (Hint: just like the response to the 1994 strike, it involves the ball...)

While he's doing this, perhaps you could research what came before the dead ball era: namely, the high-offense 1890s. Teams were taken off guard by the increase in the pitching distance and continued to play an 1880s game in a new environment. It took several seasons for adjustments, such as four-man pitching rotations and the occasional use of relief pitchers, to balance the sudden advantage that had been given to the batters. It is not surprising that 1894 would be the year in which a long hitting streak would have been most likely -- the single-season record for runs scored, 194 by Billy Hamilton, was set that year and still stands today.

The fact that you got a +5 out of such a demonstrably incorrect post is a major indictment of the baseball knowledge of the Slashdot faithful.

No, Martin is right -- the 1890s, while not as famous as Ruth and Gehrig's 1930s, were one of the most offensive eras in baseball. His simple analysis is much more forgivable than the insults you throw his way even while being completely ignorant of an entire decade of baseball history, the data from which are right on the web page you so callously direct him to visit.

Nerves

[digidave]

This doesn't take into account that once a player achieves an impressive hit streak he gets more media attention, people start asking him about Dimaggio's record, and every time he steps up to the plate he's a bit more nervous about it than the last time, making it slightly less likely that he'll get a hit.

Re:Nerves

[p0tat03]

That kind of error can be accounted for by tracking their batting averages over time. If we have a model for batting average deterioration due to stress, then the simulation will still work as a good approximation.

Re:Nerves

[Kjella]

In any case, what you are talking about would affect all players equally, therefore it would cancel itself out in their research. Not when they use it the way they use it, and say streaks of 39 to 109 is to be expected. If the difficulty increases by the length of the streak, 56 could be a far more exceptional streak than their research indicates.

Re:Nerves

[cleatsupkeep]

I remember reading an article saying that "clutch performers" don't really exist - and that the reason we believe they do is because of the same biases that make us cling to our beliefs - taking note of something when it fits your belief and tossing it away when it disagrees.

Wikipedia to the rescue (http://en.wikipedia.org/wiki/Clutch_(sports))

Some sports analysts have presented evidence that while individual plays and moments may resonate as "clutch" because of their importance, there is no such thing as "clutch ability" or an inherently clutch player. One example of such an argument is presented in the 2006 book Baseball Between the Numbers published by Baseball Prospectus, which compiles evidence that no baseball players are demonstrably consistently clutch over the course of a career, and that the numbers of allegedly clutch players in clutch situations are in fact no different from players reputed to be "chokers."[1]

The rest of the page does well with some very good examples from baseball - including Derek Jeter and Reggie Jackson (both Yankees - maybe the author was a Red Sox fan).

How to Make Baseball Even MORE Boring?

[morari]

Talk about the statistics of anyone at bat..

Re:How to Make Baseball Even MORE Boring?

[pchan-]

You don't understand. Baseball is so boring, the fans find the statistics exciting!

Re:How to Make Baseball Even MORE Boring?

[garett_spencley]

I was once at a friend's BBQ and a lot of the other guests were really into sports and talking a lot about their various sporting events etc. I made a comment about how baseball was one of those sports that is fun to play but boring as hell to watch. One of the guys responded with, simply, "I disagree". To which I replied "You're right. It's pretty boring to play too." He wasn't very amused.

Talk about a great way to make an awkward social event even more awkward :(

Re:How to Make Baseball Even MORE Boring?

[jd]

Do you have the statistics to prove that?

Re:Deistic proof?

[Loadmaster]

What would God need with an impressive hitting streak? Besides, Jesus can't hit a curve ball so he'd never make it anyway.

i came here to make an insightful comment

[circletimessquare]

unfortunately, not many of my comments are insightful, so with my batting average, you will have to refer to a parallel universe

there you will find that this comment contains something worthwhile reading. sorry

Changing game of baseball

[__aanonl8035]

One of the key points mentioned in this article is when does the hitting game streak occur? They mention that it was much more likely to occur during the early 1900's which is known as the deadball era. The baseball wasn't as springy and they tended to use the same ball during the entire game. During that time it was more efficient to try and knock the ball between the holes in the fielders and get a double or single then to try and hit it out of the park.

I think it would be more impressive to take a subset of the data, and compare from 1930 up until the present. Of course, there have been other major changes to; glove sizes, introduction of the slider for a pitch, steroid use.

too simplistic

[ndenissen]

From reading the article (which is light on the details) it seems like they used nothing but batting average, at bats, and games played.

The problem is this doesn't control for variances in the quality of pitching. The chances of going that many games without running into a hot pitcher isn't accounted for.

Imagine you average a 75% chance of getting a hit in any individual game. If you face three average pitchers, your chances are (.75)^3 but if you face a good pitcher an average pitcher and a bad pitcher it might be (.5)(.75)(1.0) which gives a different probability, despite the same average number of hits.

In order to be realistic the calculation would need to account for the deviation from average in the ability of the pitchers (which would likely be higher 100 years ago because of fewer player and segregation, and now because of expansion, as compared to the 1950s)

What they don't report is how often there are long (but not record) streaks in their model, so there is no way of knowing how accurately it reproduces reality.

Re:too simplistic

[DannyO152]

On the other hand, one doesn't get the benefit of running into the belly-itchers. My feeling is that, on average, the superstars, the ones with above 340 career averages, generally feasted on the mediocre to minor pitchers.

What this study doesn't take into account is how long it takes to live through a streak. DiMaggio needed two months. Besides the strain of day to day playing (and if it's a pennant race, you know the hot hitter is going to be in the lineup) there's also the way the weather and the light changes during the season. There used to be more day games and double-headers back in the 30s-40s-50s when batting averages were highest. Travel was by train and by bus and took longer. There seems to be a week every season when a cold or flu is making the rounds of the club. Then there's situational issues. 7th inning and behind, man on second base, the hitter is 0-3 and 30 games into the streak. I say the pitcher semi-intentionally walks the batter and amid a chorus of boos the streak goes poof. Here's another consideration, the opposing players and pitchers know the hitter has a streak when it gets past 20 games and the pitching gets a bit more careful and the batter has to extend the streak via pitchers' mistakes, and that makes it less likely.

if what I say is true, it should follow that the incidence of any consecutive games with a hit streak beyond 15 in a MLB season should be lower than the probability suggested by the league batting averages (which are depressed in the NL by pitchers and the other bottom 4 from the lineup.)

Too many assumptions?

[kevinatilusa]

From the descriptions I've seen of their research, it seems that they're treating all games identically for the purpose of determining a typical season's behavior. While this may me necessary to make the computation tractable, it's not realistic, and introduces a sizable bias towards long hitting streaks.

In reality, a league is typically very imbalanced from team to team and from pitcher to pitcher (probably even more so in the game of the early 20th century than now). It's easier to get hits off of two successive average pitchers than it is to get hits both off of a very good and a very bad pitcher. For example (to oversimplify a good deal):

Say the league is split 50/50 between "good" pitchers (pitchers you'll get a hit off of 50% of games) and "bad" pitchers (pitchers you'll get a hit off of 80% of games). In a typical 20 game stretch, you'll encounter 10 good pitchers and 10 bad ones, and your odds of getting a hit in all 20 games would be (0.50)^10(0.80)^10, about 1/9537.

Under their analyis as I understand it, they'd replace all the pitchers by mediocre pitchers who you'd get a hit off of 65% of the time, and your odds would be (0.65)^20, about 1/5517.

This one assumption almost doubled your chances of getting a hit in all 20 games.

There are other biases as well going the other way (ignoring the effect of hitting slumps, for example), but this one jumped out at me.

Re:Deistic proof?

[TheRealMindChild]

Apparently, Joeboo can't either :(

Re:Bogus

[Miseph]

No, because the probability for ANYTHING, given enough chances, is 1.

What they are actually saying is that reality appears to follow a probability bell curve.

You could also say that, in 1,230,000 years of baseball games, we could be almost certain of a hitting streak longer than 56 games.

So basically...

[davidbrit2]

They took a bunch of measured statistics, ran a simulation with outcomes biased using said statistics, and then acted surprised when the simulation results ended up pretty close to what actually happened?

Re:So basically...

[kevinatilusa]

They took a bunch of measured statistics, ran a simulation with outcomes biased using said statistics, and then acted surprised when the simulation results ended up pretty close to what actually happened? I think their point was that they took a set of numbers that were generally considered unremarkable (the overall statistical distribution of batting totals from the last 100+ years) and tried to show that a number that most people considered very unusual (the 56 game streak) was in fact also typical given this other, "unremarkable" set of data.

Anohter unreported weird fact

[mrfantasy]

In every simulation, a ground ball went between Bill Buckner's legs in the 1986 World Series.

Re:You can't do statistics with a random # generat

[kevinatilusa]

It is not mathematically sound to do statistics with a random number generator. Computers do not actually generate random numbers, but instead, they can only make pseudo-random numbers that have a certain distribution. Any 'simulation' done in this way will always have a bias. In order to get correct statistics, you must actually compute the statistics. Sure, the proper way to put it mathematically would have been "we did a Monte-Carlo based simulation of the probability distribution of the longest hitting streak under our model due to the intractability of direct computation", but this is an editorial in the New York Times, not a mathematical journal! As a side note, just because a computation is performed on a set of pseudorandom numbers does not mean it is biased...usually the whole point of pseudorandomness is that the discrepancy between computations involving them and identical computations involving true random numbers will typically be quite small.

What about slumps?

[Squirmy McPhee]

By assuming the hitter's probability of getting a hit is equal to his season average the researchers don't take into account that most, if not all, batters have a higher batting average at some points in the season than they do in others. As one with experience in Monte Carlo simulations I know that taking that into account would complicate the analysis considerably, but I suspect their results would be a bit different if they even did something as simple as using a 10-game moving average of the batter's average.

Does Joe DiMaggio's Streak Deserve an Asterisk?

[harryjohnston]

This seems relevant:

http://abcnews.go.com/Technology/WhosCounting/story?id=3694104&page=1

Disclaimer: I'm not an American, so I know next to nothing about baseball - and care less!

it's basically a sure thing.

[iminplaya]

Yeah, and so is the Cubs winning the World Series more than once in a hundred years

I've switched domains...

[xactuary]

I've switched domains to operating systems and can now say that it took 42 googleplex simulations before I found a parallel universe where Vista doesn't suck. As you would expect, that's also the only parallel universe that had Steve Jobs throwing chairs.

Re:You can't do statistics with a random # generat

[Vellmont]

Computers do not actually generate random numbers

That'll be a surprise to the multiple true random number generators build into most operating systems. There's many sources of random data in a computer. Timing between keystrokes, timing of mouse movements, network latency between packets, and of course hardware random number generators that use thermal noise as its source.

So to put it mildly, computers can, and DO generate truly random numbers that are completely unpredictable and free from bias.

(Oh, BTW, to do a Monte-Carlo simulation (which the referenced article is) you actually don't need true random numbers, you only need a pseudo-random source that's free from bias. Those pseudo-random sources do exist, and aren't that even that difficult to code.)

Comparison of Sports

[buildguy]

Interesting comparison made on this page, but I'm not sure if it is accurate. http://en.wikipedia.org/wiki/Don_Bradman#World_sport_context

Re:You can't do statistics with a random # generat

[Vellmont]

Unless you are dealing with quantum effects, you are not dealing with something truly random.


From wikipedia on "electronic (thermal) noise":

In any electronic circuit, there exist random variations in current or voltage caused by the random movement of the electrons carrying the current as they are jolted around by thermal energy.

Is that quantum mechanical enough for you?

As for network latency between packets, while it may not be random on a quantum-mechanical level, it's still unpredictable unless you can get on the same lan segment as the target computer. The keyboard timings are taken on a small enough time scale that they're quite unpredictable, and not related to the typist.

Other tidbits about DiMaggio's streak

[jocknerd]

After the streak ended, he started a new 16 game hitting streak. That means he hit safely in 72 of 73 games.

During the streak Joe DiMaggio had a batting average of .408, a slugging average of .717, he faced four (4) future hall of fame pitchers, and he played in the 1941 All-Star Game (he went one-for-four, scored a run, and drove in a run). Source is http://www.baseball-almanac.com/feats/feats3.shtml

During Joe DiMaggio's streak, Ted Williams actually had a higher batting average. William's batted .412 and finished with a .406 average for the year.

Joe DiMaggio had a 61 game hitting streak while playing for the San Francisco Seals in the Pacific Coast League in 1933.

Re:You can't do statistics with a random # generat

[CTachyon]

Modern Intel motherboards (i810 forward) and AMD motherboards (768 forward) have a hardware RNG (Random Number Generator) that IIRC is based on diode noise. That's straight up quantum randomness, and most modern Linux distros automatically detect and use it if available.

The Monroe Factor

[Slur]

Okay, okay, but what are the odds that Joe DiMaggio would have such a streak, and land Marilyn Monroe? Somebody needs to get on that simulation asap. Here are my statistics, by the way...