Scraping the MLB for Science & Glory

The Question

While there are literally TONS [1] of articles with opinions on "how to fix baseball", the main claim I want to answer is:

This question seems to have been asked every baseball season, or at least before I stopped searching the internet at a 1999 Chicago Tribune article interviewing Ted Williams about the rise in strikeouts with the "new wave" of power hitters like Mark McGuire and Sammy Sosa.

Which I get, if no one can hit the ball then baseball is really 9 dudes throwing the ball to themselves for 3 hours, which is boring.

So, with a solid question defined, let's start looking for sources of information that could answer it.

What Data Do We Need

Our question is dealing with trends in pitching and hitting statistics. So we'll need to find a source of baseball statistics going back to the beginning of baseball (anno domini ~1876) or as far back as we can find.

While there are many websites to choose from, I feel the data from the MLB is a good choice, mainly so we can say "according to the official MLB data…" before we start arguing with strangers on the internet.

Unfortunately though, as is often the case, the MLB doesn't give us a link to download their statistics for further analysis.

So we'll need to build a "web scraper" to download the data directly from their website, essentially making our own "download" button.

Finding The Data

Because our browsers are doing all the work of building the MLB's client-side web-page, we can use our browser's built-in developer tools to find where the data is coming from.

Depending on which browser you prefer, open your developer tools:

• How to open Chrome's Developer Tools.
• How to open Firefox's Developer Tools.

With the developer tools open:

• Find and open the Network tab.

For Firefox 69.0 the Network tab is in Tools → Web Developer → Network or if you're into key bindings Ctrl+Shift+E.

• Next, go to the MLB's stats page and refresh the page with the Network tab open.

You should see a list of all the requests your browser made while building the MLB's web-page. Something similar to this (I'm using Firefox 69.0):

If you're in the year 2019, our browsers made 128 individual requests to their servers (54 more than average) for information to build the web-page. Which is 127 more requests I'm willing to look through by hand, so let's start filtering these requests down:

• Click on the XHR icon to filter out unrelated requests.

The requests we're interested in are XHR (XMLHttpRequest) objects. These are requests made by the web-page's JavaScript to get data (usually in XML or JSON form) from an API.

With the requests filtered, we can start exploring for the data we want. We know we're looking for data which usually comes in either XML or JSON format, so we can ignore the "js" types.

• Click on each Response looking for the data.

I usually look at responses with the largest size first and work my way down. After a few attempts, we'll find a JSON blob with keys that match the data we want.

Huzzah! We've found the API endpoint with the data we need.

Before we move on, take note of the data's shape in the response. Notice the list rows, with all the baseball stats, is inside queryResults inside team_pitching_season_leader_master. We'll need this when we start building the web scraper later on.

Our next step is to find how we can recreate the request using python:

• Switch to that request's Headers tab and copy the URL.

This will show us the URL and the Headers we sent to the server.

If everything went according to plan, when you

• Copy the URL into a new browser tab,

you should see the same information we found in the Developer Tools just a moment ago telling us we've found the URL to the API we want.

Good Job!

Writing the Code

From our exploration above, we know the URL to our endpoint is this:

team_pitching = (
    'http://lookup-service-prod.mlb.com/'
    'json/named.team_pitching_season_leader_master.bam?'
    'season=2019&sort_order=%27asc%27&sort_column=%27whip%27'
    '&game_type=%27R%27&sport_code=%27mlb%27&recSP=1&recPP=50'
)

However, this will only download each team's stats for the 2019 season. This API uses the query string parameters as a kind of filter for the content it provides to us.

For example, changing season from 2019 to 2018 would yield us each team's pitching statistics for the 2018 season or changing game_type to "%27S%27" will give us statistics on spring training games.

So let's break team_pitching down into its component parts, allowing us to make requests for the different seasons the MLB has in their database.

We can start by separating the domain, path, and query string parameters into separate variables like this:

domain = 'https://lookup-service-prod.mlb.com'
path = '/json/named.team_pitching_season_leader_master.bam'
query_string = {
    'season': 2019,
    'sort_order': '%27asc%27',
    'sort_column': '%27whip%27',
    'game_type': '%27R%27',
    'sport_code': '%27mlb%27',
    'recSP': 1,
    'recPP': 50,
}

After looking up what the %27 means (they're URL encoded ' apostrophes), let's make our future selves happy by making a simple function to replace the %27 (with a note) and turn our query_string into this:

# words are quoted with apostrophes
quote = "'{}'".format

query_string = {
    'season': 2019,
    'sort_order': quote('asc'),
    'sort_column': quote('whip'),
    'game_type': quote('R'),
    'sport_code': quote('mlb'),
    'recSP': 1,
    'recPP': 50,
}

With our URL broken down, we can make a simple for loop to go through each year (1876 - 2019), making a request to the MLB to download the pitching statistics for that season and save the results into a list creatively called data.

import time

data = []  # we'll save all the stats here

# go through each season and make a request
for season in range(1876, 2019 + 1):
    time.sleep(1)  # rule #2 don't hammer the servers

    # update the query_string for the correct season
    query_string['season'] = season

    # make the request to the MLB
    response = requests.get(
        domain + path,
        params=query_string
    )

    # raise an error if the MLB gives
    # an invalid response back
    response.raise_for_status()

    # pull the list of stats out of the response
    stats = response.json()[
        'team_pitching_season_leader_master'
    ]['queryResults']['row']

    # add the season to each row, and
    # append it to our 'data' list
    for team in stats:
        team['season'] = season
        data.append(team)

And finally, after we've collected all the data from the MLB, we put the data into a pandas DataFrame to start the "data wrangling" phase of our journey.

import pandas as pd
df = pd.DataFrame(data)

For now though, remembering rule #3 above, we'll save our data to disk and wait until we have more time to answer the question "Are strikeouts making baseball boring".

df.to_pickle('mlb-team-pitching-statistics.pkl')

Good Job Everyone!