Python Handle Errors while trying 2 variables

Question

I'm trying to automate the gathering of mobility data from apple website with python script. One big obstacle I encountered was there are 2 variables I have to try the errors each time I run the script. The csv url structure is as following:

https://covid19-static.cdn-apple.com/covid19-mobility-data/2208HotfixDev**22**/v3/en-us/applemobilitytrends-**2022-03-03**'.csv

The 2 variables I marked bold - HotfixDev## and date. As far as I know, there's no way to know either without trying (they don't seem to update this everyday or so).

I used the following code to handle errors for trying date variable:

def get_url(estimated):
    aapl_dl_date = estimated - dt.timedelta(days=1)
    url2 = aapl_dl_date.strftime("%Y-%m-%d")
    full_url = url1+url2+url3
    
    return full_url

while True:
    try:
        url = get_url(estimated)
        df = pd.read_csv(url)
        print('using link: ',url)
        break
    except HTTPError:
        estimated = estimated - dt.timedelta(days=1)

(estimated is a variable containing today's date as string) This worked fine. Now I realized that HotfixDev ## is also a variable. Initially, my thought is for each date, do some sort of loop to go through 00, 01... 99, then if no luck goes to one day before. But because it involes error handling, I don't know how to stack it.

Any thought?

Answer 1

It turns out there is an index file that lists the location of the CSV data file at https://covid19-static.cdn-apple.com/covid19-mobility-data/current/v3/index.json. You can use this to access the CSV file programmatically, without having to loop through potential filenames.

I found this index file with the following process:

1. Visit https://covid19.apple.com/mobility
2. Press F12 to open the developer tools
3. Go to the network tab in the developer tools
4. Refresh the page

After doing that, all of the resources loaded by the page are listed in the network tab. That index.json file is one of them.

You can use this index file to load the CSV, like this:

import requests

BASE_URL = "https://covid19-static.cdn-apple.com"

def fetch_index_data():
    url = BASE_URL + "/covid19-mobility-data/current/v3/index.json"
    response = requests.get(url)
    response.raise_for_status()
    return response.json()

def fetch_csv_text(base_path, csv_path):
    url = BASE_URL + base_path + csv_path
    response = requests.get(url)
    response.raise_for_status()
    return response.text

def main():
    index_data = fetch_index_data()
    csv_text = fetch_csv_text(
        index_data["basePath"],
        index_data["regions"]["en-us"]["csvPath"]
    )
    print(csv_text[:200])

if __name__ == "__main__":
    main()

Also, it appears the same data is also available as JSON, currently at https://covid19-static.cdn-apple.com/covid19-mobility-data/2209HotfixDev8/v3/en-us/applemobilitytrends.json. This will probably be easier to work with. You can fetch it by adapting the above code slightly.

import requests

BASE_URL = "https://covid19-static.cdn-apple.com"

def fetch_index_data():
    url = BASE_URL + "/covid19-mobility-data/current/v3/index.json"
    response = requests.get(url)
    response.raise_for_status()
    return response.json()

def fetch_mobility_data(base_path, json_path):
    url = BASE_URL + base_path + json_path
    response = requests.get(url)
    response.raise_for_status()
    return response.json()

def main():
    index_data = fetch_index_data()
    mobility_data = fetch_mobility_data(
        index_data["basePath"],
        index_data["regions"]["en-us"]["jsonPath"]
    )
    print(list(mobility_data.keys()))

if __name__ == "__main__":
    main()