Somehow an error is not being picked up by try... except

Question

I am using a try... except loop to deal with opening a file that is updated throughout the day. Every now and then it would throw an error "pickle data is truncated", so I introduced the loop to try at least 100 times:

import pandas as pd

for i in range(100):
    try:
        df = pd.read_pickle('data')
        break
    except EOFError:
        time.sleep(0.01)

Somehow an error was thrown in the middle of the loop though? Its a problem because it stops my entire process which runs throughout the day.

Is there a better way of trying to open the file? Its updated every few milliseconds throughout the day, and all I want to do is retry a few milliseconds later if there is a problem.

Answer 1

A better way (although still not foolproof) to overwrite the existing file is something like this:

import pandas as pd
import os
df.to_pickle('tmp')
os.remove('data')
os.rename('tmp','data')