'is it possible to only load part of a pickle file or can I use streaming?

I have a large pickle file which is a pandas dataframe. The data is minute data measuring certain weather metrics.

There are two operations that need to be run, on a daily basis which take longer than needed & I believe that is because of how I am loading the data.

# load the pickle file
with open('my_file.pickle', 'rb') as handle:
    df = pd.read_pickle(handle)

So currently I load whole file just to get the last record in the dataframe which gives me the last time stamp of the last observation. Is there a way of loading part of the file?

Also once I have last time stamp, I then want to append the new data. Do I need to open the whole file & append the data to the dataframe or is there a better alternative?

Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution	Source

'is it possible to only load part of a pickle file or can I use streaming?

Sources

Related Questions