'How to determine the length of a very large file
I am currently working on a filter prototype that works with GNSS data.
The files that I`m supposed to work with are large (with sizes close to 60GB). Because of this, I am trying to run parallel processes on different portions of the file in order to reduce the run-time. I'm opening the file with np.fromfile which allows me to read a specified number of elements, with the possibility of adding an offset when it comes to where I want the reading operation to start from.
My question is: how can I determine the number of elements inside the file without parsing the entire file first, as this would add a lot to the execution time of the code.
An idea that I had was to return this value from the append method that python uses, since it automatically determines where the last element of the file is.
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|