'Windows and Linux file path issues using python SQL load data infile
I am working on a mysql (8) db which is too big for the 2TB linux partition size so I have moved the mysql instance onto a 16TB nvme raid under windows 10. All my other code is running on Debian 10 on a virtualbox instance and I have mapped a perm drive between the Debian VM and the nvme raid array.
I can open the database from debian and read and write as normal, so the odbc connector is working fine.
The issue here is the fact I am loading very large json log files into one table and doing it a row at a time was taking hours, so I opted to create a csv file for each log file and use LOAD DATA INFILE as part of the SQL statement.
Trouble is, when I execute the sql statement I get the 'file not found' issue, even though, looking at debug code, the file path is correct and it actually exists.
An excert from my python 3 code is:
p = f"/media/sf_unpack/{filename}"
try:
with open(p, 'r', buffering=1024 * 1024) as csvfile:
print(csvfile.read())
SQL = f"LOAD DATA INFILE '{p}' INTO TABLE xxx fields terminated by ',' LINES TERMINATED BY '\n' (field,field,.....etc) ;"
try:
mycursor.execute(SQL)
connection_object.commit()
except Exception as ex:
displayerror(ex)
This code will open the file correctly and show a value for p of /media/sf_unpack/filename.csv (which is correct).
When we get to the mycusror.execute(SQL) is raises an exception and says the directory or filename cannot be found. Interestingly, and I am sure this is the issue, the dubugger tells me the file that cannot be found is a windows version D:\media/sf_unpack/filename.csv - which looks as if it has something to do with the virtualbox mapping. i.e. p = I have tried to use the Path method from pathlib i.e p = Path(f"D:\mysql\unpack\{filename}" but that makes no difference.
I know I am doing something stupid but I am not sure what it is.
Any help would be gratefully recieved
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
Solution | Source |
---|