'Windows and Linux file path issues using python SQL load data infile

I am working on a mysql (8) db which is too big for the 2TB linux partition size so I have moved the mysql instance onto a 16TB nvme raid under windows 10. All my other code is running on Debian 10 on a virtualbox instance and I have mapped a perm drive between the Debian VM and the nvme raid array.

I can open the database from debian and read and write as normal, so the odbc connector is working fine.

The issue here is the fact I am loading very large json log files into one table and doing it a row at a time was taking hours, so I opted to create a csv file for each log file and use LOAD DATA INFILE as part of the SQL statement.

Trouble is, when I execute the sql statement I get the 'file not found' issue, even though, looking at debug code, the file path is correct and it actually exists.

An excert from my python 3 code is:

    p = f"/media/sf_unpack/{filename}"
    try:
        with open(p, 'r', buffering=1024 * 1024) as csvfile:
            print(csvfile.read())
            SQL = f"LOAD DATA INFILE '{p}' INTO TABLE xxx fields terminated by ',' LINES TERMINATED BY '\n' (field,field,.....etc) ;"
            try:
                mycursor.execute(SQL)
                connection_object.commit()
            except Exception as ex:
                displayerror(ex)

This code will open the file correctly and show a value for p of /media/sf_unpack/filename.csv (which is correct).

When we get to the mycusror.execute(SQL) is raises an exception and says the directory or filename cannot be found. Interestingly, and I am sure this is the issue, the dubugger tells me the file that cannot be found is a windows version D:\media/sf_unpack/filename.csv - which looks as if it has something to do with the virtualbox mapping. i.e. p = I have tried to use the Path method from pathlib i.e p = Path(f"D:\mysql\unpack\{filename}" but that makes no difference.

I know I am doing something stupid but I am not sure what it is.

Any help would be gratefully recieved



Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source