'Fastest way to upload a DataFrame to a table in MS SQL Server?
I've been trying to upload a huge dataframe to table in SQL Server, the dataframe itself contains 1M+ rows with more than 70+ columns, the issue is that by trying multiple codes it takes 40 minutes or more to upload it. Is there a fastest way to do so? Here is a couple of codes that I've tried to use:
Using BCPandas takes 40 minutes:
from bcpandas import SqlCreds, to_sql
creds = SqlCreds(
'server',
'schema',
'user',
'password'
)
to_sql(datadrameHuge,'table_test',creds,index = False, if_exists='append',schema='gcp')
Using SQLAlchemy takes more than an hour:
import urllib
import sqlalchemy
import pyodbc
server = 'servertest'
database = 'dbtest'
username = 'untest'
password = 'passwordtest'
driver = 'ODBC Driver 17 for SQL Server'
params = 'DRIVER='+driver + ';SERVER='+server + ';PORT=1433;DATABASE=' + database + ';UID=' + username + ';PWD=' + password
db_params = urllib.parse.quote_plus(params)
engine = sqlalchemy.create_engine("mssql+pyodbc:///?odbc_connect={}".format(db_params))
pyodbc.drivers()
from sqlalchemy import event
@event.listens_for(engine, "before_cursor_execute")
def receive_before_cursor_execute(
conn, cursor, statement, params, context, executemany
):
if executemany:
cursor.fast_executemany = True
dataframeHuge.to_sql('testtable', engine, index=False, if_exists="append", schema="dbo")
Is there a faster way to do this? Also does the connection speed affects the upload?
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|
