'Fastest way to upload a DataFrame to a table in MS SQL Server?

I've been trying to upload a huge dataframe to table in SQL Server, the dataframe itself contains 1M+ rows with more than 70+ columns, the issue is that by trying multiple codes it takes 40 minutes or more to upload it. Is there a fastest way to do so? Here is a couple of codes that I've tried to use:

Using BCPandas takes 40 minutes:

from bcpandas import SqlCreds, to_sql
creds = SqlCreds(
  'server',
  'schema',
  'user',
  'password'
  )

to_sql(datadrameHuge,'table_test',creds,index = False, if_exists='append',schema='gcp')

Using SQLAlchemy takes more than an hour:

import urllib
import sqlalchemy
import pyodbc

server = 'servertest' 
database = 'dbtest' 
username = 'untest' 
password = 'passwordtest'  
driver = 'ODBC Driver 17 for SQL Server'

params = 'DRIVER='+driver + ';SERVER='+server + ';PORT=1433;DATABASE=' + database + ';UID=' + username + ';PWD=' + password
db_params = urllib.parse.quote_plus(params)
engine = sqlalchemy.create_engine("mssql+pyodbc:///?odbc_connect={}".format(db_params))

pyodbc.drivers()



from sqlalchemy import event
@event.listens_for(engine, "before_cursor_execute")
def receive_before_cursor_execute(
       conn, cursor, statement, params, context, executemany
        ):
            if executemany:
                cursor.fast_executemany = True

dataframeHuge.to_sql('testtable', engine, index=False, if_exists="append", schema="dbo")

Is there a faster way to do this? Also does the connection speed affects the upload?



Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source