Problem on saving Spark timestamp into Azure Synapse datetime2(7)

Question

I have a database in Azure synapse with only one column with datatype datetime2(7). In Azure Databricks I have a table with the following schema.

df.schema
StructType(List(StructField(dates_tst,TimestampType,true)))

When I try to save on Synapse, I get an error message

Py4JJavaError: An error occurred while calling o535.save.: org.apache.spark.SparkException: Job aborted due to stage failure: Task 3 in stage 15.0 failed 4 times, most recent failure: Lost task 3.3 in stage 15.0 (TID 46) (10.139.64.5 executor 0): com.microsoft.sqlserver.jdbc.SQLServerException: 110802;An internal DMS error occurred that caused this operation to fail SqlNativeBufferBufferBulkCopy.WriteTdsDataToServer, error in OdbcDone: SqlState: 42000, NativeError: 4816, 'Error calling: bcp_done(this->GetHdbc()) | SQL Error Info: SrvrMsgState: 1, SrvrSeverity: 16, Error <1>: ErrorMsg: [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]Invalid column type from bcp client for colid 1. | Error calling: pConn->Done() | state: FFFF, number: 75205, active connections: 35', Connection String: Driver={pdwodbc17e};app=TypeD00-DmsNativeWriter:DB2\mpdwsvc (56768)-ODBC;autotranslate=no;trusted_connection=yes;server=\\.\pipe\DB.2-e2f5d1c1f0ba-0\sql\query;database=Distribution_24

EDIT: Runtime version 9.1 LTS (includes Apache Spark 3.1.2, Scala 2.12)

EDIT 2: It could be solved, the errors were:

• use incorrect format in write options, I was using "com.microsoft.sqlserver.jdbc.spark" and changed it to "com.databricks.spark.sqldw".
• There were also errors in the scope credentials

Answer 1

org.apache.spark.SparkException: Job aborted due to stage failure

Generally above error occurs when you perform operations on column where there is null value present.

Replace null values with valid datetime values.

Also check spark version.

Refer this SO Answer by Lyuben Todorov