Data Types at Azure Data Lake

Question

I am bringing data from SAP HANA based OLTP source to Microsoft azure based data lake using azure adf and data bricks. What should data type be at Microsoft azure based data lake for below data types at SAP HANA source:

• NVARCHAR(4)
• DECIMAL (9,5)
• Float
• int
• datetime
• nvarchar(64)
• decimal(10,2)

Answer 1

If you are using Copy activity for your copy process, in ADLS Gen2 you can store data in many different file formats like delimited, avro. paraquet, XML, etc . Depending upon which format you want the data types will change. For example, if you select delimited format, everything will be stored as string.

For Parquet, Avro, Delta, Orc formats there are few limitations in ADF while using copy activity.

For example: Parquet limitations:

ORC format limiation:

You can go through the ADF documentation for each file format to look for respective limitation with respect to datatypes.

You can also check for this documentation to explore more about the copy activity interim data types supported in ADF : ADF Copy activity Data type mapping