'Columns gets removed while loading an excel file to cosmos dB in ADF using data flow
I am new to ADF and cosmos db. I am trying to load an excel which has 34 columns from a blob to a cosmos db container using a mapping data flow . While debugging the data flow I could preview all the columns in the source and its data available in both source as well as sink. But after running the pipeline(using trigger/debug) when I check the items/documents in the cosmos db container certain columns are missing. When I do the same data load using copy activity all the columns are getting inserted and this seems to be weird. I need to change the data types of few columns before loading into the database and that's the reason I am trying to load the data using data flow activity. I have been stuck at this for so long and wasn't able to proceed further. Any help would be appreciated.
Solution 1:[1]
I had repro’d and was able to load all (34) columns into cosmos db successfully using data flow.
- Source projection
- Source data preview: Here C1 column is of string type.
- Changed the datatype of the C1 column from string to an integer.
- Sink transformation: Make sure you have included all source columns in sink mappings.
- Cosmosdb Output: It includes all 34 columns from the source and C1 as integer type.
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|---|
| Solution 1 | NiharikaMoola-MT |