'Stop Spark from serializing null fields when writing to Cosmos

I have a spark notebook that overwrites documents in Cosmos. Some of these documents have null fields which are not serialized when they are saved to Cosmos, while others do have data for those fields. When I write the dataframe back to Cosmos, I end up with documents that have a null field.

Is there a way to not serialize this column for rows for which that value is null?

Example

Document as it exists in Cosmos:

{
    "id": "7b354753-e817-41f5-848d-780dab44d179",
    "pk": "20763ae4-8394-467d-ba3c-5ab363c40f90",
    "documentType": "ExampleDocument",        
    "userId": "20763ae4-8394-467d-ba3c-5ab363c40f90",
    "label": "Needs to be updated"
}

Document after being overwritten:

{
    "id": "7b354753-e817-41f5-848d-780dab44d179",
    "pk": "20763ae4-8394-467d-ba3c-5ab363c40f90",
    "documentType": "ExampleDocument",        
    "userId": "20763ae4-8394-467d-ba3c-5ab363c40f90",
    "label": "Updated With Spark",                
    "details": null    
}

I still want details field to be serialized, but only for documents where it is not null.

azurepysparkazure-cosmosdb


Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source

Related Questions