How to solve AWS Glue pyspark script throwing retryWrite error from DocumentDB

Question

Running below code in AWS glue. Job is able to read the Data from DB but failing while writing.

An error occurred while calling o102.pyWriteDynamicFrame. Command failed with error 301: 'Retryable writes are not supported' on server :. The full response is {"ok": 0.0, "code": 301, "errmsg": "Retryable writes are not supported", "operationTime": {"$timestamp": {"t": 1647921685, "i": 1}}}

Used the catalogue DocumentDB connection in Job Details section

Tried using retryWrite=false in connection string but still getting the error

documentdb_uri = "mongodb://<host name>:27017"
documentdb_write_uri = "mongodb://<host name>:27017"

read_docdb_options = {
    "uri": documentdb_uri,
    "database": "test",
    "collection": "profiles",
    "username": "<username>",
    "password": "<password>",
    "ssl": "true",
    "ssl.domain_match": "false"
}

write_documentdb_options = {
    "uri": documentdb_write_uri,
    "database": "test",
    "collection": "collection1",
    "username": "<username>",
    "password": "<password>",
    "ssl": "true",
    "ssl.domain_match": "false"
}

# Get DynamicFrame from DocumentDB
dynamic_frame2 = glueContext.create_dynamic_frame.from_options(connection_type="documentdb",
                                                               connection_options=read_docdb_options)

# Write DynamicFrame to DocumentDB
glueContext.write_dynamic_frame.from_options(dynamic_frame2, connection_type="documentdb",
                                             connection_options=write_documentdb_options)

job.commit()

Answer 1

Solved it by downgrading Glue version from 3.0 to 2.0. In 3.0 there is no way to set retryWrite setting while using dynamic frame.

A ticket has been created in their board and it was not resolved yet. Issue in AWS board for reference - https://github.com/awslabs/aws-glue-libs/issues/111 [An error occurred while calling o365.pyWriteDynamicFrame. Command failed with error 301: 'Retryable writes are not supported' on server ****.*****.docdb.amazonaws.com:27017.]

Answer 2

The correct option is retryWrites=false and needs to be at the end of the uri.

In your case: documentdb_write_uri = "mongodb://<host name>:27017/?retryWrites=false"