How should I handle "java.net.SocketException: Connection reset" in multithread AWS S3 file upload?

Question

I have a ThreadPoolExecutorService to which I'm submitting runnable jobs that are uploading large (1-2 GB) files to Amazon's S3 file system, using the AWS Java SDK. Occasionally one of my worker threads will report a java.net.SocketException with "Connection reset" as the cause and then die.

AWS doesn't use checked exceptions so I actually can't catch SocketException directly---it must be wrapped somehow. My question is how I should deal with this problem so I can retry any problematic uploads and increase the reliability of my program.

Would the Multipart Upload API be more reliable?

Is there some exception I can reliably catch to enable retries?

Here's the stack trace. The com.example.* code is mine. Basically what the DataProcessorAWS call does is call putObject(String bucketName, String key, File file) on an instance of AmazonS3Client that's shared across threads.

14/12/11 18:43:17 INFO http.AmazonHttpClient: Unable to execute HTTP request: Connection reset
java.net.SocketException: Connection reset
    at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:118)
    at java.net.SocketOutputStream.write(SocketOutputStream.java:159)
    at sun.security.ssl.OutputRecord.writeBuffer(OutputRecord.java:377)
    at sun.security.ssl.OutputRecord.write(OutputRecord.java:363)
    at sun.security.ssl.SSLSocketImpl.writeRecordInternal(SSLSocketImpl.java:830)
    at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:801)
    at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:122)
    at org.apache.http.impl.io.AbstractSessionOutputBuffer.write(AbstractSessionOutputBuffer.java:169)
    at org.apache.http.impl.io.ContentLengthOutputStream.write(ContentLengthOutputStream.java:119)
    at org.apache.http.entity.InputStreamEntity.writeTo(InputStreamEntity.java:102)
    at com.amazonaws.http.RepeatableInputStreamRequestEntity.writeTo(RepeatableInputStreamRequestEntity.java:153)
    at org.apache.http.entity.HttpEntityWrapper.writeTo(HttpEntityWrapper.java:98)
    at org.apache.http.impl.client.EntityEnclosingRequestWrapper$EntityWrapper.writeTo(EntityEnclosingRequestWrapper.java:108)
    at org.apache.http.impl.entity.EntitySerializer.serialize(EntitySerializer.java:122)
    at org.apache.http.impl.AbstractHttpClientConnection.sendRequestEntity(AbstractHttpClientConnection.java:271)
    at org.apache.http.impl.conn.ManagedClientConnectionImpl.sendRequestEntity(ManagedClientConnectionImpl.java:197)
    at org.apache.http.protocol.HttpRequestExecutor.doSendRequest(HttpRequestExecutor.java:257)
    at com.amazonaws.http.protocol.SdkHttpRequestExecutor.doSendRequest(SdkHttpRequestExecutor.java:47)
    at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
    at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:715)
    at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:520)
    at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
    at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
    at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:685)
    at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:460)
    at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:295)
    at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3697)
    at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1434)
    at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1294)
    at com.example.DataProcessorAWS$HitWriter.close(DataProcessorAWS.java:156)
    at com.example.DataProcessorAWS$Processor.run(DataProcessorAWS.java:264)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)

Answer 1

The javadoc says that putObject() throws AmazonClientException when this kind of error occurs. AmazonClientException has a method called isRetryable(), you can try with that.

Answer 2

For that you have to use only AmazonS3Client not Trasfermanager for upload and download

U have to configure AmazonS3Client with following properties:

1. connectionTimeout=50000 in ms
2.maxConnections=500
3.socketTimeout=50000 in ms
4.maxErrorRetry=10

For upload use

AmazonS3Client.putObject(bucketName, key, inputFile);

For download use

S3Object s3Object = AmazonS3Client.getObject(new GetObjectRequest(bucketName, key));`
InputStream downloadStream  = s3Object.getObjectContent();  

and store that stream by reading bytes.

Answer 3

'Connection reset' means the connection is closed. Close the socket, or whatever higher-level construct you're using. Probably the server has decided the upload is too large, or it's overloaded, or something. Whether you can retry the operation is something only you can know.

Answer 4

In my case, I am saving a object with some metadata. During the prototype, we added a system metadata "time-created", and that consistently resulted into the similar error as above in the question. And it is not retry-able as it intends to modify a system metadata. But according to the API, as below,

Throws:

• SdkClientException - If any errors are encountered in the client while making > the request or handling the response.
• AmazonServiceException - If any errors occurred in Amazon S3 while processing > the request.

My understanding is that the expected behavior should be,

• throw one of the above exceptions, OR,
• return an 403, rather than an umbrella AmazonClientException, "Connection shutdown".

See the logs below when S3 debug is turned on,

2017-07-11 17:12:45,163 DEBUG  [org.apache.http.wire] - http-outgoing-3 >> "**[write] I/O error: Connection has been shutdown: javax.net.ssl.SSLException: java.net.SocketException: Connection reset"**
2017-07-11 17:12:45,221 DEBUG  [org.apache.http.wire] - http-outgoing-4 >> "PUT /dev-ec-dev/neon/ECTE/CPER/2017-6/ECTE_dp0p_CPER_2017-06-05.h5 HTTP/1.1[\r][\n]"
2017-07-11 17:12:45,221 DEBUG  [org.apache.http.wire] - http-outgoing-4 >> "Host: s3.data.neonscience.org[\r][\n]"
2017-07-11 17:12:45,221 DEBUG  [org.apache.http.wire] - http-outgoing-4 >> "Authorization: AWS dev-ec-owner:dE6ouKtPpP9aftLRba4OVn6DH9M=[\r][\n]"
2017-07-11 17:12:45,222 DEBUG  [org.apache.http.wire] - http-outgoing-4 >> "x-amz-meta-site: CPER[\r][\n]"
2017-07-11 17:12:45,226 DEBUG  [org.apache.http.wire] - http-outgoing-4 >> "x-amz-meta-time-created: 2017-07-11T17:12:44.904[\r][\n]"

More logs with regular logging,

2017-07-11 17:12:45,247 DEBUG  [org.apache.http.wire] - http-outgoing-4 >> "[write] I/O error: Connection has been shutdown: javax.net.ssl.SSLException: java.net.SocketException: Connection reset"
2017-07-11 17:12:45,248 ERROR  [org.battelle.neon.is.transition.ec.ECTELevelZeroPrimeJob] - Error copyDataToS3: Not getting descriptive AmazonS3Exception.
Batch runner failed.com.amazonaws.AmazonClientException: Unable to execute HTTP request: Connection reset
java.lang.RuntimeException: com.amazonaws.AmazonClientException: Unable to execute HTTP request: Connection reset
        at org.battelle.neon.is.transition.ec.ECTELevelZeroPrimeJob.runTransition(ECTELevelZeroPrimeJob.java:347)
        at org.battelle.neon.is.batch.BatchRunner.lambda$null$1(BatchRunner.java:152)
        at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)