HttpURLConnection getResponseCode() API call gets stuck for ~30 mins on POST request and then returns IOException

Question

I'm facing issue of my client app getting hanged for long duration while trying to send a POST request. My connection timeout is 5sec and read timeout is 240 sec. But, the request gets stuck in wait state for approx 30min. before returning below IOException.

SyncRESTClient|processRequest|Received IO Exception for requestURI[http://172.50.32.79:8002/BatchWrite/1/1], Exception: 
java.io.IOException: Error writing to server
    at sun.net.www.protocol.http.HttpURLConnection.writeRequests(HttpURLConnection.java:666)
    at sun.net.www.protocol.http.HttpURLConnection.writeRequests(HttpURLConnection.java:678)
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1534)
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1441)
    at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
    at SyncRESTClient.processRequest(SyncRESTClient.java:62)

POST Request Size: ~2MB

Note:

• The issue is intermittent i.e. out of 100 requests, it may/may not occur once.
• We have also sent requests with larger size than 2MB and they also got processed.

Test Environment:

• Both client and server are on different machines with 100Mbps connectivity

Code Snippet:

        HttpURLConnection connObj = null;
        URL myurl = null;
        try {
            myurl = new URL(requestInfo.getUri());
            connObj = (HttpURLConnection) myurl.openConnection();
            
            connObj.setRequestProperty("Accept", "*/*");
            // set connect timeout
            if(0 < requestInfo.getConnectTimeout()) {
                connObj.setConnectTimeout(requestInfo.getConnectTimeout());
            }
            
            // set request timeout
            if(0 < requestInfo.getRequestTimeout()) {
                connObj.setReadTimeout(requestInfo.getRequestTimeout());
            }
            
            switch(requestInfo.getMethod()) {
            case GET:
                connObj.setRequestMethod("GET");
                break;
            case POST:
                connObj.setRequestMethod("POST");
                connObj.setRequestProperty("Content-Type", "application/octet-stream");
                connObj.setDoOutput(true);
                break;
            }
            //connObj.setRequestProperty("User-Agent", "Mozilla/5.0");
            
            if(RESTRequestInfoMethodTypes.POST == requestInfo.getMethod()) {

                try (DataOutputStream wr = new DataOutputStream(connObj.getOutputStream())) {
                    wr.write(requestInfo.getData(), 0, (int)requestInfo.getDataLength());
                }
            }
            
            rv = 0;
            int statusCode = connObj.getResponseCode();      <--- STUCK HERE
            responseInfo.setHttpStatusCode(statusCode);
            
            if(HttpURLConnection.HTTP_OK == statusCode) {
                try (InputStream is = connObj.getInputStream()) {
                    //byte[] arr = is.readAllBytes(); // Available since Java9
                    //byte[] arr =  is.readNBytes(Integer.MAX_VALUE); // Available since Java11
                    /*byte[] arr = new byte[is.available()];
                    int length = is.read(arr);
                    responseInfo.setData(arr);
                    responseInfo.setLength(length);*/
                    
                    ByteArrayOutputStream buffer = new ByteArrayOutputStream();
                    int nRead;
                    byte[] data = new byte[16384];  // 16K buffer allocation for reading
                    while ((nRead = is.read(data, 0, data.length)) != -1) {
                      buffer.write(data, 0, nRead);
                    }
                    responseInfo.setData(buffer.toByteArray());
                    responseInfo.setLength(buffer.size());
                    buffer.close();
                    buffer = null;
                }
            } else {
                try (InputStream is = connObj.getErrorStream()) {
                    if(null != is) {
                        ByteArrayOutputStream buffer = new ByteArrayOutputStream();
                        int nRead;
                        byte[] data = new byte[16384];  // 16K buffer allocation for reading
                        while ((nRead = is.read(data, 0, data.length)) != -1) {
                            buffer.write(data, 0, nRead);
                        }
                        responseInfo.setData(buffer.toByteArray());
                        responseInfo.setLength(buffer.size());
                        buffer.close();
                        buffer = null;
                    }
                }
            }
            
        } catch (MalformedURLException e) {
            log.error("SyncRESTClient|processRequest|Received malformed exception for requestURI[" + requestInfo.getUri() + "], Exception: " , e);
            rv = -1;
        } catch(java.net.SocketTimeoutException e) {
            log.error("SyncRESTClient|processRequest|Request timed out for requestURI[" + requestInfo.getUri() + "] after [" + (System.currentTimeMillis()-startTime) + " ms], Exception: " + e);
            rv = -1;
            responseInfo.setHttpStatusCode(HttpURLConnection.HTTP_CLIENT_TIMEOUT);
        } catch(java.net.ConnectException e) {
            log.error("SyncRESTClient|processRequest|Failure while connecting to requestURI[" + requestInfo.getUri() + "], Exception: " + e);
            rv = -1;
            responseInfo.setHttpStatusCode(HttpURLConnection.HTTP_UNAVAILABLE);
        } catch(IOException e) {
            log.error("SyncRESTClient|processRequest|Received IO Exception for requestURI[" + requestInfo.getUri() + "], Exception: " , e);
            rv = -1;
        } catch(Exception e) {
            log.error("SyncRESTClient|processRequest|Exception while processing request[" + requestInfo.getUri() + "]: " , e);
            rv = -1;
        } catch(OutOfMemoryError e) {
            log.error("SyncRESTClient|processRequest|Received OutOfMemoryError while processing request[" + requestInfo.getUri() + "]. Error: " , e);
            rv = -1;
        } finally {
            if(null != connObj)
                connObj.disconnect();
            connObj = null;
            myurl = null;
        }

New Update:

Found the reason behind the socket getting hanged. It was observed that packet drops occurred between client and server machine for the session. Due to these packet drops, client was unable to detect that server has closed the connection by sending fin-ack packet. It resulted in client still being in established state waiting for communication.

Steps to replicate the issue:

1. Start sending data from client to server

2. While data is being transmitted, run below linux command on client machine to drop packets received from server:

   iptables -I INPUT -i "interface" -s "serverMachineIP" -j DROP

   Eg: iptables -I INPUT -i eth0 -s 172.50.31.10 -j DROP

Above steps will result in client getting stuck for long duration.

New Question: What should be done at client so that request can timeout in given interval of time?