too much waiting thread cause java heap dump in websocket client in java 8

Question

Today my java application heap dump, and I copy the dump file from server analysis using visualVM, the log look like this:

"WebSocketClient-SecureIO-1" daemon prio=5 tid=888 WAITING
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
       local variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#184
       local variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#185
    at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
       local variable: java.util.concurrent.CountDownLatch$Sync#36
    at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
       local variable: java.util.concurrent.CountDownLatch#35
    at sun.nio.ch.PendingFuture.get(PendingFuture.java:180)
    at org.apache.tomcat.websocket.AsyncChannelWrapperSecure$ReadTask.run(AsyncChannelWrapperSecure.java:269)
       local variable: sun.nio.ch.PendingFuture#47
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
       local variable: java.util.concurrent.ThreadPoolExecutor#1
       local variable: org.apache.tomcat.websocket.AsyncChannelWrapperSecure$ReadTask#6
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
       local variable: java.util.concurrent.ThreadPoolExecutor$Worker#1
    at java.lang.Thread.run(Thread.java:748)

"WebSocketClient-SecureIO-2" daemon prio=5 tid=889 WAITING
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
       local variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject#5
       local variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#114
    at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
       local variable: java.util.concurrent.LinkedBlockingQueue#1
       local variable: java.util.concurrent.atomic.AtomicInteger#56
       local variable: java.util.concurrent.locks.ReentrantLock#9
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
       local variable: java.util.concurrent.ThreadPoolExecutor$Worker#2
    at java.lang.Thread.run(Thread.java:748)

"pool-87-thread-1" prio=5 tid=890 TIMED_WAITING
    at sun.misc.Unsafe.park(Native Method)
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
       local variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$Node#183
       local variable: java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject#558
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
       local variable: java.util.concurrent.locks.ReentrantLock#3654
    at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
       local variable: java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue#1
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
       local variable: java.util.concurrent.ScheduledThreadPoolExecutor#129
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
       local variable: java.util.concurrent.ThreadPoolExecutor$Worker#3
    at java.lang.Thread.run(Thread.java:748)

seems too many thread(maybe thousands) is waithing all the time, my memory now config to 500MB, and I now I have no idea why would this happen.This my websocket connection code:

 public WebsocketClientEndpoint robotNewConnect(Long roomTypeId, String token, String userMark) {
        WebsocketClientEndpoint clientEndPoint = null;
        String websocketConnUrl = websocketUrl + "?token=" + token + "&roomTypeId=" + roomTypeId + "&robotFlag=1";
        try {
            String appMark = SessionUtil.getThreadLocal("appMark");
           
            clientEndPoint = new WebsocketClientEndpoint(new URI(websocketConnUrl));
            clientEndPoint.userSession.getUserProperties().put("userIdentity", userMark + "-" + appMark + "-" + roomTypeId);
            clientEndPoint.addMessageHandler(message -> {
                log.info("addMessageHandler:", message);
            });
        } catch (Exception e) {
            log.error("Websocket", e);
        }
        return clientEndPoint;
    }

I am searhing from internet and try to incrase my memory but problem still not resolve. what may cause this problem and what should I do to fix this?

what I have tried:

1. I follow to the souce code of tomcat-embed-websocket-9.0.30 where class WebsocketClientEndpoint belong. the connect was success, but stuck on this line code:

WsFrameClient wsFrameClient = new WsFrameClient(response, channel, wsSession, transformation);

and I step into the class and find the code was stuck in the dead lock code:

private void doResumeProcessing(boolean checkOpenOnError) {
            while (true) {
                switch (getReadState()) {
                case PROCESSING:
                    if (!changeReadState(ReadState.PROCESSING, ReadState.WAITING)) {
                        continue;
                    }
                    resumeProcessing(checkOpenOnError);
                    return;
                case SUSPENDING_PROCESS:
                    if (!changeReadState(ReadState.SUSPENDING_PROCESS, ReadState.SUSPENDED)) {
                        continue;
                    }
                    return;
                default:
                    throw new IllegalStateException(
                            sm.getString("wsFrame.illegalReadState", getReadState()));
                }
            }
        }

the read state getReadState is always PROCESSING, and the code loop forever, this is why so much waiting thread in dump file.

But now I do not know why the read state is PROCESSING and how to solve it? any one could help me?

Answer 1

I had faced similar problems with WebSocket when I was not properly closing the interrupted WebSocket connections and creating new one while retrying.

Steps I took to fix them:

1. Ensure that the WebSocketClient objects are garbage collected when connection is closed due to error. I was using Spring so I registered WebSocketClient as bean with prototype scope. So that when connection is closed Spring will do the cleanup.

    @Bean
    @Scope("prototype")
    public StandardWebSocketClient webSocketClient() throws Exception {
        StandardWebSocketClient standardWebSocketClient = new StandardWebSocketClient(clientContainer());
        standardWebSocketClient.setTaskExecutor(webSocketTaskExecutor());
        return standardWebSocketClient;
    }

    @Bean
    public AsyncListenableTaskExecutor webSocketTaskExecutor() {
        SimpleAsyncTaskExecutor executor = new SimpleAsyncTaskExecutor("webSocketTaskExecutor-");
        executor.setConcurrencyLimit(20);
        return executor;
    }

---

2. Declared ClientContainer bean with destroyMethod in annotation.

    @Bean(destroyMethod = "doStop")
    public ClientContainer clientContainer() throws Exception {
        WebSocketPolicy webSocketPolicy = WebSocketPolicy.newClientPolicy();
        webSocketPolicy.setMaxTextMessageSize(1024000); //1MB
        ClientContainer clientContainer = new ClientContainer(new SimpleContainerScope(webSocketPolicy));
        clientContainer.start();
        return clientContainer;
    }

---

FYI I was using spring-boot-starter-websocket with jetty-server.

        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-websocket</artifactId>
        </dependency>

These changes reduced the number of threads drastically.