Tomcat graceful shutdown, some request to nginx got 502

Question

Tomcat version: tomcat-embed.8.5.20 nginx version: 1.12.1 We are using Nginx and Tomcat to provide service. And our way to tomcat graceful shutdown is to register a ShutdownCallback interface, and it will be called when a designated port receive 'SHUTDOWN' command. The shutdown callback is here

@Override
    public void onShutdown() {
        try {
            long startTime = System.currentTimeMillis();
            tomcat.getConnector().pause();
            LOG.info("tomcat connector is already paused");
            Executor executor = tomcat.getConnector().getProtocolHandler().getExecutor();
            if (executor instanceof ThreadPoolExecutor) {
                try {
                    ThreadPoolExecutor threadPoolExecutor = (ThreadPoolExecutor) executor;
                    threadPoolExecutor.shutdown();
                    if (threadPoolExecutor.awaitTermination(SHUTDOWN_CHECK_MAX_TIME, TimeUnit.MILLISECONDS)) {
                        LOG.info("Tomcat thread pool closed,time:{}ms",
                                System.currentTimeMillis() - startTime);
                    }
                } catch (InterruptedException ex) {
                    Thread.currentThread().interrupt();
                } finally {
                    tomcat.stop();
                    tomcat.destroy();
                }
            }
        } catch (Throwable t) {
            LOG.error("Failed to close tomcat", t);
        }
    }

First to pause the tomcat connector to hold new request, then shutdown the thread pool of tomcat and wait it complete or reach the timeout. And then call tomcat.stop() in finally;

But by this way, I still got 502 during I stress a POST Restful api, the error in nginx error.log is 'recv() failed (104: Connection reset by peer) while reading response header from upstream'; I don't know which line of code break the connection during the nginx action called 'reading response header'.

One thing different with online environment is that nginx and tomcat will both use keepalive connection, because I stress only in my localhost; And I close nginx's keepalive config, but no difference. Thank u all