QuickFixJ SSL socket failing on Java 17

Question

I support a fix application which acts as an acceptor to a client using the QuickFixJ built-in support for SSL Sockets. When using Java 8, it all works fine and as expected.

Recently, we upgraded our application JDK to Java 17 - nothing else has changed. The client has two sessions - the first session connects fine and I can see the login exchange, heartbeats and application messages going as expected. However, when the second session attempts to connect, the first fix session disconnects with the following logged:

2022-04-21 11:18:44,646 INFO [event.xxx-yyy] - Disconnecting: Encountered END_OF_STREAM

The reconnect logic proceeds, the first session logs in again, and again when the second session attempts to connect, the first session disconnects.

After a few of these iterations, the following is logged:

    javax.net.ssl.SSLException: Improper close state: Status = OK 
            HandshakeStatus = NEED_WRAP bytesConsumed = 0 bytesProduced = 7 sequenceNumber = 0
    at org.apache.mina.filter.ssl.SslHandler.closeOutbound(SslHandler.java:496) ~[mina-core-2.1.6.jar:?]

From googling that exception, I got the impression this has something to do with TLS1.3 support in the JDK since Java 11. I have tried setting pretty much all combinations of the following settings on the application, all without any impact:

-Djdk.tls.acknowledgeCloseNotify=true 
-Djdk.tls.disabledAlgorithms=TLSv1.3 
-Dhttps.protocols=TLSv1.2 
-Djdk.tls.server.protocols=TLSv1.2 
-Djdk.tls.client.protocols=TLSv1.2

Here is the relevant portion of my QuickFix settings file:

[default]
ConnectionType=acceptor

# SSL properties
SocketConnectProtocol=SOCKET
SocketUseSSL=Y
CipherSuites=TLS_RSA_WITH_AES_256_CBC_SHA
EnabledProtocols=TLSv1.2
KeyStoreType=JKS
SocketKeyStore=data/keystore/FIX_keystore.jks
SocketKeyStorePassword=****
SocketTrustStore=data/keystore/FIX_truststore.jks
SocketTrustStorePassword=****
NeedClientAuth=Y

[session]
SocketAcceptPort=4715
SenderCompID=xxx.yyy
TargetCompID=yyy.xxx

[session]
SocketAcceptPort=4716
SenderCompID=xxx2.yyy2
TargetCompID=yyy2.xxx2

I enabled SSL logging on the jvm, and the following was logged around the time of the failure:

javax.net.ssl|DEBUG|A3|NioProcessor-3|2022-04-20 09:37:58.596 BST|ChangeCipherSpec.java:115|Produced ChangeCipherSpec message
javax.net.ssl|DEBUG|A3|NioProcessor-3|2022-04-20 09:37:58.597 BST|Finished.java:459|Produced server Finished handshake message (
"Finished": {
  "verify data": {
    0000: 16 51 52 27 65 DE B9 C8   DB EC FD A9
  }'}
)
javax.net.ssl|ALL|93|NioProcessor-7|2022-04-20 09:38:34.479 BST|SSLEngineImpl.java:825|Closing outbound of SSLEngine
javax.net.ssl|WARNING|93|NioProcessor-7|2022-04-20 09:38:34.479 BST|SSLEngineOutputRecord.java:173|outbound has closed, ignore outbound application data
javax.net.ssl|ALL|93|NioProcessor-7|2022-04-20 09:38:34.480 BST|SSLEngineImpl.java:786|Closing inbound of SSLEngine
javax.net.ssl|ERROR|93|NioProcessor-7|2022-04-20 09:38:34.480 BST|TransportContext.java:362|Fatal (INTERNAL_ERROR): closing inbound before receiving peer's close_notify (
"throwable" : {
  javax.net.ssl.SSLException: closing inbound before receiving peer's close_notify
        at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:133)
        at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:117)
        at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:357)
        at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:313)
        at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:304)
        at java.base/sun.security.ssl.SSLEngineImpl.closeInbound(SSLEngineImpl.java:796)
        at org.apache.mina.filter.ssl.SslHandler.destroy(SslHandler.java:208)
        at org.apache.mina.filter.ssl.SslFilter.sessionClosed(SslFilter.java:486)
        at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextSessionClosed(DefaultIoFilterChain.java:606)
        at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$900(DefaultIoFilterChain.java:49)
        at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.sessionClosed(DefaultIoFilterChain.java:1092)
        at org.apache.mina.core.filterchain.IoFilterAdapter.sessionClosed(IoFilterAdapter.java:98)
        at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextSessionClosed(DefaultIoFilterChain.java:606)
        at org.apache.mina.core.filterchain.DefaultIoFilterChain.fireSessionClosed(DefaultIoFilterChain.java:599)
        at org.apache.mina.core.service.IoServiceListenerSupport.fireSessionDestroyed(IoServiceListenerSupport.java:251)
        at org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.removeNow(AbstractPollingIoProcessor.java:1144)
        at org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.removeSessions(AbstractPollingIoProcessor.java:864)
        at org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.run(AbstractPollingIoProcessor.java:694)
        at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:833)}

As mentioned, all these work fine in Java 8, so I highly doubt the issue is related to a wrong client certificate or keystore password. This also only happens when the external business client connects - when I connect using our in-house test FIX client, all work as expected on Java 17.

Any ideas on what could be causing this? Is there a way to work around it?