SignalR negotiation sometimes fails on the first attempt

Question

We have a SignalR javascript client connecting to .net core hub, hosted in AWS. Both client and server use version 6.

More than one backend server may exist, so there is an internet facing Network Load Balancer forwarding the traffic to the backend servers. The NLB is configured with these options:

• Stickiness
• Preserve client IP addresses

Most of the time, everything works great: the negotiation and the connection upgrade. Sometimes, however, something strange happens: the negotiation fails (error in WebSocket transport), then the client tries again with another transport (SSE). This transport also fails and, while retrying, the client hits the other host, starting the negotiation again. Finally, the connection succeeds. All this process takes no more than 5 seconds.

This was happening in our clients, from outside, so we had set up an isolated environment to debug this situation, with the NLB and 2 backend hosts. This is internal, so no one else is connecting to it, for sure, so there is no chance hosts are overloaded. We are completely sure our IP does not change while the test is being done. We enabled client and server debug logs, shown below.

This still happens sometimes, no matter the host that we hit on the first attempt. We know that we can configure the client to skip the negotiation, but that will make us lose about 10% of our clients, because we will be limited to use the WebSockets transport.

Is this something related to the AWS NLB or the signalR protocol/libs/versions?

How can the negotiation fail if just one client is connecting and the IP does not change?

What is misconfigured in our setup?

Thanks in advance!

(logs below)

When the connection succeeds at the first attempt

Client logs
Debug: Starting connection with transfer format 'Text'.
Debug: Sending negotiation request: https://<server>...
Debug: Selecting transport 'WebSockets'.
Trace: (WebSockets transport) Connecting.
Information: WebSocket connected to wss://<server>...
Debug: The HttpConnection connected successfully.
Trace: (WebSockets transport) sending data. String data of length 32.
Trace: (WebSockets transport) data received. String data of length 3.

Server logs
[DBG] New connection r3Gv5PrBgTA2T6lijqwTrA created.
[DBG] Sending negotiation response.
[DBG] Establishing new connection.
[DBG] Socket opened using Sub-Protocol: 'null'.
[DBG] OnConnectedAsync started.
[DBG] Found protocol implementation for requested protocol: json.
[DBG] Completed connection handshake. Using HubProtocol 'json'.
[DBG] OnConnectedAsync() started for connection r3Gv5PrBgTA2T6lijqwTrA

The negotiation response sent by the server is:
  availableTransports:
    0: {transport: "WebSockets", transferFormats: ["Text", "Binary"]}
    1: {transport: "ServerSentEvents", transferFormats: ["Text"]}
    2: {transport: "LongPolling", transferFormats: ["Text", "Binary"]}
  connectionId: "r3Gv5PrBgTA2T6lijqwTrA"
  connectionToken: "tuiM4vqGsh0F4mV6mURN1w"
  negotiateVersion: 1

When the connection fails at the first attempt

Client logs
Debug: Starting connection with transfer format 'Text'.
Debug: Sending negotiation request: https://<server>...
Debug: Selecting transport 'WebSockets'.
Trace: (WebSockets transport) Connecting.
WebSocket connection to 'wss://<server>...' failed:
Information: (WebSockets transport) There was an error with the transport.
Error: Failed to start the transport 'WebSockets': Error: WebSocket failed to connect. The connection could not be found on the server, either the endpoint may not be a SignalR endpoint, the connection ID is not present on the server, or there is a proxy blocking WebSockets. If you have multiple servers check that sticky sessions are enabled.
Debug: Selecting transport 'ServerSentEvents'.
Debug: Sending negotiation request: https://<server>...
Trace: (SSE transport) Connecting.
Information: SSE connected to https://<server>...
Debug: The HttpConnection connected successfully.
Trace: (SSE transport) sending data. String data of length 32.
POST https://<server>... 404 (Not Found)
Debug: HttpConnection.stopConnection(undefined) called while in state Disconnecting.
Error: Connection disconnected with error 'Error: No Connection with that ID: Status code '404''.
Debug: Starting connection with transfer format 'Text'.
Debug: Sending negotiation request: https://<server>...
Debug: Selecting transport 'WebSockets'.
Trace: (WebSockets transport) Connecting.
Information: WebSocket connected to wss://<server>...
Debug: The HttpConnection connected successfully.
Trace: (WebSockets transport) sending data. String data of length 32.
Trace: (WebSockets transport) data received. String data of length 3.

Server logs
(server 1)
[DBG] New connection _cm5IaOtqY7tD7suKOb08Q created.
[DBG] Sending negotiation response. (1)
[DBG] New connection GuXhVydEzL-8xXcSxibysA created.
[DBG] Sending negotiation response. (2)
[DBG] Establishing new connection.
[DBG] OnConnectedAsync started.
[DBG] Failed connection handshake.
[DBG] Removing connection 4i5fzsR6wVO1Kf06iMbBuw from the list of connections.
[DBG] Removing connection vO9zLldnZO4wfh18OkdKJA from the list of connections.

The negotiation response (1) sent by the server is:
availableTransports:
  0: {transport: "WebSockets", transferFormats: ["Text", "Binary"]}
  1: {transport: "ServerSentEvents", transferFormats: ["Text"]}
  2: {transport: "LongPolling", transferFormats: ["Text", "Binary"]}
connectionId: "_cm5IaOtqY7tD7suKOb08Q"
connectionToken: "vO9zLldnZO4wfh18OkdKJA"
negotiateVersion: 1

The negotiation response (2) sent by the server is:
availableTransports:
  0: {transport: "WebSockets", transferFormats: ["Text", "Binary"]}
  1: {transport: "ServerSentEvents", transferFormats: ["Text"]}
  2: {transport: "LongPolling", transferFormats: ["Text", "Binary"]}
connectionId: "GuXhVydEzL-8xXcSxibysA"
connectionToken: "4i5fzsR6wVO1Kf06iMbBuw"
negotiateVersion: 1

(server 2)
[DBG] New connection RjoZW-BKBNMOa2UBW9yo-g created.
[DBG] Sending negotiation response.
[DBG] Establishing new connection.
[DBG] Socket opened using Sub-Protocol: 'null'.
[DBG] OnConnectedAsync started.
[DBG] Found protocol implementation for requested protocol: json.
[DBG] Completed connection handshake. Using HubProtocol 'json'.
[DBG] OnConnectedAsync() started for connection RjoZW-BKBNMOa2UBW9yo-g

The negotiation response sent by the server is:
availableTransports:
  0: {transport: "WebSockets", transferFormats: ["Text", "Binary"]}
  1: {transport: "ServerSentEvents", transferFormats: ["Text"]}
  2: {transport: "LongPolling", transferFormats: ["Text", "Binary"]}
connectionId: "RjoZW-BKBNMOa2UBW9yo-g"
connectionToken: "u4aitwut14QelHtOeBprNg"
negotiateVersion: 1