'Nifi flowfiles stucked in connection while actively balancing

I have a 3 nodes cluster(as you can see on attached images) with a connection with a Load Balance Strategy set to Partition by attribute, sometimes some flowfiles get stuck in this connection (see image1). Furthermore we have this log error in bulletin board

{{10:56:28 UTC ERROR xxxx-xxx-xx-nifi0.xxxx:8443 }}

10:56:28 UTC ERROR xxxx-xxx-xx-nifi0.xxxx:8443

Failed to communicate with Peer xxxx-xxx-xx-nifi2.xxxx:8443 when load balancing data for Connection with ID c12f3c02-9f05-1232-ffff-fffff8d925a9 due to java.io.IOException: Failed to decrypt data from Peer xxxx-xxx-xx-nifi2.xxxx:8443 because Peer unexpectedly closed connection{}

The weird thing is that only some flowfiles are stucked other flowfiles goes forward in the flows. If I restart nifi service, the problem fix automatically, but it's a very annoying problem.

I think this bug is the same of https://issues.apache.org/jira/browse/NIFI-9574 Here are the flowfile stuck in 'actively balancing' active nifi thread Nifi_version

In addition i have also this log:

2022-02-23 11:55:04,939 ERROR org.apache.nifi.controller.queue.clustered.client.async.nio.NioAsyncLoadBalanceClient: Failed to communicate with Peer nifi1:8443
2022-02-23 11:55:04,940 ERROR org.apache.nifi.controller.queue.SwappablePriorityQueue: Updated Size of Queue Unacknowledged from FlowFile Queue Size[ ActiveQueue=[2, 125681 Bytes], Swap Queue=[0, 0 Bytes], Swap Files=[0], Unacknowledged=[-2, -125681 Bytes] ] to FlowFile Queue Size[ ActiveQueue=[2, 125681 Bytes], Swap Queue=[0, 0 Bytes], Swap Files=[0], Unacknowledged=[-2, -125681 Bytes] ]
2022-02-23 11:55:04,959 ERROR org.apache.nifi.controller.queue.clustered.client.async.nio.NioAsyncLoadBalanceClient: Failed to communicate with Peer nifi1:8443

and

Failed to communicate with Peer nifi0:8443 when load balancing data for Connection with ID b1dd36e2-6aa3-10ce-ac02-a77bf0374d96 due to java.io.IOException: 
Failed to negotiate Protocol Version with Peer nifi0:8443. Recommended version 1 but instead of an ACCEPT or REJECT response got back a response of 101


Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source