JGroups FD_SOCK configuration problems

Question

I am trying to set up a cluster with JGroups using the FD_SOCK protocol but it doesn't work as I expect.

I have a JGroups configuration with the following FD_SOCK config:

<FD_SOCK start_port="33287" client_bind_port="33288" port_range="2"/>

When there are two nodes in the cluster, the TCP connections are correctly created. Let's say we have node A and B. The connections created are as follows:

• For client A to B:

  1. A:33288 -> B:33287
  2. A:33288 <- B:33287

• For client B to A:

  1. B:33288 -> A:33287
  2. B:33288 -> A:33287

However, when I try to add the third node C, the node second node B says that it can't create the client sockets to A:33287 and C:33287 because they are already in use.

Now, node C indeed created some TCP connections to node A using port 33287, but I would have expected that node B would increment the start_port="33287" to 33289 because of the configuration port_range="2".

What am I missing here?

Thanks in advance for the help.

Answer 1

Are all 3 nodes on different hosts? Port_range means that a port is skipped if already taken.

Answer 2

As described at Jgroups documentation, each server in cluster checks only its heighbor at the right side (according server list in configuration file). So, when the third server C is added, server B stops checking server A. This is right behaviour only if config file is correct. Anyway, changing config file needs all servers to be restarted.

Failure detection ports (FD_SOCK) are actual only at the moment of clean server start. In that case all ports should free. I suppose the first problem is too many server restarts without proper edit of configuration files on all servers.

On the other hand, depending on application coded properly and heavy load, ports may be changed very quickly. When application has a lot of ports in time-wait state, after restart fd_sock ports will be chosen randomly. Probably for that reason OP caught a weird message that port was already used.