Fetching nodes from external cluster failed - cross cluster replication on docker

Question

I run two docker-compose file to run two instances of elastic search and kibana and tried to create corss cluster replication with it.

Let's say the two instances are es1 and es2.

es1 docker-compose file

elasticsearch:
image: docker.elastic.co/elasticsearch/elasticsearch:8.1.2
env_file:
  - elasticsearch.env
ports:
  - 9200:9200
  - 9400:9400   
volumes:
  - ./data/elasticsearch:/usr/share/elasticsearch/data
  - ./elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml
  - ./elastic-certificates.p12:/usr/share/elasticsearch/config/elastic-certificates.p12
  - ./http.p12:/usr/share/elasticsearch/config/http.p12
environment:
  - TZ=America/Los_Angeles

es1 elasticsearch.yml file

cluster.name: "docker-cluster"
network.host: 0.0.0.0
transport.port: 9400
bootstrap.memory_lock=true
discovery.type=single-node

xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate 
xpack.security.transport.ssl.client_authentication: required
xpack.security.transport.ssl.keystore.path: http.p12
xpack.security.transport.ssl.truststore.path: elastic-certificates.p12

es2 docker-compose file

elasticsearch:
image: docker.elastic.co/elasticsearch/elasticsearch:8.1.2
env_file:
  - elasticsearch.env
ports:
  - 9300:9200
  - 9401:9300 
volumes:
  - ./data/elasticsearch:/usr/share/elasticsearch/data
  - ./elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml
  - ./elastic-certificates.p12:/usr/share/elasticsearch/config/elastic-certificates.p12
  - ./http.p12:/usr/share/elasticsearch/config/http.p12
environment:
  - TZ=America/Los_Angeles

es2 elasticsearch.yml

cluster.name: "docker-cluster"
network.host: 0.0.0.0
transport.port: 9300
bootstrap.memory_lock=true
discovery.type=single-node

xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate 
xpack.security.transport.ssl.client_authentication: required
xpack.security.transport.ssl.keystore.path: http.p12
xpack.security.transport.ssl.truststore.path: elastic-certificates.p12

When I tried to add es2 as a remote cluster to es1. I put remote cluster name as "remote" and put seed nodes as localhost:9401. As es2 transport port is exposed on 9401 port.

adding remote cluster on es1

It got failed and

Error Log:

elasticsearch_1  | {"@timestamp":"2022-05-13T13:44:29.183Z", "log.level": "WARN", "message":"fetching nodes from external cluster [remote] failed", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[1fdffc213272][generic][T#4]","log.logger":"org.elasticsearch.transport.SniffConnectionStrategy","elasticsearch.cluster.uuid":"3IIajPCkTKqorfTdpjrT5w","elasticsearch.node.id":"4QpQH_HQQZuQ7mGMq1sMMQ","elasticsearch.node.name":"1fdffc213272","elasticsearch.cluster.name":"my-awesome-elasticsearch-cluster-demo","error.type":"org.elasticsearch.transport.ConnectTransportException","error.message":"[][127.0.0.1:9401] connect_exception","error.stack_trace":"org.elasticsearch.transport.ConnectTransportException: [][127.0.0.1:9401] connect_exception\n\tat org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1107)\n\tat org.elasticsearch.action.ActionListener.lambda$toBiConsumer$0(ActionListener.java:279)\n\tat org.elasticsearch.core.CompletableContext.lambda$addListener$0(CompletableContext.java:31)\n\tat java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863)\n\tat java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841)\n\tat java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)\n\tat java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162)\n\tat org.elasticsearch.core.CompletableContext.completeExceptionally(CompletableContext.java:46)\n\tat org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$addListener$0(Netty4TcpChannel.java:63)\n\tat io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:578)\n\tat io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:571)\n\tat io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:550)\n\tat io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:491)\n\tat io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:616)\n\tat io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:609)\n\tat io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:117)\n\tat io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:321)\n\tat io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:337)\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:710)\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:623)\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:586)\n\tat io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496)\n\tat io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)\n\tat io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)\n\tat java.base/java.lang.Thread.run(Thread.java:833)\nCaused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: localhost/127.0.0.1:9401\nCaused by: java.net.ConnectException: Connection refused\n\tat java.base/sun.nio.ch.Net.pollConnect(Native Method)\n\tat java.base/sun.nio.ch.Net.pollConnectNow(Net.java:672)\n\tat java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:946)\n\tat io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:330)\n\tat io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334)\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:710)\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:623)\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:586)\n\tat io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496)\n\tat io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)\n\tat io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)\n\tat java.base/java.lang.Thread.run(Thread.java:833)\n"}

But, if I tried to connect es1 itself as a remote cluster with it's own cluster. It got connected! Like this time I put seed nodes as localhost:9400 which is the transport port of es1 itself. So I think tranport port is exposed and working. But why it is not working for other remote cluster? Did I miss anything?

I also tried same thing with es2 instance.Yet no luck, same result. es2 can't add es1 as remote cluster but can add itself as remote cluster perfectly.