MongoDB replicaset external access - keep getting internal cluster names

Question

I must be doing something terribly wrong. I have a replicaset configured using the MongoDB community operator, deployed in GKE, and exposed via LoadBalancers.

This replicaset has 3 members. I have defined the replicaSetHorizons like so:

  replicaSetHorizons:
  - mongo-replica: document-0.mydomain.com:30000
  - mongo-replica: document-1.mydomain.com:30001
  - mongo-replica: document-2.mydomain.com:30002

I then use mongosh from an external source (local computer outside of GKE) to connect:

mongosh "mongodb://<credentials>@document-0.mydomain.com:30000,document-1.mydomain.com:30001,document-2.mydomain.com:30002/admin?ssl=false&replicaSet=document"

I do not use SSL for now because I am testing this deployment. What I found is mongosh always returns this error:

MongoNetworkError: getaddrinfo ENOTFOUND document-0.document-svc.mongodb.svc.cluster.local

Can someone explain to me what I am doing wrong? Why is my internal clustername being given to mongosh to attempt the connection?

If I try to connect to a single member of the replicaset, the connection will succeed. If I run rs.conf(), I see the following (which looks correct??):

{
  _id: 'document',
  version: 1,
  term: 1,
  members: [
    {
      _id: 0,
      host: 'document-0.document-svc.mongodb.svc.cluster.local:27017',
      arbiterOnly: false,
      buildIndexes: true,
      hidden: false,
      priority: 1,
      tags: {},
      horizons: { 'mongo-replica': 'document-0.mydomain.com:30000' },
      secondaryDelaySecs: Long("0"),
      votes: 1
    },
    {
      _id: 1,
      host: 'document-1.document-svc.mongodb.svc.cluster.local:27017',
      arbiterOnly: false,
      buildIndexes: true,
      hidden: false,
      priority: 1,
      tags: {},
      horizons: { 'mongo-replica': 'document-1.mydomain.com:30001' },
      secondaryDelaySecs: Long("0"),
      votes: 1
    },
    {
      _id: 2,
      host: 'document-2.document-svc.mongodb.svc.cluster.local:27017',
      arbiterOnly: false,
      buildIndexes: true,
      hidden: false,
      priority: 1,
      tags: {},
      horizons: { 'mongo-replica': 'document-2.mydomain.com:30002' },
      secondaryDelaySecs: Long("0"),
      votes: 1
    }
  ],
  protocolVersion: Long("1"),
  writeConcernMajorityJournalDefault: true,
  settings: {
    chainingAllowed: true,
    heartbeatIntervalMillis: 2000,
    heartbeatTimeoutSecs: 10,
    electionTimeoutMillis: 10000,
    catchUpTimeoutMillis: -1,
    catchUpTakeoverDelayMillis: 30000,
    getLastErrorModes: {},
    getLastErrorDefaults: { w: 1, wtimeout: 0 },
    replicaSetId: ObjectId("62209784e8aacd8385db1609")
  }
}

Answer 1

ReplicaSetHorizons feature does not work without using SSL/TLS certificates. Quoting from Kubernetes Operator reference:

This method to use split horizons requires the Server Name Indication extension of the TLS protocol

In order to make this work, you need to include

• TLS certificate
• TLS key
• CA key

TLS Certificate must contain DNS names of all your replica sets in Subject Alternative Name (SAN) section

There is a tutorial at operator github pages. You need to complete all steps, certificate issuance cannot be skipped.

Certificate resource (using cert-manager.io CRD)

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: cert-manager-certificate
spec:
  secretName: mongodb-tls
  issuerRef:
    name: ca-issuer
    kind: Issuer
  duration: 87600h
  commonName: "*.document-svc.mongodb.svc.cluster.local"
  dnsNames:
    - "*.document-svc.mongodb.svc.cluster.local"
    - "document-0.mydomain.com"
    - "document-1.mydomain.com"
    - "document-2.mydomain.com"

MongoDBCommunity resource excerpt

spec:
  type: ReplicaSet
  ...
  replicaSetHorizons:
    - mongo-replica: document-0.mydomain.com:30000
    - mongo-replica: document-0.mydomain.com:30001
    - mongo-replica: document-0.mydomain.com:30002
  security:
    tls:
      enabled: true
      certificateKeySecretRef:
        name: mongodb-tls
      caConfigMapRef:
        name: ca-config-map

Secret mongodb-tls will by of type tls and contain ca.crt, tls.crt and tls.key fields representing Certificate Authority certificate, TLS certificate and TLS key respectively.
ConfigMap ca-config-map will contain ca.crt field only

More info at: mongodb-operator-secure-tls