MongoDB: deleted replica's still present in rs.conf() & rs.status()

Question

:-)

I am working with MongoDB (I'm new to it) and wanted to test out the replica set feature of when the primary is deleted that one of the secondaries will become a primary.

I have 3 replicas on 3 different LXD containers:

mongodb-0      10.126.63.54       <---- primary
mongodb-1      10.126.63.190
mongodb-2      10.126.63.20

I then want to simulate an unexpected shutdown of the primary so I delete the LXD container that mongodb-0 is hosted on.

After doing this I ssh into mongodb-1 and run mongosh and see that it has been elected to primary. Similarly, when I shh into mongodb-2 and run mongosh I see it has remained its status as secondary.

Yay voting a new primary works! \o/

But when I run:

1. rs.conf() I see this entry for the shutdown host:

    {
      _id: 2,
      host: '10.126.63.54:27017',
      arbiterOnly: false,
      buildIndexes: true,
      hidden: false,
      priority: 1,
      tags: {},
      secondaryDelaySecs: Long("0"),
    }

2. rs.status() I see this entry for the shutdown host:

    {
      _id: 2,
      name: '10.126.63.54:27017',
      ...
      lastHeartbeatMessage: 'Error connecting to 10.126.63.54:27017 :: caused by :: No route to host',
      ...
    }

1. Is this expected?

2. After deleting this node hours ago, this is still listed. Will they always list the removed hosts? The message from rs.status() is helpful, but I would expect rs.conf() to have the current available hosts. (maybe this is a poor expectation on my part.)

3. THE BIG QUESTION: In a production environment should I be updating something so that rs.conf() and rs.status() reflect all available hosts? Or is this information potentially helpful/important. My intuition tells me that this could potentially be helpful. But also tells me I cannot rely on rs.conf() to show me all the currently running hosts : /

Answer 1

1. Yes this is expected.
2. If you need to remove the member from the configuration you need to remove it from the replicaSet active PRIMARY with rs.remove()
3. Yes , this information is replicated to all members and need to be removed so all members to be aware. Same like when you add new member with rs.add()

This is so because you do not expect member to be removed from the configuration immediately after network interruption or server shutdown , in case member is started back he will join again automatically to the replicaSet and sync to the other members.