'Tons of “Attempting to kill operation” messages in log

I am running a Mongo (v4.4.3) sharded setup. I have two shards as PSA replica. Shard1 contains most data and is usually very busy. I noticed that there were tons of “Attempting to kill operation” in the logs. It resulted in 20-30GB of log file a day. I am pretty sure we don’t issue any opKill commands. We do a lot of upserts though. The messages would go away if we shut down the secondary. As soon as the secondary syncs up, the messages show up again. The messages are printing a few hundred times per second. It seems to use a lot more CPUs than usual.

Is it an issue? How can we turn off the messages?

One of the examples:

{
  "t": {
    "$date": "2021-02-07T10:44:39.035-05:00"
  },
  "s": "I",
  "c": "COMMAND",
  "id": 4615602,
  "ctx": "conn59830",
  "msg": "Attempting to kill operation",
  "attr": {
    "operationKey": {
      "uuid": {
        "$uuid": "3bc054fe-e267-4290-8572-7e4052399011"
      }
    }
  }
}
mongodbdatabase-administrationsharding


Solution 1:[1]

The reason behind these messages is usually read hedging -- a method which mongos instance uses to increase performance that involves secondary reads to be "forked" across all members of the replica set, using the fastest result and cancelling the others. It was introduced in MongoDB 4.4. Read hedging can be disabled in your driver (check out read preference configuration for your specific driver) or globally using the readHedgingMode parameter in mongos configuration. I must warn you, that in theory, disabling read hedging could lower performance, to the level of pre-4.4 MongoDB, but in our testing we found read hedging produces negligible performance boost.

Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source
Solution 1 onyXMaster

Related Questions