How to prevent "Given transaction number 1 does not match any in-progress transactions" with Mongoose Transactions?

Question

I am using Mongoose to access to my database. I need to use transactions to make an atomic insert-update. 95% of the time my transaction works fine, but 5% of the time an error is showing :

"Given transaction number 1 does not match any in-progress transactions"

It's very difficult to reproduce this error, so I really want to understand where it is coming from to get rid of it. I could not find a very clear explanation about this type of behaviour.

I have tried to use async/await key words on various functions. I don't know if an operation is not done in time or too soon.

Here the code I am using:

export const createMany = async function (req, res, next) {
  if (!isIterable(req.body)) {
    res.status(400).send('Wrong format of body')
    return
  }
  if (req.body.length === 0) {
    res.status(400).send('The body is well formed (an array) but empty')
    return
  }

  const session = await mongoose.startSession()
  session.startTransaction()
  try {
    const packageBundle = await Package.create(req.body, { session })
    const options = []
    for (const key in packageBundle) {
      if (Object.prototype.hasOwnProperty.call(packageBundle, key)) {
        options.push({
          updateOne: {
            filter: { _id: packageBundle[key].id },
            update: {
              $set: {
                custom_id_string: 'CAB' + packageBundle[key].custom_id.toLocaleString('en-US', {
                  minimumIntegerDigits: 14,
                  useGrouping: false
                })
              },
              upsert: true
            }
          }
        })
      }
    }
    await Package.bulkWrite(
      options,
      { session }
    )
    for (const key in packageBundle) {
      if (Object.prototype.hasOwnProperty.call(packageBundle, key)) {
        packageBundle[key].custom_id_string = 'CAB' + packageBundle[key].custom_id.toLocaleString('en-US', {
          minimumIntegerDigits: 14,
          useGrouping: false
        })
      }
    }
    res.status(201).json(packageBundle)
    await session.commitTransaction()
  } catch (error) {
    res.status(500).end()
    await session.abortTransaction()
    throw error
  } finally {
    session.endSession()
  }
}

I expect my code to add in the database and to update the entry packages in atomic way, that there is no instable database status. This is working perfectly for the main part, but I need to be sure that this bug is not showing anymore.

Answer 1

The accepted answer is great. In my case, I was running multiple transactions serially within a session. I was still facing this issue every now and then. I wrote a small helper to resolve this.

File 1:

// do some work here
await session.withTransaction(() => {});

// ensure the earlier transaction is completed
await ensureTransactionCompletion(session);

// do some more work here
await session.withTransaction(() => {});

Utils File:

async ensureTransactionCompletion(session: ClientSession, maxRetryCount: number = 50) {
        // When we are trying to split our operations into multiple transactions
        // Sometimes we are getting an error that the earlier transaction is still in progress
        // To avoid that, we ensure the earlier transaction has finished
        let count = 0;
        while (session.inTransaction()) {
            if (count >= maxRetryCount) {
                break;
            }
            // Adding a delay so that the transaction get be committed
            await new Promise(r => setTimeout(r, 100));
            count++;
        }
    }