How to get fullDocument from MongoDB changeStream when a document is deleted?

Question

My Code

I have a MongoDB with two collections, Items and Calculations.

Items
  value:  Number
  date:   Date

Calculations
  calculation:  Number
  start_date:   Date
  end_date:     Date

A Calculation is a stored calcluation based off of Item values for all Items in the DB which have dates in between the Calculation's start date and end date.

Mongo Change Streams

I figure a good way to create / update Calculations is to create a Mongo Change Stream on the Items collection which listens for changes to the Items collection to then recalculate relevant Calculations.

The issue is that according to the Mongo Change Event docs, when a document is deleted, the fullDocument field is omitted which would prevent me from accessing the deleted Item's date which would inform which Calculations should be updated.

Question

Is there any way to access the fullDocument of a Mongo Change Event that was fired due to a document deletion?

Answer 1

fullDocument is not returned when a document is deleted.

But there is a workaround.

Right before you delete the document, set a hint field. (Obviously use a name that does not collide with your current properties.)

await myCollection.updateOne({_id:theId}, {_deleting: true})
await myCollection.deleteOne({_id:theId})

This will trigger one final change event in the stream, with a hint that the document is getting deleted. Then in your stream watcher, you simple check for this value.

stream.on('change', event => {
  if (!event.fullDocument) {
    // The document was deleted
  }
  else if (event.fullDocument._deleting) {
    // The document is going to be deleted
  }
  else {
   // Document created or updated
  }

})

My oplog was not getting updated fast enough, and the update was looking up a document that was already removed, so I needed to add a small delay to get this working.

myCollection.updateOne({_id:theId}, {_deleting: true})
setTimeout( ()=> {
  myCollection.deleteOne({_id:theId})
}, 100)

If the document did not exist in the first place, it won't be updated or deleted, so nothing gets triggered.

Using TTL Indexes

Another way to make this work is to add a ttl index, and then set that field to the current time. This will trigger an update first, and then delete the document automatically.

myCollection.setIndex({_deleting:1}, {expireAfterSeconds:0})

myCollection.updateOne({_id:theId}, {$set:{_deleting:new Date()}})

The problem with this approach is that mongo prunes TTL documents only during certain intervals, 60s or more as stated in the docs, so I prefer to use the first approach.