How can AWS Kinesis Firehose lambda send update and delete requests to ElasticSearch?

Question

I'm not seeing how an AWS Kinesis Firehose lambda can send update and delete requests to ElasticSearch (AWS OpenSearch service).

Elasticsearch document APIs provides for CRUD operations: https://www.elastic.co/guide/en/elasticsearch/reference/current/docs.html

The examples I've found deals with the Create case, but doesn't show how to do delete or update requests. https://aws.amazon.com/blogs/big-data/ingest-streaming-data-into-amazon-elasticsearch-service-within-the-privacy-of-your-vpc-with-amazon-kinesis-data-firehose/ https://github.com/amazon-archives/serverless-app-examples/blob/master/python/kinesis-firehose-process-record-python/lambda_function.py

The output format in the examples do not show a way to specify create, update or delete requests:

       output_record = {
           'recordId': record['recordId'],
           'result': 'Ok',
           'data': base64.b64encode(payload)
       }

Apart from the examples, I'm not finding the definition of the output format for what the kinesis firehose lambda handler should return.

Answer 1

Firehose can only do inserts into ES. It cannot do updates and deletes. If you want to perform updates and deletes, you will need to write a lambda that is trgerred from events on the Kinesis stream

https://docs.aws.amazon.com/lambda/latest/dg/with-kinesis.html