'Commit kafka offset in Aws Lambda Function .Net Core

I'm trying to employ AWS Lambda as a consumer for Kafka topic. But I need to be able to commit offset to the topic manually, from out of the box seems it has auto-comit = true. I'm using the following structure:

        public async Task ConsumeAsync(KafkaEvent kafkaEvent, ILambdaContext context)
    {

    }

But none of the parameters provide a way to commit offset by myself. I believe I could achieve it via Confluent.Kafka library which requires GroupId to be set before commit anything in the form:

var config = new ConsumerConfig
{
    BootstrapServers = "host1:9092,host2:9092",
    GroupId = "foo",
    AutoOffsetReset = AutoOffsetReset.Earliest
};

But the problem is that when I receive event in Lambda it has the following content. It contains all but not GroupId...

    {
   "eventSource":"SelfManagedKafka",
   "bootstrapServers":"b-1.us-west-2.amazonaws.com:9092,b-1.us-west-2.amazonaws.com:9092,b-1.us-west-2.amazonaws.com:9092",
   "records":{
      "p.Vehicles-1":[
         {
            "topic":"p.Vehicles",
            "partition":2,
            "offset":151084,
            "timestamp":1652981862674,
            "timestampType":"CREATE_TIME",
            "key":"eyJpZCI6IjkxMDBGRkZGLTQ4qwktQ0I2My02Mzc2LTIyQ0E3MTJCMDAwMCJ9",
            "value":"eyJpZCQ0NkIwRDYwMDAwIiwiTWFrZSI6IjEwMiBJUk9OV09SS1MsIElOQy4iLCJNb2RlbCI6IjEwMiBJUk9OV09SS1MsIElOQy4iLCJZZWFyIjoyMDE5LCJWSU4iOiI1VkNBQ1JMVDFHSDIyMDMxNCIsIkVsZEVuYWJsZWQiOmZhbHNlLCJFQ01WSU4iOm51bGwsIkZ1ZWxUeXBlSWQiOm51bGwsIl9fb3AiOiJjIiwiX19zb3VyY2VfdHNfbXMiOjE2NTI5ODE4NDkyMzMsIl9fZGVsZXRlZCI6ImZhbHNlIiwiU3RhY2tJZCI6IjQifQ\u003d\u003d",
            "headers":[
               
            ]
         }
      ]
   }
}

Where can I get this GroupId.

c++sharp.net.net-coreapache-kafkaaws-lambdaamazon-cognitoaws-identitypools


Solution 1:[1]

https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/CognitoIdentityCredentials.html

According to this it is doing the same thing in the background.The difference in syntax for SecretKey and SecretAccesKey appears to be purely cosmetic. I have noticed the same in boto3. Regardless, you can pass either with accesskey and sessionid to authenticate other AWS services.

Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source
Solution 1 Ninad Gaikwad

Related Questions