Is it possible to publish message to Kafka topics within Async calls like Future?

Question

I want to call a method with send a kafka message to a topic inside a Future based on Failure scenario without using blocking call to maintain scalability of my application.

By using Await my Future will be blocked thus using Future will lose its meaning for Async operations. Can kafka Stream message be published inside Futures ?

Tried Await it worked , but inside Future Kafka Stream Context was passing as null

val actorTry: Future[ActorRef] =
  actorSystem.actorSelection(actorAddress).resolveOne()

actorTry.onComplete{
  case Success(value) =>
    value !! response
  case Failure(ex) =>
    context.forward(id, message,To.child(Sink))
}

Answer 1

The official Confluent java client(producer) works in an asynchronous fashion. Take a look to the example in the documentation:

def produce(producer: KafkaProducer[Bytes, Book], topic: String, book: Book): Future[RecordMetadata] = { 
    val record: ProducerRecord[Bytes, Book] = new ProducerRecord(topic, book) 

    producer.send(record, new Callback { 
      override def onCompletion(metadata: RecordMetadata, exception: Exception): Unit = Option(exception) 
        .map(ex => logger error s"fail to produce record due to: ${ex.getMessage}")
        .getOrElse(logger info s"successfully produced - ${printMetaData(metadata)}")
    })
}

So, every time you produce a record a new Java Future is created. If you need to convert the Java Future to Scala, since Scala 2.13 you can:

import scala.jdk.FutureConverters._


val scalaFuture = produceResultFuture.asScala

You don´t really need Akka actors here unless you need them in more complex scenarios.

To handle the result as Future, use pipeTo: https://doc.akka.io/docs/akka/2.5.32/futures.html#use-the-pipe-pattern

Or even yo can use mapAsync from Akka stream: https://doc.akka.io/docs/akka/current/stream/operators/Source-or-Flow/mapAsync.html