'Is there any way to read batch of events from event hub in databricks?

My Requirement is i should read 1000 messages from event hub at a time in azure databricks and do some processing and then read the next 1000 messages in the queue.

On a whole i should continously read 1000 messages at a time.

Solution 1:^[1]

You need to set value of maxEventsPerTrigger to 1000.

maxEventsPerTrigger - Rate limit on maximum number of events processed per trigger interval. The specified total number of events will be proportionally split across partitions of different volume.

Sample Code -

import org.apache.spark.eventhubs.

val cs = "<your-connection-string>"
val eventHubsConf = EventHubsConf(cs)
  .setConsumerGroup("sample-cg")
  .setMaxEventsPerTrigger(10000)

Refer - https://docs.microsoft.com/en-us/azure/databricks/spark/latest/structured-streaming/streaming-event-hubs#eventhubsconf

Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution	Source
Solution 1	AbhishekKhandave-MT

'Is there any way to read batch of events from event hub in databricks?

Solution 1:[1]

Sources

Related Questions

Solution 1:^[1]