'How to configure event rate in Flink

I have a file containing millions of entries and I need to process it with Apache Flink in combination with Kafka.

I have written a Producer which reads lines from the file and passes them to Kafka. I also have a Consumer that consumes the data from Kafka. Both of them are connectors in Apache Flink.

My question is, is there a way to configure the Producer so that I can configure it to read and pass a specific number of lines per second (such as 1000 lines per second, or 100,000 lines per second) to Kafka?

Thanks in advance :)

apache-kafkabigdataapache-flinkflink-streamingkafka-producer-api


Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source

Related Questions

Kinesis Analytics SQL query to narrow down the sensors that are not sending data

where's socketTextStream in pyflink

What is the Java version that the Flink can support in 2022?

Flink Missing Events With Windowed Processor(Event Time Windows) and Kafka Source

Apache Flink + CEP - Detect same events

Apache Flink: AWS S3 timeout exception when starting a job from a savepoint

Kafka Client Timeout of 60000ms expired before the position for partition could be determined

I'm looking for code to connect PyFlink to Pulsar

FLINK1.8 , ERROR INFO : state is larger than the maximum permitted memory-backed state. use RocksDBStateBackend not useful

Windowing on mapped stream is giving different results

What should I use instead deprecated FlinkKafkaConsumer? Scala Flink

Integration testing flink job

Cannot access Flink dashboard localhost:8081 on windows

Apache Flink - writing stream to S3 error - null uri host

Flink checkpoint not replaying the kafka events which were in process during the savepoint/checkpoint