'How to collect data with a KafkaConsumer and expose it with Flask

I have a KafkaConsumer code (in python) that looks something like:

consumer = KafkaConsumer(...)

all_messages = []
for message in consumer:
    all_messages.append(message)

I would like to know how I could combine this with a trivial Flask server code that looks something like:

@app.route("/get_all_messages",  methods=['GET'])
def get_all_messages():
    return jsonify(all_messages)

To my understanding this isn't straightforward, as both the KafkaConsumer loop and the Flask implementation do not release the thread they run on.

Is there a way to solve this, other than separating the implementation to two services? That way, the KafkaConsumer could POST the data to the Flask http server, which will also have the GET API above.



Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source