Rest API for Kafka

Question

I need to write a REST API for kafka which can read or write data from consumer/producer respectively. How can I do so?

Answer 1

This is a sample Rest API (Rest Proxy) code from Confluent. I had to type it, so it might contain some misspelling. I hope this helps you a little bit.

( Producer using REST API written in Python )

import requests
import base64
import json

url = "http://restproxy:8082/topics/my_topic"
headers = {
"Content-Type" : "application/vnd.kafka.binary.v1 + json"
   }
# Create one or more messages
payload = {"records":
       [{
           "key":base64.b64encode("firstkey"),
           "value":base64.b64encode("firstvalue")
       }]}
# Send the message
r = requests.post(url, data=json.dumps(payload), headers=headers)
if r.status_code != 200:
   print "Status Code: " + str(r.status_code)
   print r.text

(Consumer using Rest API written in Python)

import requests
import base64
import json
import sys

#Base URL for interacting with REST server
baseurl = "http://restporxy:8082/consumers/group1"

#Create the Consumer instance
print "Creating consumer instance"
payload {
    "format": "binary"
    }
headers = {
"Content-Type" : "application/vnd.kafka.v1+json"
    }
r = requests.post(baseurl, data=json.dumps(payload), headers=headers)

if r.status_code !=200:
    print "Status Code: " + str(r.status_code)
    print r.text
    sys.exit("Error thrown while creating consumer")

# Base URI is used to identify the consumer instance
base_uri = r.json()["base_uri"]

#Get the messages from the consumer
headers = {
    "Accept" : "application/vnd.kafka.binary.v1 + json"
    }

# Request messages for the instance on the Topic
r = requests.get(base_uri + "/topics/my_toopic", headers = headers, timeout =20)

if r.status_code != 200: 
    print "Status Code: " + str(r.status_code)
    print r.text
    sys.exit("Error thrown while getting message")

#Output all messages
for message in r.json():
    if message["key"] is not None:
        print "Message Key:" + base64.b64decode(message["key"])
    print "Message Value:" + base64.b64decode(message["value"])

# When we're done, delete the consumer
headers = {
    "Accept" : "application/vnd.kafka.v1+json"
    }

r = requests.delete(base_uri, headers=headers)

if r.status_code != 204: 
    print "Status Code: " + str(r.status_code)
    print r.text

Answer 2

I guess that your question was more about how to write the "server" REST API interface not the client-side (which at the end is just making HTTP requests). You can use the Strimzi HTTP bridge for example (https://github.com/strimzi/strimzi-kafka-bridge) which works stand-alone or even in Kubernetes is you are willing to deploy the cluster there (then you can use Strimzi project, https://strimzi.io/).