Topic authorization failed error while KSQL tries connecting to cluster

Question

I am trying to have my ksqldb-server docker instance up and connect to a remote Kafka cluster, but getting an error. Here are the details

docker-compose.yml

---
version: '2'

services:
  ksqldb-server:
    build:
      context: .
      dockerfile: ./Dockerfile
    hostname: ksqldb-server
    container_name: ksqldb-server-remote
    ports:
      - "8088:8088"
    environment:
      KSQL_LISTENERS: http://0.0.0.0:8088
      KSQL_BOOTSTRAP_SERVERS: broker1:port1,broker2:port2,broker3:port,broker4:port
      KSQL_KSQL_LOGGING_PROCESSING_STREAM_AUTO_CREATE: "true"
      KSQL_KSQL_LOGGING_PROCESSING_TOPIC_AUTO_CREATE: "true"
      KSQL_KSQL_LOGGING_PROCESSING_TOPIC_REPLICATION_FACTOR: 2
      KSQL__confluent-ksql-apptest__command_topic_REPLICATION_FACTOR: 2
      KSQL_TOPIC_AUTHORIZATION_CHECKS: "false"

  ksqldb-cli:
    image: confluentinc/ksqldb-cli:0.24.0
    container_name: ksqldb-cli-remote
    depends_on:
      # - brokers
      - ksqldb-server
    entrypoint: /bin/sh
    tty: true

Dockerfile

FROM confluentinc/ksqldb-server:0.24.0

ADD ksql-server.properties /etc/meta/ksql-server.properties
ADD certificates /etc/meta/certificates


RUN rm -fr /etc/ksqldb/ksql-server.properties

RUN cp -a /etc/meta/ksql-server.properties /etc/ksqldb/ksql-server.properties
ENV KSQL_BOOTSTRAP_SERVERS="broker1:port,broker2:port,broker3:port,broker4:port"
ENV KSQL_LOGGING_PROCESSING_STREAM_AUTO_CREATE="true"
ENV KSQL_LOGGING_PROCESSING_TOPIC_AUTO_CREATE="true"
ENV KSQL_LISTENERS="http://0.0.0.0:8088"
ENTRYPOINT ["/bin/sh","-c","java -cp /usr/share/java/ksqldb-rest-app/*: -Xmx3g -server -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+CMSScavengeBeforeRemark -XX:+ExplicitGCInvokesConcurrent -XX:NewRatio=1 -Djava.awt.headless=true -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dksql.log.dir=/usr/logs -Dlog4j.configuration=file:/etc/ksqldb/log4j.properties -Dksql.server.install.dir=/usr -Xlog:gc*:file=/usr/logs/ksql-server-gc.log:time,tags:filecount=10,filesize=102400 io.confluent.ksql.rest.server.KsqlServerMain /etc/meta/ksql-server.properties"]

ksql-server.properties

#------ Endpoint config -------

### HTTP  ###
listeners=http://0.0.0.0:8088



### HTTPS ###
security.protocol=SSL
ssl.truststore.location=/etc/meta/certificates/truststore.jks
ssl.truststore.password=prmcert
ssl.keystore.location=/etc/meta/certificates/keystore.jks
ssl.keystore.password=prmcert
# ssl.key.password=?

#------ Logging config -------

# Automatically create the processing log topic if it does not already exist:
ksql.logging.processing.topic.auto.create=true

# Automatically create a stream within KSQL for the processing log:
ksql.logging.processing.stream.auto.create=true

#------ Kafka -------

# The set of Kafka brokers to bootstrap Kafka cluster information from:
bootstrap.servers=elr6hz1-06-s13.uhc.com:16016,elr6hz1-06-s16.uhc.com:16016,elr6hz1-06-s17.uhc.com:16016,elr6hz1-06-s19.uhc.com:16016,elr6hz1-06-s12.uhc.com:16016

# Enable snappy compression for the Kafka producers
compression.type=snappy

#------ Schema Registry -------
ksql.service.id=apptest_

This is the error that I see

ksqldb-server-remote | [2022-04-21 04:26:48,821] ERROR Unhandled exception in server startup (io.confluent.ksql.rest.server.KsqlServerMain:97) ksqldb-server-remote | io.confluent.ksql.exception.KafkaResponseGetFailedException: Failed to set config for Kafka Topic _confluent-ksql-apptest__command_topic ksqldb-server-remote | at io.confluent.ksql.services.KafkaTopicClientImpl.addTopicConfig(KafkaTopicClientImpl.java:258) ksqldb-server-remote | at io.confluent.ksql.rest.util.KsqlInternalTopicUtils.validateTopicConfig(KsqlInternalTopicUtils.java:124) ksqldb-server-remote | at io.confluent.ksql.rest.util.KsqlInternalTopicUtils.ensureTopic(KsqlInternalTopicUtils.java:66) ksqldb-server-remote | at io.confluent.ksql.rest.server.KsqlRestApplication.registerCommandTopic(KsqlRestApplication.java:1069) ksqldb-server-remote | at io.confluent.ksql.rest.server.KsqlRestApplication.initialize(KsqlRestApplication.java:455) ksqldb-server-remote | at io.confluent.ksql.rest.server.KsqlRestApplication.startKsql(KsqlRestApplication.java:390) ksqldb-server-remote | at io.confluent.ksql.rest.server.KsqlRestApplication.startAsync(KsqlRestApplication.java:372) ksqldb-server-remote | at io.confluent.ksql.rest.server.KsqlServerMain.tryStartApp(KsqlServerMain.java:93) ksqldb-server-remote | at io.confluent.ksql.rest.server.KsqlServerMain.main(KsqlServerMain.java:68) ksqldb-server-remote | Caused by: org.apache.kafka.common.errors.TopicAuthorizationException: Topic authorization failed.

Can someone please suggest on why I am seeing this error and how to resolve this.

Answer 1

The exception shows that KSQL cannot create the command topic because lack of permissions. The exception does not have what permissions, though. You can look at the kafka-authorizer.log from the kafka broker to see what ACL was denied and add the required ACL to allow KSQL to start.