'Strimzi Kafka connector gets SocketTimeoutException while connecting to confluent schema registry

I created a Kafka connect cluster with Strimzi operator:

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnect
metadata:
  annotations:
    strimzi.io/use-connector-resources: "true"
  name: test-cluster-0
spec:
  bootstrapServers: {{.Values.bootstrapServers}}
  config:
    group.id: connect-cluster-datagen
    key.converter: io.confluent.connect.avro.AvroConverter
    key.converter.schemas.enable: true
    value.converter: io.confluent.connect.avro.AvroConverter
    value.converter.schema.registry.url: http://cp-schema-registry.my-namespace.svc:8081
    key.converter.schema.registry.url: http://cp-schema-registry.my-namespace.svc:8081
    value.converter.schemas.enable: true

    offset.storage.replication.factor: 3
    offset.storage.topic: datagen-connect-cluster-offsets

    status.storage.replication.factor: 3
    status.storage.topic: datagen-connect-cluster-status

    config.storage.replication.factor: 3
    config.storage.topic: datagen-connect-cluster-configs

  replicas: 1
  version: 3.0.0

Schema registry is created using confluent helm charts -- with following values.yaml:

cp-kafka:
  enabled: false
cp-zookeeper:
  enabled: false
cp-kafka-rest:
  enabled: false
cp-kafka-connect:
  enabled: false
cp-control-center:
  enabled: false
cp-ksql-server:
  enabled: false
cp-schema-registry:
  enabled: true
  image: confluentinc/cp-schema-registry
  kafka:
    bootstrapServers: {{.Value.bootstrapServers}}

However connectors throw exceptions like:

2022-02-15 18:07:01,939 ERROR [datagen-source-topic-connector|task-0] Failed to send HTTP request to endpoint: http://cp-schema-registry.my-namespace.svc:8081/subjects/datagen-source-topic-value/versions (io.confluent.kafka.schemaregistry.client.rest.RestService) [task-thread-datagen-source-topic-connector-0]
java.net.SocketTimeoutException: connect timed out
        at java.base/java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:399)
        at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:242)
        at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:224)
        at java.base/java.net.Socket.connect(Socket.java:609)

This question mentions that SCHEMA_REGISTRY_HOST_NAME should be resolvable. However confluent helm chart sets this to pod ip -- which should be resolvable:

  env:
  - name: SCHEMA_REGISTRY_HOST_NAME
    valueFrom:
      fieldRef:
        fieldPath: status.podIP

Am I missing a configuration parameter here? If I send normal curl requests, I see the following in schema-registry logs:

io.confluent.rest.exceptions.RestNotFoundException: Subject 'datagen-source-topic-value' not found.
        at io.confluent.kafka.schemaregistry.rest.exceptions.Errors.subjectNotFoundException(Errors.java:69)
        at io.confluent.kafka.schemaregistry.rest.resources.SubjectVersionsResource.listVersions(SubjectVersionsResource.java:197)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:124)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:167)
        at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:219)
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:79)
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:469)
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:391)
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:80)
        at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:253)
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
        at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
        at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)
        at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:232)
        at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:680)
        at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394)
        at org.glassfish.jersey.servlet.ServletContainer.serviceImpl(ServletContainer.java:386)
        at org.glassfish.jersey.servlet.ServletContainer.doFilter(ServletContainer.java:561)
        at org.glassfish.jersey.servlet.ServletContainer.doFilter(ServletContainer.java:502)
        at org.glassfish.jersey.servlet.ServletContainer.doFilter(ServletContainer.java:439)
        at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)


Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source