'Issue while starting kafka streaming from MongoDB connectors

This is my typical Setup in my MacBook

  1. Zookeeper Started - Single Node
  2. Kafka Started - Single Node
  3. MongoDB Started- 3 Replications
  4. MongoDB Connector - mongo-kafka-connect-1.7.0-confluent.jar copied to the Kafka libs directory, where all the jars lies
/Users/gaurav/Documents/Kafka/kafka_2.13-3.1.0/libs
  1. Edited the config file present in kafka configs - connect-standalone.properties

    bootstrap.servers=localhost:9092
    key.converter.schemas.enable=true
    value.converter.schemas.enable=true
    offset.storage.file.filename=/tmp/connect.offsets
    offset.flush.interval.ms=10000
    plugin.path=/Users/gaurav/Documents/Kafka/kafka_2.13-3.1.0/libs
    
  2. Added a new config file in the Kafka config directory MongoSourceConnector.properties

     
        name=mongo-source
        connector.class=com.mongodb.kafka.connect.MongoDbSourceConnector
        tasks.max=1
        topic=connect-test
        key.ignore=true
    
        connection.uri=mongodb://localhost:27017
        database=test_kafka
        collection=transaction
        max.num.retries=3
        retries.defer.timeout=5000
        type.name=kafka-connect
        schema.enable=false
       

7.Run the connect-standalone.sh (present in kafka bin location) to start kafka connect

/connect-standalone.sh ../config/connect-standalone.properties ../config/MongoSourceConnector.properties
  1. PS - I have already created topic connect-test and the mongodb database already The error I am getting is while starting kafka connect on 7th step
2022-03-18 01:14:09,992] INFO Kafka Connect started (org.apache.kafka.connect.runtime.Connect:57)
    [2022-03-18 01:14:10,001] ERROR Failed to create job for ../config/MongoSourceConnector.properties (org.apache.kafka.connect.cli.ConnectStandalone:107)
    [2022-03-18 01:14:10,002] ERROR Stopping after connector error (org.apache.kafka.connect.cli.ConnectStandalone:117)
    java.util.concurrent.ExecutionException: org.apache.kafka.connect.errors.ConnectException: Failed to find any class that implements Connector and which name matches com.mongodb.kafka.connect.MongoDbSourceConnector, available connectors are: PluginDesc{klass=class com.mongodb.kafka.connect.MongoSinkConnector, name='com.mongodb.kafka.connect.MongoSinkConnector', version='1.7.0', encodedVersion=1.7.0, type=sink, typeName='sink', location='file:/Users/gaurav/Documents/Kafka/kafka_2.13-3.1.0/libs/custom-connectors/mongo-kafka-connect-1.7.0-confluent.jar'}, PluginDesc{klass=class com.mongodb.kafka.connect.MongoSourceConnector, name='com.mongodb.kafka.connect.MongoSourceConnector', version='1.7.0', encodedVersion=1.7.0, type=source, typeName='source', location='file:/Users/gaurav/Documents/Kafka/kafka_2.13-3.1.0/libs/custom-connectors/mongo-kafka-connect-1.7.0-confluent.jar'}, PluginDesc{klass=class org.apache.kafka.connect.file.FileStreamSinkConnector, name='org.apache.kafka.connect.file.FileStreamSinkConnector', version='3.1.0', encodedVersion=3.1.0, type=sink, typeName='sink', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.file.FileStreamSourceConnector, name='org.apache.kafka.connect.file.FileStreamSourceConnector', version='3.1.0', encodedVersion=3.1.0, type=source, typeName='source', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.mirror.MirrorCheckpointConnector, name='org.apache.kafka.connect.mirror.MirrorCheckpointConnector', version='1', encodedVersion=1, type=source, typeName='source', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.mirror.MirrorHeartbeatConnector, name='org.apache.kafka.connect.mirror.MirrorHeartbeatConnector', version='1', encodedVersion=1, type=source, typeName='source', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.mirror.MirrorSourceConnector, name='org.apache.kafka.connect.mirror.MirrorSourceConnector', version='1', encodedVersion=1, type=source, typeName='source', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.tools.MockConnector, name='org.apache.kafka.connect.tools.MockConnector', version='3.1.0', encodedVersion=3.1.0, type=connector, typeName='connector', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.tools.MockSinkConnector, name='org.apache.kafka.connect.tools.MockSinkConnector', version='3.1.0', encodedVersion=3.1.0, type=sink, typeName='sink', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.tools.MockSourceConnector, name='org.apache.kafka.connect.tools.MockSourceConnector', version='3.1.0', encodedVersion=3.1.0, type=source, typeName='source', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.tools.SchemaSourceConnector, name='org.apache.kafka.connect.tools.SchemaSourceConnector', version='3.1.0', encodedVersion=3.1.0, type=source, typeName='source', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.tools.VerifiableSinkConnector, name='org.apache.kafka.connect.tools.VerifiableSinkConnector', version='3.1.0', encodedVersion=3.1.0, type=source, typeName='source', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.tools.VerifiableSourceConnector, name='org.apache.kafka.connect.tools.VerifiableSourceConnector', version='3.1.0', encodedVersion=3.1.0, type=source, typeName='source', location='classpath'}
        at org.apache.kafka.connect.util.ConvertingFutureCallback.result(ConvertingFutureCallback.java:115)
        at org.apache.kafka.connect.util.ConvertingFutureCallback.get(ConvertingFutureCallback.java:99)
        at org.apache.kafka.connect.cli.ConnectStandalone.main(ConnectStandalone.java:114)
PS - I do not want to use confluent setup now , the setup I am doing right now is totally manual       


Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source