'RocksDB NoClassDefFound - how to setup rocksDB for Kafka streams

I'm attempting to solve a problem using kstreams. I'm currently hitting this error when doing an aggregation.

Exception in thread "main" java.lang.NoClassDefFoundError: org/rocksdb/RocksDBException
    at org.apache.kafka.streams.state.internals.RocksDbWindowBytesStoreSupplier.get(RocksDbWindowBytesStoreSupplier.java:50)
    at org.apache.kafka.streams.state.internals.RocksDbWindowBytesStoreSupplier.get(RocksDbWindowBytesStoreSupplier.java:24)
    at org.apache.kafka.streams.state.internals.WindowStoreBuilder.build(WindowStoreBuilder.java:40)
    at org.apache.kafka.streams.state.internals.WindowStoreBuilder.build(WindowStoreBuilder.java:26)
    at org.apache.kafka.streams.processor.internals.InternalTopologyBuilder$StateStoreFactory.build(InternalTopologyBuilder.java:141)
    at org.apache.kafka.streams.processor.internals.InternalTopologyBuilder.buildProcessorNode(InternalTopologyBuilder.java:966)
    at org.apache.kafka.streams.processor.internals.InternalTopologyBuilder.build(InternalTopologyBuilder.java:869)
    at org.apache.kafka.streams.processor.internals.InternalTopologyBuilder.build(InternalTopologyBuilder.java:822)
    at org.apache.kafka.streams.processor.internals.InternalTopologyBuilder.build(InternalTopologyBuilder.java:805)
    at org.apache.kafka.streams.KafkaStreams.<init>(KafkaStreams.java:667)
    at org.apache.kafka.streams.KafkaStreams.<init>(KafkaStreams.java:624)
    at org.apache.kafka.streams.KafkaStreams.<init>(KafkaStreams.java:534)

My code is effectively this:

KStream<String, InputData> input = builder.stream(topicname);

KTable<Windowed<String>, CustomAgg> grouped =
                input.groupByKey()
                .windowedBy(TimeWindows.of(Duration.ofMillis(60000)))
                .aggregate(
                        CustomAgg::new,
                        (k, v, agg) -> agg.add(v),
                        Materialized.<String, CustomAgg, WindowStore<Bytes, byte[]>>as("aggs").withValueSerde(new CustomAggSerde()));
        grouped.toStream().print(Printed.toSysOut());

kafka-streams version 2.1.0

I can't seem to find any resources online on how to setup rocksDB for kafka streams - any advice would be much appreciated. (I have it installed with brew but I'm not sure how I need to point to it, any setup, does it need to be in my pom.xml file etc). Working on MacOS currently for development.

Thanks!

apache-kafka-streams

Solution 1:^[1]

You do not need to install RocksDB for Kafka Streams. RocksDB is a dependency of Kafka Streams. If you have Kafka Streams as a dependency in your build automation tool (e.g. maven or gradle), the RocksDB JAR should be automatically downloaded during a build and put onto your class path.

Without a build automation tool you probably need to put the RocksDB JAR on the class path manually. The correct version of RocksDB for Kafka Streams 2.1.0 should be 5.14.2.

The error you get seems to be a class path issue, so maybe it is related to the above.

Solution 2:^[2]

try insert below dependency in you pom.xml:

<dependency>
    <groupId>org.rocksdb</groupId>
    <artifactId>rocksdbjni</artifactId>
    <version>4.9.0</version>
</dependency>

this link might be helpful to you: https://technology.amis.nl/software-development/java/getting-started-with-kafka-streams-building-a-streaming-analytics-java-application-against-a-kafka-topic/

Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution	Source
Solution 1	Bruno Cadonna
Solution 2

'RocksDB NoClassDefFound - how to setup rocksDB for Kafka streams

Solution 1:[1]

Solution 2:[2]

Sources

Related Questions

Solution 1:^[1]

Solution 2:^[2]