Setting up Kafka

For connector version 3.2.0 and later

To use the Kafka connector, download and install Apache Kafka. Before you start configuring Kafka you also need the integration runtime and we refer to that location as <PRODUCT_HOME>.

Note: The recommended version is Kafka 2.12-2.8.2. For all available versions of Kafka that you can download, see https://kafka.apache.org/downloads. The recommended Java version is 11.

To configure the Kafka connector, copy the following client libraries from the <KAFKA_HOME>/lib directory to the <MI_HOME>/lib directory.

• kafka_2.12-2.8.2.jar
• kafka-clients-2.8.2.jar
• metrics-core-2.2.0.jar
• scala-library-2.12.13jar
• zkclient-0.10.jar
• zookeeper-3.5.9.jar

Copy the following additional client libraries to the <MI_HOME>/lib directory (can be copied from the Confluent platform):

• avro-1.11.3.jar
• common-config-5.4.0.jar
• common-utils-5.4.0.jar
• kafka-avro-serializer-5.3.0.jar
• kafka-schema-registry-client-5.3.0.jar

Navigate to <KAFKA_HOME> and run the following command to start the ZooKeeper server:

bin/zookeeper-server-start.sh config/zookeeper.properties

From the <KAFKA_HOME> directory, run the following command to start the Kafka server:

bin/kafka-server-start.sh config/server.properties

Now that you have connected to Kafka, you can start publishing and consuming messages using the Kafka brokers. For more information, see Publishing Messages using Kafka and Consuming Messages using Kafka.

For connector version 3.1.x and below

To use the Kafka connector, download and install Apache Kafka. Before you start configuring the Kafka you also need the integration runtime and we refer to that location as <PRODUCT_HOME>.

Note: The recommended version is Kafka_2.11-2.2.1. For all available versions of Kafka that you can download, see https://kafka.apache.org/downloads. The recommended Java version is 1.8.

To configure the Kafka connector, copy the following client libraries from the <KAFKA_HOME>/lib directory to the <MI_HOME>/lib directory.

• kafka_2.11-2.2.1.jar
• kafka-clients-1.0.0.jar
• metrics-core-2.2.0.jar
• scala-library-2.12.3.jar
• zkclient-0.10.jar
• zookeeper-3.4.10.jar

Copy the following additional client libraries to the <MI_HOME>/lib directory (can be copied from the Confluent platform):

• avro-1.8.1.jar
• common-config-5.4.0.jar
• common-utils-5.4.0.jar
• kafka-avro-serializer-5.3.0.jar
• kafka-schema-registry-client-5.3.0.jar

Navigate to and run the following command to start the ZooKeeper server:

bin/zookeeper-server-start.sh config/zookeeper.properties

From the directory, run the following command to start the Kafka server:

bin/kafka-server-start.sh config/server.properties

Now that you have connected to Kafka, you can start publishing and consuming messages using the Kafka brokers. For more information, see Publishing Messages using Kafka and Consuming Messages using Kafka.