Kafka Connect Installation

Jan 1, 2024

—

Standalone Mode

docker run -it -p 8083:8083 confluentinc/cp-kafka-connect bash

Edit this file to make sure bootstrap server points to the right host-name:

...
bootstrap.servers=broker:9092
...

connect-standalone /etc/kafka/connect-standalone.properties

Distributed mode

KEY_CONVERTER & VALUE_CONVERTER click here.

docker run -d -p 8083:8083 \
	-e CONNECT_BOOTSTRAP_SERVERS=broker:9092 \
    -e CONNECT_GROUP_ID=1 \
    -e CONNECT_CONFIG_STORAGE_TOPIC=connect-topics \
    -e CONNECT_OFFSET_STORAGE_TOPIC=connect-offsets \
    -e CONNECT_STATUS_STORAGE_TOPIC=connect-status \
    -e CONNECT_KEY_CONVERTER=org.apache.kafka.connect.json.JsonConverter \
    -e CONNECT_VALUE_CONVERTER=org.apache.kafka.connect.json.JsonConverter \
    -e CONNECT_REST_ADVERTISED_HOST_NAME=kconnect \
    -e CONNECT_CONFIG_STORAGE_REPLICATION_FACTOR=1 \ # Optional
    -e CONNECT_OFFSET_STORAGE_REPLICATION_FACTOR=1 \ # Optional
    -e CONNECT_STATUS_STORAGE_REPLICATION_FACTOR=1 \ # Optional
confluentinc/cp-kafka-connect

Manually:

docker run -it -p 8083:8083 confluentinc/cp-kafka-connect bash

You can put this file in any folder with any name:

bootstrap.servers=broker:9092
group.id=1
config.storage.topic=connect-topics
offset.storage.topic=connect-offsets
status.storage.topic=connect-status
key.converter=org.apache.kafka.connect.json.JsonConverter
value.converter=org.apache.kafka.connect.json.JsonConverter
rest.advertised.host.name=kconnect
rest.port=8083
plugin.path=/usr/share/java/,/usr/share/confluent-hub-components/
# config.storage.replication.factor=1
# offset.storage.replication.factor=1
# status.storage.replication.factor=1

Run it with:

connect-distributed /etc/kafka/connect-distributed.properties

References:

https://docs.confluent.io/platform/current/connect/userguide.html
https://docs.confluent.io/platform/current/connect/references/allconfigs.html
https://www.confluent.io/hub/
https://docs.confluent.io/platform/current/connect/references/restapi.html