Kafka 安裝 - Kraft Mode

Kafka - Kraft Mode 安裝步驟

Kafka

這篇文章將介紹如何在 Kraft Mode 下安裝和配置 Apache Kafka，這是一種不依賴 Zookeeper 的模式。文章將提供從開始到結束的教學。

事前準備 Preparation

準備 3 台 node:

os name ip
Rocky Linux 8.8 cns-kafka-0 192.168.50.170
Rocky Linux 8.8 cns-kafka-1 192.168.50.171
Rocky Linux 8.8 cns-kafka-2 192.168.50.172

安裝步驟

設定 hosts (在每個 Node 上執行)

透過以下指令設定 /etc/hosts:

cat << EOF >> /etc/hosts
192.168.50.170 cns-kafka-0
192.168.50.171 cns-kafka-1
192.168.50.172 cns-kafka-2
EOF

安裝 OpenJDK 11 (在每個 Node 上執行)

To use Corretto RPM repositories with the yum package manager (such as Amazon Linux AMI), import the Corretto public key and then add the repository to the system list. For most systems, you must run the following commands:

可以透過 Amazon Corretto 安裝 JAVA: https://docs.aws.amazon.com/corretto/latest/corretto-11-ug/linux-info.html

sudo rpm --import https://yum.corretto.aws/corretto.key 
sudo curl -L -o /etc/yum.repos.d/corretto.repo https://yum.corretto.aws/corretto.repo

After the repository is added, you can install Corretto 11 by running this command:

sudo yum install -y java-11-amazon-corretto-devel

Upon completion, you should able to run the follow command:

java -version

安裝 Kafka (在每個 Node 上執行)

下載 Kafka:

可以到 Kafka 頁面下載最新的 Kafka: https://kafka.apache.org/downloads

curl -OL https://downloads.apache.org/kafka/3.5.0/kafka_2.13-3.5.0.tgz

解壓縮:

tar xzf kafka_2.13-3.5.0.tgz

搬移資料夾位置:

mv kafka_2.13-3.5.0.tgz /etc/kafka

修改 server.properties (在每個 Node 上執行)

編輯 server.properties:

vi /etc/kafka/config/kraft/server.properties

修改以下內容:

如果需要修改 log 位置, 則要修改 log.dirs

key value description
node.id 1 每個 node 需要不一樣的 id, 如: cns-kafka-0 設定 1, cns-kafka-1 設定 2, cns-kafka-2 設定 3
controller.quorum.voters 1@cns-kafka-0:9093,2@cns-kafka-1:9093,3@cns-kafka-2:9093 設定每個 voters
advertised.listeners PLAINTEXT://192.168.50.170:9092 每個 node 需要提供不一樣的 listeners 給 client, 如: cns-kafka-0 設定 PLAINTEXT://192.168.50.170:9092, cns-kafka-1 設定 PLAINTEXT://192.168.50.171:9092, cns-kafka-2 設定 PLAINTEXT://192.168.50.172:9092

產生 Cluster UUID (只在其中一個 Node 上執行)

使用 kafka-storage.sh 產生 uuid

/etc/kafka/bin/kafka-storage.sh random-uuid

格式化 storage directory (在每一個 Node 上執行)

請替換 <uuid> 為剛剛產出的結果

This will format the directory that is in the log.dirs in the config/kraft/server.properties file (by default /tmp/kraft-combined-logs)

/etc/kafka/bin/kafka-storage.sh format -t <uuid> -c /etc/kafka/config/kraft/server.properties

使用 Systemd 管理 Kafka

建立 kafka.service:

cat << EOF >> kafka.service
[Unit]
Description=Kafka Server
Requires=network.target remote-fs.target
After=network.target remote-fs.target

[Service]
Type=simple
ExecStart=/bin/bash -c 'etc/kafka/bin/kafka-server-start.sh /etc/kafka/config/kraft/server.properties > /tmp/kafka.log 2>&1'
ExecStop=/etc/kafka/bin/kafka-server-stop.sh /etc/kafka/config/kraft/server.properties
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF

搬移 kafka.service 到 system 目錄:

cp kafka.service /etc/systemd/system

重啟 daemon, 設定開機啟動:

systemctl daemon-reload && systemctl enable kafka.service

啟動 Kafka:

你可以透過 /tmp/kafka.log 查看啟動 log

systemctl restart kafka.service

建立 kafka-ui

你可以透過 kafka-ui 查看 Kafka 狀態, 並且設定 service:

kafka-ui: https://github.com/provectus/kafka-ui

請先準備 image: provectuslabs/kafka-ui:latest

可以修改 KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS 指向 Kafka

# container-kafka-ui.service
# autogenerated by Podman 4.4.1
# Sat May 27 11:47:42 CST 2023

[Unit]
Description=Podman container-kafka-ui.service
Documentation=man:podman-generate-systemd(1)
Wants=network-online.target
After=network-online.target
RequiresMountsFor=%t/containers

[Service]
Environment=PODMAN_SYSTEMD_UNIT=%n
Restart=on-failure
TimeoutStopSec=70
ExecStart=/usr/bin/podman run \
        --cidfile=%t/%n.ctr-id \
        --cgroups=no-conmon \
        --rm \
        --sdnotify=conmon \
        --replace \
        --name kafka-ui \
        -d \
        -p 8080:8080 \
        -e KAFKA_CLUSTERS_0_NAME=local \
        -e KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS=192.168.50.170:9092,192.168.50.171:9092,192.168.50.172:9092 provectuslabs/kafka-ui:latest
ExecStop=/usr/bin/podman stop \
        --ignore -t 10 \
        --cidfile=%t/%n.ctr-id
ExecStopPost=/usr/bin/podman rm \
        -f \
        --ignore -t 10 \
        --cidfile=%t/%n.ctr-id
Type=notify
NotifyAccess=all

[Install]
WantedBy=default.target

結果

你可以看到 Kafka-UI 的畫面:

kafka-ui

Reference