【Kafka】安装Kafka
- 1. 安装Kafka
- 2. Kafka使用
- 2.0 集群分发脚本xsync(重要)
- 2.0.1 scp命令
- 2.0.2 rsync远程同步工具
- 2.0.3 写一个集群分发脚本xsync (Shell 脚本)
- 2.1 Zookeeper安装
- 2.2 对Kafka进行分发
- 2.2.1 执行同步脚本
- 2.2.2 三台云主机配置Kafka环境变量
1. 安装Kafka
Kafka 官网https://kafka.apache.org/
上传文件至云服务器
解压Kafka
[root@VM-8-7-centos opt]# cd /opt
[root@VM-8-7-centos opt]# mkdir moudle
[root@VM-8-7-centos opt]# cd software/
[root@VM-8-7-centos software]# tar -zxvf kafka_2.12-3.0.0.tgz -C /opt/moudle/
修改名称
[root@VM-8-7-centos moudle]# mv kafka_2.12-3.0.0/ kafka
[root@VM-8-7-centos bin]# pwd
/opt/moudle/kafka/bin
修改配置文件
[root@VM-8-7-centos kafka]# cd config/
[root@VM-8-7-centos config]# ls
connect-console-sink.properties connect-file-source.properties consumer.properties server.properties
connect-console-source.properties connect-log4j.properties kraft tools-log4j.properties
connect-distributed.properties connect-mirror-maker.properties log4j.properties trogdor.conf
connect-file-sink.properties connect-standalone.properties producer.properties zookeeper.properties
[root@VM-8-7-centos config]# vim server.properties
[root@VM-8-7-centos config]# pwd
/opt/moudle/kafka/config
修改地方1:broker.id
修改地方2:日志
修改地方3:Zookeeper
2. Kafka使用
2.0 集群分发脚本xsync(重要)
scp课程
scp安全拷贝
scp可以实现服务器与服务器之间的数据拷贝。(from server1 to server2)
2.0.1 scp命令
scp -r $pdir/$fname $user@$host:$pdir/$fname
命令 递归 要拷贝的文件路径/名称 目的地用户@主机:目的地路径/名称
[root@VM-8-7-centos moudle]# scp -r kafka/ root@139.9.133.30:/opt/moudle
The authenticity of host '139.9.133.30 (139.9.133.30)' can't be established.
ECDSA key fingerprint is SHA256:CNtTiB8P/OCKir8Dx2nVGb461U8fuwHh9v3saUBMFKo.
ECDSA key fingerprint is MD5:6f:89:b0:b1:d8:55:9c:1d:82:1e:6e:54:ac:ca:ae:fb.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '139.9.133.30' (ECDSA) to the list of known hosts.
root@139.9.133.30's password:
javax.ws.rs-api-2.1.1.jar 100% 124KB 3.5MB/s 00:00
osgi-resource-locator-1.0.3.jar 100% 19KB 996.1KB/s 00:00
zstd-jni-1.5.0-2.jar 100% 6604KB 508.0KB/s 00:13
reflections-0.9.12.jar 100% 103KB 494.4KB/s 00:00
jetty-servlet-9.4.43.v20210629.jar 100% 142KB 547.7KB/s 00:00
jakarta.annotation-api-1.3.5.jar 100% 24KB 541.1KB/s 00:00
2.0.2 rsync远程同步工具
rsync主要用于备份和镜像。具有速度快、避免复制相同内容和支持符号链接的优点。
rsync和scp区别:用rsync做文件的复制要比scp的速度快,rsync只对差异文件做更新。scp是把所有文件都复制过去。
rsync -av $pdir/$fname $user@$host:$pdir/$fname
命令 选项参数 要拷贝的文件路径/名称 目的地用户@主机:目的地路径/名称
2.0.3 写一个集群分发脚本xsync (Shell 脚本)
#!/bin/bash
#1. 判断参数个数
if [ $# -lt 1 ]
then
echo Not Enough Arguement!
exit;
fi
#2. 遍历集群所有机器
for host in 139.9.133.30 14.116.254.124
do
echo ==================== $host ====================
#3. 遍历所有目录,挨个发送
for file in $@
do
#4. 判断文件是否存在
if [ -e $file ]
then
#5. 获取父目录
pdir=$(cd -P $(dirname $file); pwd)
#6. 获取当前文件的名称
fname=$(basename $file)
ssh $host "mkdir -p $pdir"
rsync -av $pdir/$fname $host:$pdir
else
echo $file does not exists!
fi
done
done
修改脚本 xsync 具有执行权限
[atguigu@hadoop102 bin]$ chmod +x xsync
2.1 Zookeeper安装
课程:https://www.bilibili.com/video/BV1to4y1C7gw?p=7&vd_source=240d9002f7c7e3da63cd9a975639409a
// todo
2.2 对Kafka进行分发
2.2.1 执行同步脚本
2.2.2 三台云主机配置Kafka环境变量
# jdk1.8
export JAVA_HOME=/usr/local/java/jdk1.8.0_131
export CLASSPATH=$:CLASSPATH:$JAVA_HOME/lib/
export PATH=$PATH:$JAVA_HOME/bin
# Nodejs
export PATH=/usr/local/node/node-v12.16.3-linux-x64/bin:$PATH
# maven
export MAVEN_HOME=/opt/maven/apache-maven-3.6.3
export PATH=$MAVEN_HOME/bin:$PATH
# kafka
export KAFKA_HOME=/opt/module/kafka
export PATH=$PATH:$KAFKA_HOME/bin
[root@VM-8-7-centos moudle]# source /etc/profile