在Centos7.9_2207安装CDH6.3.2

news2024/11/24 8:46:49

在Centos7.9_2207安装CDH6.3.2

背景

笔者做大数据开发，实时部分一般要用到HBase、Kudu、Redis等组件来保证幂等性，为了方便，还是选用老古董CDH6.3.2【最后的免费版】做一个单节点机器，方便随时挂起。多节点虚拟机由之前的双路E5做Frp后远程调用。

物理机

资源配置

机型：惠普战99锐龙版

CPU：7840HS 8c16t

RAM：2*48G

SSD：1T+4T

很普通的集显工作站。2条DDR5的48G内存条组成96G内存。

部署好CDH常用组件后，目前的资源占用情况大概是这样：

在这里插入图片描述

经过多年的发展，轻薄本用的7840HS已经远远超越了2016年的标压6700hq，如果没有便携或者大内存的需求，上个7945HX，可以和双路E5-2696V3打的有来有回。

安装Centos7.9_2207

下载ISO镜像

https://wiki.centos.org/Download.html
http://isoredirect.centos.org/centos/7/isos/x86_64/
https://mirrors.tuna.tsinghua.edu.cn/centos/7.9.2009/isos/x86_64/

下载CDH安装包：

https://archive.cloudera.com/

资源配置

CPU：4c

RAM：48G

硬盘：300G

因为要搭一个单节点CDH，所有组件会全部在一个虚拟机启动，所以尽可能分配充足的资源。

网络配置

IP：192.168.88.52

子网掩码：255.255.255.0

网关：192.168.88.2

DNS：8.8.8.8

host：bigdata-

在这里插入图片描述

禁用IPV6。

此时：

[root@bigdata-centos ~]# hostname
bigdata-centos
[root@bigdata-centos ~]# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
[root@bigdata-centos ~]#

部署虚拟机

zhiyong
Shyf@2019

正常部署即可。

CPU性能测试

在CentOS上安装UnixBench可以按照以下步骤进行：

  1. 打开终端，使用root用户或具有sudo权限的用户登录。

  2. 确保您的系统已安装了必要的依赖关系。运行以下命令来安装这些依赖关系：

sudo yum install make gcc
mkdir -p /home/bigdata/unixbench
cd /home/bigdata/unixbench
wget https://codeload.github.com/kdlucas/byte-unixbench/zip/refs/tags/v5.1.3
unzip byte-unixbench-5.1.3.zip
cd /home/bigdata/unixbench/byte-unixbench-5.1.3/UnixBench
make ./Run

耐心等待跑完后，可以看到跑分情况，能凑合用：

[root@bigdata-centos UnixBench]# ./Run
make all
make[1]: Entering directory `/home/bigdata/unixbench/byte-unixbench-5.1.3/UnixBench'
Checking distribution of files
./pgms  exists
./src  exists
./testdir  exists
./tmp  exists
./results  exists
make[1]: Leaving directory `/home/bigdata/unixbench/byte-unixbench-5.1.3/UnixBench'
sh: 3dinfo: command not found

   #    #  #    #  #  #    #          #####   ######  #    #   ####   #    #
   #    #  ##   #  #   #  #           #    #  #       ##   #  #    #  #    #
   #    #  # #  #  #    ##            #####   #####   # #  #  #       ######
   #    #  #  # #  #    ##            #    #  #       #  # #  #       #    #
   #    #  #   ##  #   #  #           #    #  #       #   ##  #    #  #    #
    ####   #    #  #  #    #          #####   ######  #    #   ####   #    #

   Version 5.1.3                      Based on the Byte Magazine Unix Benchmark

   Multi-CPU version                  Version 5 revisions by Ian Smith,
                                      Sunnyvale, CA, USA
   January 13, 2011                   johantheghost at yahoo period com


1 x Dhrystone 2 using register variables  1 2 3 4 5 6 7 8 9 10

1 x Double-Precision Whetstone  1 2 3 4 5 6 7 8 9 10

1 x Execl Throughput  1 2 3

1 x File Copy 1024 bufsize 2000 maxblocks  1 2 3

1 x File Copy 256 bufsize 500 maxblocks  1 2 3

1 x File Copy 4096 bufsize 8000 maxblocks  1 2 3

1 x Pipe Throughput  1 2 3 4 5 6 7 8 9 10

1 x Pipe-based Context Switching  1 2 3 4 5 6 7 8 9 10

1 x Process Creation  1 2 3

1 x System Call Overhead  1 2 3 4 5 6 7 8 9 10

1 x Shell Scripts (1 concurrent)  1 2 3

1 x Shell Scripts (8 concurrent)  1 2 3

4 x Dhrystone 2 using register variables  1 2 3 4 5 6 7 8 9 10

4 x Double-Precision Whetstone  1 2 3 4 5 6 7 8 9 10

4 x Execl Throughput  1 2 3

4 x File Copy 1024 bufsize 2000 maxblocks  1 2 3

4 x File Copy 256 bufsize 500 maxblocks  1 2 3

4 x File Copy 4096 bufsize 8000 maxblocks  1 2 3

4 x Pipe Throughput  1 2 3 4 5 6 7 8 9 10

4 x Pipe-based Context Switching  1 2 3 4 5 6 7 8 9 10

4 x Process Creation  1 2 3

4 x System Call Overhead  1 2 3 4 5 6 7 8 9 10

4 x Shell Scripts (1 concurrent)  1 2 3

4 x Shell Scripts (8 concurrent)  1 2 3

========================================================================
   BYTE UNIX Benchmarks (Version 5.1.3)

   System: bigdata-centos: GNU/Linux
   OS: GNU/Linux -- 3.10.0-1160.71.1.el7.x86_64 -- #1 SMP Tue Jun 28 15:37:28 UTC 2022
   Machine: x86_64 (x86_64)
   Language: en_US.utf8 (charmap="UTF-8", collate="UTF-8")
   CPU 0: AMD Ryzen 7 7840HS w/ Radeon 780M Graphics (7585.8 bogomips)
          Hyper-Threading, x86-64, MMX, AMD MMX, Physical Address Ext, SYSENTER/SYSEXIT, SYSCALL/SYSRET
   CPU 1: AMD Ryzen 7 7840HS w/ Radeon 780M Graphics (7585.8 bogomips)
          Hyper-Threading, x86-64, MMX, AMD MMX, Physical Address Ext, SYSENTER/SYSEXIT, SYSCALL/SYSRET
   CPU 2: AMD Ryzen 7 7840HS w/ Radeon 780M Graphics (7585.8 bogomips)
          Hyper-Threading, x86-64, MMX, AMD MMX, Physical Address Ext, SYSENTER/SYSEXIT, SYSCALL/SYSRET
   CPU 3: AMD Ryzen 7 7840HS w/ Radeon 780M Graphics (7585.8 bogomips)
          Hyper-Threading, x86-64, MMX, AMD MMX, Physical Address Ext, SYSENTER/SYSEXIT, SYSCALL/SYSRET
   01:36:48 up 43 min,  3 users,  load average: 0.00, 0.01, 0.05; runlevel 5

------------------------------------------------------------------------
Benchmark Run: 三 10月 18 2023 01:36:48 - 02:05:51
4 CPUs in system; running 1 parallel copy of tests

Dhrystone 2 using register variables       45522028.9 lps   (10.0 s, 7 samples)
Double-Precision Whetstone                     3556.8 MWIPS (15.8 s, 7 samples)
Execl Throughput                               3186.7 lps   (30.0 s, 2 samples)
File Copy 1024 bufsize 2000 maxblocks        792934.6 KBps  (30.0 s, 2 samples)
File Copy 256 bufsize 500 maxblocks          215891.9 KBps  (30.0 s, 2 samples)
File Copy 4096 bufsize 8000 maxblocks       2162307.5 KBps  (30.0 s, 2 samples)
Pipe Throughput                             1915569.4 lps   (10.0 s, 7 samples)
Pipe-based Context Switching                  20637.0 lps   (10.0 s, 7 samples)
Process Creation                               8614.4 lps   (30.0 s, 2 samples)
Shell Scripts (1 concurrent)                   7681.2 lpm   (60.0 s, 2 samples)
Shell Scripts (8 concurrent)                   2594.1 lpm   (60.0 s, 2 samples)
System Call Overhead                        2884075.7 lps   (10.0 s, 7 samples)

System Benchmarks Index Values               BASELINE       RESULT    INDEX
Dhrystone 2 using register variables         116700.0   45522028.9   3900.8
Double-Precision Whetstone                       55.0       3556.8    646.7
Execl Throughput                                 43.0       3186.7    741.1
File Copy 1024 bufsize 2000 maxblocks          3960.0     792934.6   2002.4
File Copy 256 bufsize 500 maxblocks            1655.0     215891.9   1304.5
File Copy 4096 bufsize 8000 maxblocks          5800.0    2162307.5   3728.1
Pipe Throughput                               12440.0    1915569.4   1539.8
Pipe-based Context Switching                   4000.0      20637.0     51.6
Process Creation                                126.0       8614.4    683.7
Shell Scripts (1 concurrent)                     42.4       7681.2   1811.6
Shell Scripts (8 concurrent)                      6.0       2594.1   4323.4
System Call Overhead                          15000.0    2884075.7   1922.7
                                                                   ========
System Benchmarks Index Score                                        1252.4

------------------------------------------------------------------------
Benchmark Run: 三 10月 18 2023 02:05:51 - 02:34:44
4 CPUs in system; running 4 parallel copies of tests

Dhrystone 2 using register variables      151547416.3 lps   (10.0 s, 7 samples)
Double-Precision Whetstone                    14886.9 MWIPS (13.9 s, 7 samples)
Execl Throughput                              15018.6 lps   (30.0 s, 2 samples)
File Copy 1024 bufsize 2000 maxblocks        829648.1 KBps  (30.0 s, 2 samples)
File Copy 256 bufsize 500 maxblocks          217171.6 KBps  (30.0 s, 2 samples)
File Copy 4096 bufsize 8000 maxblocks       2729918.6 KBps  (30.0 s, 2 samples)
Pipe Throughput                             6317809.1 lps   (10.0 s, 7 samples)
Pipe-based Context Switching                 846011.1 lps   (10.0 s, 7 samples)
Process Creation                              54076.5 lps   (30.0 s, 2 samples)
Shell Scripts (1 concurrent)                  20820.9 lpm   (60.0 s, 2 samples)
Shell Scripts (8 concurrent)                   3192.6 lpm   (60.1 s, 2 samples)
System Call Overhead                        8667731.7 lps   (10.0 s, 7 samples)

System Benchmarks Index Values               BASELINE       RESULT    INDEX
Dhrystone 2 using register variables         116700.0  151547416.3  12986.1
Double-Precision Whetstone                       55.0      14886.9   2706.7
Execl Throughput                                 43.0      15018.6   3492.7
File Copy 1024 bufsize 2000 maxblocks          3960.0     829648.1   2095.1
File Copy 256 bufsize 500 maxblocks            1655.0     217171.6   1312.2
File Copy 4096 bufsize 8000 maxblocks          5800.0    2729918.6   4706.8
Pipe Throughput                               12440.0    6317809.1   5078.6
Pipe-based Context Switching                   4000.0     846011.1   2115.0
Process Creation                                126.0      54076.5   4291.8
Shell Scripts (1 concurrent)                     42.4      20820.9   4910.6
Shell Scripts (8 concurrent)                      6.0       3192.6   5321.0
System Call Overhead                          15000.0    8667731.7   5778.5
                                                                   ========
System Benchmarks Index Score                                        3863.4

[root@bigdata-centos UnixBench]#

准备工作

配置SSH免密

要配置CentOS上的免密登录并将密钥发送到其他机器，可以按照以下步骤进行：

  1. 在bigdata-centos机器上生成SSH密钥对。打开终端并运行以下命令：
[root@bigdata-centos ~]# ssh-keygen -t rsa	#按3次回车
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Created directory '/root/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:MLaVLFiyOmRjJ3awp9Kb3kh2Y53wSf8P3hUojvZyeho root@bigdata-centos
The key's randomart image is:
+---[RSA 2048]----+
|  . . .          |
|   o = . .       |
|  O * = +        |
| * O . *     .   |
|. = . o S . . .  |
| . + = + o .   . |
|  = + = E o   .  |
| + = . ..=oo .   |
|  o .   o*+.o    |
+----[SHA256]-----+
[root@bigdata-centos ~]# ssh-copy-id root@bigdata-centos
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
The authenticity of host 'bigdata-centos (192.168.88.52)' can't be established.
ECDSA key fingerprint is SHA256:jO6jVOMifwO+jzliWqHgSF2LU39q4UtooXWKEWsVVgc.
ECDSA key fingerprint is MD5:78:5d:df:f1:53:79:f3:73:de:ac:9d:be:dc:c4:3c:40.
Are you sure you want to continue connecting (yes/no)? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
root@bigdata-centos's password:

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh 'root@bigdata-centos'"
and check to make sure that only the key(s) you wanted were added.

[root@bigdata-centos ~]# ssh bigdata-centos
Last login: Wed Oct 18 02:09:26 2023 from 192.168.88.1
[root@bigdata-centos ~]# pwd
/root
[root@bigdata-centos ~]# exit
登出
Connection to bigdata-centos closed.
[root@bigdata-centos ~]# pwd
/root
[root@bigdata-centos ~]#

此时SSH免密配置成功。

配置NTP时钟同步

为了使用Kudu，一定要配置好NTP。

[root@bigdata-centos ~]# yum install -y ntp
已加载插件：fastestmirror, langpacks
Loading mirror speeds from cached hostfile
 * base: mirrors.bfsu.edu.cn
 * extras: mirrors.bfsu.edu.cn
 * updates: mirrors.bfsu.edu.cn
软件包 ntp-4.2.6p5-29.el7.centos.2.x86_64 已安装并且是最新版本
无须任何处理
[root@bigdata-centos ~]# vim /etc/ntp.conf
[root@bigdata-centos ~]# cat /etc/ntp.conf
# For more information about this file, see the man pages
# ntp.conf(5), ntp_acc(5), ntp_auth(5), ntp_clock(5), ntp_misc(5), ntp_mon(5).

driftfile /var/lib/ntp/drift

# Permit time synchronization with our time source, but do not
# permit the source to query or modify the service on this system.
#restrict default nomodify notrap nopeer noquery

# Permit all access over the loopback interface.  This could
# be tightened as well, but to do so would effect some of
# the administrative functions.
restrict 127.0.0.1
restrict ::1

# Hosts on local network are less restricted.
#restrict 192.168.1.0 mask 255.255.255.0 nomodify notrap

# Use public servers from the pool.ntp.org project.
# Please consider joining the pool (http://www.pool.ntp.org/join.html).
#server 0.centos.pool.ntp.org iburst
#server 1.centos.pool.ntp.org iburst
#server 2.centos.pool.ntp.org iburst
#server 3.centos.pool.ntp.org iburst

restrict 192.168.88.0 mask 255.255.255.0 nomodify notrap
#server ntp4.aliyun.com
#server 202.112.10.36
server 127.127.1.0 iburst# localclock
fudge 127.127.1.0 stratum 8

#broadcast 192.168.1.255 autokey        # broadcast server
#broadcastclient                        # broadcast client
#broadcast 224.0.1.1 autokey            # multicast server
#multicastclient 224.0.1.1              # multicast client
#manycastserver 239.255.254.254         # manycast server
#manycastclient 239.255.254.254 autokey # manycast client

# Enable public key cryptography.
#crypto

includefile /etc/ntp/crypto/pw

# Key file containing the keys and key identifiers used when operating
# with symmetric key cryptography.
keys /etc/ntp/keys

# Specify the key identifiers which are trusted.
#trustedkey 4 8 42

# Specify the key identifier to use with the ntpdc utility.
#requestkey 8

# Specify the key identifier to use with the ntpq utility.
#controlkey 8

# Enable writing of statistics records.
#statistics clockstats cryptostats loopstats peerstats

# Disable the monitoring facility to prevent amplification attacks using ntpdc
# monlist command when default restrict does not include the noquery flag. See
# CVE-2013-5211 for more details.
# Note: Monitoring will not be disabled with the limited restriction flag.
disable monitor
[root@bigdata-centos ~]# systemctl enable ntpd
Created symlink from /etc/systemd/system/multi-user.target.wants/ntpd.service to /usr/lib/systemd/system/ntpd.service.
[root@bigdata-centos ~]# systemctl start ntpd
[root@bigdata-centos ~]# hwclock --systohc --localtime
[root@bigdata-centos ~]# timedatectl
      Local time: 四 2023-10-19 15:45:54 CST
  Universal time: 四 2023-10-19 07:45:54 UTC
        RTC time: 四 2023-10-19 07:45:54
       Time zone: Asia/Shanghai (CST, +0800)
     NTP enabled: no
NTP synchronized: yes
 RTC in local TZ: yes
      DST active: n/a

Warning: The system is configured to read the RTC time in the local time zone.
         This mode can not be fully supported. It will create various problems
         with time zone changes and daylight saving time adjustments. The RTC
         time is never updated, it relies on external facilities to maintain it.
         If at all possible, use RTC in UTC by calling
         'timedatectl set-local-rtc 0'.
[root@bigdata-centos ~]# vim /etc/ntp.conf
您在 /var/spool/mail/root 中有新邮件
[root@bigdata-centos ~]# vim /etc/ntp.conf
[root@bigdata-centos ~]# systemctl enable ntpd
Created symlink from /etc/systemd/system/multi-user.target.wants/ntpd.service to /usr/lib/systemd/system/ntpd.service.
[root@bigdata-centos ~]# systemctl start ntpd
[root@bigdata-centos ~]# timedatectl
      Local time: 四 2023-10-19 16:09:23 CST
  Universal time: 四 2023-10-19 08:09:23 UTC
        RTC time: 四 2023-10-19 08:09:23
       Time zone: Asia/Shanghai (CST, +0800)
     NTP enabled: yes
NTP synchronized: yes
 RTC in local TZ: yes
      DST active: n/a

Warning: The system is configured to read the RTC time in the local time zone.
         This mode can not be fully supported. It will create various problems
         with time zone changes and daylight saving time adjustments. The RTC
         time is never updated, it relies on external facilities to maintain it.
         If at all possible, use RTC in UTC by calling
         'timedatectl set-local-rtc 0'.
[root@bigdata-centos kudu]# ntpq -p
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
*LOCAL(0)        .LOCL.           8 l    2   64    1    0.000    0.000   0.000
[root@bigdata-centos kudu]# rm -rf /home/kududata

主要是Kudu的时钟比较麻烦，所以单节点可以只配置和本机同步。

配置JDK

[root@bigdata-centos ~]# java -version
openjdk version "1.8.0_332"
OpenJDK Runtime Environment (build 1.8.0_332-b09)
OpenJDK 64-Bit Server VM (build 25.332-b09, mixed mode)
[root@bigdata-centos ~]#

这个版本的Centos7.9本身自带了JDK1.8，可以直接使用。

关闭透明大页

[root@bigdata-centos ~]# cat /sys/kernel/mm/transparent_hugepage/enabled
[always] madvise never
[root@bigdata-centos ~]# cat /sys/kernel/mm/transparent_hugepage/defrag
[always] madvise never
[root@bigdata-centos ~]# cat /etc/rc.local
#!/bin/bash
# THIS FILE IS ADDED FOR COMPATIBILITY PURPOSES
#
# It is highly advisable to create own systemd services or udev rules
# to run scripts during boot instead of using this file.
#
# In contrast to previous versions due to parallel execution during boot
# this script will NOT be run after all other services.
#
# Please note that you must run 'chmod +x /etc/rc.d/rc.local' to ensure
# that this script will be executed during boot.

touch /var/lock/subsys/local
[root@bigdata-centos ~]# cat /sys/kernel/mm/transparent_hugepage/enabled
[always] madvise never
[root@bigdata-centos ~]# cat /etc/default/grub
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX="crashkernel=auto rd.lvm.lv=centos/root rd.lvm.lv=centos/swap rhgb quiet"
GRUB_DISABLE_RECOVERY="true"
[root@bigdata-centos ~]#

显然当前版本的Centos开启了透明大页功能，需要关闭：

[root@bigdata-centos UnixBench]# echo never > /sys/kernel/mm/transparent_hugepage/enabled
[root@bigdata-centos UnixBench]# echo never > /sys/kernel/mm/transparent_hugepage/defrag
[root@bigdata-centos UnixBench]# cat /sys/kernel/mm/transparent_hugepage/enabled
always madvise [never]
[root@bigdata-centos UnixBench]# cat /sys/kernel/mm/transparent_hugepage/enabled
always madvise [never]
[root@bigdata-centos UnixBench]# vim /etc/default/grub
[root@bigdata-centos UnixBench]# cat /etc/default/grub
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX="crashkernel=auto rd.lvm.lv=centos/root rd.lvm.lv=centos/swap rhgb quiet transparent_hugepage=never"
GRUB_DISABLE_RECOVERY="true"
[root@bigdata-centos UnixBench]# cat /sys/kernel/mm/transparent_hugepage/enabled
always madvise [never]

关闭后还需要reboot重启生效。

减少swap

[root@bigdata-centos ~]# sysctl -w vm.swappiness=0
vm.swappiness = 0
[root@bigdata-centos ~]# echo 'vm.swappiness=0' >> /etc/sysctl.conf
[root@bigdata-centos ~]# sysctl -p
vm.swappiness = 0
[root@bigdata-centos ~]#

0表示只有当可⽤物理内存⼩于最⼩阈值vm.min_free_kbytes时才使⽤交换空间。

关闭安全配置

[root@bigdata-centos yum.repos.d]# systemctl disable firewalld
Removed symlink /etc/systemd/system/multi-user.target.wants/firewalld.service.
Removed symlink /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service.
[root@bigdata-centos yum.repos.d]# systemctl stop firewalld
[root@bigdata-centos yum.repos.d]# vim /etc/selinux/config
[root@bigdata-centos yum.repos.d]# cat /etc/selinux/config

# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
#     enforcing - SELinux security policy is enforced.
#     permissive - SELinux prints warnings instead of enforcing.
#     disabled - No SELinux policy is loaded.
SELINUX=disable
# SELINUXTYPE= can take one of three values:
#     targeted - Targeted processes are protected,
#     minimum - Modification of targeted policy. Only selected processes are protected.
#     mls - Multi Level Security protection.
SELINUXTYPE=targeted


[root@bigdata-centos yum.repos.d]#

此时关闭了SELinux与防火墙。

安装Http与Repo

yum install -y httpd
yum install -y createrepo
systemctl start httpd
systemctl enable httpd

# Created symlink from /etc/systemd/system/multi-user.target.wants/httpd.service to /usr/lib/systemd/system/httpd.service.

此时安装好Http与Repo。浏览器打开：

192.168.88.52:80

看到：

在这里插入图片描述

说明Http已经启动。

制作离线Yum源

[root@bigdata-centos yum.repos.d]# mkdir -p /home/bigdata/yum/offline/zhiyong-bigdata-centos
[root@bigdata-centos yum.repos.d]# vi /etc/yum.repos.d/zhiyong-bigdata-centos.repo
[root@bigdata-centos yum.repos.d]# cat /etc/yum.repos.d/zhiyong-bigdata-centos.repo
[offline]
name=zhiyong-bigdata-centos
baseurl=http://192.168.88.52:80/yum/	# file:///home/bigdata/yum/offline/zhiyong-bigdata-centos/
enabled=1
gpgcheck=0
[root@bigdata-centos repodata]# cd /home/bigdata/yum/offline/zhiyong-bigdata-centos
[root@bigdata-centos zhiyong-bigdata-centos]# ll
总用量 3599392
-rw-r--r--. 1 root root     277604 10月 18 03:40 01_mysql-community-common-5.7.16-1.el7.x86_64.rpm
-rw-r--r--. 1 root root    2237116 10月 18 03:40 02_mysql-community-libs-5.7.16-1.el7.x86_64.rpm
-rw-r--r--. 1 root root    2112700 10月 18 03:40 03_mysql-community-libs-compat-5.7.16-1.el7.x86_64.rpm
-rw-r--r--. 1 root root   25034716 10月 18 03:40 04_mysql-community-client-5.7.16-1.el7.x86_64.rpm
-rw-r--r--. 1 root root  159295840 10月 18 03:40 05_mysql-community-server-5.7.16-1.el7.x86_64.rpm
-rw-r--r--. 1 root root      14040 10月 18 03:10 allkeys.asc
-rw-r--r--. 1 root root 2082186246 10月 18 03:10 CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel
-rw-r--r--. 1 root root         40 10月 18 03:10 CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha1
-rw-r--r--. 1 root root         64 10月 18 03:10 CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha256
-rw-r--r--. 1 root root   10483568 10月 18 03:10 cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm
-rw-r--r--. 1 root root 1203832464 10月 18 03:10 cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm
-rw-r--r--. 1 root root      11488 10月 18 03:10 cloudera-manager-server-6.3.1-1466458.el7.x86_64.rpm
-rw-r--r--. 1 root root      10996 10月 18 03:10 cloudera-manager-server-db-2-6.3.1-1466458.el7.x86_64.rpm
-rw-r--r--. 1 root root   14209868 10月 18 03:10 enterprise-debuginfo-6.3.1-1466458.el7.x86_64.rpm
-rw-r--r--. 1 root root      33887 10月 18 03:10 manifest.json
-rw-r--r--. 1 root root    1006956 10月 18 03:40 mysql-connector-java-5.1.48.jar
-rw-r--r--. 1 root root  184988341 10月 18 03:10 oracle-j2sdk1.8-1.8.0+update181-1.x86_64.rpm
drwxr-xr-x. 2 root root       4096 10月 18 03:40 repodata
[root@bigdata-centos zhiyong-bigdata-centos]# createrepo .
[root@bigdata-centos zhiyong-bigdata-centos]# yum clean all
[root@bigdata-centos zhiyong-bigdata-centos]# yum makecache
[root@bigdata-centos zhiyong-bigdata-centos]# mkdir -p /var/www/html/yum
[root@bigdata-centos zhiyong-bigdata-centos]# cp /home/bigdata/yum/offline/zhiyong-bigdata-centos/* /var/www/html/yum

将离线的repo文件使用MobaXterm放置到/home/bigdata/yum/offline/zhiyong-bigdata-centos路径。此时：

http://192.168.88.52/yum/

在这里插入图片描述

已经可以看到这些文件。此时可以离线部署【由于Cloudera关闭了官网的下载链接，目前也只能离线部署】。

放置parcel包

/opt/cloudera/parcel-repo
/opt/cloudera/parcels

将CDH组件的4个包在该路径也放置一份。

安装MySQL

[root@bigdata-centos zhiyong-bigdata-centos]# pwd
/home/bigdata/yum/offline/zhiyong-bigdata-centos

rpm -qa|grep mariadb | xargs -I {} rpm -e --nodeps {}
rpm -ivh 01_mysql-community-common-5.7.16-1.el7.x86_64.rpm
rpm -ivh 02_mysql-community-libs-5.7.16-1.el7.x86_64.rpm
rpm -ivh 03_mysql-community-libs-compat-5.7.16-1.el7.x86_64.rpm
rpm -ivh 04_mysql-community-client-5.7.16-1.el7.x86_64.rpm
rpm -ivh 05_mysql-community-server-5.7.16-1.el7.x86_64.rpm

systemctl start mysqld
systemctl enable mysqld

[root@bigdata-centos zhiyong-bigdata-centos]# grep 'temporary password' /var/log/mysqld.log
2023-10-17T19:46:44.706162Z 1 [Note] A temporary password is generated for root@localhost: #DR2-C#qp1wK
[root@bigdata-centos zhiyong-bigdata-centos]# mysql -uroot -p
#DR2-C#qp1wK
alter user root@localhost identified by 'Zhiyong123#';
flush privileges;

[root@bigdata-centos yum]# pwd
/var/www/html/yum
[root@bigdata-centos yum]# cp mysql-connector-java-5.1.48.jar /usr/share/java/
[root@bigdata-centos java]# pwd
/usr/share/java
[root@bigdata-centos java]# mv mysql-connector-java-5.1.48.jar mysql-connector-java.jar

此时MySQL的密码已经修改。

创建元数据库表

mysql -uroot -pZhiyong123# -e 'create database metastore default character set utf8;'
mysql -uroot -pZhiyong123# -e "create user 'hive'@'%' identified by 'Zhiyong123#'";
mysql -uroot -pZhiyong123# -e 'create database hive default character set utf8';
mysql -uroot -pZhiyong123# -e "grant all privileges on hive.* to 'hive'@'%'";
mysql -uroot -pZhiyong123# -e "grant all privileges on metastore.* to 'hive'@'%'";
mysql -uroot -pZhiyong123# -e "create user 'amon'@'%' identified by 'Zhiyong123#'";
mysql -uroot -pZhiyong123# -e 'create database amon default character set utf8';
mysql -uroot -pZhiyong123# -e "grant all privileges on amon.* to 'amon'@'%'";
mysql -uroot -pZhiyong123# -e "create user 'rman'@'%' identified by 'Zhiyong123#'";
mysql -uroot -pZhiyong123# -e 'create database rman default character set utf8';
mysql -uroot -pZhiyong123# -e "grant all privileges on rman.* to 'rman'@'%'";
mysql -uroot -pZhiyong123# -e "create user 'sentry'@'%' identified by 'Zhiyong123#'";
mysql -uroot -pZhiyong123# -e 'create database sentry default character set utf8';
mysql -uroot -pZhiyong123# -e "grant all privileges on sentry.* to 'sentry'@'%'";
mysql -uroot -pZhiyong123# -e "create user 'nav'@'%' identified by 'Zhiyong123#'";
mysql -uroot -pZhiyong123# -e 'create database nav default character set utf8';
mysql -uroot -pZhiyong123# -e "grant all privileges on nav.* to 'nav'@'%'";
mysql -uroot -pZhiyong123# -e "create user 'navms'@'%' identified by 'Zhiyong123#'";
mysql -uroot -pZhiyong123# -e 'create database navms default character set utf8';
mysql -uroot -pZhiyong123# -e "grant all privileges on navms.* to 'navms'@'%'";
mysql -uroot -pZhiyong123# -e "create user 'cm'@'%' identified by 'Zhiyong123#'";
mysql -uroot -pZhiyong123# -e 'create database cm default character set utf8';
mysql -uroot -pZhiyong123# -e "grant all privileges on cm.* to 'cm'@'%'";
mysql -uroot -pZhiyong123# -e "create user 'oos'@'%' identified by 'Zhiyong123#'";
mysql -uroot -pZhiyong123# -e 'create database oos default character set utf8';
mysql -uroot -pZhiyong123# -e "grant all privileges on oos.* to 'oos'@'%'";
mysql -uroot -pZhiyong123# -e "create user 'hue'@'%' identified by 'Zhiyong123#'";
mysql -uroot -pZhiyong123# -e 'create database hue default character set utf8';
mysql -uroot -pZhiyong123# -e "grant all privileges on hue.* to 'hue'@'%'";

创建成功。此处建库根据需求。

安装CDH

安装CM

yum install cloudera-manager-daemons cloudera-manager-server

初始化CM

[root@bigdata-centos java]# /opt/cloudera/cm/schema/scm_prepare_database.sh mysql cm cm Zhiyong123#
JAVA_HOME=/usr/lib/jvm/jre-openjdk
Verifying that we can write to /etc/cloudera-scm-server
Creating SCM configuration file in /etc/cloudera-scm-server
Executing:  /usr/lib/jvm/jre-openjdk/bin/java -cp /usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/java/postgresql-connector-java.jar:/opt/cloudera/cm/schema/../lib/* com.cloudera.enterprise.dbutil.DbCommandExecutor /etc/cloudera-scm-server/db.properties com.cloudera.cmf.db.
Wed Oct 18 04:53:43 CST 2023 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
[                          main] DbCommandExecutor              INFO  Successfully connected to database.
All done, your SCM database is configured correctly!

此时初始化CM的元数据成功。要注意密码不能有!，可以有#。血的教训。

启动CM

systemctl start cloudera-scm-server.service

稍等片刻即可进入熟悉的CM界面。

安装组件

http://192.168.88.52:7180/cmf/login
admin
admin

在这里插入图片描述

在这里插入图片描述

当然是选择体验版。。。

在这里插入图片描述

未在已配置的存储库中找到任何 parcel。尝试在更多选项下添加一个自定义存储库。否则，您可能只能继续使用包。

还需要：

mv CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha1 CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha
systemctl restart cloudera-scm-server

在这里插入图片描述

重置Hash以后才可以下一步。

在这里插入图片描述

出现无法复制安装文件的情况。。。要取消掉存储库的所有配置项，只保留本地源，这样才能保证不会从已过时的Cloudera官网下载安装包。不然会出现：

/tmp/scm_prepare_node.k8fwJHf3
https://archive.cloudera.com/cm6/6.3.1/allkeys.asc/

类似这种的报错。

此外，由于是单节点，本身是不合理的，所以后续所有检查、告警能跳过的一律跳过。

在这里插入图片描述

在这里插入图片描述

原则就是能凑合着用即可。

后续操作

允许远程登陆MySQL

GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' IDENTIFIED BY 'Zhiyong123#' WITH GRANT OPTION;
FLUSH PRIVILEGES;

这样就可以远程登陆MySQL，方便后续查看Hive的元数据。

配置Jdbc连接

在这里插入图片描述

找一个Impala的JDBC41驱动包，传入url：

jdbc:impala://192.168.88.52:21050

后续就可以用这种方式访问Impala，间接操作Kudu和Hive，从而忽视Hue不正常的问题。

调度工具Oozie已经要淘汰了，开发环境也没有研究的必要，直接停掉。

FRP端口暴露

使用路由侠，暴露Impala的端口给外部访问。这样就可以让别的只会写Sql的Sql Boy用公网的方式连接到Impala：

jdbc:impala://zhiyong.xxxx这里是具体的网址xxxxxxx.luyouxia.net:11701
jdbc:impala://zhiyong.e3.luyouxia.net:11701

这种方式比租一个ECS要便宜些。延时90ms左右。

此时完成了CDH6.3.2单节点部署的主要操作。

Kudu无法启动问题

出现Kudu无法启动的问题，reboot重置NTP：

[root@bigdata-centos ~]# systemctl start ntpd
[root@bigdata-centos ~]# systemctl enable ntpd
Created symlink from /etc/systemd/system/multi-user.target.wants/ntpd.service to /usr/lib/systemd/system/ntpd.service.
[root@bigdata-centos ~]# ntpstat
synchronised to local net (127.127.1.0) at stratum 11
   time correct to within 7948 ms
   polling server every 64 s

之后：

gflagfile 的 Tablet Server 高级配置代码段（安全阀）	--max_clock_sync_error_usec=20000000
gflagfile 的 Master 高级配置代码段（安全阀）	--max_clock_sync_error_usec=20000000
block_cache_capacity_mb=1G
memory_limit_hard_bytes=8G

即可解决如下报错：

E1019 19:13:24.295385 15847 block_cache.cc:97] Block cache capacity exceeds the memory pressure threshold (8589934592 bytes vs. 5153960755 bytes). This will cause instability and harmful flushing behavior. Lower --block_cache_capacity_mb or raise --memory_limit_hard_bytes.

CM的报错在如下Log查看：

/var/log/cloudera-scm-server

此时可以正常使用。

也可以在Idea中使用JDBC方式访问：

package com.zhiyong.kudu;

import java.sql.*;
import java.util.ArrayList;
import java.util.LinkedHashMap;
import java.util.Map;

public class KuduImpalaJdbcDemo {
    public static void main(String[] args) throws ClassNotFoundException, SQLException {
        System.out.println("使用Impala操作Kudu");
        Class.forName("com.cloudera.impala.jdbc41.Driver");
        String url = "jdbc://192.168.88.52:21050";

        Connection connection = DriverManager.getConnection(url);

        ArrayList<Map> result = new ArrayList<>();

        Statement statement = connection.createStatement();
        ResultSet resultSet;
        ResultSetMetaData metaData;
        String sql = "select * from table_name ";

        try {

            resultSet = statement.executeQuery(sql);//结果集
            metaData = resultSet.getMetaData();//元数据集

            while (resultSet.next()){
                LinkedHashMap<Object, Object> map = new LinkedHashMap<>();
                for (int i = 0; i < metaData.getColumnCount(); i++) {
                    map.put(metaData.getColumnName(i+1),resultSet.getObject(metaData.getColumnName(i+1)));
                }
                result.add(map);
            }
        } catch (SQLException e) {
            e.printStackTrace();
        } finally {
            try {
                statement.close();
                connection.close();

            } catch (Exception e){
                e.printStackTrace();
            }
        }

        System.out.println("结束");

    }
}

基本搭建完成。

转载请注明出处：https://lizhiyong.blog.csdn.net/article/details/134433148

在这里插入图片描述

本文来自互联网用户投稿，该文观点仅代表作者本人，不代表本站立场。本站仅提供信息存储空间服务，不拥有所有权，不承担相关法律责任。如若转载，请注明出处：http://www.coloradmin.cn/o/1220179.html

如若内容造成侵权/违法违规/事实不符，请联系多彩编程网进行投诉反馈，一经查实，立即删除！

相关文章

实践小记——C#科学计数法格式化输出

实践小记——C#科学计数法格式化输出

文章速览 示例默认输出&#xff0c;不设置小数精度设置尾数部分的小数精度 总结参考文章 坚持记录实属不易&#xff0c;希望友善多金的码友能够随手点一个赞。 共同创建氛围更加良好的开发者社区&#xff01; 谢谢~ 示例 默认输出&#xff0c;不设置小数精度 private void Fo…
阅读更多...
设计模式之桥接模式--连接抽象与实现（你想知道的问题都有）

设计模式之桥接模式--连接抽象与实现（你想知道的问题都有）

目录 概述结构型设计模式桥接模式的定义桥接模式的角色和关系 版本迭代紧耦合版增加品牌两个品牌两款软件松耦合的设计版本迭代业务分析总结 问题升华抽象与实现抽象包含的一些方法或属性依赖于实现部分的接口关联关系与桥接模式桥接模式适合情况谁是实现&#xff0c;谁是抽象组…
阅读更多...
vscode设置latex

vscode设置latex

vscode配置latex 1.安装vscode,并添加环境变量路径 2.安装latex,bin文件夹添加到环境变量路径 3.vscode安装插件 4.vscode->文件->首选项->显示配置内容->setting.json文件&#xff0c;查看其位置目录&#xff0c;通过我的电脑找到此文件&#xff08;不要使用v…
阅读更多...
torch - 张量Tensor常见的形式

torch - 张量Tensor常见的形式

1.Scalar 通常就是一个数值 x tensor(42.) 输出x&#xff1a; 2.Vector 特征向量 例如&#xff1a;[-5., 2., 0.]在深度学习中通常表示特征&#xff0c;如词向量特征&#xff0c;某一维度特征等 3.Matrix 一般计算的都是矩阵&#xff0c;通常都是多维的。 可以做矩阵的…
阅读更多...
C# +.Net检验科信息管理系统源码 LIS系统源码

C# +.Net检验科信息管理系统源码 LIS系统源码

检验科信息管理系统&#xff08;LIS&#xff09; LIS系统集申请、采样、核收、计费、检验、审核、发布、质控、查询、耗材控制等检验科工作为一体的网络管理系统。它的开发和应用将加快检验科管理的统一化、网络化、标准化的进程。 主要包括以下功能&#xff1a; 1、数据采集…
阅读更多...
【Python】集合与字典

【Python】集合与字典

按照输入顺序输出 将输入的名字去重&#xff0c;同时按照输入顺序输出 sinput().split(,) blist(set(s)) bsorted(b,keys.index) print(b) 删除集合元素、更新集合 根据操作删除更新集合 update括号里可以是一个集合&#xff0c;add只能是一个元素 discard用于删除元素&#x…
阅读更多...
Fedora 项目近日发布了 Fedora Linux 39

Fedora 项目近日发布了 Fedora Linux 39

导读几经推迟之后&#xff0c;Fedora 项目近日发布了 Fedora Linux 39&#xff0c;这是红帽公司赞助的面向大众的 GNU/Linux 发行版的最新稳定版本&#xff0c;采用了最新的技术和开源应用程序。 Fedora Linux 39 由 Linux 内核 6.5 支持&#xff0c;并提供了一些最新的桌面环境…
阅读更多...
APP安全加固怎么做？加固技术、加固方法、加固方案

APP安全加固怎么做？加固技术、加固方法、加固方案

前面的文章中我们为大家介绍了移动应用安全检测的测试依据、测试方法、和测试内容&#xff0c;本文我们着重分享App安全加固的相关内容。 &#xff08;安全检测内容&#xff09; 通过前面的文章我们知道了app安全检测要去检测哪些内容&#xff0c;发现问题后我们如何去修复&am…
阅读更多...
mfc140u.dll丢失的解决方法，以及针对每个解决mfc140u.dll丢失办法的优缺点

mfc140u.dll丢失的解决方法，以及针对每个解决mfc140u.dll丢失办法的优缺点

在使用电脑的过程中&#xff0c;有时会遇到一些与动态链接库文件&#xff08;DLL&#xff09;相关的错误。其中&#xff0c;mfc140u.dll丢失是一种常见的问题&#xff0c;它可能导致应用程序无法正常运行。在本文中&#xff0c;我们将探讨关于mfc140u.dll丢失的解决办法&#x…
阅读更多...
1.数字化转型概述

1.数字化转型概述

中台数字化转型的问题 在中台数字化转型过程中&#xff0c;如何进行业务领域边界划分&#xff0c;如何完成中台领域建模实现能力复用&#xff0c;如何完成单体应用拆分利微服务设计&#xff0c;如何实现前中后台的协同设计; DDD解决的问题 DDD首先从业务领域人手&#xff0c…
阅读更多...
linux网络编程之TCP协议编程

linux网络编程之TCP协议编程

Linux网络编程之TCP协议编程 tcp协议编程模型socket函数sockaddr_inbindlistenconnect 应用服务端代码客服端代码 TCP协议编程) tcp协议编程模型 Server 1.创建socket (socket函数) 2.确定服务器协议地址簇 (struct sockaddr) 3.绑定 (bind) 4.监听 ( listen) 5.接受客户端连接…
阅读更多...
人脸识别4G执法记录仪、一体化智能AI布控球在智慧社区、智能网格中的应用

人脸识别4G执法记录仪、一体化智能AI布控球在智慧社区、智能网格中的应用

智慧社区守护者&#xff1a;人脸识别与智能监控技术的融合创新 随着城市的飞速发展和科技的不断进步&#xff0c;智慧社区和智能网格的概念已经成为现代城市管理的一个重要趋势。在这一过程中&#xff0c;人脸识别技术、4G执法记录仪以及一体化智能AI布控球等智能监控设备&…
阅读更多...
字节跳动小程序开发：探索创新的数字化世界

字节跳动小程序开发：探索创新的数字化世界

在数字化时代&#xff0c;字节跳动小程序开发成为企业数字化转型的关键一环。通过这一平台&#xff0c;企业能够借助先进的技术和丰富的功能&#xff0c;实现创新、引领市场潮流。本文将通过一些简单的技术代码示例&#xff0c;带你深入了解字节跳动小程序开发的魅力。 1. 小…
阅读更多...
无需公网IP，使用MCSM面板一键搭建我的世界Minecraft服务器联机游戏

无需公网IP，使用MCSM面板一键搭建我的世界Minecraft服务器联机游戏

文章目录 前言1.Mcsmanager安装2.创建Minecraft服务器3.本地测试联机4. 内网穿透4.1 安装cpolar内网穿透4.2 创建隧道映射内网端口 5.远程联机测试6. 配置固定远程联机端口地址6.1 保留一个固定TCP地址6.2 配置固定TCP地址 7. 使用固定公网地址远程联机 前言 MCSManager是一个…
阅读更多...
初识Scrapy：Python中的网页抓取神器

初识Scrapy：Python中的网页抓取神器

Scrapy是一个基于Python的快速、高层次的屏幕抓取和web抓取框架&#xff0c;用于抓取web站点并从页面中提取结构化的数据。它广泛应用于数据挖掘、监测和自动化测试等领域。Scrapy的强大之处在于它是一个框架&#xff0c;可以根据实际需求进行修改和扩展。 Scrapy的主要特点 …
阅读更多...
智慧物流追踪：打造未来的物流网络

智慧物流追踪：打造未来的物流网络

随着互联网和物流行业的深度融合&#xff0c;智慧物流已成为现代物流发展的新趋势。通过开发一款智能化的物流追踪app小程序&#xff0c;我们不仅可以提高物流效率&#xff0c;还可以为客户提供更加便捷的服务。本文将从市场需求、技术应用、竞争优势、行业前景等方面对智慧物流…
阅读更多...
11月24日 AI+软件研发数字峰会（AiDD）即将启航！

11月24日 AI+软件研发数字峰会（AiDD）即将启航！

▼ 伴随着人工智能&#xff08;AI&#xff0c;特别是大语言模型&#xff09;在众多行业领域的广泛应用及其带来的颠覆性变革&#xff0c;软件的开发模式、方式和实践都可能会发生巨大的变化。为助力更多企业在人工智能的浪潮中乘风破浪&#xff0c;“AI软件研发数字峰会&#x…
阅读更多...
如何快速下载微信视频号的视频？简单几步轻松搞定

如何快速下载微信视频号的视频？简单几步轻松搞定

在当今社交媒体充满活力的时代&#xff0c;微信视频号已经成为许多用户分享生活、创意和知识的新舞台。然而&#xff0c;随着互联网信息爆炸式增长&#xff0c;有时我们可能会错过一些精彩的视频内容。这时&#xff0c;如何快速下载微信视频号的视频成为了许多用户关心的话题。…
阅读更多...
企业数字化过程中数据仓库与商业智能的目标

企业数字化过程中数据仓库与商业智能的目标

当前环境下&#xff0c;各领域企业通过数字化相关的一切技术&#xff0c;以数据为基础、以用户为核心&#xff0c;创建一种新的&#xff0c;或对现有商业模式进行重塑就是数字化转型。这种数字化转型给企业带来的效果就像是一次重构&#xff0c;会对企业的业务流程、思维文化、…
阅读更多...
TCP-Modbus通信过程解析与实例演示

TCP-Modbus通信过程解析与实例演示

大家好&#xff01;今天我将简要介绍一下如何使用TCP-Modbus方式与信捷PLC&#xff08;XD5E型号&#xff09;进行通讯&#xff0c;并演示整个过程。 TCP-Modbus通信过程解析与实例演示 首先&#xff0c;我们需要在PLC软件中设置相应的IP地址。PLC作为服务器&#xff0c;我们将其…
阅读更多...
推荐文章
最新文章

Copyright @ 2022~2023