文章目录
- 前言
- 📣 1.前言概述
- 📣 2.安全设置
- 📣 3.ClickHouse安装
- ✨ 3.1 申请服务器
- ✨ 3.2 安装前准备
- ✨ 3.3 RPM安装包
- ✨ 3.4 配置文件
- ✨ 3.5 使用ClickHouse
- 📣 4.ClickHouse压测
- ✨ 4.1 下载数据
- ✨ 4.2 解压数据
- ✨ 4.3 创建数据库和表
- ✨ 4.4 插入数据
- ✨ 4.5 插入数据
前言
在上期对文章中，带大家通过华为云云耀云服务器L进行Docker的部署及应用，本次给大家首先介绍在使用华为云云耀云服务器L时，当您需要对帐号的安全信息进行设置时，可以通过"安全设置",进行相关操作，并对ClickHouse部署及压测。往期回顾：
1.华为云云耀云服务器L实例评测｜Ubuntu云服务器申请使用
2.华为云云耀云服务器L实例评测｜Ubuntu系统MySQL 8.1.0 Innovation压测
3.华为云云耀云服务器L实例评测｜Docker部署及应用
📣 1.前言概述
在上期对文章中，带大家通过华为云云耀云服务器L进行Docker的部署及应用，容器是用镜像创建的运行实例。它可以被启动、开始、停止、删除。每个容器都是相互隔离的、保证安全的平台。可以把容器看做是一个简易版的 Linux 环境。本次给大家首先介绍在使用华为云云耀云服务器L时，当您需要对帐号的安全信息进行设置时，可以通过"安全设置",进行相关操作。
📣 2.安全设置
当您需要对帐号的安全信息进行设置时，可以通过“安全设置”进行操作。“安全设置”包括“基本信息”、“敏感操作”、“登录验证策略”、“密码策略”、“访问控制”。
如何进入安全设置?所有用户均可通过控制台入口进入“安全设置”,登录华为云，在右上角单击“控制台”,在“控制台”页面，鼠标移动至右上方的用户名，在下拉列表中选择“安全设置”。
📣 3.ClickHouse安装
clickhouse简介：
ClickHouse 是俄罗斯的 Yandex 于 2016 年开源的用于在线分析处理查询（OLAP :Online Analytical Processing）MPP架构的列式存储数据库（DBMS：Database Management System），能够使用 SQL 查询实时生成分析数据报告。ClickHouse的全称是Click Stream，Data WareHouse。
clickhouse可以做用户行为分析，流批一体
线性扩展和可靠性保障能够原生支持 shard + replication
clickhouse没有走hadoop生态，采用 Local attached storage 作为存储。
✨ 3.1 申请服务器
##登录华为云云耀云服务器L实例，通过Docker容器拉取镜像，创建容器，拥有自己的云服务器Centos7.8
1)拉取镜像
docker pull daocloud.io/library/centos:centos7.8.2003
root@hcss-ecs-7c99:~# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
hello-world latest 9c7a54a9a43c 5 months ago 13.3kB
daocloud.io/library/centos centos7.8.2003 afb6fca791e0 3 years ago 203MB
2）创建容器
docker run -d --name centos7.8 -h centos7.8
-p 220:22 -p 3387:3389
–privileged=true
daocloud.io/library/centos:centos7.8.2003 /usr/sbin/init
3)进入容器
root@hcss-ecs-7c99:~# docker exec -it centos7.8 bash
[root@centos7 /]# cat /etc/redhat-release
CentOS Linux release 7.8.2003 (Core)
[root@centos7 /]# uname -r
5.15.0-60-generic
[root@centos7 /]# uname -a
Linux centos7.8 5.15.0-60-generic #66-Ubuntu SMP Fri Jan 20 14:29:49 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
✨ 3.2 安装前准备
1)指令集检查
ClickHouse可以在任何具有x86_64，AArch64或PowerPC64LE CPU架构的Linux，FreeBSD或Mac OS X上运行。
官方预构建的二进制文件通常针对x86_64进行编译，并利用SSE 4.2指令集。
检查当前CPU是否支持SSE 4.2的命令:
[root@centos7 /]# grep -q sse4_2 /proc/cpuinfo && echo “SSE 4.2 supported” || echo “SSE 4.2 not supported”
SSE 4.2 supported
2）文件数限制修改
调整CentOS系统对打开文件数的限制，在/etc/security/limits.conf和/etc/security/limits.d/20-nproc.conf文件的末尾加上
soft nofile 65536
hard nofile 65536
soft nproc 131072
hard nproc 131072
cat >> /etc/security/limits.conf <<EOF
soft nofile 65536
hard nofile 65536
soft nproc 131072
hard nproc 131072
EOF
cat >> /etc/security/limits.d/20-nproc.conf <<EOF
soft nofile 65536
hard nofile 65536
soft nproc 131072
hard nproc 131072
EOF
3） SELinux 设置
vi /etc/selinux/config
#修改
SELINUX=disabled
✨ 3.3 RPM安装包
推荐使用CentOS、RedHat和所有其他基于rpm的Linux发行版的官方预编译rpm包。
##安装依赖
yum install -y yum-utils
yum install -y libtool
yum install -y unixODBC
##需要添加官方存储库：
yum-config-manager --add-repo https://packages.clickhouse.com/rpm/clickhouse.repo
##安装
yum install -y clickhouse-server clickhouse-client
✨ 3.4 配置文件
修改配置文件，把 <listen_host>::</listen_host> 的注释打开，这样的话才能让ClickHouse被除本机以外的服务器访问
vi /etc/clickhouse-server/config.xml
cat >> /etc/clickhouse-server/config.xml <<EOF
<listen_host>::</listen_host>
EOF
在这个文件中，有ClickHouse的一些默认路径配置，比较重要的
数据文件路径：/var/lib/clickhouse/
日志文件路径：/var/log/clickhouse-server/clickhouse-server.log
✨ 3.5 使用ClickHouse
sudo /etc/init.d/clickhouse-server start
clickhouse-client # or "clickhouse-client --password" if you set up a password.
[root@centos7 /]# /etc/init.d/clickhouse-server start
chown -R clickhouse: '/var/run/clickhouse-server/'
Will run sudo --preserve-env -u 'clickhouse' /usr/bin/clickhouse-server --config-file /etc/clickhouse-server/config.xml --pid-file /var/run/clickhouse-server/clickhouse-server.pid --daemon
/bin/sh: sudo: command not found
Code: 302. DB::Exception: Child process was exited with return code 127. (CHILD_WAS_NOT_EXITED_NORMALLY) (version 23.9.1.1854 (official build))
此处解决的办法是：
[root@centos7 /]# yum insatll sudo
sudo /etc/init.d/clickhouse-server start
/etc/init.d/clickhouse-server status
##客户端登录
[root@centos7 /]# clickhouse-client
ClickHouse client version 23.9.1.1854 (official build).
Connecting to localhost:9000 as user default.
Connected to ClickHouse server version 23.9.1 revision 54466.
Warnings:
* Linux threads max count is too low. Check /proc/sys/kernel/threads-max
* Available memory at server startup is too low (2GiB).
centos7.8 :)
centos7.8 :) show databases;
SHOW DATABASES
Query id: 24cfdcc2-4e5a-46d1-922d-135cf67eb143
┌─name───────────────┐
│ INFORMATION_SCHEMA │
│ default │
│ information_schema │
│ system │
└────────────────────┘
4 rows in set. Elapsed: 0.001 sec.
##比较常用的完整命令
clickhouse-client -u root --password 123456 --port 9001 -h 127.0.0.1
📣 4.ClickHouse压测
✨ 4.1 下载数据
[root@centos7 /]# yum install -y wget
[root@centos7 /]# wget https://datasets.clickhouse.com/mgbench{1…3}.csv.xz
✨ 4.2 解压数据
[root@centos7 /]# xz -v -d mgbench{1…3}.csv.xz
mgbench1.csv.xz (1/3)
100 % 543.0 MiB / 7206.5 MiB = 0.075 98 MiB/s 1:13
mgbench2.csv.xz (2/3)
100 % 382.7 MiB / 5808.7 MiB = 0.066 99 MiB/s 0:58
mgbench3.csv.xz (3/3)
100 % 485.0 MiB / 7881.7 MiB = 0.062 100 MiB/s 1:19
✨ 4.3 创建数据库和表
[root@centos7 /]# clickhouse-client
centos7.8 :) CREATE DATABASE mgbench;
centos7.8 :) USE mgbench;
CREATE TABLE mgbench.logs1 (
log_time DateTime,
machine_name LowCardinality(String),
machine_group LowCardinality(String),
cpu_idle Nullable(Float32),
cpu_nice Nullable(Float32),
cpu_system Nullable(Float32),
cpu_user Nullable(Float32),
cpu_wio Nullable(Float32),
disk_free Nullable(Float32),
disk_total Nullable(Float32),
part_max_used Nullable(Float32),
load_fifteen Nullable(Float32),
load_five Nullable(Float32),
load_one Nullable(Float32),
mem_buffers Nullable(Float32),
mem_cached Nullable(Float32),
mem_free Nullable(Float32),
mem_shared Nullable(Float32),
swap_free Nullable(Float32),
bytes_in Nullable(Float32),
bytes_out Nullable(Float32)
)
ENGINE = MergeTree()
ORDER BY (machine_group, machine_name, log_time);
CREATE TABLE mgbench.logs2 (
log_time DateTime,
client_ip IPv4,
request String,
status_code UInt16,
object_size UInt64
)
ENGINE = MergeTree()
ORDER BY log_time;
CREATE TABLE mgbench.logs3 (
log_time DateTime64,
device_id FixedString(15),
device_name LowCardinality(String),
device_type LowCardinality(String),
device_floor UInt8,
event_type LowCardinality(String),
event_unit FixedString(1),
event_value Nullable(Float32)
)
ENGINE = MergeTree()
ORDER BY (event_type, log_time);
✨ 4.4 插入数据
clickhouse-client --query “INSERT INTO mgbench.logs1 FORMAT CSVWithNames” < mgbench1.csv
clickhouse-client --query “INSERT INTO mgbench.logs2 FORMAT CSVWithNames” < mgbench2.csv
clickhouse-client --query “INSERT INTO mgbench.logs3 FORMAT CSVWithNames” < mgbench3.csv
✨ 4.5 插入数据
Q1.1: 自午夜以来每个 Web 服务器的 CPU/网络利用率是多少？
USE mgbench;
SELECT machine_name,
MIN(cpu) AS cpu_min,
MAX(cpu) AS cpu_max,
AVG(cpu) AS cpu_avg,
MIN(net_in) AS net_in_min,
MAX(net_in) AS net_in_max,
AVG(net_in) AS net_in_avg,
MIN(net_out) AS net_out_min,
MAX(net_out) AS net_out_max,
AVG(net_out) AS net_out_avg
FROM (
SELECT machine_name,
COALESCE(cpu_user, 0.0) AS cpu,
COALESCE(bytes_in, 0.0) AS net_in,
COALESCE(bytes_out, 0.0) AS net_out
FROM logs1
WHERE machine_name IN ('anansi','aragog','urd')
AND log_time >= TIMESTAMP '2017-01-11 00:00:00'
) AS r
GROUP BY machine_name;
-- Q2：过去一个月顶级请求的平均路径深度是多少？
SELECT top_level,
AVG(LENGTH(request) - LENGTH(REPLACE(request, '/', ''))) AS depth_avg
FROM (
SELECT SUBSTRING(request FROM 1 FOR len) AS top_level,
request
FROM (
SELECT POSITION(SUBSTRING(request FROM 2), '/') AS len,
request
FROM logs2
WHERE status_code >= 200
AND status_code < 300
AND log_time >= TIMESTAMP '2012-12-01 00:00:00'
) AS r
WHERE len > 0
) AS s
WHERE top_level IN ('/about','/courses','/degrees','/events',
'/grad','/industry','/news','/people',
'/publications','/research','/teaching','/ugrad')
GROUP BY top_level
ORDER BY top_level;
-- Q3：对于每种类别的设备，每月的功耗指标是什么？
SELECT yr,
mo,
SUM(coffee_hourly_avg) AS coffee_monthly_sum,
AVG(coffee_hourly_avg) AS coffee_monthly_avg,
SUM(printer_hourly_avg) AS printer_monthly_sum,
AVG(printer_hourly_avg) AS printer_monthly_avg,
SUM(projector_hourly_avg) AS projector_monthly_sum,
AVG(projector_hourly_avg) AS projector_monthly_avg,
SUM(vending_hourly_avg) AS vending_monthly_sum,
AVG(vending_hourly_avg) AS vending_monthly_avg
FROM (
SELECT dt,
yr,
mo,
hr,
AVG(coffee) AS coffee_hourly_avg,
AVG(printer) AS printer_hourly_avg,
AVG(projector) AS projector_hourly_avg,
AVG(vending) AS vending_hourly_avg
FROM (
SELECT CAST(log_time AS DATE) AS dt,
EXTRACT(YEAR FROM log_time) AS yr,
EXTRACT(MONTH FROM log_time) AS mo,
EXTRACT(HOUR FROM log_time) AS hr,
CASE WHEN device_name LIKE 'coffee%' THEN event_value END AS coffee,
CASE WHEN device_name LIKE 'printer%' THEN event_value END AS printer,
CASE WHEN device_name LIKE 'projector%' THEN event_value END AS projector,
CASE WHEN device_name LIKE 'vending%' THEN event_value END AS vending
FROM logs3
WHERE device_type = 'meter'
) AS r
GROUP BY dt,
yr,
mo,
hr
) AS s
GROUP BY yr,
mo
ORDER BY yr,
mo;
