clickhouse MPPDB数据库--新特性使用示例

news2025/1/21 7:16:53

clickhouse 新特性：

从clickhouse 22.3至最新的版本24.3.2.23，clickhouse在快速发展中，每个版本都增加了一些新的特性，在数据写入、查询方面都有性能加速。
本文根据clickhouse blog中的clickhouse release blog中，学习并梳理了一些在实际工作中可能用到的新特性。

以下是如何基于docker，如果试用这些新性

docker run -d --name=ch -p 8123:8123 -p 9000:9000 -p 9009:9009 --ulimit nofile=262144:262144 -v D:/ch/latest/external:/external:rw -v  chlatest:/var/lib/clickhouse:rw -v D:/ch/latest/logs:/var/log/clickhouse-server:rw -v D:/ch/latest/etc/clickhouse-server:/etc/clickhouse-server:rw clickhouse/clickhouse-server:24.3.2.23

docker exec -it bash

clickhouse-client --format_csv_delimiter=','

transform函数

进行字典替换

transform(x, array_from, array_to, default)
transform(T, Array(T), Array(U), U) -> U
transform(x, array_from, array_to)

UK-house-price-dataset.csv

CREATE TABLE uk_price_paid
(
	price UInt32,
	date Date,
	postcode1 LowCardinality(String),
	postcode2 LowCardinality(String),
	type Enum8('terraced' = 1, 'semi-detached' = 2, 'detached' = 3, 'flat' = 4, 'other' = 0),
	is_new UInt8,
	duration Enum8('freehold' = 1, 'leasehold' = 2, 'unknown' = 0),
	addr1 String,
	addr2 String,
	street LowCardinality(String),
	locality LowCardinality(String),
	town LowCardinality(String),
	district LowCardinality(String),
	county LowCardinality(String)
)
ENGINE = MergeTree
ORDER BY (postcode1, postcode2, addr1, addr2);

INSERT INTO uk_price_paid
WITH
   splitByChar(' ', postcode) AS p
SELECT
	toUInt32(price_string) AS price,
	parseDateTimeBestEffortUS(time) AS date,
	p[1] AS postcode1,
	p[2] AS postcode2,
	transform(a, ['T', 'S', 'D', 'F', 'O'], ['terraced', 'semi-detached', 'detached', 'flat', 'other']) AS type,
	b = 'Y' AS is_new,
	transform(c, ['F', 'L', 'U'], ['freehold', 'leasehold', 'unknown']) AS duration, addr1, addr2, street, locality, town, district, county
FROM file('UK-house-price-dataset.csv','CSV','uuid_string String, price_string String, time String, postcode String, a String, b String, c String, addr1 String, addr2 String, street String, locality String, town String, district String, county String, d String, e String'
);


SELECT transform(number, [0, 1, 2, 3, 4, 5, 6, 7, 8, 9], ['zero', 'one', 'two', 'three', 'four', 'five', 'six', 'seven', 'eight', 'nine'], NULL) AS numbers
FROM system.numbers
LIMIT 10

读取文件

可以自动识别文件的类型，推荐字段类型

SELECT * FROM (
WITH
   splitByChar(' ', postcode) AS p
SELECT
	toUInt32(price_string) AS price,
	parseDateTimeBestEffortUS(time) AS date,
	p[1] AS postcode1,
	p[2] AS postcode2,
	transform(a, ['T', 'S', 'D', 'F', 'O'], ['terraced', 'semi-detached', 'detached', 'flat', 'other']) AS type,
	b = 'Y' AS is_new,
	transform(c, ['F', 'L', 'U'], ['freehold', 'leasehold', 'unknown']) AS duration, addr1, addr2, street, locality, town, district, county
FROM file('UK-house-price-dataset.csv','CSV','uuid_string String, price_string String, time String, postcode String, a String, b String, c String, addr1 String, addr2 String, street String, locality String, town String, district String, county String, d String, e String'
) SETTINGS format_csv_delimiter=','
) LIMIT 2;

自定义函数

根据需要，编写自定义函数

CREATE OR REPLACE TABLE line_changes
(
    version UInt32,
    line_change_type Enum('Add' = 1, 'Delete' = 2, 'Modify' = 3),
    line_number UInt32,
    line_content String,
    time datetime default now()
)
ENGINE = MergeTree
ORDER BY time;

INSERT INTO default.line_changes (version,line_change_type,line_number,line_content) VALUES
(1, 'Add'   , 1, 'ClickHouse provides SQL'),
(2, 'Add'   , 2, 'with improvements'),
(3, 'Add'   , 3, 'that makes it more friendly for analytical tasks.'),
(4, 'Add'   , 2, 'with many extensions'),
(5, 'Modify', 3, 'and powerful improvements'),
(6, 'Delete', 1, ''),
(7, 'Add'   , 1, 'ClickHouse provides a superset of SQL');

-- add a string (str) into an array (arr) at a specific position (pos)
CREATE OR REPLACE FUNCTION add AS (arr, pos, str) -> 
  arrayConcat(arraySlice(arr, 1, pos-1), [str], arraySlice(arr, pos));
  
-- delete the element at a specific position (pos) from an array (arr)
CREATE OR REPLACE FUNCTION delete AS (arr, pos) -> 
  arrayConcat(arraySlice(arr, 1, pos-1), arraySlice(arr, pos+1));

-- replace the element at a specific position (pos) in an array (arr)
CREATE OR REPLACE FUNCTION modify AS (arr, pos, str) -> 
  arrayConcat(arraySlice(arr, 1, pos-1), [str], arraySlice(arr, pos+1));

arrayFold

SELECT arrayFold((acc, v) -> (acc + v), [10, 20, 30],  0::UInt64) AS sum;


CREATE OR REPLACE VIEW text_version AS
WITH T1 AS (
  SELECT arrayZip(
    groupArray(line_change_type),
    groupArray(line_number),
    groupArray(line_content)) as line_ops
  FROM (SELECT * FROM line_changes 
        WHERE version <= {version:UInt32} ORDER BY version ASC)
)
SELECT arrayJoin(
  arrayFold((acc, v) -> 
    if(v.'change_type' = 'Add',       add(acc, v.'line_nr', v.'content'),
    if(v.'change_type' = 'Delete', delete(acc, v.'line_nr'),
    if(v.'change_type' = 'Modify', modify(acc, v.'line_nr', v.'content'), []))),
    line_ops::Array(Tuple(change_type String, line_nr UInt32, content String)),
    []::Array(String))) as lines
FROM T1;

SELECT * FROM text_version(version = 3);

Parallel window functions

窗口函数采用并行计算，性能大幅提升

SELECT
	country,
	day,
	max(tempAvg) AS temperature,
	avg(temperature) OVER (PARTITION BY country ORDER BY day ASC ROWS BETWEEN 5 PRECEDING AND CURRENT ROW) AS moving_avg_temp
FROM noaa
WHERE country != ''
GROUP BY
	country,
	date AS day
ORDER BY
	country ASC,
	day ASC

FINAL

基于FINAL及enable_vertical_final,在如下引擎
ReplacingMergeTree、 AggregatingMergeTree引擎中，可以快速查询到最新的数据

SELECT
    postcode1,
    formatReadableQuantity(avg(price))
FROM uk_property_offers FINAL
GROUP BY postcode1
ORDER BY avg(price) DESC
LIMIT 3;

SELECT
    postcode1,
    formatReadableQuantity(avg(price))
FROM uk_property_offers
GROUP BY postcode1
ORDER BY avg(price) DESC
LIMIT 3
SETTINGS enable_vertical_final = 1;

Variant Type

SET allow_experimental_variant_type=1, 
   use_variant_as_common_type = 1;
   
SELECT
    map('Hello', 1, 'World', 'Mark') AS x,
    toTypeName(x) AS type
FORMAT Vertical;


SELECT
    arrayJoin([1, true, 3.4, 'Mark']) AS value,
    toTypeName(value)

Row 1:
──────
x:    {'Hello':1,'World':'Mark'}
type: Map(String, Variant(String, UInt8))


   ┌─value─┬─toTypeName(value)─────────────────────┐
1. │ true  │ Variant(Bool, Float64, String, UInt8) │
2. │ true  │ Variant(Bool, Float64, String, UInt8) │
3. │ 3.4   │ Variant(Bool, Float64, String, UInt8) │
4. │ Mark  │ Variant(Bool, Float64, String, UInt8) │
   └───────┴───────────────────────────────────────┘

字符相似性函数

  • byteHammingDistance： the Hamming distance between two strings or vectors of equal length is the number of positions at which the corresponding symbols are different. In other words, it measures the minimum number of substitutions required to change one string into the other, or equivalently, the minimum number of errors that could have transformed one string into the other. In a more general context, the Hamming distance is one of several string metrics for measuring the edit distance between two sequences. It is named after the American mathematician Richard Hamming.

    • “karolin” and “kathrin” is 3.
    • “karolin” and “kerstin” is 3.
    • “kathrin” and “kerstin” is 4.
    • 0000 and 1111 is 4.
    • 2173896 and 2233796 is 3.

  • editDistance：a way of quantifying how dissimilar two strings (e.g., words) are to one another, that is measured by counting the minimum number of operations required to transform one string into the other.

  • damerauLevenshteinDistance: a string metric for measuring the edit distance between two sequences. Informally, the Damerau–Levenshtein distance between two words is the minimum number of operations (consisting of insertions, deletions or substitutions of a single character, or transposition of two adjacent characters) required to change one word into the other.

  • jaroWinklerSimilarity: a string metric measuring an edit distance between two sequences. It is a variant of the Jaro distance metric

  • levenshteinDistance: a string metric for measuring the edit distance between two sequences. Informally, the Damerau–Levenshtein distance between two words is the minimum number of operations (consisting of insertions, deletions or substitutions of a single character, or transposition of two adjacent characters) required to change one word into the other.

https://clickhouse.com/docs/en/sql-reference/functions/string-functions#dameraulevenshteindistance

CREATE TABLE domains
(
	`domain` String,
	`rank` Float64
)
ENGINE = MergeTree
ORDER BY domain;

INSERT INTO domains SELECT
	c2 AS domain,
	1 / c1 AS rank
FROM url('domains.csv', 'CSV');

SELECT
    domain,
    levenshteinDistance(domain, 'facebook.com') AS d1,
    damerauLevenshteinDistance(domain, 'facebook.com') AS d2,
    jaroSimilarity(domain, 'facebook.com') AS d3,
    jaroWinklerSimilarity(domain, 'facebook.com') AS d4
FROM domains
ORDER BY d1 ASC
LIMIT 10 

Query id: 6f499f27-8274-4787-819a-b510322bdce3

    ┌─domain────────┬─d1─┬─d2─┬─────────────────d3─┬─────────────────d4─┐
 1. │ facebook.com  │  0 │  0 │                  1 │                  1 │
 2. │ facebonk.com  │  1 │  1 │ 0.8838383838383838 │ 0.9303030303030303 │
 3. │ fabebook.com  │  1 │  1 │  0.914141414141414 │ 0.9313131313131312 │
 4. │ facabook.com  │  1 │  1 │ 0.9444444444444443 │  0.961111111111111 │
 5. │ facobook.com  │  1 │  1 │ 0.8535353535353535 │ 0.8974747474747474 │
 6. │ facebook1.com │  1 │  1 │ 0.9743589743589745 │ 0.9846153846153847 │
 7. │ faceook.com   │  1 │  1 │ 0.9722222222222221 │ 0.9833333333333333 │
 8. │ faacebook.com │  1 │  1 │ 0.9743589743589745 │ 0.9794871794871796 │
 9. │ faceboock.com │  1 │  1 │ 0.9326923076923077 │ 0.9596153846153846 │
10. │ facebool.com  │  1 │  1 │ 0.9444444444444443 │ 0.9666666666666666 │
    └───────────────┴────┴────┴────────────────────┴────────────────────┘

Vectorized distance functions

可以作为向量数据库使用，支持L2,cosineDistance,IP三种向量相似度的度量方法

https://clickhouse.com/blog/clickhouse-release-24-02

WITH 'dog' AS search_term,
(
  SELECT vector
  FROM glove
  WHERE word = search_term
  LIMIT 1
) AS target_vector
SELECT word, cosineDistance(vector, target_vector) AS score
FROM glove
WHERE lower(word) != lower(search_term)
ORDER BY score ASC
LIMIT 5;

WITH
	'dog' AS search_term,
	(
    	SELECT vector
    	FROM glove
    	WHERE word = search_term
    	LIMIT 1
	) AS target_vector
SELECT
	word,
	1 - dotProduct(vector, target_vector) AS score
FROM glove
WHERE lower(word) != lower(search_term)
ORDER BY score ASC
LIMIT 5;

Adaptive asynchronous inserts

Asynchronous inserts shift data batching from the client side to the server side: data from insert queries is inserted into a buffer first and then written to the database storage later or asynchronously respectively.
在这里插入图片描述

本文来自互联网用户投稿，该文观点仅代表作者本人，不代表本站立场。本站仅提供信息存储空间服务，不拥有所有权，不承担相关法律责任。如若转载，请注明出处：http://www.coloradmin.cn/o/1571602.html

如若内容造成侵权/违法违规/事实不符，请联系多彩编程网进行投诉反馈，一经查实，立即删除！

相关文章

wordpress全站开发指南-面向开发者及深度用户（全中文实操）--php函数

wordpress全站开发指南-面向开发者及深度用户（全中文实操）--php函数

php函数 wordpress会封装一部分函数&#xff0c;比如bloginfo该函数的作用是直接调用你设置的你的网站的名称 示例 This is our amazing custom theme <?php echo 22; function myfirstfunction(){ echo 33; echo "<p>Hello ,this is my first function</…
阅读更多...
Node.JS多线程PromisePool之promise-pool库实现

Node.JS多线程PromisePool之promise-pool库实现

什么是Promise Pool Map-like, concurrent promise processing for Node.js. Promise-Pool是一个用于管理并发请求的JavaScript库&#xff0c;它可以限制同时进行的请求数量&#xff0c;以避免过多的请求导致服务器压力过大。使用Promise-Pool可以方便地实现对多个异步操作的并…
阅读更多...
基于spring boot的漫画之家系统

基于spring boot的漫画之家系统

基于spring boot的漫画之家系统设计与实现 开发语言&#xff1a;Java 框架&#xff1a;springboot JDK版本&#xff1a;JDK1.8 服务器&#xff1a;tomcat7 数据库&#xff1a;mysql 5.7&#xff08;一定要5.7版本&#xff09; 数据库工具&#xff1a;Navicat11 开发软件&…
阅读更多...
云计算面临的威胁

云计算面临的威胁

目录 一、概述 二、威胁建模分析 2.1 威胁建模的概念 2.2 威胁建模起到的作用 2.3 威胁建模的流程 2.3.1 威胁建模流程图 2.3.2 威胁建模流程内容 2.3.2.1 绘制数据流图 2.3.2.2 威胁识别与分析 2.3.2.2.1 STRIDE威胁分析方法论 2.3.2.3 制定消减措施 2.3.2.3.1 消减…
阅读更多...
注解,自定义注解和元注解

注解,自定义注解和元注解

1.注解 1.1.注解概述、作用 注解&#xff08;Annotation&#xff09;&#xff0c;也叫元数据。一种代码级别的说明。它是JDK1.5及以后版本引入的一个特性&#xff0c;与类、接口、枚举是在同一个层次。它可以声明在包、类、字段、方法、局部变量、方法参数等的前面&#xff0…
阅读更多...
后端灰度发布

后端灰度发布

在软件开发中&#xff0c;"灰度"通常指的是渐进式地将新功能、更新或改进引入到生产环境中&#xff0c;但只对一小部分用户或流量进行部署和测试的过程。这种方法允许开发团队在生产环境中逐步测试新功能&#xff0c;以确保其稳定性、可靠性和用户体验&#xff0c;同…
阅读更多...
每日一题(leetcode287):寻找重复数--二分查找+思维

每日一题(leetcode287):寻找重复数--二分查找+思维

思路&#xff1a;看官方解答 class Solution { public:int findDuplicate(vector<int>& nums) {int nnums.size();int left1;int rightn-1;int ans-1;while(left<right){int mid(leftright)/2;int count0;for(int j0;j<n;j){if(nums[j]<mid){count;}}if(co…
阅读更多...
观察者模式 C++

观察者模式 C++

&#x1f442; Honey Honey - 孙燕姿 - 单曲 - 网易云音乐 目录 &#x1f33c;前言 &#x1f33c;描述 &#x1f382;问题 &#x1f4aa;解决方案 &#x1f232;现实场景 代码 场景1 -- 报纸发行 场景 解释 代码 场景2 -- 气象资料发布 场景3 -- 过红绿灯 &#x…
阅读更多...
Restful Web Service

Restful Web Service

Restful 1.特点 RESTful是一种架构风格&#xff0c;强调简单、轻量级和对资源的状态less操作。RESTful是通过HTTP协议进行通信的。RESTful的应用程序可以调用运行在不同服务器上的服务或函数。RESTful的接口通常使用JSON&#xff0c;但实际上它们都支持多种数据格式。RESTful…
阅读更多...
短信群发中链接缩短、点击量统计工具

短信群发中链接缩短、点击量统计工具

此文中介绍内容需要用到C1N短网址&#xff08;c1n.cn)。 短信营销在很多行业依然是重要的客户关系和用户增长手段。但是直接在短信中使用原始链接有以下两个问题: 1. 链接过长&#xff0c;短信计费字数增加&#xff0c;成本上升 原始链接往往会包含很多参数&#xff0c;长度比…
阅读更多...
kubernetes有ingress-controler以及没有外部loadbalancer 的情况下使用istio-gateway.

kubernetes有ingress-controler以及没有外部loadbalancer 的情况下使用istio-gateway.

那就配置一个ingress-使用已有ingress-controler代理istio-gateway class创建的gateway svc来公开。
阅读更多...
MySQL常见锁探究

MySQL常见锁探究

MySQL常见锁探究 1. 各种锁类型1.1 全局锁1.2 表级锁1.2.1 表锁1.2.2 元数据锁&#xff08;MDL&#xff09;1.2.3 意向锁1.2.4 AUTO-INC 锁 1.3 行级锁1.3.1 Record Lock1.3.2 Gap Lock1.3.3 Next-Key Lock 2. MySQL是如何加锁的&#xff1f;2.1 什么 SQL 语句会加行级锁&#…
阅读更多...
微电网优化：基于​海象优化算法（Walrus Optimization Algorithm，WOA）​的微电网优化（提供MATLAB代码）

微电网优化：基于​海象优化算法（Walrus Optimization Algorithm，WOA）​的微电网优化（提供MATLAB代码）

一、微电网优化模型 微电网是一个相对独立的本地化电力单元&#xff0c;用户现场的分布式发电可以支持用电需求。为此&#xff0c;您的微电网将接入、监控、预测和控制您本地的分布式能源系统&#xff0c;同时强化供电系统的弹性&#xff0c;保障您的用电更经济。您可以在连接…
阅读更多...
RisingWave 在品高股份 Bingo IAM 中的应用

RisingWave 在品高股份 Bingo IAM 中的应用

背景介绍 公司背景 品高股份&#xff0c;是国内专业的云计算及行业信息化服务提供商。公司成立于 2003 年&#xff0c;总部位于广州&#xff0c;下设多家子公司和分公司&#xff0c;目前员工总数近 900 人&#xff0c;其中 80 %以上是专业技术人员。 品高股份在 2008 年便开…
阅读更多...
第20次修改了可删除可持久保存的前端html备忘录:重新布局

第20次修改了可删除可持久保存的前端html备忘录:重新布局

第20次修改了可删除可持久保存的前端html备忘录&#xff1a;重新布局 <!DOCTYPE html> <html lang"zh"> <head><meta charset"UTF-8"><meta name"viewport" content"widthdevice-width, initial-scale1.0"…
阅读更多...
Tuxera2023 NTFS for Mac下载，安装和序列号激活

Tuxera2023 NTFS for Mac下载，安装和序列号激活

对于必须在Windows电脑和Mac电脑之间来回切换的Mac朋友来说&#xff0c;跨平台不兼容一直是一个巨大的障碍&#xff0c;尤其是当我们需要使用NTFS格式的硬盘在Windows和macOS之间共享文件时。因为Mac默认不支持写入NTFS磁盘。 为了解决这一问题&#xff0c;很多朋友会选择很便捷…
阅读更多...
算法刷题应用知识补充--基础算法、数据结构篇

算法刷题应用知识补充--基础算法、数据结构篇

这里写目录标题 位运算&#xff08;均是拷贝运算&#xff0c;不会影响原数据&#xff0c;这点要注意&#xff09;&、|、^位运算特性细节知识补充对于n-1的理解异或来实现数字交换找到只出现一次的数据&#xff0c;其余数据出现偶数次 >> 、<<二进制中相邻的位的…
阅读更多...
力扣---删除链表的倒数第 N 个结点

力扣---删除链表的倒数第 N 个结点

给你一个链表&#xff0c;删除链表的倒数第 n 个结点&#xff0c;并且返回链表的头结点。 示例 1&#xff1a; 输入&#xff1a;head [1,2,3,4,5], n 2 输出&#xff1a;[1,2,3,5]示例 2&#xff1a; 输入&#xff1a;head [1], n 1 输出&#xff1a;[]示例 3&#xff1a…
阅读更多...
idea 中 大于等于，不等于、小于等于等等这些符号发生了改变问题解决方法

idea 中 大于等于，不等于、小于等于等等这些符号发生了改变问题解决方法

1.问题描述 idea 中&#xff01;变为 ≠、 >变成了≥、<变成了 ≤ 等问题的解决办法 展示效果如下截图 解决方法
阅读更多...
Vue3从入门到实战：掌握状态管理库pinia（下部分）

Vue3从入门到实战：掌握状态管理库pinia（下部分）

1.storeToRefs 在Count.vue文件中 显的冗余了&#xff0c;如何更加优雅简化代码。用storeToRefs 补充&#xff1a; 为什么不用ToRefs呢&#xff1f; 使用的话会将所有数据都用ref引用包裹&#xff0c;其实方法等是没必要包裹的&#xff0c;具有一定风险 2.getters的使用 …
阅读更多...
推荐文章
最新文章

Copyright @ 2022~2023