paimon取消hive转filesystem

news2024/12/23 1:55:44

概述

公司上一版本保留了 hive ，此版优化升级后，取消 hive。

实践

关键配置

同步数据时，配置如下，将形成两个库

# ods库
CREATE CATALOG paimon WITH (
    'type' = 'paimon',
    'warehouse' = 'hdfs:///data/paimon',
    'default-database'='ods'
);
# test库
CREATE CATALOG paimon WITH (
    'type' = 'paimon',
    'warehouse' = 'hdfs:///data/paimon',
    'default-database'='test'
);

在这里插入图片描述

spark sql

下面是测试

0: jdbc:hive2://10.32.36.142:10009/> show databases;
2024-03-01 13:42:12.681 INFO KyuubiSessionManager-exec-pool: Thread-3501 org.apache.kyuubi.operation.ExecuteStatement: Processing root's query[fa620d6f-4c39-4537-bedc-e56435767194]: PENDING_STATE -> RUNNING_STATE, statement:
show databases
24/03/01 13:42:12 INFO ExecuteStatement: Processing root's query[fa620d6f-4c39-4537-bedc-e56435767194]: PENDING_STATE -> RUNNING_STATE, statement:
show databases
24/03/01 13:42:12 INFO ExecuteStatement: 
           Spark application name: kyuubi_USER_SPARK_SQL_root_default_167aca40-d767-4620-92ab-63fb0a63eb6f
                 application ID: application_1708505130791_0034
                 application web UI: http://hadoop01:8088/proxy/application_1708505130791_0034
                 master: yarn
                 deploy mode: cluster
                 version: 3.4.2
           Start time: 2024-03-01T13:41:49.685
           User: root
24/03/01 13:42:12 INFO ExecuteStatement: Execute in full collect mode
24/03/01 13:42:12 INFO CodeGenerator: Code generated in 8.594565 ms
24/03/01 13:42:12 INFO ExecuteStatement: Processing root's query[fa620d6f-4c39-4537-bedc-e56435767194]: RUNNING_STATE -> FINISHED_STATE, time taken: 0.059 seconds
2024-03-01 13:42:12.741 INFO KyuubiSessionManager-exec-pool: Thread-3501 org.apache.kyuubi.operation.ExecuteStatement: Query[fa620d6f-4c39-4537-bedc-e56435767194] in FINISHED_STATE
2024-03-01 13:42:12.742 INFO KyuubiSessionManager-exec-pool: Thread-3501 org.apache.kyuubi.operation.ExecuteStatement: Processing root's query[fa620d6f-4c39-4537-bedc-e56435767194]: RUNNING_STATE -> FINISHED_STATE, time taken: 0.06 seconds
+------------+
| namespace  |
+------------+
| default    |
| ods        |
| test       |
+------------+
3 rows selected (0.071 seconds)
0: jdbc:hive2://10.32.36.142:10009/> use test;
2024-03-01 13:42:16.561 INFO KyuubiSessionManager-exec-pool: Thread-3502 org.apache.kyuubi.operation.ExecuteStatement: Processing root's query[d10b60db-1713-4a27-816d-3b12fbbe207a]: PENDING_STATE -> RUNNING_STATE, statement:
use test
24/03/01 13:42:16 INFO ExecuteStatement: Processing root's query[d10b60db-1713-4a27-816d-3b12fbbe207a]: PENDING_STATE -> RUNNING_STATE, statement:
use test
2024-03-01 13:42:16.582 INFO KyuubiSessionManager-exec-pool: Thread-3502 org.apache.kyuubi.operation.ExecuteStatement: Processing root's query[d10b60db-1713-4a27-816d-3b12fbbe207a]: RUNNING_STATE -> FINISHED_STATE, time taken: 0.021 seconds
+---------+
| Result  |
+---------+
+---------+
No rows selected (0.037 seconds)
0: jdbc:hive2://10.32.36.142:10009/> show tables;
2024-03-01 13:42:19.547 INFO KyuubiSessionManager-exec-pool: Thread-3503 org.apache.kyuubi.operation.ExecuteStatement: Processing root's query[55057149-9a40-4fc6-8a57-0f52d73b4846]: PENDING_STATE -> RUNNING_STATE, statement:
show tables

2024-03-01 13:42:19.660 INFO KyuubiSessionManager-exec-pool: Thread-3503 org.apache.kyuubi.operation.ExecuteStatement: Processing root's query[55057149-9a40-4fc6-8a57-0f52d73b4846]: RUNNING_STATE -> FINISHED_STATE, time taken: 0.113 seconds
+------------+---------------------------+--------------+
| namespace  |         tableName         | isTemporary  |
+------------+---------------------------+--------------+
| test       | trace_log_refdes_hive_ro  | false        |
+------------+---------------------------+--------------+
1 row selected (0.127 seconds)
0: jdbc:hive2://10.32.36.142:10009/> select * from `trace_log_refdes_hive_ro$ro` limit 10;
2024-03-01 13:42:53.008 INFO KyuubiSessionManager-exec-pool: Thread-3505 org.apache.kyuubi.operation.ExecuteStatement: Processing root's query[874fc0de-8d0c-47db-b4b2-3192204d7401]: PENDING_STATE -> RUNNING_STATE, statement:
select * from `trace_log_refdes_hive_ro$ro` limit 10
2024-03-01 13:42:53.448 INFO KyuubiSessionManager-exec-pool: Thread-3505 org.apache.kyuubi.operation.ExecuteStatement: Processing root's query[874fc0de-8d0c-47db-b4b2-3192204d7401]: RUNNING_STATE -> FINISHED_STATE, time taken: 0.44 seconds
+--------------------------+------------------------+---------+----------------+----------------------+---------------+---------------------+-------+--------+-----------------------------------+----------------------+------+-------+-------+-------+-------+-------+-------------+---------+-------------------+-------+----------+-----------+---------+------------+-------------+
|          pcbid           |          rid           | refdes  | bm_circuit_no  |      timestamp       | pickupstatus  |    serial_number    | flag  | kitid  |                id                 |      createdate      | etl  | opt1  | opt2  | opt3  | opt4  | opt5  |  nozzleid   | laneno  | componentbarcode  |  pn   | lotcode  | datecode  | verdor  | workorder  |     dt      |
+--------------------------+------------------------+---------+----------------+----------------------+---------------+---------------------+-------+--------+-----------------------------------+----------------------+------+-------+-------+-------+-------+-------+-------------+---------+-------------------+-------+----------+-----------+---------+------------+-------------+
| AMV042400602033690A4HNS  | 155S00458-A12402AAAG7  | FL0906  | 2              | 2024-02-27 12:29:35  | 0             | DLCH3S010C600004Q4  | 0     | NXT    | 121BD66CFD785571E063AA2C200A7743  | 2024-02-27 12:32:10  | N    | NULL  | NULL  | NULL  | NULL  | NULL  | H29 101657  | 2       | NULL              | NULL  | NULL     | NULL      | NULL    | NULL       | 2024-02-27  |
| AMV042400602033690A4HNS  | 155S00458-A12402AAAG7  | FL1000  | 2              | 2024-02-27 12:29:35  | 0             | DLCH3S010C600004Q4  | 0     | NXT    | 121BD66CFD795571E063AA2C200A7743  | 2024-02-27 12:32:10  | N    | NULL  | NULL  | NULL  | NULL  | NULL  | H29 143893  | 2       | NULL              | NULL  | NULL     | NULL      | NULL    | NULL       | 2024-02-27  |
| AMV042400602033690A4HNS  | 155S00458-A12402AAAG7  | FL0904  | 1              | 2024-02-27 12:29:35  | 0             | DLCH3S010CH00004Q4  | 0     | NXT    | 121BD66CFD865571E063AA2C200A7743  | 2024-02-27 12:32:10  | N    | NULL  | NULL  | NULL  | NULL  | NULL  | H29 101064  | 2       | NULL              | NULL  | NULL     | NULL      | NULL    | NULL       | 2024-02-27  |
| AMV042400602033690A4HNS  | 155S00458-A12402AAAG7  | FL0911  | 11             | 2024-02-27 12:29:35  | 0             | DLCH3S010CF00004Q4  | 0     | NXT    | 121BD66CFD905571E063AA2C200A7743  | 2024-02-27 12:32:10  | N    | NULL  | NULL  | NULL  | NULL  | NULL  | H29 101657  | 2       | NULL              | NULL  | NULL     | NULL      | NULL    | NULL       | 2024-02-27  |
| AMV042400602033690A4HNS  | 155S00458-A12402AAAG7  | FL0923  | 6              | 2024-02-27 12:29:35  | 0             | DLCH3S010BL00004Q4  | 0     | NXT    | 121BD66CFD9F5571E063AA2C200A7743  | 2024-02-27 12:32:10  | N    | NULL  | NULL  | NULL  | NULL  | NULL  | H29 099800  | 2       | NULL              | NULL  | NULL     | NULL      | NULL    | NULL       | 2024-02-27  |
| AMV042400602033690A4HNS  | 155S00458-A12402AAAG7  | FL0911  | 17             | 2024-02-27 12:29:35  | 0             | DLCH3S010C900004Q4  | 0     | NXT    | 121BD66CFDA45571E063AA2C200A7743  | 2024-02-27 12:32:10  | N    | NULL  | NULL  | NULL  | NULL  | NULL  | H29 142833  | 2       | NULL              | NULL  | NULL     | NULL      | NULL    | NULL       | 2024-02-27  |
| AMV042400602033690A4HNS  | 155S00458-A12402AAAG7  | FL0902  | 16             | 2024-02-27 12:29:35  | 0             | DLCH3S010CA00004Q4  | 0     | NXT    | 121BD66CFDB25571E063AA2C200A7743  | 2024-02-27 12:32:10  | N    | NULL  | NULL  | NULL  | NULL  | NULL  | H29 102211  | 2       | NULL              | NULL  | NULL     | NULL      | NULL    | NULL       | 2024-02-27  |
| AMV042400602033690A4HNS  | 155S00458-A12402AAAG7  | FL0906  | 14             | 2024-02-27 12:29:35  | 0             | DLCH3S010CC00004Q4  | 0     | NXT    | 121BD66CFDBD5571E063AA2C200A7743  | 2024-02-27 12:32:10  | N    | NULL  | NULL  | NULL  | NULL  | NULL  | H29 102194  | 2       | NULL              | NULL  | NULL     | NULL      | NULL    | NULL       | 2024-02-27  |
| AMV042400602033690A4HNS  | 155S00458-A12402AAAG7  | FL1000  | 18             | 2024-02-27 12:29:35  | 0             | DLCH3S010C800004Q4  | 0     | NXT    | 121BD66CFDCD5571E063AA2C200A7743  | 2024-02-27 12:32:10  | N    | NULL  | NULL  | NULL  | NULL  | NULL  | H29 028445  | 2       | NULL              | NULL  | NULL     | NULL      | NULL    | NULL       | 2024-02-27  |
| AMV042400602033690A4HNS  | 155S00458-A12402AAAG7  | FL0902  | 14             | 2024-02-27 12:29:35  | 0             | DLCH3S010CC00004Q4  | 0     | NXT    | 121BD66CFDD25571E063AA2C200A7743  | 2024-02-27 12:32:10  | N    | NULL  | NULL  | NULL  | NULL  | NULL  | H29 100330  | 2       | NULL              | NULL  | NULL     | NULL      | NULL    | NULL       | 2024-02-27  |
+--------------------------+------------------------+---------+----------------+----------------------+---------------+---------------------+-------+--------+-----------------------------------+----------------------+------+-------+-------+-------+-------+-------+-------------+---------+-------------------+-------+----------+-----------+---------+------------+-------------+
10 rows selected (0.461 seconds)
0: jdbc:hive2://10.32.36.142:10009/>