kyuubi整合spark on yarn

news2024/10/7 20:30:13

目录

  • 概述
  • 实践
    • 下载
    • 配置
    • 启动
  • 结束

概述

目标:

  • 1.实现kyuubi spark on yarn
  • 2.实现 kyuubi spark on yarn 资源的动态分配

注意：版本 kyuubi 1.8.0 、 spark 3.4.2 、hadoop 3.3.6

前置准备请看如下文章

文章链接
hadoop一主三从安装链接
spark on yarn链接

实践

下载

官网下载地址

在这里插入图片描述

配置

官方文档

修改配置文件

# 需要修改的配置文件
[root@hadoop01 conf]# ls
kyuubi-defaults.conf.template  kyuubi-env.sh.template  log4j2.xml.template
[root@hadoop01 conf]# pwd
/data/hadoop/soft/apache-kyuubi-1.8.0/conf

# 第一步: 修改日志文件
[root@hadoop01 conf]# mv log4j2.xml.template log4j2.xml
# 第二步: kyuubi-env.sh
# export SPARK_HOME=/data/hadoop/soft/spark-3.4.2
[root@hadoop01 conf]# ls
kyuubi-defaults.conf.template  kyuubi-env.sh.template  log4j2.xml
[root@hadoop01 conf]# mv  kyuubi-env.sh.template kyuubi-env.sh
[root@hadoop01 conf]# vi kyuubi-env.sh 
# 第三步: kyuubi-defaults.conf
[root@hadoop01 conf]# mv kyuubi-defaults.conf.template kyuubi-defaults.conf
[root@hadoop01 conf]# vi kyuubi-defaults.conf 

kyuubi.engine.type                       SPARK_SQL
kyuubi.engine.share.level                USER
# ============ Spark Config ============
spark.master=yarn
spark.submit.deployMode=cluster
spark.driver.memory 4g
spark.executor.memory 4g
spark.executor.cores 2
#Dynamic allocation
spark.dynamicAllocation.enabled=true
##false if prefer shuffle tracking than ESS
spark.shuffle.service.enabled=true
spark.dynamicAllocation.initialExecutors=2
spark.dynamicAllocation.minExecutors=6
spark.dynamicAllocation.maxExecutors=6
spark.dynamicAllocation.executorAllocationRatio=0.5
spark.dynamicAllocation.executorIdleTimeout=200s
spark.dynamicAllocation.cachedExecutorIdleTimeout=30min
# true if prefer shuffle tracking than ESS
spark.dynamicAllocation.shuffleTracking.enabled=false
spark.dynamicAllocation.shuffleTracking.timeout=30min
spark.dynamicAllocation.schedulerBacklogTimeout=1s
spark.dynamicAllocation.sustainedSchedulerBacklogTimeout=1s
spark.cleaner.periodicGC.interval=5min
# aqe
spark.sql.adaptive.enabled=true
spark.sql.adaptive.forceApply=false
spark.sql.adaptive.logLevel=info
spark.sql.adaptive.advisoryPartitionSizeInBytes=128m
spark.sql.adaptive.coalescePartitions.enabled=true
spark.sql.adaptive.coalescePartitions.minPartitionNum=30
spark.sql.adaptive.coalescePartitions.initialPartitionNum=5120
spark.sql.adaptive.fetchShuffleBlocksInBatch=true
spark.sql.adaptive.localShuffleReader.enabled=true
spark.sql.adaptive.skewJoin.enabled=true
spark.sql.adaptive.skewJoin.skewedPartitionFactor=3
spark.sql.adaptive.skewJoin.skewedPartitionThresholdInBytes=256m
spark.sql.adaptive.nonEmptyPartitionRatioForBroadcastJoin=0.2
spark.sql.adaptive.optimizer.excludedRules
spark.sql.autoBroadcastJoinThreshold=-1
# spark 返回数据大小设定
spark.kryoserializer.buffer.max=2047m
spark.driver.maxResultSize=4096m
# paimon
#spark.sql.catalog.hive.warehouse=hdfs:///data/paimon
spark.sql.catalog.paimon=org.apache.paimon.spark.SparkCatalog
spark.sql.catalog.paimon.warehouse=hdfs:///data/paimon
spark.sql.extensions=org.apache.paimon.spark.PaimonSparkSessionExtension


# Log into Spark History Server
spark.eventLog.enabled true
spark.eventLog.dir hdfs://hadoop01:9000/spark-eventlog
spark.serializer org.apache.spark.serializer.KryoSerializer
spark.eventLog.compress true
#将 spark 历史服务器与 yarn 历史服务器进行整合，写到 HDFS 中
spark.yarn.historyServer.address hadoop01:18080

# spark UI参数
spark.ui.retainedJobs 50
spark.ui.retainedStages 300

启动

[root@hadoop01 apache-kyuubi-1.8.0]# ./bin/kyuubi start
bin/beeline -u 'jdbc:hive2://10.32.36.142:10009/' -n root
24/02/26 15:47:30 INFO Client: Application report for application_1708505130791_0007 (state: RUNNING)
24/02/26 15:47:31 INFO Client: Application report for application_1708505130791_0007 (state: RUNNING)
24/02/26 15:47:32 INFO Client: Application report for application_1708505130791_0007 (state: RUNNING)
24/02/26 15:47:33 INFO Client: Application report for application_1708505130791_0007 (state: RUNNING)
2024-02-26 15:47:35.793 INFO KyuubiSessionManager-exec-pool: Thread-58 org.apache.kyuubi.ha.client.zookeeper.ZookeeperDiscoveryClient: Get service instance:hadoop04:39670 engine id:application_1708505130791_0007 and version:1.8.0 under /kyuubi_1.8.0_USER_SPARK_SQL/root/default
24/02/26 15:47:35 INFO Client: Application report for application_1708505130791_0007 (state: RUNNING)
Connected to: Spark SQL (version 3.4.2)
Driver: Kyuubi Project Hive JDBC Client (version 1.8.0)
Beeline version 1.8.0 by Apache Kyuubi
0: jdbc:hive2://10.32.36.142:10009/> 

0: jdbc:hive2://10.32.36.142:10009/> show databases;
2024-02-26 15:48:03.130 INFO KyuubiSessionManager-exec-pool: Thread-66 org.apache.kyuubi.operation.ExecuteStatement: Processing root's query[81c88029-474d-4ed2-98fc-21013bf10cc3]: PENDING_STATE -> RUNNING_STATE, statement:
show databases
2024-02-26 15:48:03.261 INFO KyuubiSessionManager-exec-pool: Thread-66 org.apache.kyuubi.operation.ExecuteStatement: Processing root's query[81c88029-474d-4ed2-98fc-21013bf10cc3]: RUNNING_STATE -> FINISHED_STATE, time taken: 0.131 seconds
+------------+
| namespace  |
+------------+
| default    |
+------------+
1 row selected (0.34 seconds)
0: jdbc:hive2://10.32.36.142:10009/>

在这里插入图片描述
在这里插入图片描述

set spark.sql.catalog.paimon=org.apache.paimon.spark.SparkCatalog
set spark.sql.catalog.paimon.warehouse=hdfs:///data/paimon;


24/02/26 20:30:08 INFO ExecuteStatement: Execute in full collect mode
24/02/26 20:30:08 INFO V2ScanRelationPushDown: 
Pushing operators to trace_log_refdes_hive_ro
Pushed Filters: IsNotNull(id), EqualTo(id,11C0928D5A29E048E063AA2C200ABEF3)
Post-Scan Filters: isnotnull(id#354),(id#354 = 11C0928D5A29E048E063AA2C200ABEF3)
         
24/02/26 20:30:08 INFO V2ScanRelationPushDown: 
Output: pcbid#345, rid#346, refdes#347, bm_circuit_no#348, timestamp#349, pickupstatus#350, serial_number#351, flag#352, kitid#353, id#354, createdate#355, etl#356, opt1#357, opt2#358, opt3#359, opt4#360, opt5#361, nozzleid#362, laneno#363, componentbarcode#364, pn#365, lotcode#366, datecode#367, verdor#368, workorder#369, dt#370
         
24/02/26 20:30:08 INFO SparkContext: Starting job: collect at ExecuteStatement.scala:72
24/02/26 20:30:08 INFO SQLOperationListener: Query [4a52f7cf-f446-40b0-b16f-6b895ab84224]: Job 5 started with 1 stages, 1 active jobs running
24/02/26 20:30:08 INFO SQLOperationListener: Query [4a52f7cf-f446-40b0-b16f-6b895ab84224]: Stage 6.0 started with 1 tasks, 1 active stages running

三千五百万
在这里插入图片描述

主键id单笔获取

0: jdbc:hive2://10.32.36.142:10009/> select count(*) from trace_log_refdes_hive_ro where id='11C0928D5A26E048E063AA2C200ABEF3';
24/02/27 08:43:33 INFO AdaptiveSparkPlanExec: Final plan:
*(1) HashAggregate(keys=[], functions=[count(1)], output=[count(1)#523L])
+- *(1) HashAggregate(keys=[], functions=[partial_count(1)], output=[count#527L])
   +- *(1) Project
      +- *(1) Filter (isnotnull(id#505) AND (id#505 = 11C0928D5A26E048E063AA2C200ABEF3))
         +- BatchScan trace_log_refdes_hive_ro[id#505] PaimonScan: [trace_log_refdes_hive_ro], PushedFilters: [IsNotNull(id),Equal(id, 11C0928D5A26E048E063AA2C200ABEF3)] RuntimeFilters: []

24/02/27 08:43:33 INFO ExecuteStatement: Processing root's query[b4889a64-d943-41ae-819e-d8a0491c60c6]: RUNNING_STATE -> FINISHED_STATE, time taken: 1.871 seconds
24/02/27 08:43:33 INFO SQLOperationListener: Query [b4889a64-d943-41ae-819e-d8a0491c60c6]: Job 7 succeeded, 0 active jobs running
2024-02-27 08:43:33.005 INFO KyuubiSessionManager-exec-pool: Thread-496 org.apache.kyuubi.operation.ExecuteStatement: Query[b4889a64-d943-41ae-819e-d8a0491c60c6] in FINISHED_STATE
2024-02-27 08:43:33.006 INFO KyuubiSessionManager-exec-pool: Thread-496 org.apache.kyuubi.operation.ExecuteStatement: Processing root's query[b4889a64-d943-41ae-819e-d8a0491c60c6]: RUNNING_STATE -> FINISHED_STATE, time taken: 1.874 seconds
+-----------+
| count(1)  |
+-----------+
| 1         |
+-----------+
1 row selected (1.883 seconds)

非主键 count 测试

0: jdbc:hive2://10.32.36.142:10009/> select count(*) from trace_log_refdes_hive_ro where pcbid='E23MPM42201540';
24/02/27 08:41:56 INFO AdaptiveSparkPlanExec: Final plan:
*(2) HashAggregate(keys=[], functions=[count(1)], output=[count(1)#489L])
+- ShuffleQueryStage 0
   +- Exchange SinglePartition, ENSURE_REQUIREMENTS, [plan_id=109]
      +- *(1) HashAggregate(keys=[], functions=[partial_count(1)], output=[count#493L])
         +- *(1) Project
            +- *(1) Filter (isnotnull(pcbid#462) AND (pcbid#462 = E23MPM42201540))
               +- BatchScan trace_log_refdes_hive_ro[pcbid#462] PaimonScan: [trace_log_refdes_hive_ro], PushedFilters: [IsNotNull(pcbid),Equal(pcbid, E23MPM42201540)] RuntimeFilters: []

2024-02-27 08:41:56.274 INFO KyuubiSessionManager-exec-pool: Thread-495 org.apache.kyuubi.operation.ExecuteStatement: Processing root's query[f473dde1-ba80-4d3e-8dc7-fa0a3e32df1f]: RUNNING_STATE -> FINISHED_STATE, time taken: 5.368 seconds
+-----------+
| count(1)  |
+-----------+
| 108       |
+-----------+
1 row selected (5.393 seconds)

差不多6千万

24/02/27 16:54:54 INFO SQLOperationListener: Query [96e1d2f4-8230-4cc7-9b84-ce888387bb7d]: Job 2 succeeded, 0 active jobs running
24/02/27 16:54:54 INFO AdaptiveSparkPlanExec: Final plan:
*(2) HashAggregate(keys=[], functions=[count(1)], output=[count(1)#61L])
+- ShuffleQueryStage 0
   +- Exchange SinglePartition, ENSURE_REQUIREMENTS, [plan_id=52]
      +- *(1) HashAggregate(keys=[], functions=[partial_count(1)], output=[count#64L])
         +- *(1) Project
            +- BatchScan trace_log_refdes_hive_ro[] PaimonScan: [trace_log_refdes_hive_ro] RuntimeFilters: []

24/02/27 16:54:54 INFO CodeGenerator: Code generated in 7.970386 ms
24/02/27 16:54:54 INFO ExecuteStatement: Processing root's query[96e1d2f4-8230-4cc7-9b84-ce888387bb7d]: RUNNING_STATE -> FINISHED_STATE, time taken: 21.921 seconds
2024-02-27 16:54:54.490 INFO KyuubiSessionManager-exec-pool: Thread-806 org.apache.kyuubi.operation.ExecuteStatement: Query[96e1d2f4-8230-4cc7-9b84-ce888387bb7d] in FINISHED_STATE
2024-02-27 16:54:54.490 INFO KyuubiSessionManager-exec-pool: Thread-806 org.apache.kyuubi.operation.ExecuteStatement: Processing root's query[96e1d2f4-8230-4cc7-9b84-ce888387bb7d]: RUNNING_STATE -> FINISHED_STATE, time taken: 21.924 seconds
+-----------+
| count(1)  |
+-----------+
| 59480487  |
+-----------+
1 row selected (21.942 seconds)
0: jdbc:hive2://10.32.36.142:10009/> select *  from trace_log_refdes_hive_ro limit 10;
24/02/27 16:55:27 INFO ExecuteStatement: Execute in full collect mode
24/02/27 16:55:27 INFO V2ScanRelationPushDown: 
Output: pcbid#67, rid#68, refdes#69, bm_circuit_no#70, timestamp#71, pickupstatus#72, serial_number#73, flag#74, kitid#75, id#76, createdate#77, etl#78, opt1#79, opt2#80, opt3#81, opt4#82, opt5#83, nozzleid#84, laneno#85, componentbarcode#86, pn#87, lotcode#88, datecode#89, verdor#90, workorder#91, dt#92
         
2024-02-27 16:55:28.414 INFO KyuubiSessionManager-exec-pool: Thread-807 org.apache.kyuubi.operation.ExecuteStatement: Processing root's query[cadc193d-1343-4973-828c-23f23250b3d3]: RUNNING_STATE -> FINISHED_STATE, time taken: 0.735 seconds
+-----------------+------------------------------+---------+----------------+----------------------+---------------+--------------------+-------+--------+-----------------------------------+----------------------+------+-------+-------+-------+-------+-------+-----------+---------+-------------------+-------+----------+-----------+---------+------------+-------------+
|      pcbid      |             xxx              | yyyyff  | zzzzzxxxxxxxx  |      timestamp       | gggggggfffff  |   ddddddddddddd    | cccc  | aaaaa  |                id                 |      createdate      | etl  | opt1  | opt2  | opt3  | opt4  | opt5  | nozzleid  | laneno  | componentbarcode  |  pn   | xyzzzzx  | xzzzzzzz  | txxxxx  | zxxxxxxxx  |     dt      |
+-----------------+------------------------------+---------+----------------+----------------------+---------------+--------------------+-------+--------+-----------------------------------+----------------------+------+-------+-------+-------+-------+-------+-----------+---------+-------------------+-------+----------+-----------+---------+------------+-------------+
| E23MPM42203175  | 514S00292-11420240109000060  | J0200   | 5              | 2024-02-23 18:16:42  | 0             | DLC4084004RPQVLAG  | 0     | NXT    | 11C0928D5A26E048E063AA2C200ABEF3  | 2024-02-23 18:14:50  | N    | NULL  | NULL  | NULL  | NULL  | NULL  | NULL      | NULL    | NULL              | NULL  | NULL     | NULL      | NULL    | NULL       | 2024-02-23  |
| E23MPM42203175  | 514S00292-11420240109000060  | J0200   | 4              | 2024-02-23 18:16:42  | 0             | DLC4084004SPQVLAF  | 0     | NXT    | 11C0928D5A27E048E063AA2C200ABEF3  | 2024-02-23 18:14:50  | N    | NULL  | NULL  | NULL  | NULL  | NULL  | NULL      | NULL    | NULL              | NULL  | NULL     | NULL      | NULL    | NULL       | 2024-02-23  |
| E23MPM42203175  | 514S00292-11420240117000057  | J0200   | 7              | 2024-02-23 18:16:42  | 0             | DLC4084004PPQVLAJ  | 0     | NXT    | 11C0928D5A28E048E063AA2C200ABEF3  | 2024-02-23 18:14:50  | N    | NULL  | NULL  | NULL  | NULL  | NULL  | NULL      | NULL    | NULL              | NULL  | NULL     | NULL      | NULL    | NULL       | 2024-02-23  |
| E23MPM42203175  | 514S00292-11420240117000057  | J0200   | 9              | 2024-02-23 18:16:42  | 0             | DLC4084004MPQVLAL  | 0     | NXT    | 11C0928D5A29E048E063AA2C200ABEF3  | 2024-02-23 18:14:50  | N    | NULL  | NULL  | NULL  | NULL  | NULL  | NULL      | NULL    | NULL              | NULL  | NULL     | NULL      | NULL    | NULL       | 2024-02-23  |
| E23MPM42203175  | 514S00292-11420240117000056  | J0200   | 12             | 2024-02-23 18:16:42  | 0             | DLC4084004VPQVLAC  | 0     | NXT    | 11C0928D5A2CE048E063AA2C200ABEF3  | 2024-02-23 18:14:50  | N    | NULL  | NULL  | NULL  | NULL  | NULL  | NULL      | NULL    | NULL              | NULL  | NULL     | NULL      | NULL    | NULL       | 2024-02-23  |
| E23MPM42203176  | 514S00292-11420240117000056  | J0200   | 1              | 2024-02-23 18:16:42  | 0             | DLC4084005NPQVLAG  | 0     | NXT    | 11C0928D5A31E048E063AA2C200ABEF3  | 2024-02-23 18:14:50  | N    | NULL  | NULL  | NULL  | NULL  | NULL  | NULL      | NULL    | NULL              | NULL  | NULL     | NULL      | NULL    | NULL       | 2024-02-23  |
| E23MPM42203176  | 514S00292-11420240117000059  | J0200   | 4              | 2024-02-23 18:16:42  | 0             | DLC4084005GPQVLAN  | 0     | NXT    | 11C0928D5A34E048E063AA2C200ABEF3  | 2024-02-23 18:14:50  | N    | NULL  | NULL  | NULL  | NULL  | NULL  | NULL      | NULL    | NULL              | NULL  | NULL     | NULL      | NULL    | NULL       | 2024-02-23  |
| E23MPM42203176  | 514S00292-11420240116000073  | J0200   | 10             | 2024-02-23 18:16:42  | 0             | DLC4084005MPQVLAH  | 0     | NXT    | 11C0928D5A3AE048E063AA2C200ABEF3  | 2024-02-23 18:14:50  | N    | NULL  | NULL  | NULL  | NULL  | NULL  | NULL      | NULL    | NULL              | NULL  | NULL     | NULL      | NULL    | NULL       | 2024-02-23  |
| E23MPM42201540  | 117S0158-A420240115001804    | R0514   | 7              | 2024-02-23 18:16:42  | 0             | DLC40860DTVPQVLAW  | 0     | NXT    | 11C0928D5A59E048E063AA2C200ABEF3  | 2024-02-23 18:14:51  | N    | NULL  | NULL  | NULL  | NULL  | NULL  | NULL      | NULL    | NULL              | NULL  | NULL     | NULL      | NULL    | NULL       | 2024-02-23  |
| E23MPM42201540  | 117S0158-A420240115001804    | R0401   | 1              | 2024-02-23 18:16:42  | 0             | DLC40860DU4PQVLAJ  | 0     | NXT    | 11C0928D5A5DE048E063AA2C200ABEF3  | 2024-02-23 18:14:51  | N    | NULL  | NULL  | NULL  | NULL  | NULL  | NULL      | NULL    | NULL              | NULL  | NULL     | NULL      | NULL    | NULL       | 2024-02-23  |
+-----------------+------------------------------+---------+----------------+----------------------+---------------+--------------------+-------+--------+-----------------------------------+----------------------+------+-------+-------+-------+-------+-------+-----------+---------+-------------------+-------+----------+-----------+---------+------------+-------------+
10 rows selected (0.777 seconds)
0: jdbc:hive2://10.32.36.142:10009/> select *  from trace_log_refdes_hive_ro where id ='11C0928D5A5DE048E063AA2C200ABEF3';
24/02/27 16:55:49 INFO V2ScanRelationPushDown: 
Pushing operators to trace_log_refdes_hive_ro
Pushed Filters: IsNotNull(id), EqualTo(id,11C0928D5A5DE048E063AA2C200ABEF3)
Post-Scan Filters: isnotnull(id#181),(id#181 = 11C0928D5A5DE048E063AA2C200ABEF3)
         
24/02/27 16:55:49 INFO V2ScanRelationPushDown: 
Output: pcbid#172, rid#173, refdes#174, bm_circuit_no#175, timestamp#176, pickupstatus#177, serial_number#178, flag#179, kitid#180, id#181, createdate#182, etl#183, opt1#184, opt2#185, opt3#186, opt4#187, opt5#188, nozzleid#189, laneno#190, componentbarcode#191, pn#192, lotcode#193, datecode#194, verdor#195, workorder#196, dt#197
         
2024-02-27 16:55:52.191 INFO KyuubiSessionManager-exec-pool: Thread-808 org.apache.kyuubi.operation.ExecuteStatement: Processing root's query[1dd09511-efc5-4e64-ad2a-15d4b59fb553]: RUNNING_STATE -> FINISHED_STATE, time taken: 2.859 seconds
+-----------------+----------------------------+---------+----------------+----------------------+---------------+--------------------+-------+--------+-----------------------------------+----------------------+------+-------+-------+-------+-------+-------+-----------+---------+-------------------+-------+----------+-----------+---------+------------+-------------+
|      pcbid      |             xxx              | yyyyff  | zzzzzxxxxxxxx  |      timestamp       | gggggggfffff  |   ddddddddddddd    | cccc  | aaaaa  |                id                 |      createdate      | etl  | opt1  | opt2  | opt3  | opt4  | opt5  | nozzleid  | laneno  | componentbarcode  |  pn   | xyzzzzx  | xzzzzzzz  | txxxxx  | zxxxxxxxx  |     dt      |
+-----------------+----------------------------+---------+----------------+----------------------+---------------+--------------------+-------+--------+-----------------------------------+----------------------+------+-------+-------+-------+-------+-------+-----------+---------+-------------------+-------+----------+-----------+---------+------------+-------------+
| E23MPM42201540  | 117S0158-A420240115001804  | R0401   | 1              | 2024-02-23 18:16:42  | 0             | DLC40860DU4PQVLAJ  | 0     | NXT    | 11C0928D5A5DE048E063AA2C200ABEF3  | 2024-02-23 18:14:51  | N    | NULL  | NULL  | NULL  | NULL  | NULL  | NULL      | NULL    | NULL              | NULL  | NULL     | NULL      | NULL    | NULL       | 2024-02-23  |
+-----------------+----------------------------+---------+----------------+----------------------+---------------+--------------------+-------+--------+-----------------------------------+----------------------+------+-------+-------+-------+-------+-------+-----------+---------+-------------------+-------+----------+-----------+---------+------------+-------------+
1 row selected (2.875 seconds)
0: jdbc:hive2://10.32.36.142:10009/> select count(*)  from trace_log_refdes_hive_ro where pcbid ='E23MPM42203176';
2024-02-27 16:57:01.984 INFO KyuubiSessionManager-exec-pool: Thread-809 org.apache.kyuubi.operation.ExecuteStatement: Processing root's query[92377a4f-2dad-4039-be2d-4762946eaea8]: PENDING_STATE -> RUNNING_STATE, statement:
select count(*)  from trace_log_refdes_hive_ro where pcbid ='E23MPM42203176'
24/02/27 16:57:01 INFO ExecuteStatement: Processing root's query[92377a4f-2dad-
24/02/27 16:57:10 INFO SparkContext: Starting job: collect at ExecuteStatement.scala:72
24/02/27 16:57:10 INFO SQLOperationListener: Query [92377a4f-2dad-4039-be2d-4762946eaea8]: Job 6 started with 2 stages, 1 active jobs running
24/02/27 16:57:10 INFO SQLOperationListener: Query [92377a4f-2dad-4039-be2d-4762946eaea8]: Stage 8.0 started with 1 tasks, 1 active stages running
24/02/27 16:57:10 INFO SQLOperationListener: Finished stage: Stage(8, 0); Name: 'collect at ExecuteStatement.scala:72'; Status: succeeded; numTasks: 1; Took: 47 msec
24/02/27 16:57:10 INFO AdaptiveSparkPlanExec: Final plan:
*(2) HashAggregate(keys=[], functions=[count(1)], output=[count(1)#304L])
+- ShuffleQueryStage 0
   +- Exchange SinglePartition, ENSURE_REQUIREMENTS, [plan_id=115]
      +- *(1) HashAggregate(keys=[], functions=[partial_count(1)], output=[count#308L])
         +- *(1) Project
            +- *(1) Filter (isnotnull(pcbid#277) AND (pcbid#277 = E23MPM42203176))
               +- BatchScan trace_log_refdes_hive_ro[pcbid#277] PaimonScan: [trace_log_refdes_hive_ro], PushedFilters: [IsNotNull(pcbid),Equal(pcbid, E23MPM42203176)] RuntimeFilters: []

2024-02-27 16:57:10.578 INFO KyuubiSessionManager-exec-pool: Thread-809 org.apache.kyuubi.operation.ExecuteStatement: Query[92377a4f-2dad-4039-be2d-4762946eaea8] in FINISHED_STATE
2024-02-27 16:57:10.578 INFO KyuubiSessionManager-exec-pool: Thread-809 org.apache.kyuubi.operation.ExecuteStatement: Processing root's query[92377a4f-2dad-4039-be2d-4762946eaea8]: RUNNING_STATE -> FINISHED_STATE, time taken: 8.594 seconds
+-----------+
| count(1)  |
+-----------+
| 3         |
+-----------+
1 row selected (8.604 seconds)
0: jdbc:hive2://10.32.36.142:10009/>

结束

kyuubi整合spark on yarn 至此结束。

本文来自互联网用户投稿，该文观点仅代表作者本人，不代表本站立场。本站仅提供信息存储空间服务，不拥有所有权，不承担相关法律责任。如若转载，请注明出处：http://www.coloradmin.cn/o/1479447.html

如若内容造成侵权/违法违规/事实不符，请联系多彩编程网进行投诉反馈，一经查实，立即删除！

相关文章

【音视频处理】使用ffmpeg实现多个视频合成一个视频（按宫格视图）

【音视频处理】使用ffmpeg实现多个视频合成一个视频（按宫格视图）

先上结果 环境 硬件：通用PC 系统：Windows 测试有效 软件：ffmpeg 解决 0、命令 ffmpeg.exe -i input1.mp4 -i input2.mp4 -i input3.mp4 -i input4.mp4 -filter_complex "[0:v]scaleiw/2:ih/2,pad2*iw:2*ih[a]; [1:v]scaleiw/2:ih/2…
阅读更多...
看待事物的层与次 | DBA与架构的一次对话交流

看待事物的层与次 | DBA与架构的一次对话交流

前言 在计算机软件业生涯中，想必行内人或多或少都能感受到系统架构设计与数据库系统工程的重要性，也能够清晰地认识到在计算机软件行业中技术工程师这个职业所需要的专业素养和必备技能！ 背景 通过自研的数据库监控管理工具，发现 SQL Server 数据库连接数在1-2K之间，想…
阅读更多...
Java进阶-反射

Java进阶-反射

来学习一下Java的反射，通过Class实例获取class信息的方法称为反射（Reflection），内容如下 一、反射机制 1、概述 在运行状态中，对于任意一个类，都能够知道这个类的所有属性和方法；对于任意一…
阅读更多...
UDP数据报套接字编程入门

UDP数据报套接字编程入门

目录 1.TCP和UDP的特点及区别 1.1TCP的特点 1.2UDP的特点 1.3区别 2.UDP Socket的api的介绍 2.1DatagramSocket API 2.2DatagramPacket API 3.回显客户端与服务器 3.1回显服务器 3.1.1UdpEchoServer类的创建 3.1.2服务器的运行方法start() 3.1.3main部分 3.1.4.完整…
阅读更多...
LeetCode #104 二叉树的最大深度

LeetCode #104 二叉树的最大深度

104. 二叉树的最大深度 题目 二叉树的 最大深度 是指从根节点到最远叶子节点的最长路径上的节点数。 示例 1： 输入：root [3,9,20,null,null,15,7] 输出：3 示例 2： 输入：root [1,null,2] 输出：2 分析 …
阅读更多...
CMake、OpenCV 和单元测试

CMake、OpenCV 和单元测试

我写了很多关于 CMake 的文章，如果你感兴趣，可以点击以下链接阅读： CMake VS MakeCMake：在构建世界掀起风暴现代 CMake 使用技巧CMake 交叉编译CMake 生成器已开启 我们将继续对 CMake 的探索，这篇文章技术性高&…
阅读更多...
云商大宗商品使用的是什么系统，有什么功能

云商大宗商品使用的是什么系统，有什么功能

云商大宗商品使用的系统主要包括大宗商品贸易系统、大宗商品仓储管理系统。 大宗商品贸易系统则是一种基于互联网技术的交易平台，旨在为买卖双方提供高效、安全、透明的交易环境。该系统具备商品管理、交易管理、资金管理、数据分析和风险管理等功能模块&#xff0…
阅读更多...
2024年四川媒体新闻发布渠道，媒体邀约资源表

2024年四川媒体新闻发布渠道，媒体邀约资源表

传媒如春雨，润物细无声，大家好，我是51媒体网胡老师。 四川有哪些媒体新闻发布渠道，媒体邀约资源表？ 2024年四川媒体新闻发布渠道，媒体邀约资源表 四川本地媒体：如四川日报、华西都市报、成都商…
阅读更多...
2024年最好的 5 款免费的数据恢复软件

2024年最好的 5 款免费的数据恢复软件

数据意外损坏或丢失？别担心，今天我为大家带来了实用的免费硬盘数据恢复软件！ 在数字化时代，数据的重要性不言而喻。但是，由于种种原因，数据意外损坏或丢失的情况时有发生。此时，一款实用的免费…
阅读更多...
SpringBoot 自定义映射规则resultMap association一对一

SpringBoot 自定义映射规则resultMap association一对一

介绍 例：学生表，班级表，希望在查询学生的时候一起返回该学生的班级，而一个实体类封装的是一个表，如需要多表查询就需要自定义映射。 表结构 班级表 学生表 SQL语句 SELECT a.id,a.name,a.classes,b.id classes…
阅读更多...
JS(JavaScript)中如何实现，复选框checkbox多选功能

JS(JavaScript)中如何实现，复选框checkbox多选功能

起始界面&#xff1a; 代码元素&#xff1a; <p><input type"checkbox" id"checkedAll"> 全选按钮</p><p><input type"checkbox" class"cl"> 选项1</p><p><input type"checkbox&qu…
阅读更多...
张驰咨询：为什么每个应届生都应该参加六西格玛培训？

张驰咨询：为什么每个应届生都应该参加六西格玛培训？

应届生参加六西格玛培训的现象逐渐成为了一个新的趋势&#xff0c;这背后的原因不仅仅是市场对于品质管理人才的渴求&#xff0c;也源于应届生自身发展的需求。本文张驰咨询将从应届生的视角探讨参加六西格玛培训的动机&#xff0c;以及这种培训如何为他们的职业生涯铺路。 为…
阅读更多...
认识通讯协议——TCP/IP、UDP协议的区别，HTTP通讯协议的理解

认识通讯协议——TCP/IP、UDP协议的区别，HTTP通讯协议的理解

目录 引出认识通讯协议1、TCP/IP协议&#xff0c;UDP协议的区别2、HTTP通讯协议的讲解 Redis冲冲冲——缓存三兄弟&#xff1a;缓存击穿、穿透、雪崩缓存击穿缓存穿透缓存雪崩 总结 引出 认识通讯协议——TCP/IP、UDP协议的区别&#xff0c;HTTP通讯协议的理解 认识通讯协议 …
阅读更多...
做活动和会议直播，为什么要多个媒体平台同步直播？

做活动和会议直播，为什么要多个媒体平台同步直播？

传媒如春雨&#xff0c;润物细无声&#xff0c;大家好&#xff0c;我是51媒体网胡老师。 多个媒体平台同步直播活动和会议的原因主要有以下几点&#xff1a; 扩大观众覆盖面&#xff1a;不同的媒体平台拥有各自独特的用户群体&#xff0c;通过在多个媒体平台同步直播&#xff…
阅读更多...
Unity（第二十一部）动画的基础了解（感觉不了解其实也行）

Unity（第二十一部）动画的基础了解（感觉不了解其实也行）

1、动画组件老的是Animations 动画视频Play Automatically 是否自动播放Animate Physics 驱动方式&#xff0c;勾选后是物理驱动Culling Type 剔除方式 默认总是动画化就会一直执行下去&#xff0c;第二个是基于渲染播放&#xff08;离开镜头后不执行&#xff09;&#xff0c; …
阅读更多...
如何解决 C/C++ 编译器优化导致的编译BUG（程序崩溃）支援VC++/CLANG/GCC

如何解决 C/C++ 编译器优化导致的编译BUG（程序崩溃）支援VC++/CLANG/GCC

本文仅适用于&#xff0c;有愿意、爱捣鼓的童靴。 因编译器优化导致编译BUG&#xff0c;即DEBUG下面无故障稳定工作&#xff0c;但RELESE下程序会在特定函数位置上崩溃。 这要求 C/C 开发人员拥有最基本的素质&#xff0c;需要能够承受&#xff0c;逐行审视编译器输出的目标平…
阅读更多...
yolov9,使用自定义的数据训练推理

yolov9,使用自定义的数据训练推理

[源码 &#x1f40b;]( GitHub - WongKinYiu/yolov9: Implementation of paper - YOLOv9: Learning What You Want to Learn Using Programmable Gradient Information) [论文 &#x1f4d8;](arxiv.org/pdf/2402.13616.pdf) 论文摘要&#xff1a;本文介绍了一种新的目标检测…
阅读更多...
【 JS 进阶 】正则表达式

【 JS 进阶 】正则表达式

“生命是一幅抽象画&#xff0c;每一笔都是独特的经历。在色彩的交织中&#xff0c;用坚定的画笔勾勒出自己的轨迹&#xff0c;让每一幅画作都成为奋斗的印记。” - 亚历山大斯密斯 正则表达式 正则表达式&#xff08;Regular Expression&#xff09;是一种字符串匹配的模式&a…
阅读更多...
二、TensorFlow结构分析（2）

二、TensorFlow结构分析（2）

目录 1、会话 1.1 __init__(target,graphNone,configNone) 1.2 会话的run() 1.3 feed操作 TF数据流图图与TensorBoard会话张量变量OP高级API 1、会话 1.1 __init__(target,graphNone,configNone) def session_demo():# 会话的演示# Tensorflow实现加法运算a_t tf.constan…
阅读更多...
LNMP 架构

LNMP 架构

搭建 LNMP 架构需要安装 nginx mysql php 软件 linux nginx mysql php结合 在一台主机上搭建LNMP架构 一、编译安装 nginx 安装依赖包 yum -y install pcre-devel zlib-devel gcc gcc-c make 创建nginx程序用户 &#xff08;Nginx 服务程序默认以 nobody 身份运…
阅读更多...
推荐文章
最新文章

Copyright @ 2022~2023