hive中的datagrip和beeline客户端的权限问题

news2024/11/18 23:41:10

hive中的datagrip和beeline客户端的权限问题

使用ranger和kerberos配置了hadoop和hive,今天想用来测试其权限
测试xwq用户:
1.首先添加xwq用户权限,命令如下:

useradd  xwq -G hadoop
echo xwq | passwd --stdin xwq
echo 'xwq  ALL=(ALL)  NOPASSWD: NOPASSWD: ALL' >> /etc/sudoers
kadmin -padmin/admin -wNTVfPQY9kNs6  -q"addprinc -randkey xwq"
kadmin -padmin/admin -wNTVfPQY9kNs6  -q"xst -k /etc/security/keytab/xwq.keytab xwq"
chown xwq:hadoop /etc/security/keytab/xwq.keytab
chmod 660 /etc/security/keytab/xwq.keytab

2.进行认证

[root@hadoop102 keytab]# kinit xwq
Password for xwq@EXAMPLE.COM: 
[root@hadoop102 keytab]# klist 
Ticket cache: FILE:/tmp/krb5cc_0
Default principal: xwq@EXAMPLE.COM

Valid starting       Expires              Service principal
07/01/2023 10:09:21  07/02/2023 10:09:21  krbtgt/EXAMPLE.COM@EXAMPLE.COM
	renew until 07/08/2023 10:09:21

3.连接beeline客户端

[root@hadoop102 ~]# beeline 
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/module/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/ha/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Beeline version 3.1.2 by Apache Hive
beeline> !connect jdbc:hive2://hadoop102:10000/;principal=hive/hadoop102@EXAMPLE.COM
Connecting to jdbc:hive2://hadoop102:10000/;principal=hive/hadoop102@EXAMPLE.COM
Connected to: Apache Hive (version 3.1.2)
Driver: Hive JDBC (version 3.1.2)
Transaction isolation: TRANSACTION_REPEATABLE_READ
0: jdbc:hive2://hadoop102:10000/> select current_user();
INFO  : Compiling command(queryId=hive_20230701095227_419c1fe7-2f6b-47af-828c-bcf67fd6043a): select current_user()
INFO  : Concurrency mode is disabled, not creating a lock manager
INFO  : Semantic Analysis Completed (retrial = false)`在这里插入代码片`
INFO  : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:_c0, type:string, comment:null)], properties:null)
INFO  : Completed compiling command(queryId=hive_20230701095227_419c1fe7-2f6b-47af-828c-bcf67fd6043a); Time taken: 0.212 seconds
INFO  : Concurrency mode is disabled, not creating a lock manager
INFO  : Executing command(queryId=hive_20230701095227_419c1fe7-2f6b-47af-828c-bcf67fd6043a): select current_user()
INFO  : Completed executing command(queryId=hive_20230701095227_419c1fe7-2f6b-47af-828c-bcf67fd6043a); Time taken: 0.0 seconds
INFO  : OK
INFO  : Concurrency mode is disabled, not creating a lock manager
+------+
| _c0  |
+------+
| xwq  |
+------+
1 row selected (0.301 seconds)

4.执行插入语句

0: jdbc:hive2://hadoop102:10000/> insert into student values(2,'1');
INFO  : Compiling command(queryId=hive_20230701095229_d7d5807d-ff37-4aef-81d5-bc10fd929ebf): insert into student values(2,'1')
INFO  : Concurrency mode is disabled, not creating a lock manager
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:col1, type:int, comment:null), FieldSchema(name:col2, type:string, comment:null)], properties:null)
INFO  : Completed compiling command(queryId=hive_20230701095229_d7d5807d-ff37-4aef-81d5-bc10fd929ebf); Time taken: 0.318 seconds
INFO  : Concurrency mode is disabled, not creating a lock manager
INFO  : Executing command(queryId=hive_20230701095229_d7d5807d-ff37-4aef-81d5-bc10fd929ebf): insert into student values(2,'1')
INFO  : Query ID = hive_20230701095229_d7d5807d-ff37-4aef-81d5-bc10fd929ebf
INFO  : Total jobs = 1
INFO  : Launching Job 1 out of 1
INFO  : Starting task [Stage-1:MAPRED] in serial mode
ERROR : Job hasn't been submitted after 61s. Aborting it.
Possible reasons include network issues, errors in remote driver or the cluster has no available resources, etc.
Please check YARN or Spark driver's logs for further information.
The timeout is controlled by hive.spark.job.monitor.timeout.
ERROR : FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.spark.SparkTask. Spark job failed during runtime. Please check stacktrace for the root cause.
INFO  : Completed executing command(queryId=hive_20230701095229_d7d5807d-ff37-4aef-81d5-bc10fd929ebf); Time taken: 216.921 seconds
INFO  : Concurrency mode is disabled, not creating a lock manager
Error: Error while processing statement: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.spark.SparkTask. Spark job failed during runtime. Please check stacktrace for the root cause. (state=42000,code=2)
0: jdbc:hive2://hadoop102:10000/> insert into student values(2,'1');
INFO  : Compiling command(queryId=hive_20230701095708_a92293d5-eb6e-448c-b623-c5c49660ae66): insert into student values(2,'1')
INFO  : Concurrency mode is disabled, not creating a lock manager
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:col1, type:int, comment:null), FieldSchema(name:col2, type:string, comment:null)], properties:null)
INFO  : Completed compiling command(queryId=hive_20230701095708_a92293d5-eb6e-448c-b623-c5c49660ae66); Time taken: 0.28 seconds
INFO  : Concurrency mode is disabled, not creating a lock manager
INFO  : Executing command(queryId=hive_20230701095708_a92293d5-eb6e-448c-b623-c5c49660ae66): insert into student values(2,'1')
INFO  : Query ID = hive_20230701095708_a92293d5-eb6e-448c-b623-c5c49660ae66
INFO  : Total jobs = 1
INFO  : Launching Job 1 out of 1
INFO  : Starting task [Stage-1:MAPRED] in serial mode
ERROR : Job hasn't been submitted after 61s. Aborting it.
Possible reasons include network issues, errors in remote driver or the cluster has no available resources, etc.
Please check YARN or Spark driver's logs for further information.
The timeout is controlled by hive.spark.job.monitor.timeout.
ERROR : FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.spark.SparkTask. Spark job failed during runtime. Please check stacktrace for the root cause.
INFO  : Completed executing command(queryId=hive_20230701095708_a92293d5-eb6e-448c-b623-c5c49660ae66); Time taken: 181.098 seconds
INFO  : Concurrency mode is disabled, not creating a lock manager
Error: Error while processing statement: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.spark.SparkTask. Spark job failed during runtime. Please check stacktrace for the root cause. (state=42000,code=2)

任务执行失败,后面观察了yarn界面,发现是向yarn成功提交了job,job也经历 了accept到running的过程,但是最后执行失败了,日志的报错信息如下:

2023-07-01T10:16:07,513  INFO [44a5e8c7-dc6f-43f7-8a98-1037e8deffa3 HiveServer2-Handler-Pool: Thread-85] session.SessionState: Resetting thread name to  HiveServer2-Handler-Pool: Thread-85
2023-07-01T10:16:30,717 ERROR [HiveServer2-Background-Pool: Thread-158] client.SparkClientImpl: Timed out waiting for client to connect.
Possible reasons include network issues, errors in remote driver or the cluster has no available resources, etc.
Please check YARN or Spark driver's logs for further information.
java.util.concurrent.ExecutionException: java.util.concurrent.TimeoutException: Timed out waiting for client connection.
	at io.netty.util.concurrent.AbstractFuture.get(AbstractFuture.java:41) ~[netty-all-4.1.17.Final.jar:4.1.17.Final]
	at org.apache.hive.spark.client.SparkClientImpl.<init>(SparkClientImpl.java:106) ~[hive-exec-3.1.2.jar:3.1.2]
	at org.apache.hive.spark.client.SparkClientFactory.createClient(SparkClientFactory.java:88) ~[hive-exec-3.1.2.jar:3.1.2]
	at org.apache.hadoop.hive.ql.exec.spark.RemoteHiveSparkClient.createRemoteClient(RemoteHiveSparkClient.java:105) ~[hive-exec-3.1.2.jar:3.1.2]
	at org.apache.hadoop.hive.ql.exec.spark.RemoteHiveSparkClient.<init>(RemoteHiveSparkClient.java:101) ~[hive-exec-3.1.2.jar:3.1.2]
	at org.apache.hadoop.hive.ql.exec.spark.HiveSparkClientFactory.createHiveSparkClient(HiveSparkClientFactory.java:76) ~[hive-exec-3.1.2.jar:3.1.2]
	at org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionImpl.open(SparkSessionImpl.java:87) ~[hive-exec-3.1.2.jar:3.1.2]
	at org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionManagerImpl.getSession(SparkSessionManagerImpl.java:115) ~[hive-exec-3.1.2.jar:3.1.2]
	at org.apache.hadoop.hive.ql.exec.spark.SparkUtilities.getSparkSession(SparkUtilities.java:136) ~[hive-exec-3.1.2.jar:3.1.2]
	at org.apache.hadoop.hive.ql.exec.spark.SparkTask.execute(SparkTask.java:115) ~[hive-exec-3.1.2.jar:3.1.2]
	at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:205) ~[hive-exec-3.1.2.jar:3.1.2]
	at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:97) ~[hive-exec-3.1.2.jar:3.1.2]
	at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2664) ~[hive-exec-3.1.2.jar:3.1.2]
	at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:2335) ~[hive-exec-3.1.2.jar:3.1.2]
	at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:2011) ~[hive-exec-3.1.2.jar:3.1.2]
	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1709) ~[hive-exec-3.1.2.jar:3.1.2]
	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1703) ~[hive-exec-3.1.2.jar:3.1.2]
	at org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:157) ~[hive-exec-3.1.2.jar:3.1.2]
	at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:224) ~[hive-service-3.1.2.jar:3.1.2]
	at org.apache.hive.service.cli.operation.SQLOperation.access$700(SQLOperation.java:87) ~[hive-service-3.1.2.jar:3.1.2]
	at org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:316) ~[hive-service-3.1.2.jar:3.1.2]
	at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_361]
	at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_361]
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729) ~[hadoop-common-3.1.3.jar:?]
	at org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:329) ~[hive-service-3.1.2.jar:3.1.2]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_361]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_361]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_361]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_361]
	at java.lang.Thread.run(Thread.java:750) [?:1.8.0_361]
Caused by: java.util.concurrent.TimeoutException: Timed out waiting for client connection.
	at org.apache.hive.spark.client.rpc.RpcServer$2.run(RpcServer.java:172) ~[hive-exec-3.1.2.jar:3.1.2]
	at io.netty.util.concurrent.PromiseTask$RunnableAdapter.call(PromiseTask.java:38) ~[netty-all-4.1.17.Final.jar:4.1.17.Final]
	at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:120) ~[netty-all-4.1.17.Final.jar:4.1.17.Final]
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) ~[netty-all-4.1.17.Final.jar:4.1.17.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:403) ~[netty-all-4.1.17.Final.jar:4.1.17.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463) ~[netty-all-4.1.17.Final.jar:4.1.17.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) ~[netty-all-4.1.17.Final.jar:4.1.17.Final]
	... 1 more
2023-07-01T10:16:30,741 ERROR [HiveServer2-Background-Pool: Thread-158] spark.SparkTask: Failed to execute spark task, with exception 'org.apache.hadoop.hive.ql.metadata.HiveException(Failed to create Spark client for Spark session 2c551365-6d3c-458d-8d7c-3c8566d3c802)'
org.apache.hadoop.hive.ql.metadata.HiveException: Failed to create Spark client for Spark session 2c551365-6d3c-458d-8d7c-3c8566d3c802
	at org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionImpl.getHiveException(SparkSessionImpl.java:215)
	at org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionImpl.open(SparkSessionImpl.java:92)
	at org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionManagerImpl.getSession(SparkSessionManagerImpl.java:115)
	at org.apache.hadoop.hive.ql.exec.spark.SparkUtilities.getSparkSession(SparkUtilities.java:136)
	at org.apache.hadoop.hive.ql.exec.spark.SparkTask.execute(SparkTask.java:115)
	at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:205)
	at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:97)
	at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2664)
	at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:2335)
	at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:2011)
	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1709)
	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1703)
	at org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:157)
	at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:224)
	at org.apache.hive.service.cli.operation.SQLOperation.access$700(SQLOperation.java:87)
	at org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:316)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
	at org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:329)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:750)

检查了spark分配的资源和yarn的资源,资源充足,不知道是什么原因

后面用sarah用户进行测试,发现job执行成功,结果如下:

INFO  : Concurrency mode is disabled, not creating a lock manager
+--------+
|  _c0   |
+--------+
| sarah  |
+--------+
1 row selected (0.399 seconds)
0: jdbc:hive2://hadoop102:10000/> insert into student values(2,'1');
INFO  : Compiling command(queryId=hive_20230701095037_eb26098a-e4e9-438b-a33b-9bf8b6205d1f): insert into student values(2,'1')
INFO  : Concurrency mode is disabled, not creating a lock manager
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:col1, type:int, comment:null), FieldSchema(name:col2, type:string, comment:null)], properties:null)
INFO  : Completed compiling command(queryId=hive_20230701095037_eb26098a-e4e9-438b-a33b-9bf8b6205d1f); Time taken: 0.281 seconds
INFO  : Concurrency mode is disabled, not creating a lock manager
INFO  : Executing command(queryId=hive_20230701095037_eb26098a-e4e9-438b-a33b-9bf8b6205d1f): insert into student values(2,'1')
INFO  : Query ID = hive_20230701095037_eb26098a-e4e9-438b-a33b-9bf8b6205d1f
INFO  : Total jobs = 1
INFO  : Launching Job 1 out of 1
INFO  : Starting task [Stage-1:MAPRED] in serial mode
INFO  : Running with YARN Application = application_1688108003994_0006
INFO  : Kill Command = /opt/ha/hadoop/bin/yarn application -kill application_1688108003994_0006
INFO  : Hive on Spark Session Web UI URL: http://hadoop102:32853
INFO  : 
Query Hive on Spark job[0] stages: [0, 1]
INFO  : Spark job[0] status = RUNNING
INFO  : Job Progress Format
CurrentTime StageId_StageAttemptId: SucceededTasksCount(+RunningTasksCount-FailedTasksCount)/TotalTasksCount
INFO  : 2023-07-01 09:51:30,101	Stage-0_0: 0(+1)/1	Stage-1_0: 0/1	
INFO  : 2023-07-01 09:51:33,126	Stage-0_0: 0(+1)/1	Stage-1_0: 0/1	
INFO  : 2023-07-01 09:51:34,131	Stage-0_0: 1/1 Finished	Stage-1_0: 0/1	
INFO  : 2023-07-01 09:51:35,139	Stage-0_0: 1/1 Finished	Stage-1_0: 1/1 Finished	
INFO  : Spark job[0] finished successfully in 8.11 second(s)
INFO  : Starting task [Stage-0:MOVE] in serial mode
INFO  : Loading data to table default.student from hdfs://mycluster/user/hive/warehouse/student/.hive-staging_hive_2023-07-01_09-50-37_765_7507751690563815963-7/-ext-10000
INFO  : Starting task [Stage-2:STATS] in serial mode
INFO  : Completed executing command(queryId=hive_20230701095037_eb26098a-e4e9-438b-a33b-9bf8b6205d1f); Time taken: 57.47 seconds
INFO  : OK
INFO  : Concurrency mode is disabled, not creating a lock manager
No rows affected (57.758 seconds)

不知道为什么sarah用户可以成功,但是xwq用户失败了,后面有使用了hdfs进行测试,和xwq用户一样失败
如下结果:

INFO  : Concurrency mode is disabled, not creating a lock manager
+-------+
|  _c0  |
+-------+
| hdfs  |
+-------+
1 row selected (0.376 seconds)
0: jdbc:hive2://hadoop102:10000/> insert into student values(2,'1');
INFO  : Compiling command(queryId=hive_20230630172135_4dc05a0d-7783-48ec-a6d7-8b11f81f8f85): insert into student values(2,'1')
INFO  : Concurrency mode is disabled, not creating a lock manager
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:col1, type:int, comment:null), FieldSchema(name:col2, type:string, comment:null)], properties:null)
INFO  : Completed compiling command(queryId=hive_20230630172135_4dc05a0d-7783-48ec-a6d7-8b11f81f8f85); Time taken: 0.841 seconds
INFO  : Concurrency mode is disabled, not creating a lock manager
INFO  : Executing command(queryId=hive_20230630172135_4dc05a0d-7783-48ec-a6d7-8b11f81f8f85): insert into student values(2,'1')
INFO  : Query ID = hive_20230630172135_4dc05a0d-7783-48ec-a6d7-8b11f81f8f85
INFO  : Total jobs = 1
INFO  : Launching Job 1 out of 1
INFO  : Starting task [Stage-1:MAPRED] in serial mode
ERROR : FAILED: Execution Error, return code 30041 from org.apache.hadoop.hive.ql.exec.spark.SparkTask. Failed to create Spark client for Spark session 98d08d85-f20b-4ce9-8e88-aed333485cb5
INFO  : Completed executing command(queryId=hive_20230630172135_4dc05a0d-7783-48ec-a6d7-8b11f81f8f85); Time taken: 300.171 seconds
INFO  : Concurrency mode is disabled, not creating a lock manager
Error: Error while processing statement: FAILED: Execution Error, return code 30041 from org.apache.hadoop.hive.ql.exec.spark.SparkTask. Failed to create Spark client for Spark session 98d
0: jdbc:hive2://hadoop102:10000/> !quit
Closing: 0: jdbc:hive2://hadoop102:10000/;principal=hive/hadoop102@EXAMPLE.COM

分析:
看日志发现也看不出什么,但是我刚刚看yarn web页面时,突然发现了一个比较明显的错误

在这里插入图片描述
详细的bug是

Diagnostics:	
Application application_1688108003994_0009 failed 1 times (global limit =2; local limit is =1) due to AM Container for appattempt_1688108003994_0009_000001 exited with exitCode: -1000
Failing this attempt.Diagnostics: [2023-07-01 10:10:53.661]Application application_1688108003994_0009 initialization failed (exitCode=255) with output: main : command provided 0
main : run as user is xwq
main : requested yarn user is xwq
User xwq not found
For more detailed output, check the application tracking page: http://hadoop103:8088/cluster/app/application_1688108003994_0009 Then click on links to logs of each attempt.
. Failing the application.

看到這個錯誤我就知道了,之前碰到過這樣的錯誤,當時是執行一個MR任務是失敗

解決辦法
在其他節點上執行

useradd xwq  -G hadoop
echo xwq | passwd --stdin xwq
echo 'xwq  ALL=(ALL)  NOPASSWD: NOPASSWD: ALL' >> /etc/sudoers
kadmin -padmin/admin -wNTVfPQY9kNs6  -q"addprinc -randkey xwq"

然後再執行job

INFO  : Concurrency mode is disabled, not creating a lock manager
+------+
| _c0  |
+------+
| xwq  |
+------+
1 row selected (0.316 seconds)
0: jdbc:hive2://hadoop102:10000/> insert into student values(2,'1');
INFO  : Compiling command(queryId=hive_20230701104001_8b825171-b12d-416a-9044-14e40ce66b4e): insert into student values(2,'1')
INFO  : Concurrency mode is disabled, not creating a lock manager
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:col1, type:int, comment:null), FieldSchema(name:col2, type:string, comment:null)], properties:null)
INFO  : Completed compiling command(queryId=hive_20230701104001_8b825171-b12d-416a-9044-14e40ce66b4e); Time taken: 0.274 seconds
INFO  : Concurrency mode is disabled, not creating a lock manager
INFO  : Executing command(queryId=hive_20230701104001_8b825171-b12d-416a-9044-14e40ce66b4e): insert into student values(2,'1')
INFO  : Query ID = hive_20230701104001_8b825171-b12d-416a-9044-14e40ce66b4e
INFO  : Total jobs = 1
INFO  : Launching Job 1 out of 1
INFO  : Starting task [Stage-1:MAPRED] in serial mode
INFO  : Running with YARN Application = application_1688108003994_0011
INFO  : Kill Command = /opt/ha/hadoop/bin/yarn application -kill application_1688108003994_0011
INFO  : Hive on Spark Session Web UI URL: http://hadoop104:38576
INFO  : 
Query Hive on Spark job[0] stages: [0, 1]
INFO  : Spark job[0] status = RUNNING
INFO  : Job Progress Format
CurrentTime StageId_StageAttemptId: SucceededTasksCount(+RunningTasksCount-FailedTasksCount)/TotalTasksCount
INFO  : 2023-07-01 10:40:47,412	Stage-0_0: 0(+1)/1	Stage-1_0: 0/1	
INFO  : 2023-07-01 10:40:50,428	Stage-0_0: 0(+1)/1	Stage-1_0: 0/1	
INFO  : 2023-07-01 10:40:51,432	Stage-0_0: 1/1 Finished	Stage-1_0: 1/1 Finished	
INFO  : Spark job[0] finished successfully in 7.05 second(s)
INFO  : Starting task [Stage-0:MOVE] in serial mode
INFO  : Loading data to table default.student from hdfs://mycluster/user/hive/warehouse/student/.hive-staging_hive_2023-07-01_10-40-01_182_6190336792685741127-8/-ext-10000
INFO  : Starting task [Stage-2:STATS] in serial mode
INFO  : Completed executing command(queryId=hive_20230701104001_8b825171-b12d-416a-9044-14e40ce66b4e); Time taken: 50.419 seconds
INFO  : OK
INFO  : Concurrency mode is disabled, not creating a lock manager
No rows affected (50.703 seconds)
0: jdbc:hive2://hadoop102:10000/> 

執行成功

至於hdfs用戶為什麼沒有成功,是因為yarn是禁止hdfs用戶提交任務的

所以以後看hive的bug可以看log日誌,還是可以yarn web頁面Diagnostics板塊

本文来自互联网用户投稿,该文观点仅代表作者本人,不代表本站立场。本站仅提供信息存储空间服务,不拥有所有权,不承担相关法律责任。如若转载,请注明出处:http://www.coloradmin.cn/o/706727.html

如若内容造成侵权/违法违规/事实不符,请联系多彩编程网进行投诉反馈,一经查实,立即删除!

相关文章

【笔记】EDA学习笔记

网课&#xff1a;立创EDA&#xff08;专业版&#xff09;电路设计与制作快速入门 资料&#xff1a;pan.baidu.com/s/1UlcfvAZ13s_wOHo3cvNQxA?pwdb9x8 提取码&#xff1a;b9x8 解压密码&#xff1a;123456 1.创建项目 略 2.原理图设计环境设置 2.1设置 默认网格尺寸&#…

阿里云云效 流水线发布spring cloud项目及Vue项目

1. 单体spring boot 项目流水线主机部署 1.1 创建流水线 页面地址 我们选择模板,当然也可以自己新建一个 1.2 添加流水线源 这里是根据自己的代码仓库决定的,我用的gitee,需要添加服务连接 添加服务连接 新建服务授权/证书 点击新建自动跳转至gitee认证页面,同意授权即可 授权…

网安等保 | 主机安全之KylinOS银河麒麟服务器配置优化与安全加固基线文档脚本分享...

欢迎关注「全栈工程师修炼指南」公众号 点击 &#x1f447; 下方卡片 即可关注我哟! 设为「星标⭐」每天带你 基础入门 到 进阶实践 再到 放弃学习&#xff01; “ 花开堪折直须折&#xff0c;莫待无花空折枝。 ” 作者主页&#xff1a;[ https://www.weiyigeek.top ] 博客&…

线程不安全举例

1、举例说明集合类线程不安全 &#xff08;1&#xff09;查看源码可证明 看ArrayList源码 没有sync、lock&#xff0c;线程不安全 &#xff08;2&#xff09;创建多个线程写入读取数据 List<String> list new ArrayList<>(); for (int i 1; i <30 ; i) {n…

2023年Web安全学习路线总结!430页Web安全学习笔记(附PDF)

关键词&#xff1a;网络安全入门、渗透测试学习、零基础学安全、网络安全学习路线、web安全攻防笔记、渗透测试路线图 网络安全的范畴很大&#xff0c;相较于二进制安全等方向的高门槛、高要求&#xff0c;Web安全体系比较成熟&#xff0c;在现阶段来看&#xff0c;但凡有自己…

k8s下安装redis

一、使用helm安装redis 执行以下命令添加redis的repo helm repo add bitnami https://charts.bitnami.com/bitnami 创建的master和replica pod的默认size是8Gi&#xff0c;如果k8s的node没有足够的空间&#xff0c;会抛出如下错误&#xff1a;default-scheduler 0/3 nodes …

安全防御 --- SSL VPN

附&#xff1a;无线项目介绍 SSL VPN 有浏览器的设备就可以使用SSL&#xff0c;进而使用SSL VPN。无需担心客户端问题&#xff0c;所以SSL VPN也称为无客户端VPN。SSL VPN在client to lan场景下特别有优势。 实际实现过程&#xff08;基于TCP实现&#xff09; &#xff08;1&…

Nginx SSL使用自制证书

1. 生成证书 keytool -genkey -v -alias <Alias别名> -keyalg RSA -keystore <KeyStore文件> -validity <有效期> keytool -genkey -v -alias nginx -keyalg RSA -keystore nginx.keystore -validity 36500 alias别名为 nginxkeystore文件为 nginx.keystore…

C/C++内存管理(内存分布、动态内存分配、动态内存分配与释放、内存泄漏等)

喵~ 内存之5大区&#xff08;栈区、堆区、静态区、常量区、代码区&#xff09;C/C中各自的内存分配操作符内存泄露?内存泄漏检测方法 内存之5大区&#xff08;栈区、堆区、静态区、常量区、代码区&#xff09; 1、栈区&#xff08;stack&#xff09;&#xff1a;由编译器自动分…

避雷投稿!这3本SCIE期刊处于On Hold状态!有剔除风险!

今年3月份科睿唯安剔除的大量期刊中&#xff0c;有几本期刊之前被WOS数据库标记为“On Hold”状态&#xff0c;后面也出现了一些期刊解除“On Hold”状态的情况。 目前&#xff0c;WOS数据库查询有3本SCIE期刊正处于“On Hold”状态&#xff0c;这是什么信号呢&#xff1f;对于…

org.apache.thrift.transport.TTransportException: GSS initiate failed

hive配置kerberos發現錯誤&#xff0c;就所有的配置都配好了&#xff0c;但是使用datagrip連接hive時報錯如下&#xff1a; org.apache.thrift.transport.TTransportException: GSS initiate failed 後面是沒有認證的原因&#xff0c;使用kinit 認證即可 還有一種情況是 我使…

7.1下周黄金非农数据是否会影响美联储七月暂停加息?

近期有哪些消息面影响黄金走势&#xff1f;下周黄金多空该如何研判&#xff1f; ​黄金消息面解析&#xff1a;周五(6月30日)&#xff0c;现货黄金价格收盘上涨&#xff0c;盘中一度升破1920美元关口。美国疲软的个人消费支出推动了美国国债收益率的下降&#xff0c;从而导致美…

libevent实践04:监听管道第二季

一 本次实例使用函数简介 事件集合初始化&#xff1a; struct event_base *event_init(void); 示例&#xff1a; struct event_base *base event_init(); 单个事件初始化 void event_set(struct event *ev, evutil_socket_t fd, short events,void (*callback)(evutil_s…

两小时快速入门 TypeScript 基础(二)面向对象

个人简介 &#x1f440;个人主页&#xff1a; 前端杂货铺 &#x1f64b;‍♂️学习方向&#xff1a; 主攻前端方向&#xff0c;也会涉及到服务端&#xff08;Node.js 等&#xff09; &#x1f4c3;个人状态&#xff1a; 2023届本科毕业生&#xff0c;已拿多个前端 offer&#x…

【实战】 二、React 与 Hook 应用:实现项目列表 —— React17+React Hook+TS4 最佳实践,仿 Jira 企业级项目(二)

文章目录 一、项目起航&#xff1a;项目初始化与配置二、React 与 Hook 应用&#xff1a;实现项目列表1.新建文件2.状态提升3.新建utils4.Custom Hook 学习内容来源&#xff1a;React React Hook TS 最佳实践-慕课网 相对原教程&#xff0c;我在学习开始时&#xff08;2023.0…

【每天40分钟,我们一起用50天刷完 (剑指Offer)】第十二天 12/50

专注 效率 记忆 预习 笔记 复习 做题 欢迎观看我的博客&#xff0c;如有问题交流&#xff0c;欢迎评论区留言&#xff0c;一定尽快回复&#xff01;&#xff08;大家可以去看我的专栏&#xff0c;是所有文章的目录&#xff09;   文章字体风格&#xff1a; 红色文字表示&#…

服务运营|MS 论文解读:MIR 增强的Benders 分解用于具有到达率不确定性的多类服务系统人员调度问题

推文作者&#xff1a;王畅&#xff0c;蔡君洋&#xff0c;Guo 编者按 劳动力调度是服务系统运营的一个重要应用场景&#xff0c;当系统中的服务集和客户集异构且客户到达率不确定时&#xff0c;劳动力调度问题会变得特别复杂。本文因此引入了一个两阶段的随机整数规划模型&…

【静态单元格 Objective-C语言】

一、预习QQ动态 1.我们今天,是三、到四个任务,第一个任务,就是这个,预习QQ动态 OK,看到这个效果了吗 这是咱们第一个任务,第一个任务,是实现这么一个效果, 一看这个,就肯定是个啥,UITableView吧, 是不是肯定是一个TableView啊 又能滚动,又能显示一行一行的数据…

Qt QGraphicsScene、QGraphicsView类实现仪表盘

Qt QGraphicsScene、QGraphicsView类实现仪表盘 【1】UI界面设计【2】效果【3】QGraphicsScene简介【4】QGraphicsEllipseItem简介【5】QGraphicsPolygonItem简介【6】QGraphicsLineItem简介【7】QGraphicsView简介【8】仪表源码头文件源码 【1】UI界面设计 【2】效果 【3】QGr…

OpenStack(T版)——对象存储(Swift)服务介绍与安装

文章目录 OpenStack(T版)——对象存储(Swift)服务介绍与安装安装和配置(controller)准备安装和配置Swift对象存储服务组件创建账户 Ring创建容器 Ring创建对象 RingSwift 存储系统的主配置文件修改文件属主 安装和配置(compute)准备配置Swift对象存储服务组件 验证(1)加载环境变…