Ambari2.7.5安装Flink1.14

news2024/11/17 10:34:54

文章目录

    • 下载Flink配置安装源
    • 下载ambari-flink-service服务
    • 修改配置文件
    • 创建用户和组
    • 重启Ambari
    • 登录Ambari安装Flink
    • 提交Flink任务
      • Flink 直接单独提交到 On Yarn
      • 指定Flink在Yarn跑的容器运行Flink
    • 异常
      • 异常1
      • 异常2
      • 异常3

下载Flink配置安装源

wget https://archive.apache.org/dist/flink/flink-1.14.2/flink-1.14.2-bin-scala_2.12.tgz
wget https://repo.maven.apache.org/maven2/org/apache/flink/flink-shaded-hadoop-2-uber/2.8.3-10.0/flink-shaded-hadoop-2-uber-2.8.3-10.0.jar
mkdir  /var/www/html/flink
mv flink-1.14.2-bin-scala_2.12.tgz /var/www/html/flink/
mv flink-shaded-hadoop-2-uber-2.8.3-10.0.jar /var/www/html/flink/

所有稳定版本都可以在https://archive.apache.org/dist/flink/获取

下载ambari-flink-service服务

  • 查看HDP版本
[root@bigdata ~]# VERSION=`hdp-select status hadoop-client | sed 's/hadoop-client - \([0-9]\.[0-9]\).*/\1/'`
[root@bigdata~]# echo $VERSION
3.1
  • 下载ambari-flink-service服务到 ambari-server 资源目录下
# git clone https://github.com/abajwa-hw/ambari-flink-service.git   /var/lib/ambari-server/resources/stacks/HDP/$VERSION/services/FLINK

Cloning into '/var/lib/ambari-server/resources/stacks/HDP/3.1/services/FLINK'...
remote: Enumerating objects: 198, done.
remote: Counting objects: 100% (6/6), done.
remote: Compressing objects: 100% (5/5), done.
remote: Total 198 (delta 0), reused 3 (delta 0), pack-reused 192
Receiving objects: 100% (198/198), 2.09 MiB | 982.00 KiB/s, done.
Resolving deltas: 100% (89/89), done.
  • 查看是否clone成功
# ll /var/lib/ambari-server/resources/stacks/HDP/$VERSION/services/FLINK
total 32
drwxr-xr-x 2 root root 4096 Jan 13 14:01 configuration
-rw-r--r-- 1 root root  223 Jan 13 14:01 kerberos.json
-rw-r--r-- 1 root root 1777 Jan 13 14:01 metainfo.xml
drwxr-xr-x 3 root root 4096 Jan 13 14:01 package
-rwxr-xr-x 1 root root 8114 Jan 13 14:01 README.md
-rw-r--r-- 1 root root  125 Jan 13 14:01 role_command_order.json
drwxr-xr-x 2 root root 4096 Jan 13 14:01 screenshots

修改配置文件

修改配置文件

/var/lib/ambari-server/resources/stacks/HDP/3.1/services/FLINK/metainfo.xml

<services>
        <service>
            <name>FLINK</name>
            <displayName>Flink</displayName>
            <!-- 修改我们需要安装的版本号-->
            <version>1.14.2</version>

修改JAVA_HOME

vim /var/lib/ambari-server/resources/stacks/HDP/3.1/services/FLINK/configuration/flink-env.xml

env.java.home: /opt/jdk1.8.0_151

修改flink-ambari-config.xml

修改下载地址(地址为我们自己的httpd服务中放置的安装包)

vim /var/lib/ambari-server/resources/stacks/HDP/3.1/services/FLINK/configuration/flink-ambari-config.xml

  <property>
    <name>flink_download_url</name>
    <value>http://172.16.24.194/flink/flink-1.14.2-bin-scala_2.12.tgz</value>
    <description>Snapshot download location. Downloaded when setup_prebuilt is true</description>
  </property>

  <property>
    <name>flink_hadoop_shaded_jar</name>
    <value>http://172.16.24.194/flink/flink-shaded-hadoop-2-uber-2.8.3-10.0.jar</value>
    <description>Flink shaded hadoop jar download location. Downloaded when setup_prebuilt is true</description>
  </property>

创建用户和组

groupadd flink
useradd  -d /home/flink  -g flink flink

重启Ambari

ambari-server restart

登录Ambari安装Flink

在这里插入图片描述

选择Flink需要安装到哪台服务器

在这里插入图片描述

配置 Flink on yarn 故障转移方式

在Custom flink-env中添加

<property>
    <name>yarn.client.failover-proxy-provider</name>
    <value>org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider</value>
</property>

在这里插入图片描述

点击Next

在这里插入图片描述

点击DEPLOY

在这里插入图片描述

安装成功

在这里插入图片描述

在Linux服务器中配置flink环境变量

ln -s /opt/flink/bin/flink /usr/bin/flink

提交Flink任务

Flink 直接单独提交到 On Yarn

参数-m yarn-cluster

/opt/flink/bin/flink run \
-m yarn-cluster \
-p $P  \
-ys $YS \
-yjm $YJM \
-ytm $YTM \
-yt $JAR_PATH/lib \
-ynm $YNM \
-yD env.java.opts="-Dfile.encoding=UTF-8" \
-c $START_CLASS \
$JAR_PATH/$JAR_NAME.jar $1

指定Flink在Yarn跑的容器运行Flink

启动Flink后可以在yarn的ui上看到一个应用（应用有一个appId）

-yid application_1673426410002_0013 指定

/opt/flink/bin/flink run \
-yid application_1673426410002_0013 \
-ys $YS \
-yjm $YJM \
-ytm $YTM \
-yt $JAR_PATH/lib \
-ynm $YNM \
-yD env.java.opts="-Dfile.encoding=UTF-8" \
-c $START_CLASS \
$JAR_PATH/$JAR_NAME.jar $1

注意：

通过 -yt $JAR_PATH/lib指定使用的jar包，通过 -yid 提交可能会找不到Jar包

异常

异常1

stderr: 
Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/stack-hooks/before-ANY/scripts/hook.py", line 38, in <module>
    BeforeAnyHook().execute()
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 352, in execute
    method(env)
  File "/var/lib/ambari-agent/cache/stack-hooks/before-ANY/scripts/hook.py", line 31, in hook
    setup_users()
  File "/var/lib/ambari-agent/cache/stack-hooks/before-ANY/scripts/shared_initialization.py", line 50, in setup_users
    groups = params.user_to_groups_dict[user],
KeyError: u'flink'
Error: Error: Unable to run the custom hook script ['/usr/bin/python', '/var/lib/ambari-agent/cache/stack-hooks/before-ANY/scripts/hook.py', 'ANY', '/var/lib/ambari-agent/data/command-852.json', '/var/lib/ambari-agent/cache/stack-hooks/before-ANY', '/var/lib/ambari-agent/data/structured-out-852.json', 'INFO', '/var/lib/ambari-agent/tmp', 'PROTOCOL_TLSv1_2', '']
 stdout:
2023-01-13 14:22:09,665 - Stack Feature Version Info: Cluster Stack=3.1, Command Stack=None, Command Version=None -> 3.1
2023-01-13 14:22:09,676 - Using hadoop conf dir: /usr/hdp/current/hadoop-client/conf
2023-01-13 14:22:09,679 - Group['flink'] {}
2023-01-13 14:22:09,682 - Group['livy'] {}
2023-01-13 14:22:09,682 - Group['spark'] {}
2023-01-13 14:22:09,683 - Group['hdfs'] {}
2023-01-13 14:22:09,684 - Group['hadoop'] {}
2023-01-13 14:22:09,684 - Group['users'] {}
2023-01-13 14:22:09,685 - User['yarn-ats'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2023-01-13 14:22:09,687 - User['hive'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2023-01-13 14:22:09,689 - User['zookeeper'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop'], 'uid': None}
2023-01-13 14:22:09,691 - User['oozie'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop', 'users'], 'uid': None}
2023-01-13 14:22:09,693 - User['tez'] {'gid': 'hadoop', 'fetch_nonlocal_groups': True, 'groups': ['hadoop', 'users'], 'uid': None}
Error: Error: Unable to run the custom hook script ['/usr/bin/python', '/var/lib/ambari-agent/cache/stack-hooks/before-ANY/scripts/hook.py', 'ANY', '/var/lib/ambari-agent/data/command-852.json', '/var/lib/ambari-agent/cache/stack-hooks/before-ANY', '/var/lib/ambari-agent/data/structured-out-852.json', 'INFO', '/var/lib/ambari-agent/tmp', 'PROTOCOL_TLSv1_2', '']
2023-01-13 14:22:09,738 - The repository with version 3.1.5.0-152 for this command has been marked as resolved. It will be used to report the version of the component which was installed
2023-01-13 14:22:09,750 - Skipping stack-select on FLINK because it does not exist in the stack-select package structure.

Command failed after 1 tries

解决方法：

#python configs.py -u admin -p admin -n $cluster_name -l $ambari_server -t 8080 -a set -c cluster-env -k ignore_groupsusers_create -v ture

> cd /var/lib/ambari-server/resources/scripts
> python configs.py -u admin -p admin -n dev -l 172.16.24.194 -t 8080 -a set -c cluster-env -k ignore_groupsusers_create -v ture
2023-01-13 14:26:26,174 INFO ### Performing "set":
2023-01-13 14:26:26,175 INFO ### new property - "ignore_groupsusers_create":"ture"
2023-01-13 14:26:26,290 INFO ### on (Site:cluster-env, Tag:5553e181-525d-45c8-bc15-f6bdfcce607f)
2023-01-13 14:26:26,320 INFO ### PUTting json into: doSet_version1673591186320343.json
2023-01-13 14:26:26,646 INFO ### NEW Site:cluster-env, Tag:version1673591186320343

异常2

stderr: 
Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/stacks/HDP/3.1/services/FLINK/package/scripts/flink.py", line 172, in <module>
    Master().execute()
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 352, in execute
    method(env)
  File "/var/lib/ambari-agent/cache/stacks/HDP/3.1/services/FLINK/package/scripts/flink.py", line 119, in start
    Execute (cmd + format(" >> {flink_log_file}"), user=params.flink_user)
  File "/usr/lib/ambari-agent/lib/resource_management/core/base.py", line 166, in __init__
    self.env.run()
  File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 160, in run
    self.run_action(resource, action)
  File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 124, in run_action
    provider_action()
  File "/usr/lib/ambari-agent/lib/resource_management/core/providers/system.py", line 263, in action_run
    returns=self.resource.returns)
  File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 72, in inner
    result = function(command, **kwargs)
  File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 102, in checked_call
    tries=tries, try_sleep=try_sleep, timeout_kill_strategy=timeout_kill_strategy, returns=returns)
  File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 150, in _call_wrapper
    result = _call(command, **kwargs_copy)
  File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 314, in _call
    raise ExecutionFailed(err_msg, code, out, err)
resource_management.core.exceptions.ExecutionFailed: Execution of 'export HADOOP_CONF_DIR=/etc/hadoop/conf; export HADOOP_CLASSPATH=/usr/hdp/3.1.5.0-152/hadoop/conf:/usr/hdp/3.1.5.0-152/hadoop/lib/*:/usr/hdp/3.1.5.0-152/hadoop/.//*:/usr/hdp/3.1.5.0-152/hadoop-hdfs/./:/usr/hdp/3.1.5.0-152/hadoop-hdfs/lib/*:/usr/hdp/3.1.5.0-152/hadoop-hdfs/.//*:/usr/hdp/3.1.5.0-152/hadoop-mapreduce/lib/*:/usr/hdp/3.1.5.0-152/hadoop-mapreduce/.//*:/usr/hdp/3.1.5.0-152/hadoop-yarn/./:/usr/hdp/3.1.5.0-152/hadoop-yarn/lib/*:/usr/hdp/3.1.5.0-152/hadoop-yarn/.//*:/usr/hdp/3.1.5.0-152/tez/*:/usr/hdp/3.1.5.0-152/tez/lib/*:/usr/hdp/3.1.5.0-152/tez/conf:/usr/hdp/3.1.5.0-152/tez/conf_llap:/usr/hdp/3.1.5.0-152/tez/doc:/usr/hdp/3.1.5.0-152/tez/hadoop-shim-0.9.1.3.1.5.0-152.jar:/usr/hdp/3.1.5.0-152/tez/hadoop-shim-2.8-0.9.1.3.1.5.0-152.jar:/usr/hdp/3.1.5.0-152/tez/lib:/usr/hdp/3.1.5.0-152/tez/man:/usr/hdp/3.1.5.0-152/tez/tez-api-0.9.1.3.1.5.0-152.jar:/usr/hdp/3.1.5.0-152/tez/tez-common-0.9.1.3.1.5.0-152.jar:/usr/hdp/3.1.5.0-152/tez/tez-dag-0.9.1.3.1.5.0-152.jar:/usr/hdp/3.1.5.0-152/tez/tez-examples-0.9.1.3.1.5.0-152.jar:/usr/hdp/3.1.5.0-152/tez/tez-history-parser-0.9.1.3.1.5.0-152.jar:/usr/hdp/3.1.5.0-152/tez/tez-javadoc-tools-0.9.1.3.1.5.0-152.jar:/usr/hdp/3.1.5.0-152/tez/tez-job-analyzer-0.9.1.3.1.5.0-152.jar:/usr/hdp/3.1.5.0-152/tez/tez-mapreduce-0.9.1.3.1.5.0-152.jar:/usr/hdp/3.1.5.0-152/tez/tez-protobuf-history-plugin-0.9.1.3.1.5.0-152.jar:/usr/hdp/3.1.5.0-152/tez/tez-runtime-internals-0.9.1.3.1.5.0-152.jar:/usr/hdp/3.1.5.0-152/tez/tez-runtime-library-0.9.1.3.1.5.0-152.jar:/usr/hdp/3.1.5.0-152/tez/tez-tests-0.9.1.3.1.5.0-152.jar:/usr/hdp/3.1.5.0-152/tez/tez-yarn-timeline-cache-plugin-0.9.1.3.1.5.0-152.jar:/usr/hdp/3.1.5.0-152/tez/tez-yarn-timeline-history-0.9.1.3.1.5.0-152.jar:/usr/hdp/3.1.5.0-152/tez/tez-yarn-timeline-history-with-acls-0.9.1.3.1.5.0-152.jar:/usr/hdp/3.1.5.0-152/tez/tez-yarn-timeline-history-with-fs-0.9.1.3.1.5.0-152.jar:/usr/hdp/3.1.5.0-152/tez/ui:/usr/hdp/3.1.5.0-152/tez/lib/async-http-client-1.9.40.jar:/usr/hdp/3.1.5.0-152/tez/lib/commons-cli-1.2.jar:/usr/hdp/3.1.5.0-152/tez/lib/commons-codec-1.4.jar:/usr/hdp/3.1.5.0-152/tez/lib/commons-collections-3.2.2.jar:/usr/hdp/3.1.5.0-152/tez/lib/commons-collections4-4.1.jar:/usr/hdp/3.1.5.0-152/tez/lib/commons-io-2.4.jar:/usr/hdp/3.1.5.0-152/tez/lib/commons-lang-2.6.jar:/usr/hdp/3.1.5.0-152/tez/lib/commons-math3-3.1.1.jar:/usr/hdp/3.1.5.0-152/tez/lib/gcs-connector-hadoop3-1.9.17.3.1.5.0-152-shaded.jar:/usr/hdp/3.1.5.0-152/tez/lib/guava-28.0-jre.jar:/usr/hdp/3.1.5.0-152/tez/lib/hadoop-aws-3.1.1.3.1.5.0-152.jar:/usr/hdp/3.1.5.0-152/tez/lib/hadoop-azure-3.1.1.3.1.5.0-152.jar:/usr/hdp/3.1.5.0-152/tez/lib/hadoop-azure-datalake-3.1.1.3.1.5.0-152.jar:/usr/hdp/3.1.5.0-152/tez/lib/hadoop-hdfs-client-3.1.1.3.1.5.0-152.jar:/usr/hdp/3.1.5.0-152/tez/lib/hadoop-mapreduce-client-common-3.1.1.3.1.5.0-152.jar:/usr/hdp/3.1.5.0-152/tez/lib/hadoop-mapreduce-client-core-3.1.1.3.1.5.0-152.jar:/usr/hdp/3.1.5.0-152/tez/lib/hadoop-yarn-server-timeline-pluginstorage-3.1.1.3.1.5.0-152.jar:/usr/hdp/3.1.5.0-152/tez/lib/jersey-client-1.19.jar:/usr/hdp/3.1.5.0-152/tez/lib/jersey-json-1.19.jar:/usr/hdp/3.1.5.0-152/tez/lib/jettison-1.3.4.jar:/usr/hdp/3.1.5.0-152/tez/lib/jetty-server-9.3.24.v20180605.jar:/usr/hdp/3.1.5.0-152/tez/lib/jetty-util-9.3.24.v20180605.jar:/usr/hdp/3.1.5.0-152/tez/lib/jsr305-3.0.0.jar:/usr/hdp/3.1.5.0-152/tez/lib/metrics-core-3.1.0.jar:/usr/hdp/3.1.5.0-152/tez/lib/protobuf-java-2.5.0.jar:/usr/hdp/3.1.5.0-152/tez/lib/RoaringBitmap-0.4.9.jar:/usr/hdp/3.1.5.0-152/tez/lib/servlet-api-2.5.jar:/usr/hdp/3.1.5.0-152/tez/lib/slf4j-api-1.7.10.jar:/usr/hdp/3.1.5.0-152/tez/lib/tez.tar.gz; /opt/flink/bin/yarn-session.sh -d -nm flinkapp-from-ambari -n 1 -s 1 -jm 768 -tm 1024 -qu default >> /var/log/flink/flink-setup.log' returned 1. SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/flink/lib/log4j-slf4j-impl-2.16.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/hdp/3.1.5.0-152/hadoop/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]

------------------------------------------------------------
 The program finished with the following exception:

org.apache.flink.configuration.IllegalConfigurationException: JobManager memory configuration failed: Sum of configured JVM Metaspace (256.000mb (268435456 bytes)) and JVM Overhead (192.000mb (201326592 bytes)) exceed configured Total Process Memory (256.000mb (268435456 bytes)).
	at org.apache.flink.runtime.jobmanager.JobManagerProcessUtils.processSpecFromConfigWithNewOptionToInterpretLegacyHeap(JobManagerProcessUtils.java:78)
	at org.apache.flink.client.deployment.AbstractContainerizedClusterClientFactory.getClusterSpecification(AbstractContainerizedClusterClientFactory.java:43)
	at org.apache.flink.yarn.cli.FlinkYarnSessionCli.run(FlinkYarnSessionCli.java:602)
	at org.apache.flink.yarn.cli.FlinkYarnSessionCli.lambda$main$4(FlinkYarnSessionCli.java:860)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1836)
	at org.apache.flink.runtime.security.contexts.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
	at org.apache.flink.yarn.cli.FlinkYarnSessionCli.main(FlinkYarnSessionCli.java:860)
Caused by: org.apache.flink.configuration.IllegalConfigurationException: Sum of configured JVM Metaspace (256.000mb (268435456 bytes)) and JVM Overhead (192.000mb (201326592 bytes)) exceed configured Total Process Memory (256.000mb (268435456 bytes)).
	at org.apache.flink.runtime.util.config.memory.ProcessMemoryUtils.deriveJvmMetaspaceAndOverheadWithTotalProcessMemory(ProcessMemoryUtils.java:157)
	at org.apache.flink.runtime.util.config.memory.ProcessMemoryUtils.deriveProcessSpecWithTotalProcessMemory(ProcessMemoryUtils.java:114)
	at org.apache.flink.runtime.util.config.memory.ProcessMemoryUtils.memoryProcessSpecFromConfig(ProcessMemoryUtils.java:84)
	at org.apache.flink.runtime.jobmanager.JobManagerProcessUtils.processSpecFromConfig(JobManagerProcessUtils.java:83)
	at org.apache.flink.runtime.jobmanager.JobManagerProcessUtils.processSpecFromConfigWithNewOptionToInterpretLegacyHeap(JobManagerProcessUtils.java:73)
	... 8 more
 stdout:
2023-01-13 14:27:05,019 - Stack Feature Version Info: Cluster Stack=3.1, Command Stack=None, Command Version=3.1.5.0-152 -> 3.1.5.0-152
2023-01-13 14:27:05,062 - Using hadoop conf dir: /usr/hdp/3.1.5.0-152/hadoop/conf
2023-01-13 14:27:05,535 - Stack Feature Version Info: Cluster Stack=3.1, Command Stack=None, Command Version=3.1.5.0-152 -> 3.1.5.0-152
2023-01-13 14:27:05,548 - Using hadoop conf dir: /usr/hdp/3.1.5.0-152/hadoop/conf
2023-01-13 14:27:05,552 - Skipping creation of User and Group as host is sys prepped or ignore_groupsusers_create flag is on
2023-01-13 14:27:05,553 - Directory['/tmp/hbase-hbase'] {'owner': 'hbase', 'create_parents': True, 'mode': 0775, 'cd_access': 'a'}
2023-01-13 14:27:05,559 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2023-01-13 14:27:05,562 - File['/var/lib/ambari-agent/tmp/changeUid.sh'] {'content': StaticFile('changeToSecureUid.sh'), 'mode': 0555}
2023-01-13 14:27:05,564 - call['/var/lib/ambari-agent/tmp/changeUid.sh hbase'] {}
2023-01-13 14:27:05,592 - call returned (0, '1014')
2023-01-13 14:27:05,594 - Execute['/var/lib/ambari-agent/tmp/changeUid.sh hbase /home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/tmp/hbase-hbase 1014'] {'not_if': '(test $(id -u hbase) -gt 1000) || (ture)'}
2023-01-13 14:27:05,610 - Skipping Execute['/var/lib/ambari-agent/tmp/changeUid.sh hbase /home/hbase,/tmp/hbase,/usr/bin/hbase,/var/log/hbase,/tmp/hbase-hbase 1014'] due to not_if
2023-01-13 14:27:05,611 - Skipping setting dfs cluster admin and tez view acls as host is sys prepped
2023-01-13 14:27:05,611 - FS Type: HDFS
2023-01-13 14:27:05,612 - Directory['/etc/hadoop'] {'mode': 0755}
2023-01-13 14:27:05,653 - File['/usr/hdp/3.1.5.0-152/hadoop/conf/hadoop-env.sh'] {'content': InlineTemplate(...), 'owner': 'hdfs', 'group': 'hadoop'}
2023-01-13 14:27:05,655 - Writing File['/usr/hdp/3.1.5.0-152/hadoop/conf/hadoop-env.sh'] because contents don't match
2023-01-13 14:27:05,656 - Directory['/var/lib/ambari-agent/tmp/hadoop_java_io_tmpdir'] {'owner': 'hdfs', 'group': 'hadoop', 'mode': 01777}
2023-01-13 14:27:05,699 - Execute[('setenforce', '0')] {'not_if': '(! which getenforce ) || (which getenforce && getenforce | grep -q Disabled)', 'sudo': True, 'only_if': 'test -f /selinux/enforce'}
2023-01-13 14:27:05,721 - Skipping Execute[('setenforce', '0')] due to not_if
2023-01-13 14:27:05,722 - Directory['/var/log/hadoop'] {'owner': 'root', 'create_parents': True, 'group': 'hadoop', 'mode': 0775, 'cd_access': 'a'}
2023-01-13 14:27:05,730 - Directory['/var/run/hadoop'] {'owner': 'root', 'create_parents': True, 'group': 'root', 'cd_access': 'a'}
2023-01-13 14:27:05,732 - Directory['/var/run/hadoop/hdfs'] {'owner': 'hdfs', 'cd_access': 'a'}
2023-01-13 14:27:05,733 - Directory['/tmp/hadoop-hdfs'] {'owner': 'hdfs', 'create_parents': True, 'cd_access': 'a'}
2023-01-13 14:27:05,744 - File['/usr/hdp/3.1.5.0-152/hadoop/conf/commons-logging.properties'] {'content': Template('commons-logging.properties.j2'), 'owner': 'hdfs'}
2023-01-13 14:27:05,749 - File['/usr/hdp/3.1.5.0-152/hadoop/conf/health_check'] {'content': Template('health_check.j2'), 'owner': 'hdfs'}
2023-01-13 14:27:05,763 - File['/usr/hdp/3.1.5.0-152/hadoop/conf/log4j.properties'] {'content': InlineTemplate(...), 'owner': 'hdfs', 'group': 'hadoop', 'mode': 0644}
2023-01-13 14:27:05,791 - File['/usr/hdp/3.1.5.0-152/hadoop/conf/hadoop-metrics2.properties'] {'content': InlineTemplate(...), 'owner': 'hdfs', 'group': 'hadoop'}
2023-01-13 14:27:05,793 - File['/usr/hdp/3.1.5.0-152/hadoop/conf/task-log4j.properties'] {'content': StaticFile('task-log4j.properties'), 'mode': 0755}
2023-01-13 14:27:05,795 - File['/usr/hdp/3.1.5.0-152/hadoop/conf/configuration.xsl'] {'owner': 'hdfs', 'group': 'hadoop'}
2023-01-13 14:27:05,806 - File['/etc/hadoop/conf/topology_mappings.data'] {'owner': 'hdfs', 'content': Template('topology_mappings.data.j2'), 'only_if': 'test -d /etc/hadoop/conf', 'group': 'hadoop', 'mode': 0644}
2023-01-13 14:27:05,819 - File['/etc/hadoop/conf/topology_script.py'] {'content': StaticFile('topology_script.py'), 'only_if': 'test -d /etc/hadoop/conf', 'mode': 0755}
2023-01-13 14:27:05,831 - Skipping unlimited key JCE policy check and setup since the Java VM is not managed by Ambari
2023-01-13 14:27:05,851 - Skipping stack-select on FLINK because it does not exist in the stack-select package structure.
2023-01-13 14:27:06,350 - File['/opt/flink/conf/flink-conf.yaml'] {'owner': 'flink', 'content': InlineTemplate(...)}
2023-01-13 14:27:06,354 - Writing File['/opt/flink/conf/flink-conf.yaml'] because contents don't match
2023-01-13 14:27:06,356 - Execute['hadoop fs -mkdir -p /user/flink'] {'ignore_failures': True, 'user': 'hdfs'}
2023-01-13 14:27:10,231 - Execute['hadoop fs -chown flink /user/flink'] {'user': 'hdfs'}
2023-01-13 14:27:13,858 - Execute['hadoop fs -chgrp flink /user/flink'] {'user': 'hdfs'}
2023-01-13 14:27:17,779 - Execute['echo bin dir /opt/flink/bin'] {}
2023-01-13 14:27:17,789 - Execute['echo pid file /var/run/flink/flink.pid'] {}
2023-01-13 14:27:18,005 - Execute['export HADOOP_CONF_DIR=/etc/hadoop/conf; export HADOOP_CLASSPATH=/usr/hdp/3.1.5.0-152/hadoop/conf:/usr/hdp/3.1.5.0-152/hadoop/lib/*:/usr/hdp/3.1.5.0-152/hadoop/.//*:/usr/hdp/3.1.5.0-152/hadoop-hdfs/./:/usr/hdp/3.1.5.0-152/hadoop-hdfs/lib/*:/usr/hdp/3.1.5.0-152/hadoop-hdfs/.//*:/usr/hdp/3.1.5.0-152/hadoop-mapreduce/lib/*:/usr/hdp/3.1.5.0-152/hadoop-mapreduce/.//*:/usr/hdp/3.1.5.0-152/hadoop-yarn/./:/usr/hdp/3.1.5.0-152/hadoop-yarn/lib/*:/usr/hdp/3.1.5.0-152/hadoop-yarn/.//*:/usr/hdp/3.1.5.0-152/tez/*:/usr/hdp/3.1.5.0-152/tez/lib/*:/usr/hdp/3.1.5.0-152/tez/conf:/usr/hdp/3.1.5.0-152/tez/conf_llap:/usr/hdp/3.1.5.0-152/tez/doc:/usr/hdp/3.1.5.0-152/tez/hadoop-shim-0.9.1.3.1.5.0-152.jar:/usr/hdp/3.1.5.0-152/tez/hadoop-shim-2.8-0.9.1.3.1.5.0-152.jar:/usr/hdp/3.1.5.0-152/tez/lib:/usr/hdp/3.1.5.0-152/tez/man:/usr/hdp/3.1.5.0-152/tez/tez-api-0.9.1.3.1.5.0-152.jar:/usr/hdp/3.1.5.0-152/tez/tez-common-0.9.1.3.1.5.0-152.jar:/usr/hdp/3.1.5.0-152/tez/tez-dag-0.9.1.3.1.5.0-152.jar:/usr/hdp/3.1.5.0-152/tez/tez-examples-0.9.1.3.1.5.0-152.jar:/usr/hdp/3.1.5.0-152/tez/tez-history-parser-0.9.1.3.1.5.0-152.jar:/usr/hdp/3.1.5.0-152/tez/tez-javadoc-tools-0.9.1.3.1.5.0-152.jar:/usr/hdp/3.1.5.0-152/tez/tez-job-analyzer-0.9.1.3.1.5.0-152.jar:/usr/hdp/3.1.5.0-152/tez/tez-mapreduce-0.9.1.3.1.5.0-152.jar:/usr/hdp/3.1.5.0-152/tez/tez-protobuf-history-plugin-0.9.1.3.1.5.0-152.jar:/usr/hdp/3.1.5.0-152/tez/tez-runtime-internals-0.9.1.3.1.5.0-152.jar:/usr/hdp/3.1.5.0-152/tez/tez-runtime-library-0.9.1.3.1.5.0-152.jar:/usr/hdp/3.1.5.0-152/tez/tez-tests-0.9.1.3.1.5.0-152.jar:/usr/hdp/3.1.5.0-152/tez/tez-yarn-timeline-cache-plugin-0.9.1.3.1.5.0-152.jar:/usr/hdp/3.1.5.0-152/tez/tez-yarn-timeline-history-0.9.1.3.1.5.0-152.jar:/usr/hdp/3.1.5.0-152/tez/tez-yarn-timeline-history-with-acls-0.9.1.3.1.5.0-152.jar:/usr/hdp/3.1.5.0-152/tez/tez-yarn-timeline-history-with-fs-0.9.1.3.1.5.0-152.jar:/usr/hdp/3.1.5.0-152/tez/ui:/usr/hdp/3.1.5.0-152/tez/lib/async-http-client-1.9.40.jar:/usr/hdp/3.1.5.0-152/tez/lib/commons-cli-1.2.jar:/usr/hdp/3.1.5.0-152/tez/lib/commons-codec-1.4.jar:/usr/hdp/3.1.5.0-152/tez/lib/commons-collections-3.2.2.jar:/usr/hdp/3.1.5.0-152/tez/lib/commons-collections4-4.1.jar:/usr/hdp/3.1.5.0-152/tez/lib/commons-io-2.4.jar:/usr/hdp/3.1.5.0-152/tez/lib/commons-lang-2.6.jar:/usr/hdp/3.1.5.0-152/tez/lib/commons-math3-3.1.1.jar:/usr/hdp/3.1.5.0-152/tez/lib/gcs-connector-hadoop3-1.9.17.3.1.5.0-152-shaded.jar:/usr/hdp/3.1.5.0-152/tez/lib/guava-28.0-jre.jar:/usr/hdp/3.1.5.0-152/tez/lib/hadoop-aws-3.1.1.3.1.5.0-152.jar:/usr/hdp/3.1.5.0-152/tez/lib/hadoop-azure-3.1.1.3.1.5.0-152.jar:/usr/hdp/3.1.5.0-152/tez/lib/hadoop-azure-datalake-3.1.1.3.1.5.0-152.jar:/usr/hdp/3.1.5.0-152/tez/lib/hadoop-hdfs-client-3.1.1.3.1.5.0-152.jar:/usr/hdp/3.1.5.0-152/tez/lib/hadoop-mapreduce-client-common-3.1.1.3.1.5.0-152.jar:/usr/hdp/3.1.5.0-152/tez/lib/hadoop-mapreduce-client-core-3.1.1.3.1.5.0-152.jar:/usr/hdp/3.1.5.0-152/tez/lib/hadoop-yarn-server-timeline-pluginstorage-3.1.1.3.1.5.0-152.jar:/usr/hdp/3.1.5.0-152/tez/lib/jersey-client-1.19.jar:/usr/hdp/3.1.5.0-152/tez/lib/jersey-json-1.19.jar:/usr/hdp/3.1.5.0-152/tez/lib/jettison-1.3.4.jar:/usr/hdp/3.1.5.0-152/tez/lib/jetty-server-9.3.24.v20180605.jar:/usr/hdp/3.1.5.0-152/tez/lib/jetty-util-9.3.24.v20180605.jar:/usr/hdp/3.1.5.0-152/tez/lib/jsr305-3.0.0.jar:/usr/hdp/3.1.5.0-152/tez/lib/metrics-core-3.1.0.jar:/usr/hdp/3.1.5.0-152/tez/lib/protobuf-java-2.5.0.jar:/usr/hdp/3.1.5.0-152/tez/lib/RoaringBitmap-0.4.9.jar:/usr/hdp/3.1.5.0-152/tez/lib/servlet-api-2.5.jar:/usr/hdp/3.1.5.0-152/tez/lib/slf4j-api-1.7.10.jar:/usr/hdp/3.1.5.0-152/tez/lib/tez.tar.gz; /opt/flink/bin/yarn-session.sh -d -nm flinkapp-from-ambari -n 1 -s 1 -jm 768 -tm 1024 -qu default >> /var/log/flink/flink-setup.log'] {'user': 'flink'}
2023-01-13 14:27:22,136 - Skipping stack-select on FLINK because it does not exist in the stack-select package structure.

Command failed after 1 tries

解决方法：

在flink-evn.xml中添加

jobmanager.memory.process.size: 1600m
taskmanager.memory.process.size: 1728m

在这里插入图片描述

保存后重启Flink

异常3

stderr: 
Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/stacks/HDP/3.1/services/FLINK/package/scripts/flink.py", line 172, in <module>
    Master().execute()
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 352, in execute
    method(env)
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 980, in restart
    self.stop(env)
  File "/var/lib/ambari-agent/cache/stacks/HDP/3.1/services/FLINK/package/scripts/flink.py", line 98, in stop
    pid = str(sudo.read_file(status_params.flink_pid_file))
  File "/usr/lib/ambari-agent/lib/resource_management/core/sudo.py", line 151, in read_file
    with open(filename, "rb") as fp:
IOError: [Errno 2] No such file or directory: u'/var/run/flink/flink.pid'

下载javax.ws.rs-api-2.0.jar放到/opt/flink/lib目录下

cd /opt/flink/lib
wget https://repo1.maven.org/maven2/javax/ws/rs/javax.ws.rs-api/2.0/javax.ws.rs-api-2.0.jar
chown flink.flink javax.ws.rs-api-2.0.jar

解决方法

su hdfs
cd /opt/flink/bin
./yarn-session.sh -n 1 -s 1 -jm 768 -tm 1024 -qu default -nm flinkapp-from-ambari -d >> /var/log/hadoop/hdfs/flink-setup.log

参考：

https://blog.csdn.net/qq_36048223/article/details/116114765

本文来自互联网用户投稿，该文观点仅代表作者本人，不代表本站立场。本站仅提供信息存储空间服务，不拥有所有权，不承担相关法律责任。如若转载，请注明出处：http://www.coloradmin.cn/o/160430.html

如若内容造成侵权/违法违规/事实不符，请联系多彩编程网进行投诉反馈，一经查实，立即删除！

相关文章

Goby+AWVS 联动

Goby+AWVS 联动

系列文章 AWVS安装与激活 AWVS扫描Web应用程序 AWVS扫描报告分析 GobyAWVS 联动 1.Goby简介 Goby是针对目标企业梳理最全面的工具&#xff0c;同构goby可以清晰的扫描出ip地址开放的端口&#xff0c;以及端口对应的服务&#xff0c;于此同事会根据开放的端口及应用进行实战…
阅读更多...
分享116个ASP源码，总有一款适合您

分享116个ASP源码，总有一款适合您

ASP源码 分享116个ASP源码&#xff0c;总有一款适合您 116个ASP源码下载链接&#xff1a;https://pan.baidu.com/s/1LEs24-feWjvhac6vzyVecg?pwdnosx 提取码&#xff1a;nosx 下面是文件的名字&#xff0c;我放了一些图片&#xff0c;文章里不是所有的图主要是放不下...&am…
阅读更多...
2.线性表

2.线性表

##线性结构 基本特点&#xff1a;结构中各元素之间满足线性关系。 线性关系&#xff1a;数据元素之间存在一对一的关系 1.存在唯一的开始元素 2.存在唯一的终止元素 3.除了开始元素和终止元素&#xff0c;其他元素均有且仅有一个直接前驱元素和一个直接后驱元素。 所有元素可排…
阅读更多...
Sass进阶指南 -- 写出更优雅的样式表

Sass进阶指南 -- 写出更优雅的样式表

我以为已经会了&#xff0c;之前在公司写项目基本都是用sass写样式&#xff0c;十分顺手。直到有段时间&#xff0c;我准备参考Element Plus来设计自己组件库的工程结构&#xff0c;看到Element Plus那些优雅的sass用法时&#xff0c;我开始为我的浅薄和无知感到羞愧。这便开始…
阅读更多...
Python学习笔记——集合

Python学习笔记——集合

集合&#xff08;set&#xff09;是一个无序的不重复元素序列。可以使用大括号 { } 或者 set() 函数创建集合&#xff0c;注意&#xff1a;创建一个空集合必须用 set() 而不是 { }&#xff0c;因为 { } 是用来创建一个空字典。创建格式&#xff1a;parame {value01,value02,..…
阅读更多...
LeetCode[313]超级丑数

LeetCode[313]超级丑数

难度&#xff1a;中等题目&#xff1a;超级丑数 是一个正整数&#xff0c;并满足其所有质因数都出现在质数数组 primes中。给你一个整数 n和一个整数数组 primes&#xff0c;返回第 n个 超级丑数 。题目数据保证第 n个 超级丑数 在 32-bit 带符号整数范围内。示例 1&#xff1a…
阅读更多...
C++ 类和对象（三）

C++ 类和对象（三）

类和对象&#xff08;一&#xff09; 类和对象&#xff08;二&#xff09; 日期类 目录 一. 再谈构造函数 1.构造函数体赋值 2.初始化列表 3.隐式类型转换与explicit关键字 4.C11优化 二. static成员 1.概念 2.特性 三. 友元 1.友元函数 2.友元类 四. 内部…
阅读更多...
Java 如何设计一款小游戏详细讲解(二)

Java 如何设计一款小游戏详细讲解(二)

继续上面(一)&#xff0c;Java 如何设计一款小游戏详细讲解(一)步骤步骤一接下来我们主要完成GameJFrame&#xff0c;就是拼图游戏的界面及代码。这里接下来完成菜单的设置&#xff0c;将主界面的代码放到一个方法中&#xff0c;像下面这样&#xff0c;接下来我们书写菜单的代码…
阅读更多...
基于Python实现种差值方法（完整代码详细教程）

基于Python实现种差值方法（完整代码详细教程）

三种插值方法都是使用Python自己实现的。1.1 最近邻插值寻找每个中心点周围的八个点中有无未丢失的点&#xff0c;如果有的话就赋值为第一个找到的点&#xff0c;如果没有就扩大范围再次寻找&#xff0c;在最大范围内都找不到的话就跳过。1.2 双线性插值使用解方程的方法求解&a…
阅读更多...
（十八）Threads异步和多线程（Thread、Threadpool、Task）-语言进阶2

（十八）Threads异步和多线程（Thread、Threadpool、Task）-语言进阶2

Threads异步和多线程-语言进阶2一、Thread1. 线程启动2. 线程等待3.前台线程/后台线程4.扩展thread封装回调二、Threadpool1.线程池2.线程池使用3.ManualResetEvent 线程池等待三、Task1. Task启动方式2.waitall 、waitany1.waitall2.waitany3.WaitAll、waitany场景4. 应用&…
阅读更多...
Spring为什么这么火 之 Spring的创建及存储、获取Bean对象

Spring为什么这么火 之 Spring的创建及存储、获取Bean对象

目录 1、创建Spring项目 1.1、创建一个Maven项目 1.2、添加Spring框架支持 1.3、添加启动类 2、存储Bean对象 2.1、创建Bean对象 2.2、将Bean对象注册到Spring容器中 3、获取、使用Bean对象 3.1、得到Spring上下文对象 使用ApplicationContext作为Spring的上下文 【更…
阅读更多...
服务器相关命令(docker相关)

服务器相关命令(docker相关)

一:安装docker 之前安装过旧版本&#xff0c;使用以下命令可以卸载(整个copy过去): yum remove docker \docker-client \docker-client-latest \docker-common \docker-latest \docker-latest-logrotate \docker-logrotate \docker-selinux \docker-engine-selinux \docker-e…
阅读更多...
AOP面向切面编程

AOP面向切面编程

AOP 面向切面编程 AOP是什么 AOP 为 Aspect Oriented Programming 的缩写&#xff0c;意为&#xff1a;面向切面编程&#xff0c;通过预编译方式和运行期间动态代理实现程序功能的统一维护的一种技术。 AOP 是 OOP 的延续&#xff0c;是软件开发中的一个热点&#xff0c;是 j…
阅读更多...
【Linux】-- 环境变量

【Linux】-- 环境变量

目录 环境变量 常见环境变量 HOME 和环境变量相关的命令 通过代码如何获取环境变量 环境变量参数 通过第三方变量environ获取 通过getenv函数来特定访问获取 通过setenv函数来特定访问获取 环境变量 命令&#xff1a;which ls 将命令ls的完整路径写入到标准输出。 #in…
阅读更多...
用R语言理解洛必达法则

用R语言理解洛必达法则

文章目录5 洛必达法则极限的种类洛必达法则作用于幂函数5 洛必达法则 极限的种类 令NNN为常数&#xff0c;则常规的极限运算大致有以下几种 ∞N∞∞⋇N∞(N̸0)N∔∞∞N−∞−∞N/∞0N/0∞N∞∞(N̸1)∞N∞(N̸0)\begin{matrix} &\infty\pm N\infty\quad&\infty\divi…
阅读更多...
MySQL版本由5.7.37更新到5.7.39

MySQL版本由5.7.37更新到5.7.39

一、前景 由于mysql5.7.37存在漏洞&#xff0c;影响系统安全&#xff0c;所以需要将mysql版本升级到5.7的最新版本5.7.39。 二、步骤 1、下载5.7.39的安装包&#xff1a; 下载链接如下&#xff1a; https://downloads.mysql.com/archives/get/p/23/file/mysql-5.7.39-1.el…
阅读更多...
node基础知识

node基础知识

node基础知识 node在真实项目中的应用 webpack基于node环境 用nodejs的语法合并压缩打包 js放到客户端浏览器中执行 放到服务器端运行&#xff1a;java因为jdk php因为tomcat c#因为有.net framework 项目架构1&#xff1a;中小型项目-基于nodejs构建全栈 项目架构2&#xff1a…
阅读更多...
mysql转DM达梦数据库+springboot兼容DM数据库+springboot兼容activity5.22.0

mysql转DM达梦数据库+springboot兼容DM数据库+springboot兼容activity5.22.0

由于现在做的项目中需要针对数据库进行国产化操作&#xff0c;最终完成从mysql到达梦的迁移&#xff0c;记录整合迁移记录如下&#xff1a;安装初始化达梦数据库&#xff08;傻瓜式安装即可&#xff09;安装达梦数据库&#xff08;windows、linux&#xff09;初始化数据库实例关…
阅读更多...
代码质量与安全 | 新时代：2023年商业软件开发的五大关键目标

代码质量与安全 | 新时代：2023年商业软件开发的五大关键目标

进入2023年&#xff0c;技术趋势仍然聚焦于人工智能、边缘智能和气候变化领域&#xff0c;但供应链增速放缓和日益增长的消费者需求阻碍着创新的步伐。为了在行业中保持竞争力&#xff0c;并实现软件工程预定目标&#xff0c;软件开发领导者需要主动制定预算和时间管理计划&…
阅读更多...
Java设计模式-模板模式Template

Java设计模式-模板模式Template

介绍 模板方法模式&#xff08;Template Method Pattern&#xff09;&#xff0c;又叫模板模式(Template Pattern)&#xff0c;z 在一个抽象类公开定义了执行。它的方法的模板。它的子类可以按需要重写方法实现&#xff0c;但调用将以抽象类中定义的方式进行。简单说&#xff…
阅读更多...
推荐文章
最新文章

Copyright @ 2022~2023