seatunnel-2.3.2 doris数据同步到hive(cdh-6.3.2)首次运行报错解决,解决的报错如下:
1、java.lang.NoClassDefFoundError: org/apache/hadoop/hive/metastore/api/MetaException
2、java.lang.NoClassDefFoundError: org/apache/thrift/TBase
3、java.lang.NoClassDefFoundError:org/apache/hadoop/hive/conf/HiveConf
4、java.lang.NoClassDefFoundError:com/facebook/fb303/FacebookService$Iface
5、java.lang.OutOfMemoryError: Java heap space
目录
- 1、java.lang.NoClassDefFoundError:org/apache/hadoop/hive/metastore/api/MetaException
- 1.1 解决办法
- 2、java.lang.NoClassDefFoundError: org/apache/thrift/TBase
- 2.1 解决办法
- 3、java.lang.NoClassDefFoundError:org/apache/hadoop/hive/conf/HiveConf
- 3.1 解决办法
- 4、java.lang.NoClassDefFoundError:com/facebook/fb303/FacebookService$Iface
- 4.1 解决办法
- 5、java.lang.OutOfMemoryError: Java heap space
- 5.1 解决办法
1、java.lang.NoClassDefFoundError:org/apache/hadoop/hive/metastore/api/MetaException
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/hive/metastore/api/MetaException
at org.apache.seatunnel.connectors.seatunnel.hive.config.HiveConfig.getTableInfo(HiveConfig.java:59)
at org.apache.seatunnel.connectors.seatunnel.hive.sink.HiveSink.prepare(HiveSink.java:123)
at org.apache.seatunnel.engine.core.parse.JobConfigParser.parseSink(JobConfigParser.java:190)
at org.apache.seatunnel.engine.core.parse.JobConfigParser.parseSinks(JobConfigParser.java:162)
at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parseSink(MultipleTableJobConfigParser.java:515)
at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parse(MultipleTableJobConfigParser.java:170)
at org.apache.seatunnel.engine.client.job.JobExecutionEnvironment.getLogicalDag(JobExecutionEnvironment.java:155)
at org.apache.seatunnel.engine.client.job.JobExecutionEnvironment.execute(JobExecutionEnvironment.java:147)
at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:140)
at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hive.metastore.api.MetaException
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at org.apache.seatunnel.engine.common.loader.SeaTunnelBaseClassLoader.loadClassWithoutExceptionHandling(SeaTunnelBaseClassLoader.java:56)
at org.apache.seatunnel.engine.common.loader.SeaTunnelChildFirstClassLoader.loadClassWithoutExceptionHandling(SeaTunnelChildFirstClassLoader.java:86)
at org.apache.seatunnel.engine.common.loader.SeaTunnelBaseClassLoader.loadClass(SeaTunnelBaseClassLoader.java:47)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 11 more
1.1 解决办法
是缺少对应的包,去hive里面的lib将包复制到seatunnel的包下面即可
hive-metastore-2.1.1-cdh6.3.2.jar # cdh版本的记得将这个也复杂上
hive-metastore.jar
2、java.lang.NoClassDefFoundError: org/apache/thrift/TBase
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/thrift/TBase
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:411)
at org.apache.seatunnel.engine.common.loader.SeaTunnelBaseClassLoader.loadClassWithoutExceptionHandling(SeaTunnelBaseClassLoader.java:56)
at org.apache.seatunnel.engine.common.loader.SeaTunnelChildFirstClassLoader.loadClassWithoutExceptionHandling(SeaTunnelChildFirstClassLoader.java:86)
at org.apache.seatunnel.engine.common.loader.SeaTunnelBaseClassLoader.loadClass(SeaTunnelBaseClassLoader.java:47)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at org.apache.seatunnel.connectors.seatunnel.hive.config.HiveConfig.getTableInfo(HiveConfig.java:59)
at org.apache.seatunnel.connectors.seatunnel.hive.sink.HiveSink.prepare(HiveSink.java:123)
at org.apache.seatunnel.engine.core.parse.JobConfigParser.parseSink(JobConfigParser.java:190)
at org.apache.seatunnel.engine.core.parse.JobConfigParser.parseSinks(JobConfigParser.java:162)
at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parseSink(MultipleTableJobConfigParser.java:515)
at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parse(MultipleTableJobConfigParser.java:170)
at org.apache.seatunnel.engine.client.job.JobExecutionEnvironment.getLogicalDag(JobExecutionEnvironment.java:155)
at org.apache.seatunnel.engine.client.job.JobExecutionEnvironment.execute(JobExecutionEnvironment.java:147)
at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:140)
at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34)
Caused by: java.lang.ClassNotFoundException: org.apache.thrift.TBase
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
2.1 解决办法
是缺少对应的包,去hive里面的lib将包复制到seatunnel的包下面即可
libthrift-0.9.3-1.jar
3、java.lang.NoClassDefFoundError:org/apache/hadoop/hive/conf/HiveConf
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/hive/conf/HiveConf
at org.apache.seatunnel.connectors.seatunnel.hive.utils.HiveMetaStoreProxy.<init>(HiveMetaStoreProxy.java:48)
at org.apache.seatunnel.connectors.seatunnel.hive.utils.HiveMetaStoreProxy.getInstance(HiveMetaStoreProxy.java:74)
at org.apache.seatunnel.connectors.seatunnel.hive.config.HiveConfig.getTableInfo(HiveConfig.java:59)
at org.apache.seatunnel.connectors.seatunnel.hive.sink.HiveSink.prepare(HiveSink.java:123)
at org.apache.seatunnel.engine.core.parse.JobConfigParser.parseSink(JobConfigParser.java:190)
at org.apache.seatunnel.engine.core.parse.JobConfigParser.parseSinks(JobConfigParser.java:162)
at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parseSink(MultipleTableJobConfigParser.java:515)
at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parse(MultipleTableJobConfigParser.java:170)
at org.apache.seatunnel.engine.client.job.JobExecutionEnvironment.getLogicalDag(JobExecutionEnvironment.java:155)
at org.apache.seatunnel.engine.client.job.JobExecutionEnvironment.execute(JobExecutionEnvironment.java:147)
at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:140)
at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hive.conf.HiveConf
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at org.apache.seatunnel.engine.common.loader.SeaTunnelBaseClassLoader.loadClassWithoutExceptionHandling(SeaTunnelBaseClassLoader.java:56)
at org.apache.seatunnel.engine.common.loader.SeaTunnelChildFirstClassLoader.loadClassWithoutExceptionHandling(SeaTunnelChildFirstClassLoader.java:86)
at org.apache.seatunnel.engine.common.loader.SeaTunnelBaseClassLoader.loadClass(SeaTunnelBaseClassLoader.java:47)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 13 more
3.1 解决办法
是缺少对应的包,去hive里面的lib将包复制到seatunnel的包下面即可
hive-common-2.1.1-cdh6.3.2.jar #cdh版本的记得复制过来
hive-common.jar
hive-exec-2.1.1-cdh6.3.2.jar #cdh版本的记得复制过来
hive-exec.jar
4、java.lang.NoClassDefFoundError:com/facebook/fb303/FacebookService$Iface
Exception in thread "main" java.lang.NoClassDefFoundError: com/facebook/fb303/FacebookService$Iface
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at org.apache.seatunnel.connectors.seatunnel.hive.utils.HiveMetaStoreProxy.<init>(HiveMetaStoreProxy.java:58)
at org.apache.seatunnel.connectors.seatunnel.hive.utils.HiveMetaStoreProxy.getInstance(HiveMetaStoreProxy.java:74)
at org.apache.seatunnel.connectors.seatunnel.hive.config.HiveConfig.getTableInfo(HiveConfig.java:59)
at org.apache.seatunnel.connectors.seatunnel.hive.sink.HiveSink.prepare(HiveSink.java:123)
at org.apache.seatunnel.engine.core.parse.JobConfigParser.parseSink(JobConfigParser.java:190)
at org.apache.seatunnel.engine.core.parse.JobConfigParser.parseSinks(JobConfigParser.java:162)
at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parseSink(MultipleTableJobConfigParser.java:515)
at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parse(MultipleTableJobConfigParser.java:170)
at org.apache.seatunnel.engine.client.job.JobExecutionEnvironment.getLogicalDag(JobExecutionEnvironment.java:155)
at org.apache.seatunnel.engine.client.job.JobExecutionEnvironment.execute(JobExecutionEnvironment.java:147)
at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:140)
at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34)
Caused by: java.lang.ClassNotFoundException: com.facebook.fb303.FacebookService$Iface
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 25 more
4.1 解决办法
是缺少对应的包,这个我的hive里面的libthrift-0.9.3-1.ja
r包里面没有,所以包了这个错,去下面这个地址直接下载即可
下载地址:https://repo1.maven.org/maven2/org/apache/thrift/libfb303/0.9.3/libfb303-0.9.3.jar
5、java.lang.OutOfMemoryError: Java heap space
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:379)
at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:230)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_set_ugi(ThriftHiveMetastore.java:4129)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.set_ugi(ThriftHiveMetastore.java:4115)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:563)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:303)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:219)
at org.apache.seatunnel.connectors.seatunnel.hive.utils.HiveMetaStoreProxy.<init>(HiveMetaStoreProxy.java:58)
at org.apache.seatunnel.connectors.seatunnel.hive.utils.HiveMetaStoreProxy.getInstance(HiveMetaStoreProxy.java:74)
at org.apache.seatunnel.connectors.seatunnel.hive.config.HiveConfig.getTableInfo(HiveConfig.java:59)
at org.apache.seatunnel.connectors.seatunnel.hive.sink.HiveSink.prepare(HiveSink.java:123)
at org.apache.seatunnel.engine.core.parse.JobConfigParser.parseSink(JobConfigParser.java:190)
at org.apache.seatunnel.engine.core.parse.JobConfigParser.parseSinks(JobConfigParser.java:162)
at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parseSink(MultipleTableJobConfigParser.java:515)
at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parse(MultipleTableJobConfigParser.java:170)
at org.apache.seatunnel.engine.client.job.JobExecutionEnvironment.getLogicalDag(JobExecutionEnvironment.java:155)
at org.apache.seatunnel.engine.client.job.JobExecutionEnvironment.execute(JobExecutionEnvironment.java:147)
at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:140)
at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34)
5.1 解决办法
去seatunnel的config里面增加对应值,截图如下:
将值修改大一点就可以了