hadoop测试环境sqoop使用

news2025/2/27 4:37:58

提示：文章写完后，目录可以自动生成，如何生成可参考右边的帮助文档

Sqoop看这篇文章就够了_must contain '$conditions' in where clause._SoWhat1412的博客-CSDN博客

大数据环境

C:\Windows\System32\drivers\etc

修改ip和hostname的对应关系

127.0.0.1 view-localhost # view localhost server

192.168.52.150 hadoop01 hadoop01.itcast.cn

192.168.52.151 hadoop02 hadoop02.itcast.cn

检查环境

ping 通环境测试

检查hadoop的版本

[root@hadoop01 ~]# hadoop version

Hadoop 3.0.0-cdh6.2.1

Source code repository http://github.com/cloudera/hadoop -r 7ce7dcdb76fd7f700cd7749bbc6390f76ff2abae

Compiled by jenkins on 2019-09-11T08:04Z

Compiled with protoc 2.5.0

From source with checksum b6cc25ff15ff575f159489abbee34c5

This command was run using /opt/cloudera/parcels/CDH-6.2.1-1.cdh6.2.1.p0.1425774/jars/hadoop-common-3.0.0-cdh6.2.1.jar

检查hive的版本

[root@hadoop01 ~]# hive --version

WARNING: Use "yarn jar" to launch YARN applications.

SLF4J: Class path contains multiple SLF4J bindings.

SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.2.1-1.cdh6.2.1.p0.1425774/jars/log4j-slf4j-impl-2.8.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.2.1-1.cdh6.2.1.p0.1425774/jars/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.

SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]

Hive 2.1.1-cdh6.2.1

Subversion file:///container.redhat7/build/cdh/hive/2.1.1-cdh6.2.1/rpm/BUILD/hive-2.1.1-cdh6.2.1 -r fc0582dc4d2e2342d1eef8cb702bad68e1e2cdd2

Compiled by jenkins on Wed Sep 11 01:31:00 PDT 2019

From source with checksum 6bf22b68335469e1b010bcfecc8ea570

检查 hive服务是否启动了

[root@hadoop01 ~]# hive

WARNING: Use "yarn jar" to launch YARN applications.

SLF4J: Class path contains multiple SLF4J bindings.

SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.2.1-1.cdh6.2.1.p0.1425774/jars/log4j-slf4j-impl-2.8.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.2.1-1.cdh6.2.1.p0.1425774/jars/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.

SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]

Logging initialized using configuration in jar:file:/opt/cloudera/parcels/CDH-6.2.1-1.cdh6.2.1.p0.1425774/jars/hive-common-2.1.1-cdh6.2.1.jar!/hive-log4j2.properties Async: false

WARNING: Hive CLI is deprecated and migration to Beeline is recommended.

hive>

WARNING: Hive CLI is deprecated and migration to Beeline is recommended.

hive>

> show databases;

default

test

Time taken: 4.039 seconds, Fetched: 2 row(s)

hive>

检查 sqoop是否安装

[root@hadoop01 ~]# sqoop help

Warning: /opt/cloudera/parcels/CDH-6.2.1-1.cdh6.2.1.p0.1425774/bin/../lib/sqoop/../accumulo does not exist! Accumulo imports will fail.

Please set $ACCUMULO_HOME to the root of your Accumulo installation.

SLF4J: Class path contains multiple SLF4J bindings.

SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.2.1-1.cdh6.2.1.p0.1425774/jars/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.2.1-1.cdh6.2.1.p0.1425774/jars/log4j-slf4j-impl-2.8.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.

SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]

23/09/20 13:48:09 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7-cdh6.2.1

usage: sqoop COMMAND [ARGS]

Available commands:

codegen Generate code to interact with database records

create-hive-table Import a table definition into Hive

eval Evaluate a SQL statement and display the results

export Export an HDFS directory to a database table

help List available commands

import Import a table from a database to HDFS

import-all-tables Import tables from a database to HDFS

import-mainframe Import datasets from a mainframe server to HDFS

job Work with saved jobs

list-databases List available databases on a server

list-tables List available tables in a database

merge Merge results of incremental imports

metastore Run a standalone Sqoop metastore

version Display version information

See 'sqoop help COMMAND' for information on a specific command.

检查mysql数据库

[root@hadoop01 ~]# mysql -uroot -p123456

mysql: [Warning] Using a password on the command line interface can be insecure.

Welcome to the MySQL monitor. Commands end with ; or \g.

Your MySQL connection id is 70

Server version: 5.7.26 MySQL Community Server (GPL)

Oracle is a registered trademark of Oracle Corporation and/or its

affiliates. Other names may be trademarks of their respective

owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql>

mysql> show databases;

+--------------------+

| Database |

+--------------------+

| information_schema |

| hive |

| hue |

| mysql |

| nev |

| oozie |

| performance_schema |

| scm |

| sys |

| test |

+--------------------+

10 rows in set (0.00 sec)

sqoop list-databases --connect jdbc:mysql://192.168.52.150:3306/ --username root --password 123456

测试sqoop数据库的连接

[root@hadoop01 ~]# sqoop list-databases --connect jdbc:mysql://192.168.52.150:3306/ --username root --password 123456

Warning: /opt/cloudera/parcels/CDH-6.2.1-1.cdh6.2.1.p0.1425774/bin/../lib/sqoop/../accumulo does not exist! Accumulo imports will fail.

Please set $ACCUMULO_HOME to the root of your Accumulo installation.

SLF4J: Class path contains multiple SLF4J bindings.

SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.2.1-1.cdh6.2.1.p0.1425774/jars/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.2.1-1.cdh6.2.1.p0.1425774/jars/log4j-slf4j-impl-2.8.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.

SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]

23/09/20 13:54:17 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7-cdh6.2.1

23/09/20 13:54:17 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.

23/09/20 13:54:17 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.

Wed Sep 20 13:54:18 CST 2023 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.

information_schema

hive

hue

mysql

nev

oozie

performance_schema

scm

sys

test

[root@hadoop01 ~]#

练习环境部分信息