十一、hadoop应用

news2024/9/29 11:37:04

1.上传数据集

27.19.74.143,2015/3/30 17:38,/static/image/common/faq.gif
110.52.250.126,2015/3/30 17:38,/data/cache/style_1_widthauto.css?y7a
27.19.74.143,2015/3/30 17:38,/static/image/common/hot_1.gif
27.19.74.143,2015/3/30 17:38,/static/image/common/hot_2.gif
27.19.74.143,2015/3/30 17:38,/static/image/filetype/common.gif
110.52.250.126,2015/3/30 17:38,/source/plugin/wsh_wx/img/wsh_zk.css
110.52.250.126,2015/3/30 17:38,/data/cache/style_1_forum_index.css?y7a
110.52.250.126,2015/3/30 17:38,/source/plugin/wsh_wx/img/wx_jqr.gif
27.19.74.143,2015/3/30 17:38,/static/image/common/recommend_1.gif
110.52.250.126,2015/3/30 17:38,/static/image/common/logo.png
27.19.74.143,2015/3/30 17:38,/data/attachment/common/c8/common_2_verify_icon.png
110.52.250.126,2015/3/30 17:38,/static/js/logging.js?y7a
8.35.201.144,2015/3/30 17:38,/uc_server/avatar.php?uid=29331&size=middle
27.19.74.143,2015/3/30 17:38,/data/cache/common_smilies_var.js?y7a
27.19.74.143,2015/3/30 17:38,/static/image/common/pn.png
27.19.74.143,2015/3/30 17:38,/static/image/common/swfupload.swf?preventswfcaching=1369906718144
27.19.74.143,2015/3/30 17:38,/static/image/editor/editor.gif
8.35.201.165,2015/3/30 17:38,/uc_server/data/avatar/000/05/94/42_avatar_middle.jpg
8.35.201.164,2015/3/30 17:38,/uc_server/data/avatar/000/03/13/51_avatar_middle.jpg
8.35.201.163,2015/3/30 17:38,/uc_server/data/avatar/000/04/87/94_avatar_middle.jpg
8.35.201.165,2015/3/30 17:38,/uc_server/data/avatar/000/01/01/03_avatar_middle.jpg
8.35.201.160,2015/3/30 17:38,/uc_server/data/avatar/000/04/12/85_avatar_middle.jpg
8.35.201.164,2015/3/30 17:38,/uc_server/avatar.php?uid=53635&size=middle
8.35.201.163,2015/3/30 17:38,/static/image/common/arw_r.gif
8.35.201.166,2015/3/30 17:38,/static/image/common/px.png
8.35.201.144,2015/3/30 17:38,/static/image/common/pmto.gif
8.35.201.161,2015/3/30 17:38,/static/image/common/search.png
8.35.201.163,2015/3/30 17:38,/uc_server/avatar.php?uid=57232&size=middle
8.35.201.164,2015/3/30 17:38,/uc_server/data/avatar/000/05/83/35_avatar_middle.jpg
8.35.201.160,2015/3/30 17:38,/uc_server/data/avatar/000/01/54/22_avatar_middle.jpg

2.启动Hadoop

3. 创建文件夹并上传文件

4.查看数据集

5.在hive中创建数据库

启动数据库

启动hive

创建数据库

CREATE EXTERNAL TABLE IF NOT EXISTS dblab.bigdata_user (
    ip STRING,
    event_time STRING,
    url STRING
)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY ','
STORED AS TEXTFILE
LOCATION '/bigdata/dataset';

查看表的属性

查看表的结构

查看表的前10条数据

6.数据分析

简单查询

查询前20条ip和event_time

统计表中的数据

7.数据迁移

hive创建临时表

CREATE EXTERNAL TABLE dblab.user_action(ip STRING,event_time STRING,url STRING) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' STORED AS TEXTFILE;

查看创建的表

hadoop dfs -ls /user/hive/warehouse/dblab.db/user_action

导入数据

查看数据

登录MySQL创建数据表

CREATE TABLE user_action (ip VARCHAR(50), even_time VARCHAR(50),url VARCHAR(100)) ENGINE=InnoDB DEFAULT CHARSET=utf8;

sqoop迁移数据

sqoop export --connect jdbc:mysql://localhost:3306/dblab --username root -P --table user_action --export-dir /user/hive/warehouse/dblab.db/user_action --input-fields-terminated-by ','