学习大数据DAY58 增量抽取数据表

news2024/9/22 19:26:58

作业

1 SQL 优化的常见写法有哪些 - 面试经常被问
使用索引：合理创建和使用索引是提高查询效率的关键。索引可以加速数据的检
索速度，但是索引也会占用额外的存储空间，并且在插入、删除和更新操作时会
有额外的开销。
避免全表扫描：尽量避免在 WHERE 子句中使用不等于(!=)、非索引列的函数或者
IS NULL 等操作，因为这些操作往往会导致数据库引擎放弃索引而进行全表扫描。
优化 JOIN 操作：确保在 JOIN 操作中使用的字段上有索引，并且尽可能地减少
JOIN 的数量，使用 INNER JOIN 替代 LEFT JOIN 来减少额外的行返回。
限制结果集大小：在可能的情况下，使用 WHERE 子句来限制结果集的大小。如果
只需要结果集的一部分数据，可以使用 LIMIT 或 TOP 关键字来限制返回的行数。
分批处理大结果集：如果查询返回的结果集非常大，考虑使用分批处理的方式来
逐步获取数据，例如使用游标或分页查询。
使用 EXPLAIN 分析查询：在执行查询前使用 EXPLAIN（MySQL）或类似的工具来
分析查询执行计划，可以帮助识别问题所在，从而优化查询语句。
-- oracle
explain plan for sql 语句
-- mysql
explain sql 语句
减少 SELECT * 的使用：避免使用 SELECT * 来获取所有列，尽量只选择需要的
列，这样可以减少数据传输的量并提高查询效率。
注意数据类型的一致性：在 WHERE 子句中进行比较时，确保数据类型的一致性，
以避免不必要的类型转换操作。
优化子查询：如果可能，尽量将子查询改写成 JOIN 操作，因为子查询可能会导
致查询效率低下。
使用临时表和表变量：对于复杂查询，合理使用临时表或表变量可以帮助存储中
间结果，减少不必要的计算。 2 编写 python 辅助脚本
可以使用 CDH02 作为开发机器 可以不用打开自己的虚拟机
在 SSH 配置文件中添加：（注意要放在最上面）
进入后，在里面创建自己的文件：
如何连接 CDH 的 Hive
把驱动
共享\项目课工具\hive-jdbc-uber-2.6.5.0-292.jar 复制到自己的电脑，位置随意
访问 CDH6 的网页地址
协作开发一定要注意
●不要随便删除别人的数据库和表
●不要随便删除 HDFS 别人的数据路径
快速生成配置文件
python /opt/datax/bin/datax.py -r mysqlreader -w hdfswriter
添加属性：
u_accept_m_inc: 
{
"job": {
"content": [
{
"reader": {
"name": "mysqlreader",
"parameter": {
"connection": [
{
"jdbcUrl": [
"jdbc:mysql://zhiyun.pub:233
06/erp"
],"querySql": [
"select * from u_accept_m"
]
}
],
"password": "zhiyun",
"username": "zhiyun"
}
},
"writer": {
"name": "hdfswriter",
"parameter": {
"column": [
{"name":"id","type":"int"},
{"name":"billcode","type":"string"},
{"name":"acceptno","type":"string"},
{"name":"receiptno","type":"string"}
,
{"name":"orderno","type":"string"},
{"name":"pactno","type":"string"},
{"name":"busno","type":"string"},
{"name":"vendorno","type":"string"},
{"name":"paytype","type":"string"},
{"name":"paydate","type":"string"},
{"name":"buyer","type":"string"},
{"name":"checker1","type":"string"},
{"name":"checkbit1","type":"string"}
,
{"name":"checker2","type":"string"},
{"name":"checkbit2","type":"string"}
,
{"name":"checker3","type":"string"},
{"name":"checkbit3","type":"string"}
,
{"name":"checker4","type":"string"},
{"name":"checkbit4","type":"string"}
,
{"name":"status","type":"string"},
{"name":"accepttype","type":"string"}
,
{"name":"createuser","type":"string"}
,{"name":"createtime","type":"string"}
,
{"name":"notes","type":"string"},
{"name":"stamp","type":"string"},
{"name":"execdate","type":"string"},
{"name":"sendno","type":"string"},
{"name":"distno","type":"string"},
{"name":"whlno","type":"string"},
{"name":"dept","type":"string"},
{"name":"bak1","type":"string"},
{"name":"bak2","type":"string"},
{"name":"bak3","type":"string"},
{"name":"bak4","type":"string"},
{"name":"bak5","type":"string"},
{"name":"bak6","type":"string"},
{"name":"bak7","type":"string"},
{"name":"store_status","type":"strin
g"},
{"name":"bak88","type":"string"},
{"name":"consingerid","type":"string
"},
{"name":"flag_status","type":"string
"},
{"name":"flag_date","type":"string"}
,
{"name":"ownerid","type":"string"},
{"name":"userdeptno","type":"string"}
,
{"name":"bill_source","type":"string
"},
{"name":"puramt","type":"string"},
{"name":"vendorsaler","type":"string
"},
{"name":"yycbillno","type":"string"}
,
{"name":"yycstatus","type":"string"}
,
{"name":"yycvendornotes","type":"str
ing"},
{"name":"yycexecdate","type":"string
"},
{"name":"wms_flag","type":"string"},{"name":"init_createtime","type":"st
ring"},
{"name":"init_createuser","type":"st
ring"},
{"name":"msfx_upflag","type":"string
"},
{"name":"vendor_address","type":"str
ing"}
],
"defaultFS": "hdfs://cdh02:8020",
"fieldDelimiter": "\t",
"fileName": "u_accept_m_inc.data",
"fileType": "orc",
"path":
"/zhiyun/shihaihong/tmp/u_accept_m_inc",
"writeMode": "truncate"
}
}
}
],
"setting": {
"speed": {
"channel": 2
}
}
}
}
u_accept_c_inc.json: 
{
"job": {
"content": [
{
"reader": {
"name": "mysqlreader",
"parameter": {
"connection": [
{
"jdbcUrl": ["jdbc:mysql://zhiyun.pub:233
06/erp"
],
"querySql": [
"select * from u_accept_c"
]
}
],
"password": "zhiyun",
"username": "zhiyun"
}
},
"writer": {
"name": "hdfswriter",
"parameter": {
"column": [
{"name":"id","type":"int"},
{"name":"acceptno","type":"string"},
{"name":"idno","type":"string"},
{"name":"direct","type":"string"},
{"name":"wareid","type":"string"},
{"name":"stallno","type":"string"},
{"name":"wareqty","type":"string"},
{"name":"purprice","type":"string"},
{"name":"purtax","type":"string"},
{"name":"makeno","type":"string"},
{"name":"makedate","type":"string"},
{"name":"invalidate","type":"string"}
,
{"name":"acb_batchno","type":"string
"},
{"name":"acb_no","type":"string"},
{"name":"acb_idno","type":"string"},
{"name":"acb_qty","type":"string"},
{"name":"cker1","type":"string"},
{"name":"cker2","type":"string"},
{"name":"cker3","type":"string"},
{"name":"notes","type":"string"},
{"name":"leastpriceo","type":"string
"},
{"name":"leastpricen","type":"string
"},{"name":"saleprice","type":"string"}
,
{"name":"whlpriceo","type":"string"}
,
{"name":"whlpricen","type":"string"}
,
{"name":"purpriceo","type":"string"}
,
{"name":"rowid","type":"string"},
{"name":"indentqty","type":"string"}
,
{"name":"invalidate_char","type":"st
ring"},
{"name":"bak1","type":"string"},
{"name":"bak2","type":"string"},
{"name":"bak3","type":"string"},
{"name":"bak4","type":"string"},
{"name":"bak5","type":"string"},
{"name":"bak6","type":"string"},
{"name":"bak7","type":"string"},
{"name":"bak8","type":"string"},
{"name":"maxqty","type":"string"},
{"name":"midqty","type":"string"},
{"name":"batchno_act","type":"string
"},
{"name":"idno_act","type":"string"},
{"name":"indentprice","type":"string
"},
{"name":"prodid","type":"string"},
{"name":"seal_stall","type":"string"}
,
{"name":"flag2","type":"string"},
{"name":"indentno","type":"string"},
{"name":"checkstallno","type":"strin
g"},
{"name":"chkcont","type":"string"},
{"name":"chkresult","type":"string"}
,
{"name":"backprice","type":"string"}
,
{"name":"bioavailability","type":"st
ring"},{"name":"sterilemakeno","type":"stri
ng"},
{"name":"chk","type":"string"},
{"name":"groupid","type":"string"},
{"name":"distprice","type":"string"}
,
{"name":"bak9","type":"string"},
{"name":"bak10","type":"string"},
{"name":"purprice_no","type":"string
"},
{"name":"sterileinvalidate","type":"
string"},
{"name":"storeqty","type":"string"},
{"name":"barcode","type":"string"},
{"name":"check_qty","type":"string"}
,
{"name":"check_nook_qty","type":"str
ing"},
{"name":"reason","type":"string"},
{"name":"prod_addid","type":"string"}
,
{"name":"douchecker1","type":"string
"},
{"name":"douchecker2","type":"string
"},
{"name":"tallyqty","type":"string"},
{"name":"tally_checknoqty","type":"s
tring"},
{"name":"checkno_notes","type":"stri
ng"},
{"name":"unqualified","type":"string
"},
{"name":"eq_no","type":"string"},
{"name":"sterilemakedate","type":"st
ring"},
{"name":"tally_idno","type":"string"}
,
{"name":"tally_makeno","type":"strin
g"},
{"name":"tally_stallno","type":"stri
ng"},
{"name":"trayno","type":"string"},
{"name":"bp_id","type":"string"},{"name":"packqty","type":"string"},
{"name":"distcount","type":"string"}
,
{"name":"payeetype","type":"string"}
,
{"name":"wareqty_bak","type":"string
"},
{"name":"scan_flag","type":"string"}
,
{"name":"quality_standard","type":"s
tring"}
],
"defaultFS": "hdfs://cdh02:8020",
"fieldDelimiter": "\t",
"fileName": "u_accept_c_inc.data",
"fileType": "orc",
"path":
"/zhiyun/shihaihong/tmp/u_accept_c_inc",
"writeMode": "truncate"
}
}
}
],
"setting": {
"speed": {
"channel": 2
}
}
}
}
数据列的处理
编写两个 python 辅助脚本，自动打印出 datax 和 hive 的字段信息
u_accept_m_inc.py: 
#!/bin/python3
# 自动打印 datax 和 hive 的字段信息
ddl='''
`id` int(10) unsigned NOT NULL AUTO_INCREMENT,`billcode` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`acceptno` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`receiptno` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`orderno` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`pactno` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`busno` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`vendorno` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`paytype` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`paydate` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`buyer` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`checker1` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`checkbit1` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`checker2` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`checkbit2` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`checker3` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`checkbit3` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`checker4` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`checkbit4` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`status` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`accepttype` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`createuser` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`createtime` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`notes` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`stamp` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`execdate` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`sendno` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`distno` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`whlno` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`dept` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`bak1` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`bak2` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`bak3` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`bak4` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`bak5` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`bak6` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`bak7` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`store_status` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`bak88` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`consingerid` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`flag_status` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`flag_date` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`ownerid` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`userdeptno` varchar(255) COLLATE utf8_bin DEFAULT NULL,`bill_source` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`puramt` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`vendorsaler` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`yycbillno` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`yycstatus` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`yycvendornotes` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`yycexecdate` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`wms_flag` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`init_createtime` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`init_createuser` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`msfx_upflag` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`vendor_address` varchar(255) COLLATE utf8_bin DEFAULT NULL,
'''
# 切割字符串
lines=ddl.strip().split("\n")
fileds=[]
# print(lines)
for line in lines:
# print(line)
data=line.strip().split(" ")
# mysql 的类型
field_name=data[0].replace('`',"")
#数据类型
field_type=data[1].split("(")[0]
#转换成 hive 类型
field_hive_type="string"
if field_type=="int" or field_type=="tinyint" or
field_type=="bigint":
field_hive_type="int"
if field_type=="float" or field_type=="double":
field_hive_type="float"
fileds.append([field_name,field_hive_type])
print("=============== 打印 datax 的列信息 ===============")
for filed in fileds:line='{"name":"'+filed[0]+'","type":"'+filed[1]+'"},'
print(line)
print("=============== 打印 hive 的列信息 ===============")
for filed in fileds:
line=f"{filed[0]} {filed[1]},"
print(line)
hive 建表： 
create database if not exists ods_shihaihong location
"/zhiyun/shihaihong/ods";
-- 增量表
create external table if not exists ods_shihaihong.u_accept_m_inc(
id int,
billcode string,
acceptno string,
receiptno string,
orderno string,
pactno string,
busno string,
vendorno string,
paytype string,
paydate string,
buyer string,
checker1 string,
checkbit1 string,
checker2 string,
checkbit2 string,
checker3 string,
checkbit3 string,
checker4 string,
checkbit4 string,
status string,
accepttype string,
createuser string,
notes string,
stamp string,
execdate string,
sendno string,distno string,
whlno string,
dept string,
bak1 string,
bak2 string,
bak3 string,
bak4 string,
bak5 string,
bak6 string,
bak7 string,
store_status string,
bak88 string,
consingerid string,
flag_status string,
flag_date string,
ownerid string,
userdeptno string,
bill_source string,
puramt string,
vendorsaler string,
yycbillno string,
yycstatus string,
yycvendornotes string,
yycexecdate string,
wms_flag string,
init_createtime string,
init_createuser string,
msfx_upflag string,
vendor_address string
) partitioned by (createtime string)
row format delimited fields terminated by "\t"
lines terminated by "\n"
stored as orc
location "/zhiyun/shihaihong/ods/u_accept_m_inc";
抽取数据：
hadoop fs -mkdir -p /zhiyun/shihaihong/tmp/u_accept_m_inc
python
/opt/datax/bin/datax.py
/zhiyun/shihaihong/jobs/u_accept_m_inc.json
加载脚本：
进入 hive 后：
load data inpath
"/zhiyun/shihaihong/tmp/u_accept_m_inc/*"
overwrite into table ods_shihaihong.u_accept_m_inc;
验证：
show partitions ods_shihaihong.u_accept_m_inc ;
select count ( 1 ) from ods_shihaihong.u_accept_m_inc where createtime =
"2016-11-10" ;
select * from ods_shihaihong.u_accept_m_inc where createtime =
"2016-11-10" limit 5 ;
编写调度脚本：
u_accept_m_inc.sh: 
#!/bin/bash
day=$(date -d "yesterday" +%Y-%m-%d)if [ $1 != "" ]; then
day=$1;
fi;
echo "抽取的日期为 $day"
echo "生成增量配置文件"
mkdir -p /zhiyun/shihaihong/jobs
echo '{
"job": {
"content": [
{
"reader": {
"name": "mysqlreader",
"parameter": {
"connection": [
{
"jdbcUrl": [
"jdbc:mysql://zhiyun.pub:233
06/erp"
],
"querySql": [
"select * from u_accept_m"
]
}
],
"password": "zhiyun",
"username": "zhiyun"
}
},
"writer": {
"name": "hdfswriter",
"parameter": {
"column": [
{"name":"id","type":"int"},
{"name":"billcode","type":"string"},
{"name":"acceptno","type":"string"},
{"name":"receiptno","type":"string"}
,
{"name":"orderno","type":"string"},
{"name":"pactno","type":"string"},{"name":"busno","type":"string"},
{"name":"vendorno","type":"string"},
{"name":"paytype","type":"string"},
{"name":"paydate","type":"string"},
{"name":"buyer","type":"string"},
{"name":"checker1","type":"string"},
{"name":"checkbit1","type":"string"}
,
{"name":"checker2","type":"string"},
{"name":"checkbit2","type":"string"}
,
{"name":"checker3","type":"string"},
{"name":"checkbit3","type":"string"}
,
{"name":"checker4","type":"string"},
{"name":"checkbit4","type":"string"}
,
{"name":"status","type":"string"},
{"name":"accepttype","type":"string"}
,
{"name":"createuser","type":"string"}
,
{"name":"createtime","type":"string"}
,
{"name":"notes","type":"string"},
{"name":"stamp","type":"string"},
{"name":"execdate","type":"string"},
{"name":"sendno","type":"string"},
{"name":"distno","type":"string"},
{"name":"whlno","type":"string"},
{"name":"dept","type":"string"},
{"name":"bak1","type":"string"},
{"name":"bak2","type":"string"},
{"name":"bak3","type":"string"},
{"name":"bak4","type":"string"},
{"name":"bak5","type":"string"},
{"name":"bak6","type":"string"},
{"name":"bak7","type":"string"},
{"name":"store_status","type":"strin
g"},
{"name":"bak88","type":"string"},
{"name":"consingerid","type":"string
"},{"name":"flag_status","type":"string
"},
{"name":"flag_date","type":"string"}
,
{"name":"ownerid","type":"string"},
{"name":"userdeptno","type":"string"}
,
{"name":"bill_source","type":"string
"},
{"name":"puramt","type":"string"},
{"name":"vendorsaler","type":"string
"},
{"name":"yycbillno","type":"string"}
,
{"name":"yycstatus","type":"string"}
,
{"name":"yycvendornotes","type":"str
ing"},
{"name":"yycexecdate","type":"string
"},
{"name":"wms_flag","type":"string"},
{"name":"init_createtime","type":"st
ring"},
{"name":"init_createuser","type":"st
ring"},
{"name":"msfx_upflag","type":"string
"},
{"name":"vendor_address","type":"str
ing"}
],
"defaultFS": "hdfs://cdh02:8020",
"fieldDelimiter": "\t",
"fileName": "u_accept_m_inc.data",
"fileType": "orc",
"path":
"/zhiyun/shihaihong/tmp/u_accept_m_inc",
"writeMode": "truncate"
}
}
}
],
"setting": {
"speed": {"channel": 2
}
}
}
}' > /zhiyun/shihaihong/jobs/u_accept_m_inc.json
echo "开始抽取"
hadoop fs -mkdir -p /zhiyun/shihaihong/tmp/u_accept_m_inc
python /opt/datax/bin/datax.py
/zhiyun/shihaihong/jobs/u_accept_m_inc.json
echo "hive 建表"
beeline -u jdbc:hive2://localhost:10000 -n root -p 123 -e '
create database if not exists ods_shihaihong location
"/zhiyun/shihaihong/ods";
-- 增量表
create external table if not exists ods_shihaihong.u_accept_m_inc(
id int unsigned,
billcode
string,
acceptno
string,
receiptno
string,
orderno string,
pactno string,
busno
string,
vendorno
string,
paytype string,
paydate string,
buyer
string,
checker1
string,
checkbit1
string,
checker2
string,
checkbit2
string,
checker3
string,
checkbit3
string,
checker4
string,
checkbit4
string,
status string,
accepttype string,
createuser string,
notes
string,
stamp
string,
execdate
string,
sendno string,distno string,
whlno
string,
dept
string,
bak1
string,
bak2
string,
bak3
string,
bak4
string,
bak5
string,
bak6
string,
bak7
string,
store_status
string,
bak88
string,
consingerid string,
flag_status string,
flag_date
string,
ownerid string,
userdeptno string,
bill_source string,
puramt string,
vendorsaler string,
yycbillno
string,
yycstatus
string,
yycvendornotes string,
yycexecdate string,
wms_flag
string,
init_createtime string,
init_createuser string,
msfx_upflag string,
vendor_address string
) partitioned by (createtime string)
row format delimited fields terminated by "\t"
lines terminated by "\n"
stored as orc
location "/zhiyun/shihaihong/ods/u_accept_m_inc";
'
echo "加载数据"
beeline -u jdbc:hive2://localhost:10000 -n root -p 123 -e "
load data inpath \"/zhiyun/shihaihong/tmp/u_accept_m_inc/*\"
overwrite into table ods_shihaihong.u_accept_m_inc
partition(createtime='$day');
"echo "验证数据"
beeline -u jdbc:hive2://localhost:10000 -n root -p 123 -e "
-- 验证分区
show partitions ods_shihaihong.u_accept_m_inc;
-- 验证分区的总数
select count(1) from ods_shihaihong.u_accept_m_inc where
createtime = \"$day\";
-- 验证数据
select * from ods_shihaihong.u_accept_m_inc where createtime =
\"$day\" limit 5;
"
echo "抽取完成"
用同样的方式导入另一张表：
u_accept_c_inc.py: 
#!/bin/python3
# 自动打印 datax 和 hive 的字段信息
ddl='''
`id` int(11) unsigned NOT NULL AUTO_INCREMENT,
`acceptno` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`idno` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`direct` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`wareid` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`stallno` varchar(255) COLLATE utf8_bin NOT NULL,
`wareqty` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`purprice` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`purtax` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`makeno` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`makedate` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`invalidate` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`acb_batchno` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`acb_no` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`acb_idno` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`acb_qty` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`cker1` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`cker2` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`cker3` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`notes` varchar(255) COLLATE utf8_bin DEFAULT NULL,`leastpriceo` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`leastpricen` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`saleprice` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`whlpriceo` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`whlpricen` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`purpriceo` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`rowid` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`indentqty` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`invalidate_char` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`bak1` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`bak2` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`bak3` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`bak4` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`bak5` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`bak6` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`bak7` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`bak8` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`maxqty` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`midqty` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`batchno_act` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`idno_act` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`indentprice` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`prodid` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`seal_stall` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`flag2` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`indentno` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`checkstallno` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`chkcont` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`chkresult` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`backprice` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`bioavailability` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`sterilemakeno` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`chk` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`groupid` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`distprice` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`bak9` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`bak10` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`purprice_no` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`sterileinvalidate` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`storeqty` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`barcode` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`check_qty` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`check_nook_qty` varchar(255) COLLATE utf8_bin DEFAULT NULL,`reason` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`prod_addid` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`douchecker1` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`douchecker2` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`tallyqty` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`tally_checknoqty` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`checkno_notes` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`unqualified` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`eq_no` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`sterilemakedate` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`tally_idno` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`tally_makeno` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`tally_stallno` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`trayno` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`bp_id` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`packqty` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`distcount` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`payeetype` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`wareqty_bak` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`scan_flag` varchar(255) COLLATE utf8_bin DEFAULT NULL,
`quality_standard` varchar(255) COLLATE utf8_bin DEFAULT NULL
'''
# 切割字符串
lines=ddl.strip().split("\n")
fileds=[]
# print(lines)
for line in lines:
# print(line)
data=line.strip().split(" ")
# mysql 的类型
field_name=data[0].replace('`',"")
#数据类型
field_type=data[1].split("(")[0]
#转换成 hive 类型
field_hive_type="string"if field_type=="int" or field_type=="tinyint" or
field_type=="bigint":
field_hive_type="int"
if field_type=="float" or field_type=="double":
field_hive_type="float"
fileds.append([field_name,field_hive_type])
print("=============== 打印 datax 的列信息 ===============")
for filed in fileds:
line='{"name":"'+filed[0]+'","type":"'+filed[1]+'"},'
print(line)
print("=============== 打印 hive 的列信息 ===============")
for filed in fileds:
line=f"{filed[0]} {filed[1]},"
print(line)
u_accept_c_inc.json:
{
"job": {
"content": [
{
"reader": {
"name": "mysqlreader",
"parameter": {
"connection": [
{
"jdbcUrl": [
"jdbc:mysql://zhiyun.pub:233
06/erp"
],
"querySql": [
"select u_accept_c.* from
u_accept_c left join u_accept_m on
u_accept_c.acceptno=u_accept_m.acceptno"
]
}
],
"password": "zhiyun",
"username": "zhiyun"}
},
"writer": {
"name": "hdfswriter",
"parameter": {
"column": [
{"name":"id","type":"int"},
{"name":"acceptno","type":"string"},
{"name":"idno","type":"string"},
{"name":"direct","type":"string"},
{"name":"wareid","type":"string"},
{"name":"stallno","type":"string"},
{"name":"wareqty","type":"string"},
{"name":"purprice","type":"string"},
{"name":"purtax","type":"string"},
{"name":"makeno","type":"string"},
{"name":"makedate","type":"string"},
{"name":"invalidate","type":"string"}
,
{"name":"acb_batchno","type":"string
"},
{"name":"acb_no","type":"string"},
{"name":"acb_idno","type":"string"},
{"name":"acb_qty","type":"string"},
{"name":"cker1","type":"string"},
{"name":"cker2","type":"string"},
{"name":"cker3","type":"string"},
{"name":"notes","type":"string"},
{"name":"leastpriceo","type":"string
"},
{"name":"leastpricen","type":"string
"},
{"name":"saleprice","type":"string"}
,
{"name":"whlpriceo","type":"string"}
,
{"name":"whlpricen","type":"string"}
,
{"name":"purpriceo","type":"string"}
,
{"name":"rowid","type":"string"},
{"name":"indentqty","type":"string"}
,{"name":"invalidate_char","type":"st
ring"},
{"name":"bak1","type":"string"},
{"name":"bak2","type":"string"},
{"name":"bak3","type":"string"},
{"name":"bak4","type":"string"},
{"name":"bak5","type":"string"},
{"name":"bak6","type":"string"},
{"name":"bak7","type":"string"},
{"name":"bak8","type":"string"},
{"name":"maxqty","type":"string"},
{"name":"midqty","type":"string"},
{"name":"batchno_act","type":"string
"},
{"name":"idno_act","type":"string"},
{"name":"indentprice","type":"string
"},
{"name":"prodid","type":"string"},
{"name":"seal_stall","type":"string"}
,
{"name":"flag2","type":"string"},
{"name":"indentno","type":"string"},
{"name":"checkstallno","type":"strin
g"},
{"name":"chkcont","type":"string"},
{"name":"chkresult","type":"string"}
,
{"name":"backprice","type":"string"}
,
{"name":"bioavailability","type":"st
ring"},
{"name":"sterilemakeno","type":"stri
ng"},
{"name":"chk","type":"string"},
{"name":"groupid","type":"string"},
{"name":"distprice","type":"string"}
,
{"name":"bak9","type":"string"},
{"name":"bak10","type":"string"},
{"name":"purprice_no","type":"string
"},
{"name":"sterileinvalidate","type":"
string"},{"name":"storeqty","type":"string"},
{"name":"barcode","type":"string"},
{"name":"check_qty","type":"string"}
,
{"name":"check_nook_qty","type":"str
ing"},
{"name":"reason","type":"string"},
{"name":"prod_addid","type":"string"}
,
{"name":"douchecker1","type":"string
"},
{"name":"douchecker2","type":"string
"},
{"name":"tallyqty","type":"string"},
{"name":"tally_checknoqty","type":"s
tring"},
{"name":"checkno_notes","type":"stri
ng"},
{"name":"unqualified","type":"string
"},
{"name":"eq_no","type":"string"},
{"name":"sterilemakedate","type":"st
ring"},
{"name":"tally_idno","type":"string"}
,
{"name":"tally_makeno","type":"strin
g"},
{"name":"tally_stallno","type":"stri
ng"},
{"name":"trayno","type":"string"},
{"name":"bp_id","type":"string"},
{"name":"packqty","type":"string"},
{"name":"distcount","type":"string"}
,
{"name":"payeetype","type":"string"}
,
{"name":"wareqty_bak","type":"string
"},
{"name":"scan_flag","type":"string"}
,
{"name":"quality_standard","type":"s
tring"}
],"defaultFS": "hdfs://cdh02:8020",
"fieldDelimiter": "\t",
"fileName": "u_accept_c_inc.data",
"fileType": "orc",
"path":
"/zhiyun/shihaihong/tmp/u_accept_c_inc",
"writeMode": "truncate"
}
}
}
],
"setting": {
"speed": {
"channel": 2
}
}
}
}
u_accept_c_inc.sh: 
#!/bin/bash
day=$(date -d "yesterday" +%Y-%m-%d)
if [ $1 != "" ]; then
day=$1;
fi;
echo "抽取的日期为 $day"
echo "生成增量配置文件"
mkdir -p /zhiyun/shihaihong/jobs
echo '
{
"job": {
"content": [
{
"reader": {"name": "mysqlreader",
"parameter": {
"connection": [
{
"jdbcUrl": [
"jdbc:mysql://zhiyun.pub:233
06/erp?useSSL=false"
],
"querySql": [
"select a.* from u_accept_c
a left join u_accept_m b on a.acceptno =b.acceptno where
createtime between '\'$day 00:00:00\'' and '\'$day 23:59:59\''"
]
}
],
"password": "zhiyun",
"username": "zhiyun"
}
},
"writer": {
"name": "hdfswriter",
"parameter": {
"column": [
{"name":"id","type":"int"},
{"name":"acceptno","type":"string"},
{"name":"idno","type":"string"},
{"name":"direct","type":"string"},
{"name":"wareid","type":"string"},
{"name":"stallno","type":"string"},
{"name":"wareqty","type":"string"},
{"name":"purprice","type":"string"},
{"name":"purtax","type":"string"},
{"name":"makeno","type":"string"},
{"name":"makedate","type":"string"},
{"name":"invalidate","type":"string"}
,
{"name":"acb_batchno","type":"string
"},
{"name":"acb_no","type":"string"},
{"name":"acb_idno","type":"string"},
{"name":"acb_qty","type":"string"},
{"name":"cker1","type":"string"},
{"name":"cker2","type":"string"},{"name":"cker3","type":"string"},
{"name":"notes","type":"string"},
{"name":"leastpriceo","type":"string
"},
{"name":"leastpricen","type":"string
"},
{"name":"saleprice","type":"string"}
,
{"name":"whlpriceo","type":"string"}
,
{"name":"whlpricen","type":"string"}
,
{"name":"purpriceo","type":"string"}
,
{"name":"rowid","type":"string"},
{"name":"indentqty","type":"string"}
,
{"name":"invalidate_char","type":"st
ring"},
{"name":"bak1","type":"string"},
{"name":"bak2","type":"string"},
{"name":"bak3","type":"string"},
{"name":"bak4","type":"string"},
{"name":"bak5","type":"string"},
{"name":"bak6","type":"string"},
{"name":"bak7","type":"string"},
{"name":"bak8","type":"string"},
{"name":"maxqty","type":"string"},
{"name":"midqty","type":"string"},
{"name":"batchno_act","type":"string
"},
{"name":"idno_act","type":"string"},
{"name":"indentprice","type":"string
"},
{"name":"prodid","type":"string"},
{"name":"seal_stall","type":"string"}
,
{"name":"flag2","type":"string"},
{"name":"indentno","type":"string"},
{"name":"checkstallno","type":"strin
g"},
{"name":"chkcont","type":"string"},{"name":"chkresult","type":"string"}
,
{"name":"backprice","type":"string"}
,
{"name":"bioavailability","type":"st
ring"},
{"name":"sterilemakeno","type":"stri
ng"},
{"name":"chk","type":"string"},
{"name":"groupid","type":"string"},
{"name":"distprice","type":"string"}
,
{"name":"bak9","type":"string"},
{"name":"bak10","type":"string"},
{"name":"purprice_no","type":"string
"},
{"name":"sterileinvalidate","type":"
string"},
{"name":"storeqty","type":"string"},
{"name":"barcode","type":"string"},
{"name":"check_qty","type":"string"}
,
{"name":"check_nook_qty","type":"str
ing"},
{"name":"reason","type":"string"},
{"name":"prod_addid","type":"string"}
,
{"name":"douchecker1","type":"string
"},
{"name":"douchecker2","type":"string
"},
{"name":"tallyqty","type":"string"},
{"name":"tally_checknoqty","type":"s
tring"},
{"name":"checkno_notes","type":"stri
ng"},
{"name":"unqualified","type":"string
"},
{"name":"eq_no","type":"string"},
{"name":"sterilemakedate","type":"st
ring"},
{"name":"tally_idno","type":"string"}
,{"name":"tally_makeno","type":"strin
g"},
{"name":"tally_stallno","type":"stri
ng"},
{"name":"trayno","type":"string"},
{"name":"bp_id","type":"string"},
{"name":"packqty","type":"string"},
{"name":"distcount","type":"string"}
,
{"name":"payeetype","type":"string"}
,
{"name":"wareqty_bak","type":"string
"},
{"name":"scan_flag","type":"string"}
,
{"name":"quality_standard","type":"s
tring"}
],
"defaultFS": "hdfs://cdh02:8020",
"fieldDelimiter": "\t",
"fileName": "u_accept_c_inc.data",
"fileType": "orc",
"path":
"/zhiyun/shihaihong/tmp/u_accept_c_inc",
"writeMode": "truncate"
}
}
}
],
"setting": {
"speed": {
"channel": 2
}
}
}
}
' > /zhiyun/shihaihong/jobs/u_accept_c_inc.json
echo "开始抽取"
hadoop fs -mkdir -p /zhiyun/shihaihong/tmp/u_accept_c_inc
python /opt/datax/bin/datax.py
/zhiyun/shihaihong/jobs/u_accept_c_inc.jsonecho "hive 建表"
beeline -u jdbc:hive2://localhost:10000 -n root -p 123 -e '
create database if not exists ods_shihaihong location
"/zhiyun/shihaihong/ods";
-- 增量表
create external table if not exists ods_shihaihong.u_accept_c_inc(
id int,
acceptno string,
idno string,
direct string,
wareid string,
stallno string,
wareqty string,
purprice string,
purtax string,
makeno string,
makedate string,
invalidate string,
acb_batchno string,
acb_no string,
acb_idno string,
acb_qty string,
cker1 string,
cker2 string,
cker3 string,
notes string,
leastpriceo string,
leastpricen string,
saleprice string,
whlpriceo string,
whlpricen string,
purpriceo string,
rowid string,
indentqty string,
invalidate_char string,
bak1 string,
bak2 string,
bak3 string,
bak4 string,
bak5 string,
bak6 string,
bak7 string,
bak8 string,maxqty string,
midqty string,
batchno_act string,
idno_act string,
indentprice string,
prodid string,
seal_stall string,
flag2 string,
indentno string,
checkstallno string,
chkcont string,
chkresult string,
backprice string,
bioavailability string,
sterilemakeno string,
chk string,
groupid string,
distprice string,
bak9 string,
bak10 string,
purprice_no string,
sterileinvalidate string,
storeqty string,
barcode string,
check_qty string,
check_nook_qty string,
reason string,
prod_addid string,
douchecker1 string,
douchecker2 string,
tallyqty string,
tally_checknoqty string,
checkno_notes string,
unqualified string,
eq_no string,
sterilemakedate string,
tally_idno string,
tally_makeno string,
tally_stallno string,
trayno string,
bp_id string,
packqty string,
distcount string,payeetype string,
wareqty_bak string,
scan_flag string,
quality_standard string
) partitioned by (dt string)
row format delimited fields terminated by "\t"
lines terminated by "\n"
stored as orc
location "/zhiyun/shihaihong/ods/u_accept_c_inc";
'
echo "加载数据"
beeline -u jdbc:hive2://localhost:10000 -n root -p 123 -e "
load data inpath '/zhiyun/shihaihong/tmp/u_accept_c_inc/*'
overwrite into table ods_shihaihong.u_accept_c_inc
partition(dt='$day');
"
echo "验证数据"
beeline -u jdbc:hive2://localhost:10000 -n root -p 123 -e "
show partitions ods_shihaihong.u_accept_c_inc;
select count(1) from ods_shihaihong.u_accept_c_inc where dt =
'$day';
select * from ods_shihaihong.u_accept_c_inc where dt = '$day' limit
5;
"
echo "抽取完成"
3 完成两张表的增量处理, 部署两个调度任务
3.1 每个任务的日期需要传参 3 个不同的日期, 不能报错
3.2 抽取的 SQL 不能是全表扫描
u_accepts_m_inc:
2018-03-12:
2020-01-01:
2015-07-02:
u_accept_c_inc:
2016-10-10:
2016-11-11:
2017-10-10:

本文来自互联网用户投稿，该文观点仅代表作者本人，不代表本站立场。本站仅提供信息存储空间服务，不拥有所有权，不承担相关法律责任。如若转载，请注明出处：http://www.coloradmin.cn/o/2139199.html

如若内容造成侵权/违法违规/事实不符，请联系多彩编程网进行投诉反馈，一经查实，立即删除！

相关文章

Git+Jenkins 基本使用(Basic Usage of Git+Jenkins)

Git+Jenkins 基本使用(Basic Usage of Git+Jenkins)

💝💝💝欢迎来到我的博客，很高兴能够在这里和您见面！希望您在这里可以感受到一份轻松愉快的氛围，不仅可以获得有趣的内容和知识，也可以畅所欲言、分享您的想法和见解。 推荐:Linux运维老纪的首页…
阅读更多...
Linux命令分享 四 （ubuntu 16.04）（vi操作文件）

Linux命令分享 四 （ubuntu 16.04）（vi操作文件）

1、su 切换用户 su - 用户名 切换到该用户并将目录切换至该用户的主目录 **注意该语句执行后需要输入密码，输入密码时终端不回显（不会显示你输入的密码），输完直接回车即可 su 用户名 切换用户但不切换目录 su - root su root **注…
阅读更多...
你可能遗漏的一些C#/.NET/.NET Core知识点

你可能遗漏的一些C#/.NET/.NET Core知识点

前言 在这个快速发展的技术世界中，时常会有一些重要的知识点、信息或细节被忽略或遗漏。《C#/.NET/.NET Core拾遗补漏》专栏我们将探讨一些可能被忽略或遗漏的重要知识点、信息或细节，以帮助大家更全面地了解这些技术栈的特性和发展方向。 拾遗补漏Git…
阅读更多...
NISP 一级 | 5.3 电子邮件安全

NISP 一级 | 5.3 电子邮件安全

关注这个证书的其他相关笔记：NISP 一级 —— 考证笔记合集-CSDN博客 0x01：电子邮件安全威胁 电子邮件（Electronic mail，Email）是一种用电子手段提供信息交换的服务方式，是因特网上应用最为广泛的服务之一。…
阅读更多...
【原创 架构设计】多级缓存的应用、常见问题与解决方式

【原创 架构设计】多级缓存的应用、常见问题与解决方式

1. 简介 多级缓存是一种常见的性能优化手段，对于多级缓存一般意义上的理解具体主要实现其实指的就是本地缓存和分布式缓存。 本地缓存一般采用Caffeine或者Guava Cache来进行实现，而分布式缓存一般采用Redis来进行实现。 2. 业务流程 业务线程先在本…
阅读更多...
聚类_K均值

聚类_K均值

import numpy as np import matplotlib.pyplot as plt from sklearn.datasets import make_blobs1.数据预处理 #创建基于高斯分布的样本点, x是点的坐标，y是所属聚类值 x, y make_blobs(n_samples100, centers6, random_state100, cluster_std0.6) # 设置图形尺寸…
阅读更多...
整数在内存中的存储原码反码补码

整数在内存中的存储原码反码补码

目录 1.整数在内存中以二进制的形式存在 1.1（正数存储情况） 1.2 负数存储情况 1.3整数的补码如何得到原码 2.无符号整数的原反补码 小心！VS2022不可直接接触，否则！没这个必要，方源面色淡然一把抓住&am…
阅读更多...
Linux网络——守护进程、会话、进程组

Linux网络——守护进程、会话、进程组

文章目录 会话守护进程编写守护进程的注意事项编写样例 会话 会话是session，代表的是客户端与服务器的一次交互过程，我们可以简单理解为，当我们打开一个终端，在用户登录时，就是创建了一个会话 一般来说会话都与各自的…
阅读更多...
62. 不同路径、64. 最小路径和

62. 不同路径、64. 最小路径和

思路 dp：代表到达当前位置的总方式 初始化：第一行的位置dp[0][j]：当前位置只能由左边的位置向右移动得到 所以只有1种方式 d[0][j]1, d[0][0]1 第一列的位置 dp[i][0]：当前位置只能由上一个位置向下移动得到 除此之外的位置可以由…
阅读更多...
python画图|同时输出二维和三维图

python画图|同时输出二维和三维图

前面已经学习了如何输出二维图和三维图，部分文章详见下述链接： python画图|极坐标下的3D surface-CSDN博客 python画图|垂线标记系列_如何用pyplot画垂直x轴的线-CSDN博客 有时候也需要同时输出二位和三维图，因此有必要学习一下。 【1】…
阅读更多...
三明儿童自闭症寄宿制学校：关爱、教育、成长一站式服务

三明儿童自闭症寄宿制学校：关爱、教育、成长一站式服务

三明儿童自闭症寄宿制学校的启示：广州星贝育园——自闭症儿童的一站式成长乐园 在探讨自闭症儿童教育的道路上，寄宿制学校以其独特的优势，为孩子们提供了集关爱、教育、成长于一体的全方位服务。虽然文章开头提及了“三明儿童自闭症寄宿制学…
阅读更多...
AI预测体彩排3采取888=3策略+和值012路或胆码测试9月15日升级新模型预测第81弹

AI预测体彩排3采取888=3策略+和值012路或胆码测试9月15日升级新模型预测第81弹

经过80多期的测试，当然有很多彩友也一直在观察我每天发的预测结果，得到了一个非常有价值的信息，那就是9码定位的命中率非常高，已到达90%的命中率，这给喜欢打私菜的朋友提供了极高价值的预测结果~当然了，大部…
阅读更多...
请求响应-05.请求-日期参数JSON参数

请求响应-05.请求-日期参数JSON参数

一.日期参数 当浏览器发起的请求参数类型是日期参数时，我们通常使用LocalDateTime对象来接收，前面使用DateTimeFormat注解来完成日期的格式转换（日期时间格式有多种，需要哪种就设置为哪种：如yyyy-MM-dd HH:mm:ss&…
阅读更多...
C++比大小游戏

C++比大小游戏

目录 开头程序程序的流程图程序游玩的效果下一篇博客要说的东西 开头 大家好&#xff0c;我叫这是我58。 程序 #include <iostream> #include <Windows.h> using namespace std; int main() {int ir 1;char chparr[2] { 0 };int ip1 0;int ip2 0;int i 1;c…
阅读更多...
【ARM】Cache深度解读

【ARM】Cache深度解读

Cache的基本概念和使用场景 不同的Master硬件共享数据时—invalid cache 外设和DDR之间没有cache&#xff0c;所以外设直接把数据写入DDR中&#xff0c;但是cpu和DDR之间有cache&#xff0c;cpu会首先访问cache&#xff0c;如果命中直接从cache中拿数据&#xff0c;但是此时的…
阅读更多...
完整版订单超时自动取消功能

完整版订单超时自动取消功能

前几天对实习还是继续学习技术产生了抉择&#xff0c;问了一个前辈&#xff0c;他抛给我一个问题&#xff0c;怎么做15分钟订单自动取消&#xff0c;我说然后到时间之后&#xff0c;自动执行这个订单关闭业务&#xff0c;比如把锁了的库存给解开等等操作&#xff0c;然后在数据…
阅读更多...
后续学习规划 ----含我个人的学习路线，经历及感受

后续学习规划 ----含我个人的学习路线，经历及感受

目前的基础 开发相关&#xff08;最重要&#xff09; 1.Java SE 从入门到起飞 2.Java Web开发 3.苍穹外卖 以上三个是和开发相关的基础。 我是按照书写的顺序学习的&#xff0c;有需要的朋友可以参考。 计算机相关 其他的话&#xff0c;都是比较久远的了。隔得时间一年半…
阅读更多...
【大数据方案】智慧大数据平台总体建设方案书（word原件）

【大数据方案】智慧大数据平台总体建设方案书（word原件）

第1章 总体说明 1.1 建设背景 1.2 建设目标 1.3 项目建设主要内容 1.4 设计原则 第2章 对项目的理解 2.1 现状分析 2.2 业务需求分析 2.3 功能需求分析 第3章 大数据平台建设方案 3.1 大数据平台总体设计 3.2 大数据平台功能设计 3.3 平台应用 第4章 政策标准保障体系 4.1 政策…
阅读更多...
nginx: [emerg] unknown “connection_upgrade“ variable 解决与思考

nginx: [emerg] unknown “connection_upgrade“ variable 解决与思考

问题,如下图&#xff1a; 一天更新完主分支后启动nginx,结果报错&#xff1a;nginx: [emerg] unknown "connection_upgrade" variable 解决方法 网上查&#xff0c;发现是nginx配置文件出了问题&#xff0c;将下面map代码块补上即可。 http { map $http_upgrade …
阅读更多...
深入理解算法效率：时间复杂度与空间复杂度

深入理解算法效率：时间复杂度与空间复杂度

目录 引言 一、算法效率的基础 二、时间复杂度 1.概念 2.常见类型 1.O(1) — 常数阶 2.O(n) — 线性阶 3.O(n^2) — 平方阶 4.O(2^&#x1d45b;) — 指数阶 5.O(log &#x1d45b;) — 对数阶 3.总结 三、空间复杂度 1.概念 2.常见类型 1.O(1) — 常数阶 2.…
阅读更多...
推荐文章
最新文章

Copyright @ 2022~2023