Datax助力轻松迁移SQLServer数据至GreatSQL

news2024/10/2 8:29:18

1.环境说明

1.1源端SQLSserver

版本IP端口
Microsoft SQL Server 2017192.168.140.1601433

1.2目标端GreatSQL

版本IP端口
GreatSQL-8.0.32192.168.139.863308

2.安装环境

2.1安装SQLServer环境

环境说明：借助Docker使用镜像启动数据库

2.1.1安装docker

1.安装基础软件包

$ yum install -y wget net-tools nfs-utils lrzsz gcc gcc-c++ make cmake libxml2-devel openssl-devel curl curl-devel unzip sudo ntp libaio-devel wget vim ncurses-devel autoconf automake zlib-devel python-devel epel-release openssh-server socat ipvsadm conntrack yum-utils

2.配置 docker-ce 国内 yum 源（阿里云） 

$ yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo

3.安装 docker 依赖包 

$ yum install -y  device-mapper-persistent-data lvm2

4.安装 docker-ce 

$ yum install docker-ce -y

5.启动容器

$ systemctl start docker && systemctl enable docker
2.1.2 拉取镜像
$ docker pull mcr.microsoft.com/mssql/server:2017-latest
2.1.3运行容器
$ docker run -e "ACCEPT_EULA=Y" -e "SA_PASSWORD=********" \
-p 1433:1433 --name sqlserver2017 \
-d mcr.microsoft.com/mssql/server:2017-latest

此处记得密码设置为复杂密码

参数解释：

  • -e "ACCEPT_EULA=Y"：默认选择同意协议许可证

  • -e "SA_PASSWORD=****":设置连接密码，密码不能太短太简单，否则不满足sqlserver密码规范，容器会停止运行

  • -p 1433:1433: 宿主机端口映射到容器端口（前者为宿主机）

  • --name sqlserver2017：容器别名

  • -d: 后台运行

  • mcr.microsoft.com/mssql/server:2017-latest：镜像名称:标签

2.1.4使用数据库

1.进入容器

$ docker exec -it sqlserver2017 bash

2.连接数据库

$ /opt/mssql-tools/bin/sqlcmd -S localhost -U SA -P "********"

3.查询数据库

1> select name from sys.Databases;
2> go

4.创建数据库

1> create database testdb;
2> go

5.创建表并插入数据

use testdb
create table t1(id int)
go
Insert into t1 values(1),(2)
go

2.2安装GreatSQL环境

使用Docker镜像进行安装，直接拉取GreatSQL镜像即可

$ docker pull greatsql/greatsql

并创建GreatSQL容器

$ docker run -d --name greatsql --hostname=greatsql -e MYSQL_ALLOW_EMPTY_PASSWORD=1 greatsql/greatsql

2.3安装datax

DataX安装需要依赖的环境

  • JDK(1.8以上，推荐1.8)

  • Python(推荐Python2.6.X及以上)

安装步骤，解压即用，但是解压完成不做其他操作运行job会报错

$ cd /soft

$ ll
total 3764708
-rw-r--r--  1 root root  853734462 Dec  9 04:06 datax.tar.gz

$ tar xf datax.tar.gz

$ python /soft/datax/bin/datax.py /soft/datax/job/job.json 

DataX (DATAX-OPENSOURCE-3.0), From Alibaba !
Copyright (C) 2010-2017, Alibaba Group. All Rights Reserved.
2023-07-19 11:19:17.483 [main] WARN  ConfigParser - 插件[streamreader,streamwriter]加载失败，1s后重试... Exception:Code:[Common-00], Describe:[您提供的配置文件存在错误信息，请检查您的作业配置 .] - 配置信息错误，您提供的配置文件[/soft/datax/plugin/reader/._mysqlreader/plugin.json]不存在. 请检查您的配置文件. 
2023-07-19 11:19:18.488 [main] ERROR Engine - 
经DataX智能分析,该任务最可能的错误原因是:
com.alibaba.datax.common.exception.DataXException: Code:[Common-00], Describe:[您提供的配置文件存在错误信息，请检查您的作业配置 .] - 配置信息错误，您提供的配置文件[/soft/datax/plugin/reader/._mysqlreader/plugin.json]不存在. 请检查您的配置文件.
at com.alibaba.datax.common.exception.DataXException.asDataXException(DataXException.java:26)
at com.alibaba.datax.common.util.Configuration.from(Configuration.java:95)
at com.alibaba.datax.core.util.ConfigParser.parseOnePluginConfig(ConfigParser.java:153)
at com.alibaba.datax.core.util.ConfigParser.parsePluginConfig(ConfigParser.java:125)
at com.alibaba.datax.core.util.ConfigParser.parse(ConfigParser.java:63)
at com.alibaba.datax.core.Engine.entry(Engine.java:137)
at com.alibaba.datax.core.Engine.main(Engine.java:204)

解决报错:删除plugin目录下以及plugin/reader和plugin/writer目录下的所有._开头的文件

需要删除三个目录下的隐藏文件

  • plugin/
  • plugin/reader/
  • plugin/writer/
$ rm -rf /opt/app/datax/plugin/._*
$ rm -rf /opt/app/datax/plugin/reader/._*
$ rm -rf /opt/app/datax/plugin/writer/._*

运行一个测例，检测datax是否安装成功

$ python /soft/datax/bin/datax.py  /soft/datax/job/job.json 

DataX (DATAX-OPENSOURCE-3.0), From Alibaba !
Copyright (C) 2010-2017, Alibaba Group. All Rights Reserved.
 2023-07-19 11:22:12.298 [main] INFO  VMInfo - VMInfo# operatingSystem class => sun.management.OperatingSystemImpl
2023-07-19 11:22:12.305 [main] INFO  Engine - the machine info  => 
osInfo: Oracle Corporation 1.8 25.251-b08
jvmInfo: Linux amd64 4.19.25-200.1.el7.bclinux.x86_64
cpu num: 48
totalPhysicalMemory: -0.00G
freePhysicalMemory: -0.00G
maxFileDescriptorCount: -1
currentOpenFileDescriptorCount: -1
GC Names [PS MarkSweep, PS Scavenge]

MEMORY_NAME                    | allocation_size                | init_size                      
PS Eden Space                  | 256.00MB                       | 256.00MB                       
Code Cache                     | 240.00MB                       | 2.44MB                         
Compressed Class Space         | 1,024.00MB                     | 0.00MB                         
PS Survivor Space              | 42.50MB                        | 42.50MB                        
PS Old Gen                     | 683.00MB                       | 683.00MB                       
Metaspace                      | -0.00MB                        | 0.00MB  

2023-07-19 11:22:12.320 [main] INFO  Engine - 
{"content":[{"reader":{"name":"streamreader",
"parameter":{"column":[
{"type":"string","value":"DataX"},
{"type":"long","value":19890604},
{"type":"date","value":"1989-06-04 00:00:00"},
{"type":"bool","value":true},
{"type":"bytes","value":"test"}
],"sliceRecordCount":100000}
},"writer":{"name":"streamwriter","parameter":{"encoding":"UTF-8","print":false}}}],
"setting":{"errorLimit":{"percentage":0.02,"record":0},
"speed":{"byte":10485760}}}

2023-07-19 11:22:12.336 [main] WARN  Engine - prioriy set to 0, because NumberFormatException, the value is: null
2023-07-19 11:22:12.337 [main] INFO  PerfTrace - PerfTrace traceId=job_-1, isEnable=false, priority=0
2023-07-19 11:22:12.338 [main] INFO  JobContainer - DataX jobContainer starts job.
2023-07-19 11:22:12.339 [main] INFO  JobContainer - Set jobId = 0
2023-07-19 11:22:12.352 [job-0] INFO  JobContainer - jobContainer starts to do prepare ...
2023-07-19 11:22:12.352 [job-0] INFO  JobContainer - DataX Reader.Job [streamreader] do prepare work .
2023-07-19 11:22:12.352 [job-0] INFO  JobContainer - DataX Writer.Job [streamwriter] do prepare work .
2023-07-19 11:22:12.352 [job-0] INFO  JobContainer - jobContainer starts to do split ...
2023-07-19 11:22:12.353 [job-0] INFO  JobContainer - Job set Max-Byte-Speed to 10485760 bytes.
2023-07-19 11:22:12.354 [job-0] INFO  JobContainer - DataX Reader.Job [streamreader] splits to [1] tasks.
2023-07-19 11:22:12.354 [job-0] INFO  JobContainer - DataX Writer.Job [streamwriter] splits to [1] tasks.
2023-07-19 11:22:12.371 [job-0] INFO  JobContainer - jobContainer starts to do schedule ...
2023-07-19 11:22:12.375 [job-0] INFO  JobContainer - Scheduler starts [1] taskGroups.
2023-07-19 11:22:12.376 [job-0] INFO  JobContainer - Running by standalone Mode.
2023-07-19 11:22:12.384 [taskGroup-0] INFO  TaskGroupContainer - taskGroupId=[0] start [1] channels for [1] tasks.
2023-07-19 11:22:12.388 [taskGroup-0] INFO  Channel - Channel set byte_speed_limit to -1, No bps activated.
2023-07-19 11:22:12.388 [taskGroup-0] INFO  Channel - Channel set record_speed_limit to -1, No tps activated.
2023-07-19 11:22:12.396 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] taskId[0] attemptCount[1] is started
2023-07-19 11:22:12.697 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] taskId[0] is successed, used[302]ms
2023-07-19 11:22:12.698 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] completed it's tasks.
2023-07-19 11:22:22.402 [job-0] INFO  StandAloneJobContainerCommunicator - Total 100000 records, 2600000 bytes | Speed 253.91KB/s, 10000 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 0.020s |  All Task WaitReaderTime 0.033s | Percentage 100.00%
2023-07-19 11:22:22.402 [job-0] INFO  AbstractScheduler - Scheduler accomplished all tasks.
2023-07-19 11:22:22.402 [job-0] INFO  JobContainer - DataX Writer.Job [streamwriter] do post work.
2023-07-19 11:22:22.403 [job-0] INFO  JobContainer - DataX Reader.Job [streamreader] do post work.
2023-07-19 11:22:22.403 [job-0] INFO  JobContainer - DataX jobId [0] completed successfully.
2023-07-19 11:22:22.403 [job-0] INFO  HookInvoker - No hook invoked, because base dir not exists or is a file: /soft/datax/hook
2023-07-19 11:22:22.404 [job-0] INFO  JobContainer - 
 [total cpu info] => 
averageCpu                     | maxDeltaCpu                    | minDeltaCpu                    
-1.00%                         | -1.00%                         | -1.00%
 [total gc info] => 
 NAME                 | totalGCCount       | maxDeltaGCCount    | minDeltaGCCount    | totalGCTime        | maxDeltaGCTime     | minDeltaGCTime     
 PS MarkSweep         | 0                  | 0                  | 0                  | 0.000s             | 0.000s             | 0.000s             
 PS Scavenge          | 0                  | 0                  | 0                  | 0.000s             | 0.000s             | 0.000s             

2023-07-19 11:22:22.404 [job-0] INFO  JobContainer - PerfTrace not enable!
2023-07-19 11:22:22.404 [job-0] INFO  StandAloneJobContainerCommunicator - Total 100000 records, 2600000 bytes | Speed 253.91KB/s, 10000 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 0.020s |  All Task WaitReaderTime 0.033s | Percentage 100.00%
2023-07-19 11:22:22.406 [job-0] INFO  JobContainer - 
任务启动时刻                    : 2023-07-19 11:22:12
任务结束时刻                    : 2023-07-19 11:22:22
任务总计耗时                    :                 10s
任务平均流量                    :          253.91KB/s
记录写入速度                    :          10000rec/s
读出记录总数                    :              100000
读写失败总数                    :                   0

3.SQLServer2GreatSQL全量迁移

3.1 源端（SQLServer）造测试数据

$ docker exec -it 47bd0ed79c26 /bin/bash

$ /opt/mssql-tools/bin/sqlcmd -S localhost -U SA -P "********"

1> create database testdb
1> use testdb
1> insert into t1 values(1),(2),(3);
2> go
1> select * from t1;
2> go
id         
\-----------
​          1
​          2
​          3

3.2 目标端（GreatSQL）创建表结构

greatsql> create database testdb;
greatsql> use testdb;
greatsql> create table t1 (id int primary key);

3.3 编写Datax的job文件

$ cat /soft/datax/job/sqlserver_to_greatsql.json
{
​    "job": {
​        "content": [
​            {
​                "reader": {
​                    "name": "sqlserverreader",
​                    "parameter": {
​                        "connection": [
​                            {
​                                "jdbcUrl": ["jdbc:sqlserver://127.0.0.1:1433;DatabaseName=testdb"],
​                                "table": ["t1"]
​                            }
​                        ],
​                        "password": "********",
​                        "username": "SA",
​                        "column": ["*"]
​                    }
​                },
​                "writer": {
​                    "name": "mysqlwriter",
​                    "parameter": {
​                        "column": ["*"],
​                        "connection": [
​                            {
​                                "jdbcUrl": "jdbc:mysql://10.17.139.86:3308/testdb",
​                                "table": ["t1"]
​                            }
​                       ],
​                        "password": "******",
​                        "session": [],
​                        "username": "admin",
​                        "writeMode": "insert"
​                    }
​                }
​            }
​        ],
​        "setting": {
​            "speed": {
​                "channel": "5"
​            }
​        }
​    }
}

3.4 运行Datax迁移任务

$ python /soft/datax/bin/datax.py /soft/datax/job/sqlserver_to_greatsql.json 

DataX (DATAX-OPENSOURCE-3.0), From Alibaba !
Copyright (C) 2010-2017, Alibaba Group. All Rights Reserved.

2023-11-28 09:58:44.087 [main] INFO  VMInfo - VMInfo# operatingSystem class => sun.management.OperatingSystemImpl
2023-11-28 09:58:44.104 [main] INFO  Engine - the machine info  => 
osInfo: Oracle Corporation 1.8 25.181-b13
jvmInfo: Linux amd64 3.10.0-957.el7.x86_64

cpu num: 8
totalPhysicalMemory: -0.00G
freePhysicalMemory: -0.00G
maxFileDescriptorCount: -1
currentOpenFileDescriptorCount: -1
GC Names [PS MarkSweep, PS Scavenge]
MEMORY_NAME                    | allocation_size                | init_size                      
PS Eden Space                  | 256.00MB                       | 256.00MB                       
Code Cache                     | 240.00MB                       | 2.44MB                         
Compressed Class Space         | 1,024.00MB                     | 0.00MB                         
PS Survivor Space              | 42.50MB                        | 42.50MB                        
PS Old Gen                     | 683.00MB                       | 683.00MB                       
Metaspace                      | -0.00MB                        | 0.00MB                         

2023-11-28 09:58:44.137 [main] INFO  Engine - 
{
"content":[
{"reader":{
"name":"sqlserverreader",
"parameter":{
"column":["*"],
"connection":[
{"jdbcUrl":["jdbc:sqlserver://127.0.0.1:1433;DatabaseName=testdb"],
"table":["t1"]}],
"password":"*************",
"username":"SA"}},

"writer":{"name":"mysqlwriter","parameter":{"column":["*"],

"connection":[{"jdbcUrl":"jdbc:mysql://10.17.139.86:3308/testdb",
"table":["t1"]}],
"password":"********",
"session":[],
"username":"admin",
"writeMode":"insert"}}}],
"setting":{"speed":{"channel":"5"}}}

2023-11-28 09:58:44.176 [main] WARN  Engine - prioriy set to 0, because NumberFormatException, the value is: null
2023-11-28 09:58:44.179 [main] INFO  PerfTrace - PerfTrace traceId=job_-1, isEnable=false, priority=0
2023-11-28 09:58:44.180 [main] INFO  JobContainer - DataX jobContainer starts job.
2023-11-28 09:58:44.183 [main] INFO  JobContainer - Set jobId = 0
2023-11-28 09:58:44.542 [job-0] INFO  OriginalConfPretreatmentUtil - Available jdbcUrl:jdbc:sqlserver://127.0.0.1:1433;DatabaseName=testdb.
2023-11-28 09:58:44.544 [job-0] WARN  OriginalConfPretreatmentUtil - 您的配置文件中的列配置存在一定的风险. 因为您未配置读取数据库表的列，当您的表字段个数、类型有变动时，可能影响任务正确性甚至会运行出错。请检查您的配置并作出修改.
Loading class `com.mysql.jdbc.Driver'. This is deprecated. The new driver class is `com.mysql.cj.jdbc.Driver'. The driver is automatically registered via the SPI and manual loading of the driver class is generally unnecessary.
2023-11-28 09:58:45.099 [job-0] INFO  OriginalConfPretreatmentUtil - table:[t1] all columns:[id].
2023-11-28 09:58:45.099 [job-0] WARN  OriginalConfPretreatmentUtil - 您的配置文件中的列配置信息存在风险. 因为您配置的写入数据库表的列为*，当您的表字段个数、类型有变动时，可能影响任务正确性甚至会运行出错。请检查您的配置并作出修改.
2023-11-28 09:58:45.102 [job-0] INFO  OriginalConfPretreatmentUtil - Write data [
insert INTO %s (id) VALUES(?)
], which jdbcUrl like:[jdbc:mysql://10..17.139.86:16310/testdb?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true]
2023-11-28 09:58:45.103 [job-0] INFO  JobContainer - jobContainer starts to do prepare ...
2023-11-28 09:58:45.103 [job-0] INFO  JobContainer - DataX Reader.Job [sqlserverreader] do prepare work 
2023-11-28 09:58:45.104 [job-0] INFO  JobContainer - DataX Writer.Job [mysqlwriter] do prepare work .
2023-11-28 09:58:45.104 [job-0] INFO  JobContainer - jobContainer starts to do split ...
2023-11-28 09:58:45.105 [job-0] INFO  JobContainer - Job set Channel-Number to 5 channels.
2023-11-28 09:58:45.112 [job-0] INFO  JobContainer - DataX Reader.Job [sqlserverreader] splits to [1] tasks.
2023-11-28 09:58:45.114 [job-0] INFO  JobContainer - DataX Writer.Job [mysqlwriter] splits to [1] tasks.
2023-11-28 09:58:45.135 [job-0] INFO  JobContainer - jobContainer starts to do schedule ...
2023-11-28 09:58:45.139 [job-0] INFO  JobContainer - Scheduler starts [1] taskGroups.
2023-11-28 09:58:45.142 [job-0] INFO  JobContainer - Running by standalone Mode.
2023-11-28 09:58:45.151 [taskGroup-0] INFO  TaskGroupContainer - taskGroupId=[0] start [1] channels for [1] tasks.
2023-11-28 09:58:45.157 [taskGroup-0] INFO  Channel - Channel set byte_speed_limit to -1, No bps activated.
2023-11-28 09:58:45.158 [taskGroup-0] INFO  Channel - Channel set record_speed_limit to -1, No tps activated.
2023-11-28 09:58:45.173 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] taskId[0] attemptCount[1] is started
2023-11-28 09:58:45.181 [0-0-0-reader] INFO  CommonRdbmsReader$Task - Begin to read record by Sql: [select * from t1 
] jdbcUrl:[jdbc:sqlserver://127.0.0.1:1433;DatabaseName=testdb].
2023-11-28 09:58:45.398 [0-0-0-reader] INFO  CommonRdbmsReader$Task - Finished read record by Sql: [select * from t1 
] jdbcUrl:[jdbc:sqlserver://127.0.0.1:1433;DatabaseName=testdb].
2023-11-28 09:58:45.454 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] taskId[0] is successed, used[284]ms
2023-11-28 09:58:45.455 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] completed it's tasks.
2023-11-28 09:58:55.175 [job-0] INFO  StandAloneJobContainerCommunicator - Total 3 records, 3 bytes | Speed 0B/s, 0 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 0.000s |  All Task WaitReaderTime 0.000s | Percentage 100.00%
2023-11-28 09:58:55.175 [job-0] INFO  AbstractScheduler - Scheduler accomplished all tasks.
2023-11-28 09:58:55.175 [job-0] INFO  JobContainer - DataX Writer.Job [mysqlwriter] do post work.
2023-11-28 09:58:55.176 [job-0] INFO  JobContainer - DataX Reader.Job [sqlserverreader] do post work.
2023-11-28 09:58:55.176 [job-0] INFO  JobContainer - DataX jobId [0] completed successfully.
2023-11-28 09:58:55.176 [job-0] INFO  HookInvoker - No hook invoked, because base dir not exists or is a file: /soft/datax/hook
2023-11-28 09:58:55.177 [job-0] INFO  JobContainer - 
 [total cpu info] => 
averageCpu                     | maxDeltaCpu                    | minDeltaCpu                    
-1.00%                         | -1.00%                         | -1.00%
 [total gc info] => 
 NAME                 | totalGCCount       | maxDeltaGCCount    | minDeltaGCCount    | totalGCTime        | maxDeltaGCTime     | minDeltaGCTime     
 PS MarkSweep         | 1                  | 1                  | 1                  | 0.061s             | 0.061s             | 0.061s             
 PS Scavenge          | 1                  | 1                  | 1                  | 0.039s             | 0.039s             | 0.039s             
2023-11-28 09:58:55.177 [job-0] INFO  JobContainer - PerfTrace not enable!
2023-11-28 09:58:55.177 [job-0] INFO  StandAloneJobContainerCommunicator - Total 3 records, 3 bytes | Speed 0B/s, 0 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 0.000s |  All Task WaitReaderTime 0.000s | Percentage 100.00%
2023-11-28 09:58:55.179 [job-0] INFO  JobContainer - 
任务启动时刻                    : 2023-11-28 09:58:44
任务结束时刻                    : 2023-11-28 09:58:55
任务总计耗时                    :                 10s
任务平均流量                    :                0B/s
记录写入速度                    :              0rec/s
读出记录总数                    :                   3
读写失败总数                    :                   0

3.5 到目标端验证数据

greatsql> select * from t1;
+----+
| id |
+----+
|  1 |
|  2 |
|  3 |
+----+
3 rows in set (0.01 sec)

4. SQLServer to GreatSQL增量迁移

4.1 源端（SQLServer）创建测试数据

2> create table t2 (id int,createtime datetime);
3> go
1> insert into t2 values(1,GETDATE());
2> g
(1 rows affected)
1> insert into t2 values(2,GETDATE());
2> go
(1 rows affected)
1> insert into t2 values(3,GETDATE());
2> go
(1 rows affected)
1> insert into t2 values(4,GETDATE());
2> go
(1 rows affected)
1> insert into t2 values(5,GETDATE());
2> go
(1 rows affected)
1> insert into t2 values(6,GETDATE());
2> go
(1 rows affected)
1> select * from t2;
2> go
id          createtime             
---------- -----------------------
​          1 2023-11-28 02:18:20.790
​          2 2023-11-28 02:18:27.040
​          3 2023-11-28 02:18:32.103
​          4 2023-11-28 02:18:37.690
​          5 2023-11-28 02:18:41.450
​          6 2023-11-28 02:18:46.330

4.2 编写Datax的全量迁移job文件

$ cat sqlserver_to_greatsql_inc.json 
{
​    "job": {
​        "content": [
​            {
​                "reader": {
​                    "name": "sqlserverreader",
​                    "parameter": {
​                        "connection": [
​                            {
​                                "jdbcUrl": ["jdbc:sqlserver://127.0.0.1:1433;DatabaseName=testdb"],
​                                "table": ["t2"]
​                            }
​                        ],
​                        "password": "********",
​                        "username": "SA",
​                        "column": ["*"]
​                    }
​                },
​                "writer": {
​                    "name": "mysqlwriter",
​                    "parameter": {
​                        "column": ["*"],
​                        "connection": [
​                            {
​                                "jdbcUrl": "jdbc:mysql://10.17.139.86:3308/testdb",
​                                "table": ["t2"]
​                            }
​                        ],
​                        "password": "!QAZ2wsx",
​                        "session": [],
​                        "username": "admin",
​                        "writeMode": "insert"
​                    }
​                }
​            }
​        ],
​        "setting": {
​            "speed": {
​                "channel": "5"
​            }
​        }
​    }
}

4.3 运行Datax全量迁移任务

$ python /soft/datax/bin/datax.py /soft/datax/job/sqlserver_to_greatsql_inc.json 

 DataX (DATAX-OPENSOURCE-3.0), From Alibaba !
Copyright (C) 2010-2017, Alibaba Group. All Rights Reserved.
2023-11-28 10:19:59.279 [main] INFO  VMInfo - VMInfo# operatingSystem class => sun.management.OperatingSystemImpl
2023-11-28 10:19:59.286 [main] INFO  Engine - the machine info  => 
osInfo: Oracle Corporation 1.8 25.181-b13
jvmInfo: Linux amd64 3.10.0-957.el7.x86_64
cpu num: 8
totalPhysicalMemory: -0.00G
freePhysicalMemory: -0.00G
maxFileDescriptorCount: -1
currentOpenFileDescriptorCount: -1
GC Names [PS MarkSweep, PS Scavenge]

MEMORY_NAME                    | allocation_size                | init_size                      
PS Eden Space                  | 256.00MB                       | 256.00MB                       
Code Cache                     | 240.00MB                       | 2.44MB                         
Compressed Class Space         | 1,024.00MB                     | 0.00MB                         
PS Survivor Space              | 42.50MB                        | 42.50MB                        
PS Old Gen                     | 683.00MB                       | 683.00MB                       
Metaspace                      | -0.00MB                        | 0.00MB                         

2023-11-28 10:19:59.302 [main] INFO  Engine - 
{"content":[{"reader":{"name":"sqlserverreader","parameter":{"column":[
"*"],"connection":[{"jdbcUrl":["jdbc:sqlserver://127.0.0.1:1433;DatabaseName=testdb"],
"table":["t2"]}],"password":"*************","username":"SA"}},
"writer":{"name":"mysqlwriter","parameter":{"column":["*"],
"connection":[{"jdbcUrl":"jdbc:mysql://10..17.139.86:16310/testdb","table":["t2"]}],
"password":"********",
"session":[],
"username":"admin",
"writeMode":"insert"}}}],
"setting":{"speed":{"channel":"5"}}}

2023-11-28 10:19:59.319 [main] WARN  Engine - prioriy set to 0, because NumberFormatException, the value is: null
2023-11-28 10:19:59.321 [main] INFO  PerfTrace - PerfTrace traceId=job_-1, isEnable=false, priority=0
2023-11-28 10:19:59.321 [main] INFO  JobContainer - DataX jobContainer starts job.
2023-11-28 10:19:59.324 [main] INFO  JobContainer - Set jobId = 0
2023-11-28 10:19:59.629 [job-0] INFO  OriginalConfPretreatmentUtil - Available jdbcUrl:jdbc:sqlserver://127.0.0.1:1433;DatabaseName=testdb.
2023-11-28 10:19:59.630 [job-0] WARN  OriginalConfPretreatmentUtil - 您的配置文件中的列配置存在一定的风险. 因为您未配置读取数据库表的列，当您的表字段个数、类型有变动时，可能影响任务正确性甚至会运行出错。请检查您的配置并作出修改.
Loading class `com.mysql.jdbc.Driver'. This is deprecated. The new driver class is `com.mysql.cj.jdbc.Driver'. The driver is automatically registered via the SPI and manual loading of the driver class is generally unnecessary.
2023-11-28 10:20:00.027 [job-0] INFO  OriginalConfPretreatmentUtil - table:[t2] all columns:[
id,createtime].
2023-11-28 10:20:00.027 [job-0] WARN  OriginalConfPretreatmentUtil - 您的配置文件中的列配置信息存在风险. 因为您配置的写入数据库表的列为*，当您的表字段个数、类型有变动时，可能影响任务正确性甚至会运行出错。请检查您的配置并作出修改.
2023-11-28 10:20:00.029 [job-0] INFO  OriginalConfPretreatmentUtil - Write data [
insert INTO %s (id,createtime) VALUES(?,?)
], which jdbcUrl like:[jdbc:mysql://10..17.139.86:16310/testdb?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true]
2023-11-28 10:20:00.030 [job-0] INFO  JobContainer - jobContainer starts to do prepare ...
2023-11-28 10:20:00.031 [job-0] INFO  JobContainer - DataX Reader.Job [sqlserverreader] do prepare work .
2023-11-28 10:20:00.031 [job-0] INFO  JobContainer - DataX Writer.Job [mysqlwriter] do prepare work .
2023-11-28 10:20:00.032 [job-0] INFO  JobContainer - jobContainer starts to do split ...
2023-11-28 10:20:00.032 [job-0] INFO  JobContainer - Job set Channel-Number to 5 channels.
2023-11-28 10:20:00.037 [job-0] INFO  JobContainer - DataX Reader.Job [sqlserverreader] splits to [1] tasks.
2023-11-28 10:20:00.038 [job-0] INFO  JobContainer - DataX Writer.Job [mysqlwriter] splits to [1] tasks.
2023-11-28 10:20:00.060 [job-0] INFO  JobContainer - jobContainer starts to do schedule ...
2023-11-28 10:20:00.063 [job-0] INFO  JobContainer - Scheduler starts [1] taskGroups.
2023-11-28 10:20:00.066 [job-0] INFO  JobContainer - Running by standalone Mode.
2023-11-28 10:20:00.073 [taskGroup-0] INFO  TaskGroupContainer - taskGroupId=[0] start [1] channels for [1] tasks.
2023-11-28 10:20:00.080 [taskGroup-0] INFO  Channel - Channel set byte_speed_limit to -1, No bps activated.
2023-11-28 10:20:00.080 [taskGroup-0] INFO  Channel - Channel set record_speed_limit to -1, No tps activated.
2023-11-28 10:20:00.093 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] taskId[0] attemptCount[1] is started
2023-11-28 10:20:00.101 [0-0-0-reader] INFO  CommonRdbmsReader$Task - Begin to read record by Sql: [select * from t2 
] jdbcUrl:[jdbc:sqlserver://127.0.0.1:1433;DatabaseName=testdb].
2023-11-28 10:20:00.262 [0-0-0-reader] INFO  CommonRdbmsReader$Task - Finished read record by Sql: [select * from t2 
] jdbcUrl:[jdbc:sqlserver://127.0.0.1:1433;DatabaseName=testdb].
2023-11-28 10:20:00.334 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] taskId[0] is successed, used[243]ms
2023-11-28 10:20:00.335 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] completed it's tasks.
2023-11-28 10:20:10.087 [job-0] INFO  StandAloneJobContainerCommunicator - Total 6 records, 54 bytes | Speed 5B/s, 0 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 0.000s |  All Task WaitReaderTime 0.000s | Percentage 100.00%
2023-11-28 10:20:10.088 [job-0] INFO  AbstractScheduler - Scheduler accomplished all tasks.
2023-11-28 10:20:10.088 [job-0] INFO  JobContainer - DataX Writer.Job [mysqlwriter] do post work.
2023-11-28 10:20:10.089 [job-0] INFO  JobContainer - DataX Reader.Job [sqlserverreader] do post work.
2023-11-28 10:20:10.090 [job-0] INFO  JobContainer - DataX jobId [0] completed successfully.
2023-11-28 10:20:10.091 [job-0] INFO  HookInvoker - No hook invoked, because base dir not exists or is a file: /soft/datax/hook
2023-11-28 10:20:10.094 [job-0] INFO  JobContainer - 
 [total cpu info] => 
averageCpu                     | maxDeltaCpu                    | minDeltaCpu                    
-1.00%                         | -1.00%                         | -1.00%                      
 [total gc info] => 
 NAME                 | totalGCCount       | maxDeltaGCCount    | minDeltaGCCount    | totalGCTime        | maxDeltaGCTime     | minDeltaGCTime     
 PS MarkSweep         | 1                  | 1                  | 1                  | 0.034s             | 0.034s             | 0.034s             
 PS Scavenge          | 1                  | 1                  | 1                  | 0.031s             | 0.031s             | 0.031s             

2023-11-28 10:20:10.094 [job-0] INFO  JobContainer - PerfTrace not enable!
2023-11-28 10:20:10.095 [job-0] INFO  StandAloneJobContainerCommunicator - Total 6 records, 54 bytes | Speed 5B/s, 0 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 0.000s |  All Task WaitReaderTime 0.000s | Percentage 100.00%
2023-11-28 10:20:10.097 [job-0] INFO  JobContainer - 
任务启动时刻                    : 2023-11-28 10:19:59
任务结束时刻                    : 2023-11-28 10:20:10
任务总计耗时                    :                 10s
任务平均流量                    :                5B/s
记录写入速度                    :              0rec/s
读出记录总数                    :                   6
读写失败总数                    :                   0

4.4 验证全量迁移的数据

greatsql> select * from t2;
+----+---------------------+
| id | createtime          |
+----+---------------------+
|  1 | 2023-11-28 02:18:21 |
|  2 | 2023-11-28 02:18:27 |
|  3 | 2023-11-28 02:18:32 |
|  4 | 2023-11-28 02:18:38 |
|  5 | 2023-11-28 02:18:41 |
|  6 | 2023-11-28 02:18:46 |
+----+---------------------+

还可以用 checksum table x 来验证，大表就不能全表select * 了

4.5 源端（SQLServer）插入增量数据

2> insert into t2 values(7,'202311-28 03:18:46.330');
3> go
Changed database context to 'jem_db'.
(1 rows affected)
1> insert into t2 values(8,'2023-11-28 03:20:46.330');
2> go
(1 rows affected)
1> insert into t2 values(9,'2023-11-28 03:25:46.330');
2> go
(1 rows affected)
1> insert into t2 values(10,'2023-11-28 03:30:46.330');
2> go
(1 rows affected)
1> select * from t2;
2> go
id          createtime             
----------- -----------------------
​          1 2023-11-28 02:18:20.790
​          2 2023-11-28 02:18:27.040
​          3 2023-11-28 02:18:32.103
​          4 2023-11-28 02:18:37.690
​          5 2023-11-28 02:18:41.450
​          6 2023-11-28 02:18:46.330
​          7 2023-11-28 03:18:46.330
​          8 2023-11-28 03:20:46.330
​          9 2023-11-28 03:25:46.330
​         10 2023-11-28 03:30:46.330

4.6 编写Datax增量迁移job文件

$ cat sqlserver_to_greatsql_inc.json 
{
​    "job": {
​        "content": [
​            {
​                "reader": {
​                    "name": "sqlserverreader",
​                    "parameter": {
​                        "connection": [
​                            {
​                                "jdbcUrl": ["jdbc:sqlserver://127.0.0.1:1433;DatabaseName=testdb"],
​                                "table": ["t2"]
​                            }
​                        ],
​                        "password": "********",
​                        "username": "SA",
​                        "column": ["*"],
​                        "where":"createtime > '${start_time}' and createtime < '${end_time}'"
​                    }
​                },
​                "writer": {
​                    "name": "mysqlwriter",
​                    "parameter": {
​                        "column": ["*"],
​                        "connection": [
​                            {
​                                "jdbcUrl": "jdbc:mysql://10..17.139.86:16310/testdb",
​                                "table": ["t2"]
​                            }
​                        ],
​                        "password": "!QAZ2wsx",
​                        "session": [],
​                        "username": "admin",
​                        "writeMode": "insert"
​                    }
​                }
​            }
​        ],
​        "setting": {
​            "speed": {
​                "channel": "5"
​            }
​        }
​    }
}

4.7 运行Datax增量迁移任务

$ python /soft/datax/bin/datax.py /soft/datax/job/sqlserver_to_mysql_inc.json -p "-Dstart_time='2023-11-28 03:17:46.330' -Dend_time='2023-11-28 03:31:46.330'"

DataX (DATAX-OPENSOURCE-3.0), From Alibaba !
Copyright (C) 2010-2017, Alibaba Group. All Rights Reserved.

2023-11-28 10:29:24.492 [main] INFO  VMInfo - VMInfo# operatingSystem class => sun.management.OperatingSystemImpl

2023-11-28 10:29:24.504 [main] INFO  Engine - the machine info  => 

osInfo: Oracle Corporation 1.8 25.181-b13
jvmInfo: Linux amd64 3.10.0-957.el7.x86_64
cpu num: 8

totalPhysicalMemory: -0.00G
freePhysicalMemory: -0.00G
maxFileDescriptorCount: -1
currentOpenFileDescriptorCount: -1
GC Names [PS MarkSweep, PS Scavenge]
MEMORY_NAME                    | allocation_size                | init_size                      
PS Eden Space                  | 256.00MB                       | 256.00MB                       
Code Cache                     | 240.00MB                       | 2.44MB                         
Compressed Class Space         | 1,024.00MB                     | 0.00MB                         
PS Survivor Space              | 42.50MB                        | 42.50MB                        
PS Old Gen                     | 683.00MB                       | 683.00MB                       
Metaspace                      | -0.00MB                        | 0.00MB                         

2023-11-28 10:29:24.524 [main] INFO  Engine - 
{"content":[{"reader":{"name":"sqlserverreader","parameter":{"column":["*"],
"connection":[{"jdbcUrl":["jdbc:sqlserver://127.0.0.1:1433;DatabaseName=testdb"],
"table":["t2"]}],"password":"*************","username":"SA",
"where":"createtime > '2023-11-28 03:17:46.330' and createtime < '2023-11-28 03:31:46.330'"}},
"writer":{"name":"mysqlwriter","parameter":{"column":["*"],"connection":[{"jdbcUrl":"jdbc:mysql://10..17.139.86:16310/testdb","table":["t2"]}],
"password":"********",
"session":[],
"username":"admin",
"writeMode":"insert"}}}],
"setting":{"speed":{"channel":"5"}}}

2023-11-28 10:29:24.542 [main] WARN  Engine - prioriy set to 0, because NumberFormatException, the value is: null
2023-11-28 10:29:24.544 [main] INFO  PerfTrace - PerfTrace traceId=job_-1, isEnable=false, priority=0
2023-11-28 10:29:24.544 [main] INFO  JobContainer - DataX jobContainer starts job.
2023-11-28 10:29:24.546 [main] INFO  JobContainer - Set jobId = 0
2023-11-28 10:29:24.830 [job-0] INFO  OriginalConfPretreatmentUtil - Available jdbcUrl:jdbc:sqlserver://127.0.0.1:1433;DatabaseName=testdb.
2023-11-28 10:29:24.831 [job-0] WARN  OriginalConfPretreatmentUtil - 您的配置文件中的列配置存在一定的风险. 因为您未配置读取数据库表的列，当您的表字段个数、类型有变动时，可能影响任务正确性甚至会运行出错。请检查您的配置并作出修改.
Loading class `com.mysql.jdbc.Driver'. This is deprecated. The new driver class is `com.mysql.cj.jdbc.Driver'. The driver is automatically registered via the SPI and manual loading of the driver class is generally unnecessary.
2023-11-28 10:29:25.113 [job-0] INFO  OriginalConfPretreatmentUtil - table:[t2] all columns:[id,createtime].
2023-11-28 10:29:25.113 [job-0] WARN  OriginalConfPretreatmentUtil - 您的配置文件中的列配置信息存在风险. 因为您配置的写入数据库表的列为*，当您的表字段个数、类型有变动时，可能影响任务正确性甚至会运行出错。请检查您的配置并作出修改.
2023-11-28 10:29:25.115 [job-0] INFO  OriginalConfPretreatmentUtil - Write data [
insert INTO %s (id,createtime) VALUES(?,?)
], which jdbcUrl like:[jdbc:mysql://10..17.139.86:16310/testdb?yearIsDateType=false&zeroDateTimeBehavior=convertToNull&tinyInt1isBit=false&rewriteBatchedStatements=true]
2023-11-28 10:29:25.116 [job-0] INFO  JobContainer - jobContainer starts to do prepare ...
2023-11-28 10:29:25.117 [job-0] INFO  JobContainer - DataX Reader.Job [sqlserverreader] do prepare work .
2023-11-28 10:29:25.117 [job-0] INFO  JobContainer - DataX Writer.Job [mysqlwriter] do prepare work .
2023-11-28 10:29:25.118 [job-0] INFO  JobContainer - jobContainer starts to do split ...
2023-11-28 10:29:25.118 [job-0] INFO  JobContainer - Job set Channel-Number to 5 channels.
2023-11-28 10:29:25.123 [job-0] INFO  JobContainer - DataX Reader.Job [sqlserverreader] splits to [1] tasks.
2023-11-28 10:29:25.124 [job-0] INFO  JobContainer - DataX Writer.Job [mysqlwriter] splits to [1] tasks.
2023-11-28 10:29:25.146 [job-0] INFO  JobContainer - jobContainer starts to do schedule ...
2023-11-28 10:29:25.150 [job-0] INFO  JobContainer - Scheduler starts [1] taskGroups.
2023-11-28 10:29:25.153 [job-0] INFO  JobContainer - Running by standalone Mode.
2023-11-28 10:29:25.159 [taskGroup-0] INFO  TaskGroupContainer - taskGroupId=[0] start [1] channels for [1] tasks.
2023-11-28 10:29:25.165 [taskGroup-0] INFO  Channel - Channel set byte_speed_limit to -1, No bps activated.
2023-11-28 10:29:25.165 [taskGroup-0] INFO  Channel - Channel set record_speed_limit to -1, No tps activated.
2023-11-28 10:29:25.176 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] taskId[0] attemptCount[1] is started
2023-11-28 10:29:25.183 [0-0-0-reader] INFO  CommonRdbmsReader$Task - Begin to read record by Sql: [select * from t2 where (createtime > '2023-11-28 03:17:46.330' and createtime < '2023-11-28 03:31:46.330')
] jdbcUrl:[jdbc:sqlserver://127.0.0.1:1433;DatabaseName=testdb].
2023-11-28 10:29:25.344 [0-0-0-reader] INFO  CommonRdbmsReader$Task - Finished read record by Sql: [select * from t2 where (createtime > '2023-11-28 03:17:46.330' and createtime < '2023-11-28 03:31:46.330')
] jdbcUrl:[jdbc:sqlserver://127.0.0.1:1433;DatabaseName=testdb].
2023-11-28 10:29:25.606 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] taskId[0] is successed, used[431]ms
2023-11-28 10:29:25.607 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] completed it's tasks.
2023-11-28 10:29:35.173 [job-0] INFO  StandAloneJobContainerCommunicator - Total 4 records, 37 bytes | Speed 3B/s, 0 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 0.000s |  All Task WaitReaderTime 0.000s | Percentage 100.00%
2023-11-28 10:29:35.173 [job-0] INFO  AbstractScheduler - Scheduler accomplished all tasks.
2023-11-28 10:29:35.174 [job-0] INFO  JobContainer - DataX Writer.Job [mysqlwriter] do post work.
2023-11-28 10:29:35.175 [job-0] INFO  JobContainer - DataX Reader.Job [sqlserverreader] do post work.
2023-11-28 10:29:35.175 [job-0] INFO  JobContainer - DataX jobId [0] completed successfully.
2023-11-28 10:29:35.177 [job-0] INFO  HookInvoker - No hook invoked, because base dir not exists or is a file: /soft/datax/hook
2023-11-28 10:29:35.179 [job-0] INFO  JobContainer - 

 [total cpu info] => 
averageCpu                     | maxDeltaCpu                    | minDeltaCpu                    
-1.00%                         | -1.00%                         | -1.00%

 [total gc info] => 
 NAME                 | totalGCCount       | maxDeltaGCCount    | minDeltaGCCount    | totalGCTime        | maxDeltaGCTime     | minDeltaGCTime     
 PS MarkSweep         | 1                  | 1                  | 1                  | 0.052s             | 0.052s             | 0.052s             
 PS Scavenge          | 1                  | 1                  | 1                  | 0.024s             | 0.024s             | 0.024s             
2023-11-28 10:29:35.180 [job-0] INFO  JobContainer - PerfTrace not enable!
2023-11-28 10:29:35.181 [job-0] INFO  StandAloneJobContainerCommunicator - Total 4 records, 37 bytes | Speed 3B/s, 0 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 0.000s |  All Task WaitReaderTime 0.000s | Percentage 100.00%

2023-11-28 10:29:35.183 [job-0] INFO  JobContainer - 
任务启动时刻                    : 2023-11-28 10:29:24
任务结束时刻                    : 2023-11-28 10:29:35
任务总计耗时                    :                 10s
任务平均流量                    :                3B/s
记录写入速度                    :              0rec/s
读出记录总数                    :                   4
读写失败总数                    :                   0

4.8 到目标端（GreatSQL）验证增量数据

greatsql> select * from t2;

+----+---------------------+
| id | createtime          |
+----+---------------------+
|  1 | 2023-11-28 02:18:21 |
|  2 | 2023-11-28 02:18:27 |
|  3 | 2023-11-28 02:18:32 |
|  4 | 2023-11-28 02:18:38 |
|  5 | 2023-11-28 02:18:41 |
|  6 | 2023-11-28 02:18:46 |
|  7 | 2023-11-28 03:18:46 |
|  8 | 2023-11-28 03:20:46 |
|  9 | 2023-11-28 03:25:46 |
| 10 | 2023-11-28 03:30:46 |
+----+---------------------+
10 rows in set (0.00 sec)

增量迁移总结：通过增加过滤条件达到增量迁移的目的。主要是通过过滤条件过滤掉了全量迁移的数据，进而变相的完成了增量迁移。

Enjoy GreatSQL :)

关于 GreatSQL

GreatSQL是适用于金融级应用的国内自主开源数据库，具备高性能、高可靠、高易用性、高安全等多个核心特性，可以作为MySQL或Percona Server的可选替换，用于线上生产环境，且完全免费并兼容MySQL或Percona Server。

相关链接： GreatSQL社区 Gitee GitHub Bilibili

GreatSQL社区：

image

社区有奖建议反馈： https://greatsql.cn/thread-54-1-1.html

社区博客有奖征稿详情： https://greatsql.cn/thread-100-1-1.html

（对文章有疑问或者有独到见解都可以去社区官网提出或分享哦~）

技术交流群：

微信&QQ群：

QQ群：533341697

微信群：添加GreatSQL社区助手（微信号：wanlidbc ）好友，待社区助手拉您进群。

本文来自互联网用户投稿，该文观点仅代表作者本人，不代表本站立场。本站仅提供信息存储空间服务，不拥有所有权，不承担相关法律责任。如若转载，请注明出处：http://www.coloradmin.cn/o/1613657.html

如若内容造成侵权/违法违规/事实不符，请联系多彩编程网进行投诉反馈，一经查实，立即删除！

相关文章

【prometheus】k8s集群部署AlertManager实现邮件和钉钉告警

【prometheus】k8s集群部署AlertManager实现邮件和钉钉告警

目录 一、AlertManager概述 1.1 alertmanager简介 1.2 AlertManager核心概念 1.2.1 分组 1.2.2 抑制 1.2.3 静默 1.2.4 客户的行为 1.2.5 高可用性 二、Alertmanager部署邮箱告警 2.1 邮箱配置 2.2 Alertmanager global和route路由配置 2.3 部署prometheus和alertM…
阅读更多...
如何熟悉一个陌生的业务系统

如何熟悉一个陌生的业务系统

目录 一、业务层面 1.1 业务背景 1.2 业务系统概括 1.3 使用情况 1.4 业务流程演练 二、技术层面 2.1 技术架构 2.2 存储层 2.3 系统交互 2.4 运维部署 2.5 系统问题 三、总结 不知道你有没有这样的经历&#xff0c;以前维护某个系统的同事因为某些原因不在维护这个系统了&a…
阅读更多...
真实世界的密码学（三）

真实世界的密码学（三）

原文&#xff1a;annas-archive.org/md5/655c944001312f47533514408a1a919a 译者&#xff1a;飞龙 协议&#xff1a;CC BY-NC-SA 4.0 第十一章&#xff1a;用户认证 本章涵盖了 认证人员和数据之间的区别 用户认证&#xff0c;根据密码或密钥对用户进行身份验证。 用户辅助认…
阅读更多...
4.21java聊天室项目小结

4.21java聊天室项目小结

基本完成了用户的登录注册功能&#xff0c;可以实现用户账号登录和邮箱登录功能&#xff0c;忘记密码通过邮箱发送验证码找回&#xff0c;注册账号功能&#xff0c;并传递给客户端更新数据库的表内容 注册功能&#xff1a; 注册成功后密码进行MD5加密并通过服务器保存到数据库…
阅读更多...
Python 密码学实用指南（全）

Python 密码学实用指南（全）

原文&#xff1a;zh.annas-archive.org/md5/fe5e9f4d664790ea92fb33d78ca9108d 译者&#xff1a;飞龙 协议&#xff1a;CC BY-NC-SA 4.0 前言 密码学在保护关键系统和敏感信息方面有着悠久而重要的历史。本书将向您展示如何使用 Python 加密、评估、比较和攻击数据。总的来说&…
阅读更多...
Java学习Go（入门）

Java学习Go（入门）

下载Go 《官网下载golang》 直接点Download&#xff0c;然后根据你自己的操作系统进行下载&#xff0c;我这里以win10为例 安装go 默认安装到C:\Program Files\Go&#xff0c;这里我们可以选择安装到其他盘&#xff0c;也可以选择默认安装。初学者建议直接一路next。 安装完…
阅读更多...
07节-51单片机-矩阵键盘

07节-51单片机-矩阵键盘

文章目录 1矩阵键盘原理2.扫描的概念3.弱上拉4.实战-实现矩阵键盘对应按钮按下显示对应值4.1配置代码模板 5.键盘锁 1矩阵键盘原理 在键盘中按键数量较多时&#xff0c;为了减少I/O口的占用&#xff0c;通常将按键排列成矩阵形式 采用逐行或逐列的“扫描”&#xff0c;就可以读…
阅读更多...
2024团体设计天梯赛（部分题解）

2024团体设计天梯赛（部分题解）

目录 个人总结: L1-1 编程解决一切&#xff08;5分&#xff09; AC代码: L1-2 再进去几个人&#xff08;5分&#xff09; AC代码: L1- 3 帮助色盲&#xff08;10分&#xff09; AC代码: L1- 4 四项全能&#xff08;10分&#xff09; 代码:(9分) L1-4 别再来这么多猫娘了…
阅读更多...
【大语言模型LLM】-大语言模型乐园，高效办公不迷路！

【大语言模型LLM】-大语言模型乐园，高效办公不迷路！

&#x1f525;博客主页&#xff1a;西瓜WiFi &#x1f3a5;系列专栏&#xff1a;《大语言模型》 ❤️感谢大家点赞&#x1f44d; 收藏⭐ 评论⭐ &#x1f3a5;大语言模型LLM基础-系列文章&#xff1a; 【大语言模型LLM】-大语言模型如何编写Prompt? 【大语言模型LLM】-如何…
阅读更多...
Pytorch第一部分数据模块

Pytorch第一部分数据模块

数据划分&#xff1a; 从数据集中将数据划分为训练集&#xff0c;测试集&#xff0c;验证集 # -*- coding: utf-8 -*- """ # file name : 1_split_dataset.py # author : tingsongyu # date : 2019-09-07 10:08:00 # brief : 将数据集划分为训…
阅读更多...
20.Unity飞机大战游戏

20.Unity飞机大战游戏

1任务&#xff1a;使背景图动起来 2任务&#xff1a;飞机换帧动画 3任务&#xff1a;让飞机发射子弹 4任务&#xff1a;敌机出现 5任务&#xff1a;控制飞机 6任务&#xff1a;游戏碰撞逻辑 7任务&#xff1a;另外两种类型的敌机 8任务&#xff1a;拾取奖励物品换枪 9…
阅读更多...
230基于matlab的布谷鸟（COA）多目标优化算法

230基于matlab的布谷鸟（COA）多目标优化算法

基于matlab的布谷鸟&#xff08;COA&#xff09;多目标优化算法&#xff0c;以 满意度、成本、时间、质量为目标的多目标优化求解代码。程序已调通&#xff0c;可直接运行。 230 matlab 布谷鸟&#xff08;COA&#xff09;多目标优化 - 小红书 (xiaohongshu.com)
阅读更多...
操作符不存在：sde.st_geometry ^ ！sde.st_geometry建议 SQL函 数st_intersects在内联inlining期间

操作符不存在：sde.st_geometry ^ ！sde.st_geometry建议 SQL函 数st_intersects在内联inlining期间

操作符不存在&#xff1a;sde.st_geometry ^ &#xff01;sde.st_geometry建议 SQL函 数st_intersects在内联inlining期间 问题&#xff1a;最近在使用SQL图形处理函数处理图形时&#xff0c;莫名奇妙报如下错误&#xff0c;甚是费解 于是开始四处"寻医问药" 1、nav…
阅读更多...
叶子相似的树

叶子相似的树

题目链接 叶子相似的树 题目描述 注意点 给定的两棵树结点数在 [1, 200] 范围内给定的两棵树上的值在 [0, 200] 范围内 解答思路 深度优先遍历按顺序找到两棵树各自的叶子节点并存储到两个list中&#xff0c;随后比较两个list是否相同即可 代码 /*** Definition for a b…
阅读更多...
mysql基础20——数据备份

mysql基础20——数据备份

数据备份 数据备份有2种 一种是物理备份 一种是逻辑备份 物理备份 物理备份 通过把数据文件复制出来 达到备份的目的 用得比较少 逻辑备份 逻辑备份 把描述数据库结构和内容的信息保存起来 达到备份的目的 是免费的 数据备份工具 mysqldump &#xff08;3种模式&#x…
阅读更多...
力扣283. 移动零

力扣283. 移动零

Problem: 283. 移动零 文章目录 题目描述思路复杂度Code 题目描述 思路 1.定义一个int类型变量index初始化为0&#xff1b; 2.遍历nums当当前的元素nums[i]不为0时使nums[i]赋值给nums[index]&#xff1b; 3.从index开始将nums中置对应位置的元素设为0&#xff1b; 复杂度 时间…
阅读更多...
【数据结构-树和二叉树-森林-哈夫曼树】

【数据结构-树和二叉树-森林-哈夫曼树】

目录 1 树1.1 树的描述&#xff08;基本术语&#xff09; 2 二叉树&#xff08;树的度最大为2&#xff09;2.1 注意事项-五种基本形态2.2 二叉树的抽象数据类型定义 3 二叉树的性质3.1 两种特殊形式的二叉树-重点会计算3.2 题目练习&#xff1a; 4 二叉树的存储结构4.1 顺序存储…
阅读更多...
卷积神经网络（CNN）基础

卷积神经网络（CNN）基础

目录 卷积神经网络介绍 卷积神经网络原理 卷积层&#xff1a;通过在原始图片上平移来提取特征 激活层&#xff1a;增加非线性分割能力 池化层polling&#xff08;下采样层&#xff09;&#xff1a;减少学习参数&#xff0c;去掉不重要的样本&#xff0c;降低网络的复杂度 卷…
阅读更多...
面试（06）————MySQL篇

面试（06）————MySQL篇

目录 问题一&#xff1a;在MySQL中&#xff0c;如何定位慢查询&#xff1f; 方案一&#xff1a;开源工具 方案二&#xff1a;MySQL自带慢日志 模拟面试 问题二&#xff1a;这个SQL语句执行很慢&#xff0c;如何分析的呐&#xff1f; 模拟面试 问题三&#xff1a;了解过索引…
阅读更多...
【GlobalMapper精品教程】076：基于高程和影像数据创建电子沙盘（真实三维地形）

【GlobalMapper精品教程】076：基于高程和影像数据创建电子沙盘（真实三维地形）

影像与数字高程模型叠加，可以构建三维真是地形。本文讲解在Globalmapper中基于高程和影像数据创建电子沙盘（真实三维地形）。 文章目录 一、加载数据二、创建三维网格三、三维叠加显示一、加载数据 本实验的数据（配套实验数据资料包中的data076.rar，订阅专栏，获取全文及数…
阅读更多...
推荐文章
最新文章

Copyright @ 2022~2023