前言
服务器说明
| 主机名称 | 操作系统 | 说明 |
|---|---|---|
| 192.168.11.82 | Ubuntu 22.04 | 主库所在服务器 |
| 192.168.11.28 | Oracle Linux Server 8.7 | 从库所在服务器 |
版本说明
<span style="color:#000000"><span style="background-color:#ffffff"><code class="language-plaintext language-txt">MySQL版本：MySQL_8.0.32
Canal版本：Canal_1.1.7
//我的canal安装部署在192.168.11.82上，当然你也可以部署在其它的服务器上
Java版本：1.8.0.362
</code></span></span>配置MySQL8.0数据库
修改MySQL配置文件
1、Ubuntu系统下MySQL配置文件位置
<span style="color:#000000"><span style="background-color:#ffffff"><code class="language-shell">cd /etc/mysql/mysql.conf.d
vi mysqld.cnf
</code></span></span>2、CentOS系统下MySQL配置文件位置
<span style="color:#000000"><span style="background-color:#ffffff"><code class="language-shell">vi /etc/my.cnf
</code></span></span>3、添加如下配置，开启MySQL binlog功能
<span style="color:#000000"><span style="background-color:#ffffff"><code class="language-powershell"><span style="color:#008000"># 服务编号, 与其它节点不冲突即可</span>
server_id=<span style="color:#880000">1</span>
log_bin=binlog
binlog_format=ROW
</code></span></span>Canal简介
关于canal简介，这里就不再阐述，具体可以参看官方文档介绍，地址如下：
https://github.com/alibaba/canal/wiki/简介
主库服务器操作
启动MySQL8.0数据库
<span style="color:#000000"><span style="background-color:#ffffff"><code class="language-powershell">systemctl <span style="color:#0000ff">start</span> mysql 或者
systemctl <span style="color:#0000ff">start</span> mysqld.service
Ubuntu 使用前者
</code></span></span>
创建账号密码
<span style="color:#000000"><span style="background-color:#ffffff"><code class="language-sql">mysql<span style="color:#ab5656">></span> <span style="color:#0000ff">CREATE</span> <span style="color:#0000ff">USER</span> canal IDENTIFIED <span style="color:#0000ff">BY</span> <span style="color:#a31515">'canal'</span>;
mysql<span style="color:#ab5656">></span> <span style="color:#0000ff">GRANT</span> <span style="color:#0000ff">SELECT</span>, <span style="color:#0000ff">SHOW</span> <span style="color:#0000ff">VIEW</span>, REPLICATION SLAVE, REPLICATION CLIENT <span style="color:#0000ff">ON</span> <span style="color:#ab5656">*</span>.<span style="color:#ab5656">*</span> <span style="color:#0000ff">TO</span> <span style="color:#a31515">'canal'</span>@<span style="color:#a31515">'%'</span>;
mysql<span style="color:#ab5656">></span> FLUSH PRIVILEGES;
</code></span></span>canal数据同步服务器操作
创建canal文件夹
<span style="color:#000000"><span style="background-color:#ffffff"><code class="language-shell">cd /usr/local
mkdir canal
cd canal
mkdir canal-package canal-adapter canal-deployer
</code></span></span>安装canal deployer和canal adapter
官网下载地址
<span style="color:#000000"><span style="background-color:#ffffff"><code class="language-plaintext language-txt">https://github.com/alibaba/canal/releases/tag/canal-1.1.7-alpha-2
</code></span></span>
只需要下载标红框的两个文件即可。复制到canal-package文件夹下。
解压
<span style="color:#000000"><span style="background-color:#ffffff"><code class="language-shell">tar -zxvf canal.adapter-1.1.7-SNAPSHOT.tar.gz -C /usr/local/canal/canal-adapter
tar -zxvf canal.deployer-1.1.7-SNAPSHOT.tar.gz -C /usr/local/canal/canal-deployer
</code></span></span>配置和启动canal-deployer
配置canal-deployer
由于此次同步为MySQL数据库间的数据同步，所以只需修改 instance.properties 即可。
<span style="color:#000000"><span style="background-color:#ffffff"><code class="language-shell">cd /usr/local/canal/canal-deployer/conf/example
vi instance.properties
</code></span></span>修改内容如下：
<span style="color:#000000"><span style="background-color:#ffffff"><code class="language-php"><span style="color:#008000"># 修改说明</span>
第一个框：主库所在服务器IP
第二个框：主库数据库账号密码
第三个框：规则如下：
instance.properties中同步数据表默认为同步数据库下所有的表信息，具体配置如图第三个框：
<span style="color:#008000"># 若需要同步某几张表，可以参考如下配置：</span>
<span style="color:#008000"># 同步某数据库test1下的user表,test2数据库下的所有表，所有库下所有表数据</span>
canal.instance.filter.regex=test1.user,test2\\..*,.*\\..*
</code></span></span>启动canal-deployer
<span style="color:#000000"><span style="background-color:#ffffff"><code class="language-shell">cd /usr/local/canal/canal-deployer/bin
./startup.sh
</code></span></span>查看日志确定是否启动成功
<span style="color:#000000"><span style="background-color:#ffffff"><code class="language-shell">cd /usr/local/canal/canal-deployer/logs/example
cat example.log
</code></span></span>我遇到的几个错误情况，仅供参考：
canal-deployer启动之后，如果在 logs 文件夹下没有 example 文件，参考如下情况说明：
1、查看 /usr/local/canal/canal-deployer/bin 文件夹下，是否存在.pid的文件。
2、查看logs文件夹下的canal文件夹下的canal_stdout.log文件，命令如下：
<span style="color:#000000"><span style="background-color:#ffffff"><code class="language-shell">cat /usr/local/canal/canal-deployer/logs/canal/canal_stdout.log
</code></span></span>若出现如下信息：
解决方案：（在此强烈建议系统中只安装jdk8或者jdk11，切不可采用jenv管理jdk，安装多个版本）
<span style="color:#000000"><span style="background-color:#ffffff"><code class="language-shell">cd /usr/local/canal/canal-deployer/bin
./stop.sh
vi startup.sh
<span style="color:#2b91af"># </span>删除报错信息包含的参数（该错误信息可能会出现多次，出现哪个删除哪个即可）
./startup.sh
</code></span></span>重新启动：
<span style="color:#000000"><span style="background-color:#ffffff"><code class="language-shell">cd /usr/local/canal/canal-deployer/bin
./startup.sh
</code></span></span>直到出现以下信息：
<span style="color:#000000"><span style="background-color:#ffffff"><code class="language-shell"><span style="color:#2b91af"># </span>打开日志文件
cat /usr/local/canal/canal-deployer/logs/example/example.log
<span style="color:#2b91af"># </span>出现以下信息说明canal-deployer启动成功
[destination = example , address = /192.168.11.82:3306 , EventParser] WARN c.a.o.c.p.inbound.mysql.rds.RdsBinlogEventParserProxy - ---> find start position successfully, EntryPosition[included=false,journalName=binlog.000040,position=65224673,serverId=1,gtid=,timestamp=1682062760000] cost : 1331ms , the next step is binlog dump
</code></span></span>配置canal-adapter
修改配置文件
<span style="color:#000000"><span style="background-color:#ffffff"><code class="language-shell">cd /usr/local/canal/canal-adapter/conf
vi application.yml
</code></span></span><span style="color:#000000"><span style="background-color:#ffffff"><code class="language-shell">server:
port: 8081
spring:
jackson:
date-format: yyyy-MM-dd HH:mm:ss
time-zone: GMT+8
default-property-inclusion: non_null
canal.conf:
mode: tcp #tcp kafka rocketMQ rabbitMQ
flatMessage: true
zookeeperHosts:
syncBatchSize: 1000
retries: -1
timeout:
accessKey:
secretKey:
consumerProperties:
# canal tcp consumer
# 修改位置1：Canal-deployer所在主机IP
canal.tcp.server.host: 127.0.0.1:11111
canal.tcp.zookeeper.hosts:
canal.tcp.batch.size: 500
canal.tcp.username:
canal.tcp.password:
# kafka consumer
kafka.bootstrap.servers: 127.0.0.1:9092
kafka.enable.auto.commit: false
kafka.auto.commit.interval.ms: 1000
kafka.auto.offset.reset: latest
kafka.request.timeout.ms: 40000
kafka.session.timeout.ms: 30000
kafka.isolation.level: read_committed
kafka.max.poll.records: 1000
# rocketMQ consumer
rocketmq.namespace:
rocketmq.namesrv.addr: 127.0.0.1:9876
rocketmq.batch.size: 1000
rocketmq.enable.message.trace: false
rocketmq.customized.trace.topic:
rocketmq.access.channel:
rocketmq.subscribe.filter:
# rabbitMQ consumer
rabbitmq.host:
rabbitmq.virtual.host:
rabbitmq.username:
rabbitmq.password:
rabbitmq.resource.ownerId:
<span style="color:#2b91af"># </span>修改位置：添加源库配置信息，此处为同步同库下所有表信息
srcDataSources:
defaultDS:
url: jdbc:mysql://192.168.11.82:3306/mynet?useUnicode=true&characterEncoding=utf8&autoReconnect=true&useSSL=false
username: ymliu
password: ymliu2023
canalAdapters:
- instance: example # canal instance Name or mq topic name
groups:
- groupId: g1
outerAdapters:
- name: logger
- name: rdb
key: mysql1
properties:
jdbc.driverClassName: com.mysql.jdbc.Driver
jdbc.url: jdbc:mysql://192.168.11.28:3306/mynet?useUnicode=true&characterEncoding=utf8&autoReconnect=true&useSSL=false
jdbc.username: ymliu
jdbc.password: ymliu2023
druid.stat.enable: false
druid.stat.slowSqlMillis: 1000
<span style="color:#2b91af"># </span> - name: rdb
<span style="color:#2b91af"># </span> key: oracle1
<span style="color:#2b91af"># </span> properties:
<span style="color:#2b91af"># </span> jdbc.driverClassName: oracle.jdbc.OracleDriver
<span style="color:#2b91af"># </span> jdbc.url: jdbc:oracle:thin:@localhost:49161:XE
<span style="color:#2b91af"># </span> jdbc.username: mytest
<span style="color:#2b91af"># </span> jdbc.password: m121212
<span style="color:#2b91af"># </span> - name: rdb
<span style="color:#2b91af"># </span> key: postgres1
<span style="color:#2b91af"># </span> properties:
<span style="color:#2b91af"># </span> jdbc.driverClassName: org.postgresql.Driver
<span style="color:#2b91af"># </span> jdbc.url: jdbc:postgresql://localhost:5432/postgres
<span style="color:#2b91af"># </span> jdbc.username: postgres
<span style="color:#2b91af"># </span> jdbc.password: 121212
<span style="color:#2b91af"># </span> threads: 1
<span style="color:#2b91af"># </span> commitSize: 3000
<span style="color:#2b91af"># </span> - name: hbase
<span style="color:#2b91af"># </span> properties:
<span style="color:#2b91af"># </span> hbase.zookeeper.quorum: 127.0.0.1
<span style="color:#2b91af"># </span> hbase.zookeeper.property.clientPort: 2181
<span style="color:#2b91af"># </span> zookeeper.znode.parent: /hbase
<span style="color:#2b91af"># </span> - name: es
<span style="color:#2b91af"># </span> hosts: 127.0.0.1:9300 <span style="color:#008000"># 127.0.0.1:9200 for rest mode</span>
<span style="color:#2b91af"># </span> properties:
<span style="color:#2b91af"># </span> mode: transport <span style="color:#008000"># or rest</span>
<span style="color:#2b91af"># </span> <span style="color:#008000"># security.auth: test:123456 # only used for rest mode</span>
<span style="color:#2b91af"># </span> cluster.name: elasticsearch
<span style="color:#2b91af"># </span> - name: kudu
<span style="color:#2b91af"># </span> key: kudu
<span style="color:#2b91af"># </span> properties:
<span style="color:#2b91af"># </span> kudu.master.address: 127.0.0.1 <span style="color:#008000"># ',' split multi address</span>
<span style="color:#2b91af"># </span> - name: phoenix
<span style="color:#2b91af"># </span> key: phoenix
<span style="color:#2b91af"># </span> properties:
<span style="color:#2b91af"># </span> jdbc.driverClassName: org.apache.phoenix.jdbc.PhoenixDriver
<span style="color:#2b91af"># </span> jdbc.url: jdbc:phoenix:127.0.0.1:2181:/hbase/db
<span style="color:#2b91af"># </span> jdbc.username:
<span style="color:#2b91af"># </span> jdbc.password:
</code></span></span>修改canal-adapter/conf/rdb文件夹下的yml文件
<span style="color:#000000"><span style="background-color:#ffffff"><code class="language-shell">cd /usr/local/canal/canal-adapter/conf/rdb
vi mytest_user.yml
</code></span></span>同步数据库下的某张表，例如同步mytest数据库下的user表，操作如下：
<span style="color:#000000"><span style="background-color:#ffffff"><code class="language-shell">dataSourceKey: defaultDS # 源数据源的key, 对应上面配置的srcDataSources中的值
destination: example # cannal的instance或者MQ的topic
groupId: g1 # 对应MQ模式下的groupId, 只会同步对应groupId的数据
outerAdapterKey: mysql1 # adapter key, 对应上面配置outAdapters中的key
concurrent: true # 是否按主键hash并行同步, 并行同步的表必须保证主键不会更改及主键不能为其他同步表的外键!
dbMapping:
database: test # 源数据源的database/schema
table: user # 源数据源表名
targetTable: test.user # 目标数据源的库名.表名
targetPk: # 主键映射
id: id # 如果是复合主键可以换行映射多个
mapAll: true # 是否整表映射, 要求源表和目标表字段名一模一样 (如果targetColumns也配置了映射, 则以targetColumns配置为准)
<span style="color:#2b91af"> #</span>targetColumns: <span style="color:#008000"># 字段映射, 格式: 目标表字段: 源表字段, 如果字段名一样源表字段名可不填</span>
<span style="color:#2b91af"> # </span> <span style="color:#0000ff">id</span>:
<span style="color:#2b91af"> # </span> name:
<span style="color:#2b91af"> # </span> role_id:
<span style="color:#2b91af"> # </span> c_time:
<span style="color:#2b91af"> # </span> test1:
</code></span></span>同步数据库下所有表数据，例如：同步mytest数据库下所有表数据，操作如下：
<span style="color:#000000"><span style="background-color:#ffffff"><code class="language-shell">dataSourceKey: defaultDS
destination: example
groupId: g1
outerAdapterKey: mysql1
concurrent: true
dbMapping:
mirrorDb: true
database: test # 该数据库名称修改为你的数据库名称
</code></span></span>启动canal-adapter
<span style="color:#000000"><span style="background-color:#ffffff"><code class="language-shell">cd /usr/local/canal/canal-adapter/bin
./startup.sh
</code></span></span>出现错误，排查方式同canal-deployer
查看日志信息
<span style="color:#000000"><span style="background-color:#ffffff"><code class="language-powershell"><span style="color:#0000ff">cd</span> /usr/local/canal/canal<span style="color:#a31515">-adapter</span>/logs/adapter
<span style="color:#0000ff">cat</span> adapter.log
</code></span></span>补充说明：
1、只有canal-adapter成功启动并正确连接canal-deployer后，canal-deployer才会读取binlog信息。/usr/local/canal/canal-deployer/conf/example 文件夹下才会出现meta.dat文件。
2、canal-adapter启动以后会出现127.0.0.1：3306/canal_manage连接失败信息，该信息是因为我们没有安装启动前端页面所致，可以忽略，不影响数据同步。
3、以上所有内容均为自己实操结果。
![[JavaWeb]【七】web后端开发-MYSQL](https://img-blog.csdnimg.cn/047b3f06ee414579b642eb9a2905dee0.png)
![[SWPUCTF 2022 新生赛]ez_ez_php](https://img-blog.csdnimg.cn/862b4ad38c66404b9903333c40c05296.png)
