运行canal-adapter
- 1. 下载canal源码
- 1.1 下载源码并安装好环境
- 1.2 查看目录结构
- 2.项目运行
- 2.1 项目打包
- 2.2 项目打包可能遇到的问题:
- 1.Failure to find com.alibaba.otter:connector.tcp:jar:jar-with-dependencies:1.1.5
- 2.com.alibaba.druid.pool.DruidDataSource cannot be cast to com.alibaba.druid.pool.DruidDataSource
- 3.配置application.yml
- 4.运行项目
- 4.1 运行启动类:CanalAdapterApplication.java
- 4.2 Config dir not found 报错解决
- 5.可直接运行的项目源码
省流:最后面有能直接跑的源码,可以直接下载运行
1. 下载canal源码
1.1 下载源码并安装好环境
官网地址:https://github.com/alibaba/canal/releases/tag/canal-1.1.5
服务端的部署参考我之前写过的文档: canal实时同步mysql数据到elasticsearch,注意只需要去部署一个服务端
保证以下组件已经安装运行
组件 | 版本 |
---|---|
mysql | 8.0.23 |
elasticsearch | 7.9.0 |
1.2 查看目录结构
下载解压后的目录结构如下,我们当前只关注客户端的代码
2.项目运行
使用idea打开项目,由于需求是往es同步数据,因此需要关注以下三个模块
2.1 项目打包
打包顺序如下,务必不要打乱顺序,模块之间存在依赖
- escore->mvn clean install
- es7x->mvn clean install
- launcher->mvn clean install
2.2 项目打包可能遇到的问题:
1.Failure to find com.alibaba.otter:connector.tcp:jar:jar-with-dependencies:1.1.5
[ERROR] Failed to execute goal on project client-adapter.launcher: Could not resolve dependencies for project com.alibaba.otter:client-adapter.launcher:jar:1.1.5: Failure to find com.alibaba.otter:connector.tcp:jar:jar-with-dependencies:1.1.5 in https://maven.aliyun.com/repository/public was cached in the local repository, resolution will not be reattempted until the update interval of aliyunmaven has elapsed or updates are forced -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
原因:connector.tcp maven下载不到导致的,tmd吊jar包找了半天才找到,还要劳资扫码关注才能下载
链接:https://pan.baidu.com/s/1Biw-Gwum-vPaqfWhqc_gGQ?pwd=of2e
提取码:of2e
解决方案:下载jar后放到报错信息中maven仓库的位置
2.com.alibaba.druid.pool.DruidDataSource cannot be cast to com.alibaba.druid.pool.DruidDataSource
ERROR c.a.o.canal.adapter.launcher.loader.CanalAdapterLoader - Load canal adapter: es7 failed
java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassCastException: com.alibaba.druid.pool.DruidDataSource cannot be cast to com.alibaba.druid.pool.DruidDataSource
at com.alibaba.otter.canal.client.adapter.es7x.ES7xAdapter.init(ES7xAdapter.java:54) ~[client-adapter.es7x-1.1.5-jar-with-dependencies.jar:na]
at co
Caused by: java.lang.RuntimeException: java.lang.ClassCastException: com.alibaba.druid.pool.DruidDataSource cannot be cast to com.alibaba.druid.pool.DruidDataSource
at com.alibaba.otter.canal.client.adapter.es.core.ESAdapter.init(ESAdapter.java:83) ~[client-adapter.es7x-1.1.5-jar-with-dependencies.jar:na]
at com.alibaba.otter.canal.client.adapter.es7x.ES7xAdapter.init(ES7xAdapter.java:52) ~[client-adapter.es7x-1.1.5-jar-with-dependencies.jar:na]
... 42 common frames omitted
Caused by: java.lang.ClassCastException: com.alibaba.druid.pool.DruidDataSource cannot be cast to com.alibaba.druid.pool.DruidDataSource
at com.alibaba.otter.canal.client.adapter.es.core.ESAdapter.addSyncConfigToCache(ESAdapter.java:146) ~[client-adapter.es7x-1.1.5-jar-with-dependencies.jar:na]
at com.alibaba.otter.canal.client.adapter.es.core.ESAdapter.init(ESAdapter.java:75) ~[client-adapter.es7x-1.1.5-jar-with-dependencies.jar:na]
... 43 common frames omitted
2023-05-15 21:29:59.969 [main] INFO c.alibaba.otter.canal.connector.core.spi.ExtensionLoader - extension classpath dir: D:\IdeaProjects\canal\canal-canal-1.1.5\client-adapter\launcher\target\canal-adapter\plugin
2023-05-15 21:29:59.985 [main] INFO c.a.o.canal.adapter.launcher.loader.CanalAdapterLoader - Start adapter for canal-client mq topic: cqu-g1 succeed
2023-05-15 21:29:59.985 [main] INFO c.a.o.canal.adapter.launcher.loader.CanalAdapterService - ## the canal client adapters are running now ......
2023-05-15 21:29:59.985 [Thread-9] INFO c.a.otter.canal.adapter.launcher.loader.AdapterProcessor - =============> Start to connect destination: cqu <=============
2023-05-15 21:29:59.989 [main] INFO org.apache.coyote.http11.Http11NioProtocol - Starting ProtocolHandler ["http-nio-8081"]
2023-05-15 21:29:59.989 [main] INFO org.apache.tomcat.util.net.NioSelectorPool - Using a shared selector for servlet write/read
2023-05-15 21:30:00.023 [main] INFO o.s.boot.web.embedded.tomcat.TomcatWebServer - Tomcat started on port(s): 8081 (http) with context path ''
2023-05-15 21:30:00.026 [main] INFO c.a.otter.canal.adapter.launcher.CanalAdapterApplication - Started CanalAdapterApplication in 3.208 seconds (JVM running for 4.171)
2023-05-15 21:30:00.100 [Thread-9] INFO c.a.otter.canal.adapter.launcher.loader.AdapterProcessor - =============> Subscribe destination: cqu succeed <=============
原因:common和escore的druid包冲突
解决方案:
限制escore的pom文件中的druid的作用域,定位到 client-adapter.escore 模块的 pom.xml
上图红框出修改为
<dependency>
<groupId>com.alibaba</groupId>
<artifactId>druid</artifactId>
<scope>provided</scope>
</dependency>
3.配置application.yml
server:
port: 8081
spring:
jackson:
date-format: yyyy-MM-dd HH:mm:ss
time-zone: GMT+8
default-property-inclusion: non_null
canal.conf:
mode: tcp #tcp kafka rocketMQ rabbitMQ
flatMessage: true
zookeeperHosts:
syncBatchSize: 10000
retries: 0
timeout:
accessKey:
secretKey:
consumerProperties:
# canal tcp consumer
canal.tcp.server.host: 192.168.0.10:11111 #canal服务端地址
canal.tcp.zookeeper.hosts:
canal.tcp.batch.size: 5000
canal.tcp.username:
canal.tcp.password:
srcDataSources:
defaultDS:
url: jdbc:mysql://192.168.0.10:3306/dev_db?useUnicode=true #测试数据库连接
username: canal #数据库账号
password: canal #数据库密码
canalAdapters:
- instance: cqu # canal instance Name or mq topic name
groups:
- groupId: g1
outerAdapters:
- name: es7 # ES同步适配器
hosts: http://192.168.0.10:11700 # ES连接地址
properties:
mode: rest # 模式可选transport(9300) 或者 rest(9200)
cluster.name: es-single # ES集群名称, 与es目录下 elasticsearch.yml文件cluster.name对应
4.运行项目
4.1 运行启动类:CanalAdapterApplication.java
4.2 Config dir not found 报错解决
运行时可能会有的报错:
ERROR c.a.o.canal.adapter.launcher.loader.CanalAdapterLoader - Load canal adapter: es7 failed
java.lang.RuntimeException: java.lang.RuntimeException: java.lang.RuntimeException: Config dir not found.
at com.alibaba.otter.canal.client.adapter.es7x.ES7xAdapter.init(ES7xAdapter.java:54) ~[client-adapter.es7x-1.1.5-jar-with-dependencies.jar:na]
at com.alibaba.otter.canal.adapter.launcher.loader.CanalAdapterLoader.loadAdapter(CanalAdapterLoader.java:225) [classes/:na]
at com.alibaba.otter.canal.adapter.launcher.loader.CanalAdapterLoader.init(CanalAdapterLoader.java:56) [classes/:na]
at com.alibaba.otter.canal.adapter.launcher.loader.CanalAdapterService.init(CanalAdapterService.java:60) [classes/:na]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_221]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_221]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_221]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_221]
原因:launcher模块下缺少es7文件夹
解决:将es7x下的es7文件夹复制到launcher的resources目录下
target中生成的文件
重新运行,即可运行成功
5.可直接运行的项目源码
以下代码在运行前,需要在根目录执行mvn clean install 命令,然后修改application.yml的配置后就可以直接运行。
链接:https://pan.baidu.com/s/1sKOdIdAMGkgHP1XddWfGLA?pwd=yw33
提取码:yw33