问题: 启动maxwell后出现数据更新后就出现以下报错。
13:29:14,727 ERROR MaxwellKafkaProducer - TimeoutException @ Position[BinlogPosition[binlog.000002:12215591], lastHeartbeat=1700717043797] -- maxWellData: medical:consultation:[(id,212)]
13:29:14,728 ERROR MaxwellKafkaProducer - Expiring 35 record(s) for maxWellData-0: 30005 ms has passed since batch creation plus linger time
排查过程:
1.排查maxwell:配置maxwell输出到控制台,发现采集数据后正常输出,说明maxwell配置正常。
[root@VM-4-10-centos maxwell]# bin/maxwell --user='maxwell' \
> --password='maxwell' \
> --host='10.0.4.10' \
> --producer=stdout
2.使用logstash采集数据到kafka
logstash使用说明:https://blog.csdn.net/m0_52606060/article/details/134499513?spm=1001.2014.3001.5501
input {
generator {
message => '14.49.42.25 - - [12/May/2019:01:24:44 +0000] "GET /articles/ppp-over-ssh/ HTTP/1.1" 200 18586 "-" "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2b1) Gecko/20091014 Firefox/3.6b1 GTB5"'
count => 1
}
}
output {
stdout {
codec => rubydebug
}
kafka {
bootstrap_servers => "101.91.153.39:9092,61.171.111.6:9092,61.171.100.138:9092"
codec => json_lines
topic_id => "maxWellData"
}
}