k8s 部署 seata1.6.0 集群 基于 nacos 注册中心 + mysql 数据库
大纲
- 1 镜像制作
- 2 准备configmap
- 3 准备deploy 部署文件
- 4 部署seata到k8s
镜像制作
下载seata 选择1.6.0。下载后得到 seata-server-1.6.0.zip 已经上传到百度云盘
下载地址:http://seata.io/zh-cn/blog/download.html
使用unzip 解压 seata-server-1.6.0.zip
unzip seata-server-1.6.0.zip
解压后得到如下文件夹
seata文件夹内部包含
- Dockerfile 创建docker镜像的文件
- LICENSE
- bin seata启动脚本
- conf seata配置
- ext 性能监控
- lib 依赖的jar包
- logs 日志放置位置
- script 一些脚本 例如数据库脚本 k8s脚本
- target seata程序所在包 脚本最终会执行 seata-server.jar
可以直接Dockerfile来创建镜像
docker build -t my-seata-server .
docker tag my-seata-server registry.cn-hangzhou.aliyuncs.com/jimliu/my-seata-server:1.6.0
docker push registry.cn-hangzhou.aliyuncs.com/jimliu/my-seata-server:1.6.0
创建好镜像后推送到私库备用
准备configmap
注意:
- 1 也可以不使用configmap 因为seata1.6.0无法reload配置文件
- 2 假定mysql数据库配置以及nacos配置以完成,
关于nacos 与 mysql配置可参考
可参考 《seata1.6.0 单机,集群搭建 基于nacos注册中心 mysql数据库》
如果要实现配置文件修改后自动重启,可以使用linux-inotify监听文件变化后重启
可参考 《linux-inotify工具监控文件状态变化总结》
《k8s 部署nginx 实现集群统一配置,自动更新nginx.conf配置文件 总结》
准备两个configmap
- 1 保存seata application 配置文件 (configmap-for-server.yaml)
- 2 保存seata logback 配置文件 (configmap-for-logger.yaml)
主要是把 application.yml logback-spring.xml 和 logback文件夹中的配置文件交由configmap管理
configmap-for-server.yaml 内容如下 保存server相关配置文件
# 将配置放置在configmap中统一管理
apiVersion: v1
kind: ConfigMap
metadata:
name: seata-server-config
data:
application.yml: |
server:
port: 17091 #修改控制台端口
spring:
application:
name: seata-server
logging:
config: classpath:logback-spring.xml
file:
path: ${user.home}/logs/seata
extend:
logstash-appender:
destination: 127.0.0.1:4560
kafka-appender:
bootstrap-servers: 127.0.0.1:9092
topic: logback_to_logstash
console:
user:
username: seata
password: seata
seata:
config:
type: nacos
nacos:
server-addr: 192.168.0.206:8248
namespace: 8422b4bf-8b04-406e-8295-39d4968b2ec4
group: DEFAULT_GROUP
data-id: seata.properties
#username:
#password:
registry:
type: nacos
nacos:
application: seata-server
server-addr: 192.168.0.206:8248
namespace: 8422b4bf-8b04-406e-8295-39d4968b2ec4
group: DEFAULT_GROUP
cluster: default
#username:
#password:
security:
secretKey: SeataSecretKey0c382ef121d778043159209298fd40bf3850a017
tokenValidityInMilliseconds: 1800000
ignore:
urls: /,/**/*.css,/**/*.js,/**/*.html,/**/*.map,/**/*.svg,/**/*.png,/**/*.ico,/console-fe/public/**,/api/v1/auth/login
logback-spring.xml: |
<?xml version="1.0" encoding="UTF-8"?>
<configuration scan="true" scanPeriod="60 seconds" debug="false">
<!-- Context listeners -->
<contextListener class="io.seata.server.logging.listener.SystemPropertyLoggerContextListener"/>
<!-- The conversion rules are copied from `defaults.xml` in the `spring-boot-xxx.jar` -->
<conversionRule conversionWord="clr" converterClass="org.springframework.boot.logging.logback.ColorConverter" />
<conversionRule conversionWord="wex" converterClass="org.springframework.boot.logging.logback.WhitespaceThrowableProxyConverter" />
<conversionRule conversionWord="wEx" converterClass="org.springframework.boot.logging.logback.ExtendedWhitespaceThrowableProxyConverter" />
<!-- The custom conversion rules -->
<conversionRule conversionWord="wEx2" converterClass="io.seata.server.logging.logback.ExtendedWhitespaceThrowableProxyConverter"/>
<!-- common properties -->
<springProperty name="PORT" source="server.port" defaultValue="7091"/>
<springProperty name="APPLICATION_NAME" source="spring.application.name" defaultValue="seata-server"/>
<!-- console-appender -->
<include resource="logback/console-appender.xml"/>
<!-- file-appender -->
<include resource="logback/file-appender.xml"/>
<!-- logstash-appender: off by default -->
<!--<include resource="logback/logstash-appender.xml"/>-->
<!-- kafka-appender: off by default -->
<!--<include resource="logback/kafka-appender.xml"/>-->
<root level="INFO">
<!-- console-appender -->
<appender-ref ref="CONSOLE"/>
<!-- file-appender -->
<appender-ref ref="FILE_ALL"/>
<appender-ref ref="FILE_WARN"/>
<appender-ref ref="FILE_ERROR"/>
<!-- logstash-appender: off by default -->
<!--<appender-ref ref="LOGSTASH"/>-->
<!-- kafka-appender: off by default -->
<!--<appender-ref ref="KAFKA"/>-->
</root>
</configuration>
configmap-for-logger.yaml 内容如下 保存logback相关配置文件 (注意 kafka-appender.xml logstash-appender.xml 并没有使用 这里为了演示统一也加入了管理)
# 将配置放置在configmap中统一管理
apiVersion: v1
kind: ConfigMap
metadata:
name: seata-server-log-config
data:
console-appender.xml: |
<?xml version="1.0" encoding="UTF-8"?>
<included>
<!-- console-appender properties -->
<springProperty name="CONSOLE_LOG_PATTERN" source="logging.pattern.console"
defaultValue="%clr(%d{HH:mm:ss.SSS}){faint} %clr(%5p) %clr(---){faint} %clr([%25.25t]){faint} %clr(%-40.40logger{39}){cyan} %clr(:){faint} %m%n%wEx2"/>
<appender name="CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
<encoder>
<Pattern>${CONSOLE_LOG_PATTERN}</Pattern>
<charset>UTF-8</charset>
</encoder>
</appender>
</included>
file-appender.xml: |
<?xml version="1.0" encoding="UTF-8"?>
<included>
<!-- file-appender properties -->
<springProperty name="LOG_FILE_PATH" source="logging.file.path"
defaultValue="${user.home}/logs/seata"/>
<springProperty name="FILE_LOG_PATTERN" source="logging.pattern.file"
defaultValue="%d{yyyy-MM-dd HH:mm:ss.SSS} %5p --- [%30.30t] %-40.40logger{39} : %m%n%wEx2"/>
<!--ALL-->
<appender name="FILE_ALL" class="ch.qos.logback.core.rolling.RollingFileAppender">
<file>${LOG_FILE_PATH}/${APPLICATION_NAME:-seata-server}.${RPC_PORT}.all.log</file>
<append>true</append>
<rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">
<fileNamePattern>${LOG_FILE_PATH}/history/${APPLICATION_NAME:-seata-server}.${RPC_PORT}.all.%d{yyyy-MM-dd}.%i.log.gz</fileNamePattern>
<maxFileSize>2GB</maxFileSize>
<MaxHistory>7</MaxHistory>
<totalSizeCap>7GB</totalSizeCap>
<cleanHistoryOnStart>true</cleanHistoryOnStart>
</rollingPolicy>
<encoder>
<Pattern>${FILE_LOG_PATTERN}</Pattern>
<charset>UTF-8</charset>
</encoder>
</appender>
<!--WARN-->
<appender name="FILE_WARN" class="ch.qos.logback.core.rolling.RollingFileAppender">
<filter class="ch.qos.logback.classic.filter.LevelFilter">
<level>WARN</level>
<onMatch>ACCEPT</onMatch>
<onMismatch>DENY</onMismatch>
</filter>
<file>${LOG_FILE_PATH}/${APPLICATION_NAME:-seata-server}.${RPC_PORT}.warn.log</file>
<append>true</append>
<rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">
<fileNamePattern>${LOG_FILE_PATH}/history/${APPLICATION_NAME:-seata-server}.${RPC_PORT}.warn.%d{yyyy-MM-dd}.%i.log.gz</fileNamePattern>
<maxFileSize>2GB</maxFileSize>
<MaxHistory>7</MaxHistory>
<totalSizeCap>7GB</totalSizeCap>
<cleanHistoryOnStart>true</cleanHistoryOnStart>
</rollingPolicy>
<encoder>
<Pattern>${FILE_LOG_PATTERN}</Pattern>
<charset>UTF-8</charset>
</encoder>
</appender>
<!--ERROR-->
<appender name="FILE_ERROR" class="ch.qos.logback.core.rolling.RollingFileAppender">
<filter class="ch.qos.logback.classic.filter.LevelFilter">
<level>ERROR</level>
<onMatch>ACCEPT</onMatch>
<onMismatch>DENY</onMismatch>
</filter>
<file>${LOG_FILE_PATH}/${APPLICATION_NAME:-seata-server}.${RPC_PORT}.error.log</file>
<append>true</append>
<rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">
<fileNamePattern>${LOG_FILE_PATH}/history/${APPLICATION_NAME:-seata-server}.${RPC_PORT}.error.%d{yyyy-MM-dd}.%i.log.gz</fileNamePattern>
<maxFileSize>2GB</maxFileSize>
<MaxHistory>7</MaxHistory>
<totalSizeCap>7GB</totalSizeCap>
<cleanHistoryOnStart>true</cleanHistoryOnStart>
</rollingPolicy>
<encoder>
<Pattern>${FILE_LOG_PATTERN}</Pattern>
<charset>UTF-8</charset>
</encoder>
</appender>
</included>
kafka-appender.xml: |
<?xml version="1.0" encoding="UTF-8"?>
<included>
<!-- kafka-appender properties -->
<springProperty name="KAFKA_BOOTSTRAP_SERVERS" source="logging.extend.kafka-appender.bootstrap-servers"
defaultValue="127.0.0.1:9092"/>
<springProperty name="KAFKA_TOPIC" source="logging.extend.kafka-appender.topic"
defaultValue="logback_to_logstash"/>
<appender name="KAFKA" class="com.github.danielwegener.logback.kafka.KafkaAppender">
<encoder>
<!--<pattern>%d{yyyy-MM-dd HH:mm:ss.SSS}|%p|${APPLICATION_NAME:-seata-server}|${RPC_PORT:-0}|%t|%logger|%X{X-TX-XID:-}|%X{X-TX-BRANCH-ID:-}|%m|%wex</pattern>-->
<pattern>{
"@timestamp": "%d{yyyy-MM-dd HH:mm:ss.SSS}",
"level":"%p",
"app_name":"${APPLICATION_NAME:-seata-server}",
"PORT": ${RPC_PORT:-0},
"thread_name": "%t",
"logger_name": "%logger",
"X-TX-XID": "%X{X-TX-XID:-}",
"X-TX-BRANCH-ID": "%X{X-TX-BRANCH-ID:-}",
"message": "%m",
"stack_trace": "%wex"
}
</pattern>
</encoder>
<topic>${KAFKA_TOPIC}</topic>
<keyingStrategy class="com.github.danielwegener.logback.kafka.keying.NoKeyKeyingStrategy"/>
<deliveryStrategy class="com.github.danielwegener.logback.kafka.delivery.AsynchronousDeliveryStrategy"/>
<producerConfig>bootstrap.servers=${KAFKA_BOOTSTRAP_SERVERS}</producerConfig>
<producerConfig>acks=0</producerConfig>
<producerConfig>linger.ms=1000</producerConfig>
<producerConfig>max.block.ms=0</producerConfig>
</appender>
</included>
logstash-appender.xml: |
<?xml version="1.0" encoding="UTF-8"?>
<included>
<!-- logstash-appender properties -->
<springProperty name="LOGSTASH_DESTINATION" source="logging.extend.logstash-appender.destination"
defaultValue="127.0.0.1:4560"/>
<appender name="LOGSTASH" class="net.logstash.logback.appender.LogstashTcpSocketAppender">
<!-- the TCP address of the logstash -->
<destination>${LOGSTASH_DESTINATION}</destination>
<!--<encoder charset="UTF-8" class="net.logstash.logback.encoder.LogstashEncoder">-->
<encoder charset="UTF-8" class="io.seata.server.logging.logback.appender.EnhancedLogstashEncoder">
<!-- the global custom fields -->
<customFields>
{
"app_name": "${APPLICATION_NAME:-seata-server}"
}
</customFields>
<!-- Exclude the provider of data `@version` -->
<excludeProvider>net.logstash.logback.composite.LogstashVersionJsonProvider</excludeProvider>
<!-- Exclude providers that are not currently needed, reduce some performance loss. -->
<excludeProvider>net.logstash.logback.composite.loggingevent.JsonMessageJsonProvider</excludeProvider>
<excludeProvider>net.logstash.logback.composite.loggingevent.TagsJsonProvider</excludeProvider>
<excludeProvider>net.logstash.logback.composite.loggingevent.LogstashMarkersJsonProvider</excludeProvider>
<excludeProvider>net.logstash.logback.composite.loggingevent.ArgumentsJsonProvider</excludeProvider>
</encoder>
</appender>
</included>
准备deploy 部署文件
准备部署两个seata服务来实现集群
由于业务服务部署在k8s集群外部无法访问k8s集群内部ip,seata注册到nacos需要使用独立的主机ip, 所以需要两个Deployment来部署seata 无法直接使用 --replicas=2
如果业务服务,nacos ,seata 都在k8s集群内部 ,则直接使用一个Deployment 指定–replicas=2
deploy.yaml 内容如下
# deploy1
apiVersion: apps/v1
kind: Deployment
metadata:
name: seata-server-deploy-1
labels:
k8s-app: seata-server-1
spec:
replicas: 1
selector:
matchLabels:
k8s-seata-node1: node1
template:
metadata:
labels:
k8s-seata-node1: node1
k8s-app: seata-server
spec:
imagePullSecrets:
- name: myaliyunsecret
containers:
- name: seata-server-run
image: registry.cn-hangzhou.aliyuncs.com/jimliu/my-seata-server:1.6.0 #使用开始创建的镜像
env:
- name: SEATA_IP #环境变量中配置一个IP 用于注册中心
value: "192.168.0.54"
- name: SERVER_NODE #指定seata-server节点ID, 默认为 根据ip生成 (可以不配置)
value: "1"
- name: SEATA_PORT
value: "5959" #注意环境变量中是数字也需要使用“”
volumeMounts:
- name: seata-config
mountPath: /seata-server/conf #使用configMap挂载配置文件
- name: seata-log-config
mountPath: /seata-server/conf/logback #使用configMap挂载配置文件
volumes:
- name: seata-config
configMap:
name: seata-server-config
- name: seata-log-config
configMap:
name: seata-server-log-config
---
# deploy2
apiVersion: apps/v1
kind: Deployment
metadata:
name: seata-server-deploy-2
labels:
k8s-app: seata-server-2
spec:
replicas: 1
selector:
matchLabels:
k8s-seata-node2: node2
template:
metadata:
labels:
k8s-seata-node2: node2
k8s-app: seata-server
spec:
imagePullSecrets:
- name: myaliyunsecret
containers:
- name: seata-server-run
image: registry.cn-hangzhou.aliyuncs.com/jimliu/my-seata-server:1.6.0 #使用开始创建的镜像
env:
- name: SEATA_IP #环境变量中配置一个IP 用于注册中心
value: "192.168.0.124"
- name: SERVER_NODE #指定seata-server节点ID, 默认为 根据ip生成 (可以不配置)
value: "2"
- name: SEATA_PORT
value: "5959" #注意环境变量中是数字也需要使用“”
volumeMounts:
- name: seata-config
mountPath: /seata-server/conf #使用configMap挂载配置文件
- name: seata-log-config
mountPath: /seata-server/conf/logback #使用configMap挂载配置文件
volumes:
- name: seata-config
configMap:
name: seata-server-config
- name: seata-log-config
configMap:
name: seata-server-log-config
---
# service
apiVersion: v1
kind: Service
metadata:
name: seata-server
labels:
k8s-app: seata-server
spec:
type: NodePort
ports:
- port: 5959
nodePort: 5959
targetPort: 5959
protocol: TCP
name: http1
- port: 17091
nodePort: 17091 #for seata 控制台
targetPort: 17091
protocol: TCP
name: http2
selector:
k8s-app: seata-server
部署seata到k8s
kubectl apply -f configmap-for-logger.yaml
kubectl apply -f configmap-for-server.yaml
kubectl apply -f deploy.yaml
在nacos上可以看到 seata已经成功注册
控制台可以正常访问
启动seata 容器时注意关注k8s内是否有LimitRange限制,seata默认启动使用1G内存