一、前言

  由于公司的要求下要使Nacos集群以Oracle作为外部数据源，前期咱们已经阐述了如何在本地搭建（Nacos集群搭建【Oracle作外部数据源】），本次将带领大家在k8s上部署Nacos集群并以Oracle作为外部数据源。

二、软件包

• nacos-finder-plugin.tar官方镜像包
• nacos-server-oracle.tar自定义镜像包

说明：nacos-server-oracle.tar该镜像包经过本人dockerfile构建修改、加入了oracle相关变量、参数、以及支持oracle 11g的二次编译nacos-server.jar程序包。

三、环境

• k8s1.23.1
• nacos-server2.3.2
• Oracle 11g RAC集群
  

四、部署

1.创建命名空间

ls
cat 00-ns.yaml 

apiVersion: v1
kind: Namespace
metadata:
  name: nacoscluster
  labels:
    app: nacoscluster
    
kubectl apply -f 00-ns.yaml

2.创建存储类

cat 00-sc.yaml

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: local-storage2
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer

kubectl apply -f 00-sc.yaml

3.创建pv卷

 cat pv.yaml 
apiVersion: v1
kind: PersistentVolume
metadata:
  name: local-storage-nacos-pv-0
  namespace: nacoscluster     # 空间名称
  labels:
    name: local-storage-nacos-pv-0
spec:
  capacity:
    storage: 100Gi
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: local-storage2
  local:
    path: /srv/nacos_data #存储卷的实际位置，与nfs暴露的文件地址相同
  nodeAffinity:
    required:
      nodeSelectorTerms:        # 节点选择
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - node3            # 根据自己节点名称
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: local-storage-nacos-pv-1
  namespace: nacoscluster     # 空间名称
  labels:
    name: local-storage-nacos-pv-1
spec:
  capacity:
    storage: 100Gi
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: local-storage2
  local:
    path: /srv/nacos_data #存储卷的实际位置，与nfs暴露的文件地址相同
  nodeAffinity:
    required:
      nodeSelectorTerms:        # 节点选择
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - node4            # 根据自己节点名称
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: local-storage-nacos-pv-2
  namespace: nacoscluster     # 空间名称
  labels:
    name: local-storage-nacos-pv-2
spec:
  capacity:
    storage: 100Gi
  accessModes:
  - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: local-storage2
  local:
    path: /srv/nacos_data #存储卷的实际位置，与nfs暴露的文件地址相同
  nodeAffinity:
    required:
      nodeSelectorTerms:        # 节点选择
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - node5            # 根据自己节点名称

kubectl apply -f  pv.yaml

注意：需要在对应节点(例：node3、node4、node5)创建该目录/srv/nacos_data

3.给相应节点打标签

# 用于nacos容器绑定到指定节点（可选）
kubectl label nodes node3 app=nacos
kubectl label nodes node4 app=nacos
kubectl label nodes node5 app=nacos

4.创建nacos集群

cat nacos-pvc.yaml

###使用自建数据库；使用Ingress发布配置后台###
---
apiVersion: v1
kind: Service
metadata:
  name: nacos-headless
  namespace: nacoscluster
  labels:
    app: nacos-headless
spec:
  type: ClusterIP
  clusterIP: None
  ports:
    - name: tcp-8848
      protocol: TCP
      port: 8848
      targetPort: 8848
    - name: tcp-9848
      protocol: TCP
      port: 9848
      targetPort: 9848
    - name: tcp-9849
      protocol: TCP
      port: 9849
      targetPort: 9849
    - name: tcp-7848
      protocol: TCP
      port: 7848
      targetPort: 7848
  selector:
    app: nacos
---
# 修改以下数据库配置
apiVersion: v1
kind: ConfigMap
metadata:
  name: nacos-cm
  namespace: nacoscluster
data:
  oracle.db.name: "traffic_dev"
  oracle.host: "192.168.200.122"
  oracle.password: "Sjgj20241107"
  oracle.port: '1521'
  oracle.user: "zkdn_os_init"
  nacos-auth-enable: 'true'
  nacos-auth-identity-key: nacosPlus496
  nacos-auth-identity-value: nacosPlus628
  nacos-auth-token: TmFjb3NQbHVzMDEyMzQ1Njc4OTAxMjM0NTY3ODk5ODc2NTQzMjEwMDEyMzQ1Njc4OTk4NzY1NDMyMTAwMTIzNDU2Nzg5
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: nacos
  namespace: nacoscluster
spec:
  serviceName: nacos-headless
  replicas: 3
  template:
    metadata:
      labels:
        app: nacos
      annotations:
        pod.alpha.kubernetes.io/initialized: "true"
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchExpressions:
                  - key: "app"
                    operator: In
                    values:
                      - nacos
              topologyKey: "kubernetes.io/hostname"
      initContainers:
        - name: peer-finder-plugin-install
          image: '192.168.200.133:8080/sjgj/nacos/nacos-peer-finder-plugin:1.1'
          resources: {}
          volumeMounts:
            - name: pv-storage-elastic-master
              mountPath: /home/nacos/plugins/peer-finder
              subPath: peer-finder
      containers:
        - name: nacos
          imagePullPolicy: Always
          image: 192.168.200.133:8080/sjgj/nacos/nacos-server:v2.3.2
          resources:
            requests:
              memory: "2Gi"
              cpu: "500m"
          ports:
            - name: client-port
              containerPort: 8848
              protocol: TCP
            - name: client-rpc
              containerPort: 9848
              protocol: TCP
            - name: raft-rpc
              containerPort: 9849
              protocol: TCP
            - name: old-raft-rpc
              containerPort: 7848
              protocol: TCP
          env:
            - name: NACOS_REPLICAS
              value: "3"
            - name: ORACLE_SERVICE_HOST
              valueFrom:
                configMapKeyRef:
                  name: nacos-cm
                  key: oracle.host
            - name: ORACLE_SERVICE_DB_NAME
              valueFrom:
                configMapKeyRef:
                  name: nacos-cm
                  key: oracle.db.name
            - name: ORACLE_SERVICE_PORT
              valueFrom:
                configMapKeyRef:
                  name: nacos-cm
                  key: oracle.port
            - name: ORACLE_SERVICE_USER
              valueFrom:
                configMapKeyRef:
                  name: nacos-cm
                  key: oracle.user
            - name: ORACLE_SERVICE_PASSWORD
              valueFrom:
                configMapKeyRef:
                  name: nacos-cm
                  key: oracle.password
            - name: MODE
              value: "cluster"
            - name: SPRING_DATASOURCE_PLATFORM
              value: oracle
            - name: NACOS_APPLICATION_PORT
              value: "8848"
            - name: PREFER_HOST_MODE
              value: "hostname"
            - name: NACOS_SERVERS
              value: "nacos-0.nacos-headless.nacoscluster.svc.cluster.local:8848 nacos-1.nacos-headless.nacoscluster.svc.cluster.local:8848 nacos-2.nacos-headless.nacoscluster.svc.cluster.local:8848"
            - name: NACOS_AUTH_ENABLE
              valueFrom:
                configMapKeyRef:
                  name: nacos-cm
                  key: nacos-auth-enable
            - name: NACOS_AUTH_IDENTITY_KEY
              valueFrom:
                configMapKeyRef:
                  name: nacos-cm
                  key: nacos-auth-identity-key
            - name: NACOS_AUTH_IDENTITY_VALUE
              valueFrom:
                configMapKeyRef:
                  name: nacos-cm
                  key: nacos-auth-identity-value
            - name: NACOS_AUTH_TOKEN
              valueFrom:
                configMapKeyRef:
                  name: nacos-cm
                  key: nacos-auth-token
          volumeMounts:
            - name: pv-storage-elastic-master
              mountPath: /home/nacos/peer-finder
  volumeClaimTemplates:
  - metadata:
      name: pv-storage-elastic-master
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: "local-storage2"
      resources:
        requests:
          storage: 100Gi
  selector:
    matchLabels:
      app: nacos

kubectl apply -f nacos-pvc.yaml 

5.创建服务端口

cat nacos-service.yaml

#特别强调一点，k8s的版本不同ingress配置的一些细节不同，具体可以看看官网，
#我展示的配置信息，只能保证1.23.1版本的k8s可以使用。
#nacos-k8s里也有ingress的配置，可以参考一下
#./nacos-k8s/deploy/nacos/nacos-no-pvc-ingress.yaml
apiVersion: v1
kind: Service
metadata:
  name: nacos
  namespace: nacoscluster
spec:
  ports:
    - name: tcp-8848
      protocol: TCP
      port: 8848
      targetPort: 8848
    - name: tcp-9848
      protocol: TCP
      port: 9848
      targetPort: 9848
    - name: tcp-9849
      protocol: TCP
      port: 9849
      targetPort: 9849
    - name: tcp-7848
      protocol: TCP
      port: 7848
      targetPort: 7848
  selector:
    app: nacos
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: nacos
  namespace: nacoscluster
  labels:
    nacos: ingress-http
  annotations:
    kubernetes.io/ingress.class: nginx
spec:
  rules:
  - host: nacos.hdjsjb.com #自定义,ingress规定只能使用域名，没有的可以去修改host文件
    http:
      paths:
      - path: / #自定义,建议就这样，不然在springCloud服务注册时报错405
        pathType: Prefix #必须配置匹配策略
        backend:
          service: 
            name: nacos-headless
            port: 
              number: 8848

kubectl apply -f nacos-service.yaml

说明：可以在kuboard上查看

6.Nacos镜像内application.properties配置展示

cat application.properties

# nacos-oracle-config
# spring
server.servlet.contextPath=${SERVER_SERVLET_CONTEXTPATH:/nacos}
server.contextPath=/nacos
server.port=${NACOS_APPLICATION_PORT:8848}
server.tomcat.accesslog.max-days=30
server.tomcat.accesslog.pattern=%h %l %u %t "%r" %s %b %D %{User-Agent}i %{Request-Source}i
server.tomcat.accesslog.enabled=${TOMCAT_ACCESSLOG_ENABLED:false}
server.error.include-message=ALWAYS
# default current work dir
server.tomcat.basedir=file:.
#*************** Config Module Related Configurations ***************#
### Deprecated configuration property, it is recommended to use `spring.sql.init.platform` replaced.
#spring.datasource.platform=${SPRING_DATASOURCE_PLATFORM:}
nacos.plugin.datasource.log.enabled=true

##### If use Oracle as datasource:
spring.sql.init.platform=${SPRING_DATASOURCE_PLATFORM:}
nacos.cmdb.dumpTaskInterval=3600
nacos.cmdb.eventTaskInterval=10
nacos.cmdb.labelTaskInterval=300
nacos.cmdb.loadDataAtStart=false
db.num=${ORACLE_DATABASE_NUM:1}
db.url.0=jdbc:oracle:thin:@//${ORACLE_SERVICE_HOST}:${ORACLE_SERVICE_PORT:1521}/${ORACLE_SERVICE_DB_NAME}
db.user.0=${ORACLE_SERVICE_USER}
db.password.0=${ORACLE_SERVICE_PASSWORD}
## DB connection pool settings
db.pool.config.connectionTimeout=${DB_POOL_CONNECTION_TIMEOUT:30000}
db.pool.config.validationTimeout=10000
db.pool.config.maximumPoolSize=20
db.pool.config.minimumIdle=2
db.pool.config.driver-class-name=oracle.jdbc.OracleDriver
db.testQuery=select 1 from dual
### The auth system to use, currently only 'nacos' and 'ldap' is supported:
nacos.core.auth.enabled=${NACOS_AUTH_ENABLE:false}
nacos.core.auth.system.type=${NACOS_AUTH_SYSTEM_TYPE:nacos}
### worked when nacos.core.auth.system.type=nacos
### The token expiration in seconds:
nacos.core.auth.plugin.nacos.token.expire.seconds=${NACOS_AUTH_TOKEN_EXPIRE_SECONDS:18000}
### The default token:
nacos.core.auth.plugin.nacos.token.secret.key=${NACOS_AUTH_TOKEN:}
### Turn on/off caching of auth information. By turning on this switch, the update of auth information would have a 15 seconds delay.
nacos.core.auth.caching.enabled=${NACOS_AUTH_CACHE_ENABLE:false}
nacos.core.auth.enable.userAgentAuthWhite=${NACOS_AUTH_USER_AGENT_AUTH_WHITE_ENABLE:false}
nacos.core.auth.server.identity.key=${NACOS_AUTH_IDENTITY_KEY:}
nacos.core.auth.server.identity.value=${NACOS_AUTH_IDENTITY_VALUE:}
## spring security config
### turn off security
nacos.security.ignore.urls=${NACOS_SECURITY_IGNORE_URLS:/,/error,/**/*.css,/**/*.js,/**/*.html,/**/*.map,/**/*.svg,/**/*.png,/**/*.ico,/console-fe/public/**,/v1/auth/**,/v1/console/health/**,/actuator/**,/v1/console/server/**}

# metrics for elastic search
management.metrics.export.elastic.enabled=false
management.metrics.export.influx.enabled=false
nacos.naming.distro.taskDispatchThreadCount=10
nacos.naming.distro.taskDispatchPeriod=200
nacos.naming.distro.batchSyncKeyCount=1000
nacos.naming.distro.initDataRatio=0.9
nacos.naming.distro.syncRetryDelay=5000
nacos.naming.data.warmup=true
nacos.console.ui.enabled=true
nacos.core.param.check.enabled=true

五、测试

用户名/密码：nacos/nacos
http://< IP >:8848/nacos/#/login

六、结束语

  到此k8s上Nacos集群部署Oracle作外部数据源使用讲解就完毕啦，如果有其它问题欢迎大家评论留言，如果觉得还可以，那么给作者点赞、收藏、关注吧！‌