问题描述
在两台机器上，分别为服务器A与主机B，其中服务器A上部署OAI核心网，主机B上部署容器版ueransim。两机器用一根网线连接。两台机器的连接如此：但是ueransim无法接入核心网。
解决思路1——机器网卡设置到192.168.70.0网段
将机器物理网卡网段设置到与核心网所在网段一致，192.168.70.x，ueransim直接接入。
发现1
两机器分别设置如下：
核心网A：
ens38:
addresses: [192.168.70.12/24]
dhcp4: no
gateway4: 192.168.70.1
nameservers:
addresses: [8.8.8.8,144.144.144.144,255.255.255.255]
主机B：
ens39:
addresses: [192.168.70.13/24]
dhcp4: no
gateway4: 192.168.70.1
nameservers:
addresses: [8.8.8.8,144.144.144.144,255.255.255.255]
两者可以互相ping通，但开启核心网后却无法互相ping通
进行如下修改，将对方网口作为自己的网关后，在开通核心网后可以ping通，但依旧无法ping通核心网
核心网A：
ens38:
addresses: [192.168.70.12/24]
dhcp4: no
gateway4: 192.168.70.12
nameservers:
addresses: [8.8.8.8,144.144.144.144,255.255.255.255]
主机B：
ens39:
addresses: [192.168.70.13/24]
dhcp4: no
gateway4: 192.168.70.13
nameservers:
addresses: [8.8.8.8,144.144.144.144,255.255.255.255]
经过在服务器A抓包，发现服务器A的192.168.70.12上有抓到来自192.168.70.13寻找192.168.70.1的ARP，但是却没有成功找到的返回
有来自192.168.70.13的寻找192.168.70.1的icmp包，但只有request， no response found
然后通过如下方法解决，让主机B可以ping到192.168.70.1：
在主机B上添加到192.168.70.0/24网段的路由：
sudo route add -net 192.168.70.0/24 gw 192.168.70.12
在服务器A添加到主机B的路由：
sudo route add -host 192.168.70.13 gw 192.168.70.12
然后启动ueransim测试：
无法启动，ueransim需要接入demo-oai等网桥，貌似无法接入另一台机器上的网桥。
但这样的网络配置可以作为非容器化ueransim的一种配置，中间不用换网段到192.168.12.x
解决思路2 docker-swarm
Docker Compose和Docker Swarm旨在实现完全集成，这意味着您可以将Compose应用程序指向Swarm集群，并使它们都像使用单个Docker主机一样正常工作。引用
manager是我们管理集群的入口，我们的docker命令都是在manager上执行，node节点上是不能执行dockr命令的
这里我将服务器A作为管理节点，主机B作为工作节点
在服务器A上，通过ssh接入想作为管理节点的服务器，然后执行swarm init命令，指定swarm管理机的IP地址：
#安装docker-ce
sudo apt-get update
sudo apt-get install docker-ce
#指定为管理节点，该节点ip为192.168.12.3
lab@lab-virtual-machine:~$ docker swarm init --advertise-addr 192.168.12.3
出现以下提示
Swarm initialized: current node (uowf6o6oftuh47y2qjgm7jqeh) is now a manager.
To add a worker to this swarm, run the following command:
docker swarm join --token SWMTKN-1-2f3q3ajwud8mg8xw62zbs7k0hpw9ypss7qgqcyjdzw5dns4qxf-a9mnk42uf9tjh58yj8s00t4ta 192.168.12.3:2377
To add a manager to this swarm, run 'docker swarm join-token manager' and follow the instructions.
其中这里表示如何加入工作节点：
docker swarm join --token SWMTKN-1-2f3q3ajwud8mg8xw62zbs7k0hpw9ypss7qgqcyjdzw5dns4qxf-a9mnk42uf9tjh58yj8s00t4ta 192.168.12.3:2377
在主机B上将主机B设置为工作节点：
docker swarm join --token SWMTKN-1-2f3q3ajwud8mg8xw62zbs7k0hpw9ypss7qgqcyjdzw5dns4qxf-a9mnk42uf9tjh58yj8s00t4ta 192.168.12.3:2377
出现以下提示：
This node joined a swarm as a worker.
到此搭建了一个简单的swarm集群。
查看接入的节点：
lab@lab-virtual-machine:~$ docker node ls
查看创建的网络：
docker network ls
创建一个新网络：
网络模式需要选择overlay
docker network create -d overlay testnet
可是OAI核心网的网络模式是bridge，是否可以将其变成overlay？
将所有docker-compose-basic-vpp-nrf.yaml中的网络由bridge改为overlay：
networks:
public_net:
driver: overlay
name: demo-oai-public-net
ipam:
config:
- subnet: 192.168.70.0/24
driver_opts:
com.docker.network.bridge.name: "demo-oai"
public_net_access:
driver: overlay
name: oai-public-access
ipam:
config:
- subnet: 192.168.72.0/24
driver_opts:
com.docker.network.bridge.name: "cn5g-access"
public_net_core:
driver: overlay
name: oai-public-core
ipam:
config:
- subnet: 192.168.73.0/24
driver_opts:
com.docker.network.bridge.name: "cn5g-core"
然后使用docker stack启动
之前 swarm 集群中docker service create一次只能部署一个微服务，我们可以使用 docker stack + compose 一次启动多个服务
stack相关操作：
deploy Deploy a new stack or update an existing stack
ls List stacks
ps List the tasks in the stack
rm Remove one or more stacks
services List the services in the stack
lab@lab-virtual-machine:~/oai-cn5g-fed/docker-compose$ docker stack deploy --compose-file docker-compose-basic-vpp-nrf.yaml oai
查看运行的服务：
docker service ls
查看容器运行状态：
docker service ps 容器名字
启动ueransim;
lab@lab-virtual-machine:~/oai-cn5g-fed/docker-compose$ sudo docker stack deploy --compose-file docker-compose-ueransim-vpp.yaml ueransim
[sudo] password for lab:
WARN[0000] network public_net_access: network.external.name is deprecated in favor of network.name
WARN[0000] network public_net: network.external.name is deprecated in favor of network.name
Ignoring unsupported options: privileged
Ignoring deprecated options:
container_name: Setting the container name is not supported.
Error response from daemon: 2 matches found based on name: network demo-oai-public-net is ambiguous
其中前两个警告WARN【0000】是由于yaml文件网络external的格式不对，更改为如下后会消失：
networks:
public_net:
external: true
name: demo-oai-public-net
public_net_access:
external: true
name: oai-public-access
后面的警告Error response from daemon: 2 matches found based on name: network demo-oai-public-net is ambiguous是由于机器上docker network里的demo-oai-public-net有local与swarm两种
将其中local的部分删掉：
docker network rm f6dd0838cf68
docker network rm 34b108d99783
docker network rm f9b3e7f7b442
再次启动ueransim：
lab@lab-virtual-machine:~/oai-cn5g-fed/docker-compose$ sudo docker stack deploy --compose-file docker-compose-ueransim-vpp.yaml ueransim
Ignoring unsupported options: privileged
Ignoring deprecated options:
container_name: Setting the container name is not supported.
Creating service ueransim_ueransim
但是查看ueransim的日志，发现未能启动成功：
甚至连gnb都没有成功接入OAI核心网
lab@lab-virtual-machine:~/oai-cn5g-fed/docker-compose$ docker service logs 5iarfozcjdvi
WARNING: Error loading config file: /home/lab/.docker/config.json: open /home/lab/.docker/config.json: permission denied
ueransim_ueransim.1.4wxgx35hss0j@lab-virtual-machine | Now setting these variables '@GTP_IP@ @IGNORE_STREAM_IDS@ @LINK_IP@ @MCC@ @MNC@ @NCI@ @NGAP_IP@ @NGAP_PEER_IP@ @SD_0@ @SD_1@ @SD_2@ @SST_0@ @SST_1@ @SST_2@ @TAC@'
ueransim_ueransim.1.4wxgx35hss0j@lab-virtual-machine | Now setting these variables '@AMF_VALUE@ @APN@ @GNB_IP_ADDRESS@ @IMEI@ @IMEI_SV@ @IMSI@ @KEY@ @MCC@ @MNC@ @OP@ @OP_TYPE@ @PDU_TYPE@ @SD_C@ @SD_D@ @SD_R@ @SST_C@ @SST_D@ @SST_R@'
ueransim_ueransim.1.4wxgx35hss0j@lab-virtual-machine | Done setting the configuration
ueransim_ueransim.1.4wxgx35hss0j@lab-virtual-machine | ### Running ueransim ###
ueransim_ueransim.1.4wxgx35hss0j@lab-virtual-machine | Running gnb
ueransim_ueransim.1.4wxgx35hss0j@lab-virtual-machine | UERANSIM v3.2.5
ueransim_ueransim.1.4wxgx35hss0j@lab-virtual-machine | [2023-04-17 07:30:40.606] [rls-udp] [error] RLS failure [Socket bind failed: Cannot assign requested address]
ueransim_ueransim.1.4wxgx35hss0j@lab-virtual-machine | [2023-04-17 07:30:40.607] [sctp] [info] Trying to establish SCTP connection... (192.168.70.132:38412)
ueransim_ueransim.1.4wxgx35hss0j@lab-virtual-machine | [2023-04-17 07:30:40.607] [gtp] [error] GTP/UDP task could not be created. Socket bind failed: Cannot assign requested address
ueransim_ueransim.1.jygw0gvxxm32@lab-virtual-machine | Now setting these variables '@GTP_IP@ @IGNORE_STREAM_IDS@ @LINK_IP@ @MCC@ @MNC@ @NCI@ @NGAP_IP@ @NGAP_PEER_IP@ @SD_0@ @SD_1@ @SD_2@ @SST_0@ @SST_1@ @SST_2@ @TAC@'
ueransim_ueransim.1.4wxgx35hss0j@lab-virtual-machine | [2023-04-17 07:30:40.613] [sctp] [error] Binding to 192.168.70.141:0 failed. SCTP bind failed: Cannot assign requested address
ueransim_ueransim.1.jygw0gvxxm32@lab-virtual-machine | Now setting these variables '@AMF_VALUE@ @APN@ @GNB_IP_ADDRESS@ @IMEI@ @IMEI_SV@ @IMSI@ @KEY@ @MCC@ @MNC@ @OP@ @OP_TYPE@ @PDU_TYPE@ @SD_C@ @SD_D@ @SD_R@ @SST_C@ @SST_D@ @SST_R@'
ueransim_ueransim.1.4wxgx35hss0j@lab-virtual-machine | Running ue
ueransim_ueransim.1.jygw0gvxxm32@lab-virtual-machine | Done setting the configuration
ueransim_ueransim.1.4wxgx35hss0j@lab-virtual-machine | UERANSIM v3.2.5
ueransim_ueransim.1.4wxgx35hss0j@lab-virtual-machine | [2023-04-17 07:30:41.589] [nas] [info] UE switches to state [MM-DEREGISTERED/PLMN-SEARCH]
ueransim_ueransim.1.jygw0gvxxm32@lab-virtual-machine | ### Running ueransim ###
ueransim_ueransim.1.4wxgx35hss0j@lab-virtual-machine | [2023-04-17 07:30:43.791] [nas] [error] PLMN selection failure, no cells in coverage
ueransim_ueransim.1.4wxgx35hss0j@lab-virtual-machine | [2023-04-17 07:30:45.993] [nas] [error] PLMN selection failure, no cells in coverage
ueransim_ueransim.1.jygw0gvxxm32@lab-virtual-machine | Running gnb
ueransim_ueransim.1.4wxgx35hss0j@lab-virtual-machine | [2023-04-17 07:30:46.592] [rrc] [warning] Acceptable cell selection failed, no cell is in coverage
ueransim_ueransim.1.jygw0gvxxm32@lab-virtual-machine | UERANSIM v3.2.5
ueransim_ueransim.1.4wxgx35hss0j@lab-virtual-machine | [2023-04-17 07:30:46.592] [rrc] [error] Cell selection failure, no suitable or acceptable cell found
ueransim_ueransim.1.4wxgx35hss0j@lab-virtual-machine | [2023-04-17 07:30:47.094] [nas] [info] UE switches to state [MM-DEREGISTERED/NO-CELL-AVAILABLE]
ueransim_ueransim.1.jygw0gvxxm32@lab-virtual-machine | [2023-04-17 07:31:53.735] [rls-udp] [error] RLS failure [Socket bind failed: Cannot assign requested address]
ueransim_ueransim.1.4wxgx35hss0j@lab-virtual-machine | [2023-04-17 07:31:16.604] [rrc] [warning] Acceptable cell selection failed, no cell is in coverage
ueransim_ueransim.1.4wxgx35hss0j@lab-virtual-machine | [2023-04-17 07:31:16.605] [rrc] [error] Cell selection failure, no suitable or acceptable cell found
ueransim_ueransim.1.jygw0gvxxm32@lab-virtual-machine | [2023-04-17 07:31:53.737] [sctp] [info] Trying to establish SCTP connection... (192.168.70.132:38412)
ueransim_ueransim.1.4wxgx35hss0j@lab-virtual-machine | [2023-04-17 07:31:16.823] [nas] [error] PLMN selection failure, no cells in coverage
ueransim_ueransim.1.jygw0gvxxm32@lab-virtual-machine | [2023-04-17 07:31:53.737] [gtp] [error] GTP/UDP task could not be created. Socket bind failed: Cannot assign requested address
ueransim_ueransim.1.jygw0gvxxm32@lab-virtual-machine | [2023-04-17 07:31:53.743] [sctp] [error] Binding to 192.168.70.141:0 failed. SCTP bind failed: Cannot assign requested address
ueransim_ueransim.1.jygw0gvxxm32@lab-virtual-machine | Running ue
ueransim_ueransim.1.jygw0gvxxm32@lab-virtual-machine | UERANSIM v3.2.5
ueransim_ueransim.1.jygw0gvxxm32@lab-virtual-machine | [2023-04-17 07:31:54.719] [nas] [info] UE switches to state [MM-DEREGISTERED/PLMN-SEARCH]
ueransim_ueransim.1.jygw0gvxxm32@lab-virtual-machine | [2023-04-17 07:31:56.921] [nas] [error] PLMN selection failure, no cells in coverage
ueransim_ueransim.1.jygw0gvxxm32@lab-virtual-machine | [2023-04-17 07:31:59.124] [nas] [error] PLMN selection failure, no cells in coverage
ueransim_ueransim.1.jygw0gvxxm32@lab-virtual-machine | [2023-04-17 07:31:59.722] [rrc] [warning] Acceptable cell selection failed, no cell is in coverage
ueransim_ueransim.1.jygw0gvxxm32@lab-virtual-machine | [2023-04-17 07:31:59.722] [rrc] [error] Cell selection failure, no suitable or acceptable cell found
ueransim_ueransim.1.jygw0gvxxm32@lab-virtual-machine | [2023-04-17 07:32:00.225] [nas] [info] UE switches to state [MM-DEREGISTERED/NO-CELL-AVAILABLE]
ueransim_ueransim.1.jygw0gvxxm32@lab-virtual-machine | [2023-04-17 07:32:29.734] [rrc] [warning] Acceptable cell selection failed, no cell is in coverage
删除oai与ueransim服务：
docker stack rm oai
docker stack rm ueransim
将ueransim与oai合并到一起启动：
创建文件1.yaml
version: '3.8'
services:
mysql:
container_name: "mysql"
image: mysql:8.0
volumes:
- ./database/oai_db2.sql:/docker-entrypoint-initdb.d/oai_db.sql
- ./healthscripts/mysql-healthcheck2.sh:/tmp/mysql-healthcheck.sh
environment:
- TZ=Europe/Paris
- MYSQL_DATABASE=oai_db
- MYSQL_USER=test
- MYSQL_PASSWORD=test
- MYSQL_ROOT_PASSWORD=linux
healthcheck:
test: /bin/bash -c "/tmp/mysql-healthcheck.sh"
interval: 10s
timeout: 5s
retries: 30
networks:
public_net:
ipv4_address: 192.168.70.131
oai-udr:
container_name: "oai-udr"
image: oaisoftwarealliance/oai-udr:v1.5.0
environment:
- TZ=Europe/Paris
- UDR_NAME=OAI_UDR
- UDR_INTERFACE_NAME_FOR_NUDR=eth0
- MYSQL_IPV4_ADDRESS=192.168.70.131
- MYSQL_USER=test
- MYSQL_PASS=test
- MYSQL_DB=oai_db
- WAIT_MYSQL=120
- USE_FQDN_DNS=yes
- REGISTER_NRF=yes
- NRF_IPV4_ADDRESS=192.168.70.130
- NRF_FQDN=oai-nrf
depends_on:
- mysql
- oai-nrf
networks:
public_net:
ipv4_address: 192.168.70.136
oai-udm:
container_name: "oai-udm"
image: oaisoftwarealliance/oai-udm:v1.5.0
environment:
- TZ=Europe/Paris
- UDM_NAME=OAI_UDM
- SBI_IF_NAME=eth0
- REGISTER_NRF=yes
- USE_FQDN_DNS=yes
- UDR_IP_ADDRESS=192.168.70.136
- UDR_FQDN=oai-udr
- NRF_IPV4_ADDRESS=192.168.70.130
- NRF_FQDN=oai-nrf
depends_on:
- oai-udr
networks:
public_net:
ipv4_address: 192.168.70.137
oai-ausf:
container_name: "oai-ausf"
image: oaisoftwarealliance/oai-ausf:v1.5.0
environment:
- TZ=Europe/Paris
- AUSF_NAME=OAI_AUSF
- SBI_IF_NAME=eth0
- USE_FQDN_DNS=yes
- REGISTER_NRF=yes
- UDM_IP_ADDRESS=192.168.70.137
- UDM_FQDN=oai-udm
- NRF_IPV4_ADDRESS=192.168.70.130
- NRF_FQDN=oai-nrf
depends_on:
- oai-udm
networks:
public_net:
ipv4_address: 192.168.70.138
oai-nrf:
container_name: "oai-nrf"
image: oaisoftwarealliance/oai-nrf:v1.5.0
environment:
- TZ=Europe/Paris
- NRF_INTERFACE_NAME_FOR_SBI=eth0
networks:
public_net:
ipv4_address: 192.168.70.130
oai-amf:
container_name: "oai-amf"
image: oaisoftwarealliance/oai-amf:v1.5.0
environment:
- TZ=Europe/paris
- MCC=208
- MNC=95
- REGION_ID=128
- AMF_SET_ID=1
- SERVED_GUAMI_MCC_0=208
- SERVED_GUAMI_MNC_0=95
- SERVED_GUAMI_REGION_ID_0=128
- SERVED_GUAMI_AMF_SET_ID_0=1
- SERVED_GUAMI_MCC_1=460
- SERVED_GUAMI_MNC_1=11
- SERVED_GUAMI_REGION_ID_1=10
- SERVED_GUAMI_AMF_SET_ID_1=1
- PLMN_SUPPORT_MCC=208
- PLMN_SUPPORT_MNC=95
- PLMN_SUPPORT_TAC=0xa000
# Slice 0 (222, 123)
- SST_0=222
- SD_0=123
# Slice 0 (128, 12)
- SST_1=128
- SD_1=12
- AMF_INTERFACE_NAME_FOR_NGAP=eth0
- AMF_INTERFACE_NAME_FOR_N11=eth0
# One single SMF instance
- SMF_INSTANCE_ID_0=1
- SMF_FQDN_0=oai-smf
- SMF_IPV4_ADDR_0=192.168.70.133
- SELECTED_0=true
- NF_REGISTRATION=yes
- SMF_SELECTION=yes
- USE_FQDN_DNS=yes
- EXTERNAL_AUSF=yes
- EXTERNAL_UDM=no
- EXTERNAL_NSSF=no
- NRF_IPV4_ADDRESS=192.168.70.130
- NRF_FQDN=oai-nrf
- AUSF_IPV4_ADDRESS=192.168.70.138
- AUSF_FQDN=oai-ausf
- UDM_IPV4_ADDRESS=192.168.70.137
- UDM_FQDN=oai-udm
- INT_ALGO_LIST=[ "NIA1" , "NIA2"]
- CIPH_ALGO_LIST=[ "NEA1" , "NEA2"]
depends_on:
- mysql
- vpp-upf
- oai-ext-dn
- oai-ausf
networks:
public_net:
ipv4_address: 192.168.70.132
oai-smf:
container_name: "oai-smf"
image: oaisoftwarealliance/oai-smf:v1.5.0
environment:
- TZ=Europe/Paris
- SMF_INTERFACE_NAME_FOR_N4=eth0
- SMF_INTERFACE_NAME_FOR_SBI=eth0
- DEFAULT_DNS_IPV4_ADDRESS=172.21.3.100
- DEFAULT_DNS_SEC_IPV4_ADDRESS=8.8.8.8
- AMF_IPV4_ADDRESS=192.168.70.132
- AMF_FQDN=oai-amf
- UDM_IPV4_ADDRESS=192.168.70.137
- UDM_FQDN=oai-udm
- UPF_IPV4_ADDRESS=192.168.70.201
- UPF_FQDN_0=vpp-upf.node.5gcn.mnc95.mcc208.3gppnetwork.org
- NRF_IPV4_ADDRESS=192.168.70.130
- NRF_FQDN=oai-nrf
- DEFAULT_CSCF_IPV4_ADDRESS=127.0.0.1 # only needed when ims is being used
- USE_LOCAL_SUBSCRIPTION_INFO=yes #Set to yes if SMF uses local subscription information instead of from an UDM
- REGISTER_NRF=yes
- DISCOVER_UPF=yes
- USE_FQDN_DNS=yes
- USE_NETWORK_INSTANCE=yes
- ENABLE_USAGE_REPORTING=yes
# Slice 0 (1, 0xFFFFFF)
- DNN_NI0=oai
- TYPE0=IPv4
- DNN_RANGE0=12.1.1.151 - 12.1.1.253
- NSSAI_SST0=1
- SESSION_AMBR_UL0=200Mbps
- SESSION_AMBR_DL0=400Mbps
# Slice 1 (1, 1)
- DNN_NI1=oai.ipv4
- TYPE1=IPv4
- DNN_RANGE1=12.1.1.51 - 12.1.1.150
- NSSAI_SST1=1
- NSSAI_SD1=1
- SESSION_AMBR_UL1=100Mbps
- SESSION_AMBR_DL1=200Mbps
# Slice 2 (222, 123)
- DNN_NI2=default
- TYPE2=IPv4
- DNN_RANGE2=12.1.1.2 - 12.1.1.50
- NSSAI_SST2=222
- NSSAI_SD2=123
- SESSION_AMBR_UL2=50Mbps
- SESSION_AMBR_DL2=100Mbps
# Slice 3 for ims
- DNN_NI3=ims
- TYPE3=IPv4v6
- DNN_RANGE3=14.1.1.2 - 14.1.1.253
extra_hosts:
- "vpp-upf.node.5gcn.mnc95.mcc208.3gppnetwork.org:192.168.70.201"
depends_on:
- oai-amf
networks:
public_net:
ipv4_address: 192.168.70.133
vpp-upf:
privileged: true
container_name: "vpp-upf"
image: oaisoftwarealliance/oai-upf-vpp:v1.5.0
environment:
- IF_1_IP=192.168.70.201
- IF_1_TYPE=N4
- IF_2_IP=192.168.72.201
- IF_2_TYPE=N3
- IF_2_NWI=access.oai.org
- IF_3_IP=192.168.73.201
- IF_3_TYPE=N6
- IF_3_IP_REMOTE=192.168.73.135 # EXT-DN IP Address
- IF_3_NWI=internet.oai.org
- NAME=VPP-UPF
- MNC=95
- MCC=208
- REALM=3gppnetwork.org
- VPP_MAIN_CORE=0
- VPP_CORE_WORKER=1
# - VPP_PLUGIN_PATH=/usr/lib64/vpp_plugins/ # RHEL7
- VPP_PLUGIN_PATH=/usr/lib/x86_64-linux-gnu/vpp_plugins/ # Ubntu18.04
- SNSSAI_SD=123
- SNSSAI_SST=222
- DNN=default
- REGISTER_NRF=yes
- NRF_IP_ADDR=192.168.70.130
- NRF_PORT=80
- HTTP_VERSION=1
depends_on:
- oai-nrf
healthcheck:
test: /bin/bash -c "pgrep vpp"
interval: 10s
timeout: 5s
retries: 5
networks:
public_net:
ipv4_address: 192.168.70.134
public_net_access:
ipv4_address: 192.168.72.134
public_net_core:
ipv4_address: 192.168.73.134
oai-ext-dn:
privileged: true
init: true
container_name: "oai-ext-dn"
image: oaisoftwarealliance/trf-gen-cn5g:latest
entrypoint: /bin/bash -c \
"iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE;"\
"ip route add 12.1.1.0/24 via 192.168.73.201 dev eth0; ip route; sleep infinity"
command: ["/bin/bash", "-c", "trap : SIGTERM SIGINT; sleep infinity & wait"]
healthcheck:
test: /bin/bash -c "iptables -L -t nat | grep MASQUERADE"
interval: 10s
timeout: 5s
retries: 5
networks:
public_net_core:
ipv4_address: 192.168.73.135
ueransim:
container_name: ueransim
image: ueransim:latest
privileged: true
environment:
# GNB Congig Parameters
- MCC=208
- MNC=95
- NCI=0x000000010
- TAC=0xa000
- LINK_IP=192.168.70.141
- NGAP_IP=192.168.70.141
- GTP_IP=192.168.72.141
- NGAP_PEER_IP=192.168.70.132
- SST=222
- SD=123
- IGNORE_STREAM_IDS=true
# UE Config Parameters
- NUMBER_OF_UE=1
- IMSI=208950000000041
- KEY=0C0A34601D4F07677303652C0462535B
- OP=63bfa50ee6523365ff14c1f45f88737d
- OP_TYPE=OPC
- AMF_VALUE=8000
- IMEI=356938035643803
- IMEI_SV=0035609204079514
- GNB_IP_ADDRESS=192.168.70.141
- PDU_TYPE=IPv4
- APN=default
- SST_0=222
- SD_0=123
- SST_C=222
- SD_C=123
- SST_D=222
- SD_D=123
healthcheck:
test: /bin/bash -c "ifconfig uesimtun0"
interval: 10s
timeout: 5s
retries: 5
networks:
public_net:
ipv4_address: 192.168.70.141
public_net_access:
ipv4_address: 192.168.72.141
networks:
public_net:
driver: overlay
name: demo-oai-public-net
ipam:
config:
- subnet: 192.168.70.0/24
driver_opts:
com.docker.network.bridge.name: "demo-oai"
public_net_access:
driver: overlay
name: oai-public-access
ipam:
config:
- subnet: 192.168.72.0/24
driver_opts:
com.docker.network.bridge.name: "cn5g-access"
public_net_core:
driver: overlay
name: oai-public-core
ipam:
config:
- subnet: 192.168.73.0/24
driver_opts:
com.docker.network.bridge.name: "cn5g-core"
启动1.yaml：
lab@lab-virtual-machine:~/oai-cn5g-fed/docker-compose$ docker stack deploy --compose-file 1.yaml oaiue
WARNING: Error loading config file: /home/lab/.docker/config.json: open /home/lab/.docker/config.json: permission denied
Ignoring unsupported options: privileged
Ignoring deprecated options:
container_name: Setting the container name is not supported.
Creating network oai-public-access
Creating network oai-public-core
Creating network demo-oai-public-net
Creating service oaiue_ueransim
Creating service oaiue_vpp-upf
Creating service oaiue_oai-amf
Creating service oaiue_oai-udr
Creating service oaiue_oai-udm
Creating service oaiue_oai-ausf
Creating service oaiue_oai-nrf
Creating service oaiue_mysql
Creating service oaiue_oai-ext-dn
Creating service oaiue_oai-smf
这样运行的容器乱七八糟的分布在多个机器上，需要通过如下方式指定不同容器运行的节点：
利用docker-compose.yml文件中的deploy.placement.constraints属性
#指定manage或worker
deploy:
replicas: 1
placement:
constraints:
## 常用方式指定 manager或者work节点
- node.role == worker
#指定主机名
deploy:
replicas: 1
placement:
constraints:
- node.hostname == docker1
指定标签的方法：
node label 可以给一个或多个机器打上一个标签，然后再compose文件中指定节点标签，就可以部署在打上对应标签的机器上。具体步骤如下：
现在主节点机器上对所有节点机器进行打标签操作，命令如下：
docker node update --label-add role=标签名称 主机名
例如我要对一个主机名为docker1的机器打上一个标签为db的标签
docker node update --label-add role=db docker1
deploy:
replicas: 1
placement:
constraints:
- node.labels.role == db
除此以外还有指定节点ID的方式
这里我采用指定manager与worker的方式：
将核心网部署在manager节点上，将ueransim部署在worker节点上
version: '3.8'
services:
mysql:
container_name: "mysql"
image: mysql:8.0
volumes:
- ./database/oai_db2.sql:/docker-entrypoint-initdb.d/oai_db.sql
- ./healthscripts/mysql-healthcheck2.sh:/tmp/mysql-healthcheck.sh
environment:
- TZ=Europe/Paris
- MYSQL_DATABASE=oai_db
- MYSQL_USER=test
- MYSQL_PASSWORD=test
- MYSQL_ROOT_PASSWORD=linux
healthcheck:
test: /bin/bash -c "/tmp/mysql-healthcheck.sh"
interval: 10s
timeout: 5s
retries: 30
networks:
public_net:
ipv4_address: 192.168.70.131
deploy:
mode: global
placement:
constraints:
- node.role == manager
oai-udr:
container_name: "oai-udr"
image: oaisoftwarealliance/oai-udr:v1.5.0
environment:
- TZ=Europe/Paris
- UDR_NAME=OAI_UDR
- UDR_INTERFACE_NAME_FOR_NUDR=eth0
- MYSQL_IPV4_ADDRESS=192.168.70.131
- MYSQL_USER=test
- MYSQL_PASS=test
- MYSQL_DB=oai_db
- WAIT_MYSQL=120
- USE_FQDN_DNS=yes
- REGISTER_NRF=yes
- NRF_IPV4_ADDRESS=192.168.70.130
- NRF_FQDN=oai-nrf
depends_on:
- mysql
- oai-nrf
networks:
public_net:
ipv4_address: 192.168.70.136
deploy:
mode: global
placement:
constraints:
- node.role == manager
oai-udm:
container_name: "oai-udm"
image: oaisoftwarealliance/oai-udm:v1.5.0
environment:
- TZ=Europe/Paris
- UDM_NAME=OAI_UDM
- SBI_IF_NAME=eth0
- REGISTER_NRF=yes
- USE_FQDN_DNS=yes
- UDR_IP_ADDRESS=192.168.70.136
- UDR_FQDN=oai-udr
- NRF_IPV4_ADDRESS=192.168.70.130
- NRF_FQDN=oai-nrf
depends_on:
- oai-udr
networks:
public_net:
ipv4_address: 192.168.70.137
deploy:
mode: global
placement:
constraints:
- node.role == manager
oai-ausf:
container_name: "oai-ausf"
image: oaisoftwarealliance/oai-ausf:v1.5.0
environment:
- TZ=Europe/Paris
- AUSF_NAME=OAI_AUSF
- SBI_IF_NAME=eth0
- USE_FQDN_DNS=yes
- REGISTER_NRF=yes
- UDM_IP_ADDRESS=192.168.70.137
- UDM_FQDN=oai-udm
- NRF_IPV4_ADDRESS=192.168.70.130
- NRF_FQDN=oai-nrf
depends_on:
- oai-udm
networks:
public_net:
ipv4_address: 192.168.70.138
deploy:
mode: global
placement:
constraints:
- node.role == manager
oai-nrf:
container_name: "oai-nrf"
image: oaisoftwarealliance/oai-nrf:v1.5.0
environment:
- TZ=Europe/Paris
- NRF_INTERFACE_NAME_FOR_SBI=eth0
networks:
public_net:
ipv4_address: 192.168.70.130
deploy:
mode: global
placement:
constraints:
- node.role == manager
oai-amf:
container_name: "oai-amf"
image: oaisoftwarealliance/oai-amf:v1.5.0
environment:
- TZ=Europe/paris
- MCC=208
- MNC=95
- REGION_ID=128
- AMF_SET_ID=1
- SERVED_GUAMI_MCC_0=208
- SERVED_GUAMI_MNC_0=95
- SERVED_GUAMI_REGION_ID_0=128
- SERVED_GUAMI_AMF_SET_ID_0=1
- SERVED_GUAMI_MCC_1=460
- SERVED_GUAMI_MNC_1=11
- SERVED_GUAMI_REGION_ID_1=10
- SERVED_GUAMI_AMF_SET_ID_1=1
- PLMN_SUPPORT_MCC=208
- PLMN_SUPPORT_MNC=95
- PLMN_SUPPORT_TAC=0xa000
# Slice 0 (222, 123)
- SST_0=222
- SD_0=123
# Slice 0 (128, 12)
- SST_1=128
- SD_1=12
- AMF_INTERFACE_NAME_FOR_NGAP=eth0
- AMF_INTERFACE_NAME_FOR_N11=eth0
# One single SMF instance
- SMF_INSTANCE_ID_0=1
- SMF_FQDN_0=oai-smf
- SMF_IPV4_ADDR_0=192.168.70.133
- SELECTED_0=true
- NF_REGISTRATION=yes
- SMF_SELECTION=yes
- USE_FQDN_DNS=yes
- EXTERNAL_AUSF=yes
- EXTERNAL_UDM=no
- EXTERNAL_NSSF=no
- NRF_IPV4_ADDRESS=192.168.70.130
- NRF_FQDN=oai-nrf
- AUSF_IPV4_ADDRESS=192.168.70.138
- AUSF_FQDN=oai-ausf
- UDM_IPV4_ADDRESS=192.168.70.137
- UDM_FQDN=oai-udm
- INT_ALGO_LIST=[ "NIA1" , "NIA2"]
- CIPH_ALGO_LIST=[ "NEA1" , "NEA2"]
depends_on:
- mysql
- vpp-upf
- oai-ext-dn
- oai-ausf
networks:
public_net:
ipv4_address: 192.168.70.132
deploy:
mode: global
placement:
constraints:
- node.role == manager
oai-smf:
container_name: "oai-smf"
image: oaisoftwarealliance/oai-smf:v1.5.0
environment:
- TZ=Europe/Paris
- SMF_INTERFACE_NAME_FOR_N4=eth0
- SMF_INTERFACE_NAME_FOR_SBI=eth0
- DEFAULT_DNS_IPV4_ADDRESS=172.21.3.100
- DEFAULT_DNS_SEC_IPV4_ADDRESS=8.8.8.8
- AMF_IPV4_ADDRESS=192.168.70.132
- AMF_FQDN=oai-amf
- UDM_IPV4_ADDRESS=192.168.70.137
- UDM_FQDN=oai-udm
- UPF_IPV4_ADDRESS=192.168.70.201
- UPF_FQDN_0=vpp-upf.node.5gcn.mnc95.mcc208.3gppnetwork.org
- NRF_IPV4_ADDRESS=192.168.70.130
- NRF_FQDN=oai-nrf
- DEFAULT_CSCF_IPV4_ADDRESS=127.0.0.1 # only needed when ims is being used
- USE_LOCAL_SUBSCRIPTION_INFO=yes #Set to yes if SMF uses local subscription information instead of from an UDM
- REGISTER_NRF=yes
- DISCOVER_UPF=yes
- USE_FQDN_DNS=yes
- USE_NETWORK_INSTANCE=yes
- ENABLE_USAGE_REPORTING=yes
# Slice 0 (1, 0xFFFFFF)
- DNN_NI0=oai
- TYPE0=IPv4
- DNN_RANGE0=12.1.1.151 - 12.1.1.253
- NSSAI_SST0=1
- SESSION_AMBR_UL0=200Mbps
- SESSION_AMBR_DL0=400Mbps
# Slice 1 (1, 1)
- DNN_NI1=oai.ipv4
- TYPE1=IPv4
- DNN_RANGE1=12.1.1.51 - 12.1.1.150
- NSSAI_SST1=1
- NSSAI_SD1=1
- SESSION_AMBR_UL1=100Mbps
- SESSION_AMBR_DL1=200Mbps
# Slice 2 (222, 123)
- DNN_NI2=default
- TYPE2=IPv4
- DNN_RANGE2=12.1.1.2 - 12.1.1.50
- NSSAI_SST2=222
- NSSAI_SD2=123
- SESSION_AMBR_UL2=50Mbps
- SESSION_AMBR_DL2=100Mbps
# Slice 3 for ims
- DNN_NI3=ims
- TYPE3=IPv4v6
- DNN_RANGE3=14.1.1.2 - 14.1.1.253
extra_hosts:
- "vpp-upf.node.5gcn.mnc95.mcc208.3gppnetwork.org:192.168.70.201"
depends_on:
- oai-amf
networks:
public_net:
ipv4_address: 192.168.70.133
deploy:
mode: global
placement:
constraints:
- node.role == manager
vpp-upf:
privileged: true
container_name: "vpp-upf"
image: oaisoftwarealliance/oai-upf-vpp:v1.5.0
environment:
- IF_1_IP=192.168.70.201
- IF_1_TYPE=N4
- IF_2_IP=192.168.72.201
- IF_2_TYPE=N3
- IF_2_NWI=access.oai.org
- IF_3_IP=192.168.73.201
- IF_3_TYPE=N6
- IF_3_IP_REMOTE=192.168.73.135 # EXT-DN IP Address
- IF_3_NWI=internet.oai.org
- NAME=VPP-UPF
- MNC=95
- MCC=208
- REALM=3gppnetwork.org
- VPP_MAIN_CORE=0
- VPP_CORE_WORKER=1
# - VPP_PLUGIN_PATH=/usr/lib64/vpp_plugins/ # RHEL7
- VPP_PLUGIN_PATH=/usr/lib/x86_64-linux-gnu/vpp_plugins/ # Ubntu18.04
- SNSSAI_SD=123
- SNSSAI_SST=222
- DNN=default
- REGISTER_NRF=yes
- NRF_IP_ADDR=192.168.70.130
- NRF_PORT=80
- HTTP_VERSION=1
depends_on:
- oai-nrf
healthcheck:
test: /bin/bash -c "pgrep vpp"
interval: 10s
timeout: 5s
retries: 5
networks:
public_net:
ipv4_address: 192.168.70.134
public_net_access:
ipv4_address: 192.168.72.134
public_net_core:
ipv4_address: 192.168.73.134
deploy:
mode: global
placement:
constraints:
- node.role == manager
oai-ext-dn:
privileged: true
init: true
container_name: "oai-ext-dn"
image: oaisoftwarealliance/trf-gen-cn5g:latest
entrypoint: /bin/bash -c \
"iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE;"\
"ip route add 12.1.1.0/24 via 192.168.73.201 dev eth0; ip route; sleep infinity"
command: ["/bin/bash", "-c", "trap : SIGTERM SIGINT; sleep infinity & wait"]
healthcheck:
test: /bin/bash -c "iptables -L -t nat | grep MASQUERADE"
interval: 10s
timeout: 5s
retries: 5
networks:
public_net_core:
ipv4_address: 192.168.73.135
deploy:
mode: global
placement:
constraints:
- node.role == manager
ueransim:
container_name: ueransim
image: ueransim:latest
privileged: true
environment:
# GNB Congig Parameters
- MCC=208
- MNC=95
- NCI=0x000000010
- TAC=0xa000
- LINK_IP=192.168.70.141
- NGAP_IP=192.168.70.141
- GTP_IP=192.168.72.141
- NGAP_PEER_IP=192.168.70.132
- SST=222
- SD=123
- IGNORE_STREAM_IDS=true
# UE Config Parameters
- NUMBER_OF_UE=1
- IMSI=208950000000041
- KEY=0C0A34601D4F07677303652C0462535B
- OP=63bfa50ee6523365ff14c1f45f88737d
- OP_TYPE=OPC
- AMF_VALUE=8000
- IMEI=356938035643803
- IMEI_SV=0035609204079514
- GNB_IP_ADDRESS=192.168.70.141
- PDU_TYPE=IPv4
- APN=default
- SST_0=222
- SD_0=123
- SST_C=222
- SD_C=123
- SST_D=222
- SD_D=123
healthcheck:
test: /bin/bash -c "ifconfig uesimtun0"
interval: 10s
timeout: 5s
retries: 5
networks:
public_net:
ipv4_address: 192.168.70.141
public_net_access:
ipv4_address: 192.168.72.141
deploy:
mode: global
placement:
constraints:
- node.role == worker
networks:
public_net:
driver: overlay
name: demo-oai-public-net
ipam:
config:
- subnet: 192.168.70.0/24
driver_opts:
com.docker.network.bridge.name: "demo-oai"
public_net_access:
driver: overlay
name: oai-public-access
ipam:
config:
- subnet: 192.168.72.0/24
driver_opts:
com.docker.network.bridge.name: "cn5g-access"
public_net_core:
driver: overlay
name: oai-public-core
ipam:
config:
- subnet: 192.168.73.0/24
driver_opts:
com.docker.network.bridge.name: "cn5g-core"
经过较长时间的等待后，启动结果如下：
核心网网元正常启动，ueransim未成功
对ueransim进行重启：
docker service ls
docker service update --force 07ms7mdsk907
重启后依旧无法接入
将ueransim与核心网都部署到服务器A上，依旧同样的报错
再尝试将所有容器全部部署到主机B上
依旧不行
结点服务的可视化界面管理visualizer：
docker pull dockersamples/visualizer:latest
docker run -itd --name visualizer -p 8099:8080 -e HOST=192.168.12.3 -e PORT=8080 -v /var/run/docker.sock:/var/run/docker.sock dockersamples/visualizer:latest
然后在浏览器输入 192.168.12.3：8090访问
reference1
reference2
reference3：docker-compose+swarm
