如此简单的K8S,来玩下pv和pvc，利用nfs来实现持久化存储(内网环境，非常详细)
k8s很简单，怎么个简单法呢，来给小编一起再来复习一边吧。今天主要来了解下pv和pvc的概念，小编也是当过多次的面试官，小编悄悄的告诉你这个pv和pvc的区别也是小编经常面试的时候提问的哈，多么巧妙的问题，有没有接触过k8s，一问也就露馅了，好了，下面开始进入正题
当然为了更加友好的区分pv和pvc，咱们借助一下最简单的nfs存储，不要问小编为什么要用nfs而不用其他的呢，主要是因为“懒”，懒可以产生一切。
稿件非常不易，一直写到了半夜3点，还请大家多多支持，谢谢
| 主机名 | 系统版本 | 作用 | ip | 备注 |
|---|---|---|---|---|
| localhost | centos7.5 | 捞取nfs得rpm包和pull镜像打包镜像的机器 | 192.168.3.129 | 确保可以联网，捞取下来rpm包后，上传到内网得Nfs服务器上面进行rpm安装，确保和nfs系统版本是一致得哈 |
| nfs-a | centos7.5 | nfs服务器 | 10.245.4.5 | 专门用来存储代码等数据的存储服务器 |
| k8s-master1 | centos7.5 | k8s-master1 | 10.245.4.1 | k8s的master节点 |
| k8s-node1 | centos7.5 | k8s-node1 | 10.245.4.3 | k8s的node节点1 |
| k8s-node2 | centos7.5 | k8s-node2 | 10.245.4.4 | k8s的node节点2 |
小编本稿件所涉及的文件资料下载地址
链接：https://pan.baidu.com/s/1z--LApr2kMbGVBnN28YnNg
提取码：72uk
--来自百度网盘超级会员V4的分享
- pv 持久化存储，对存储的资源对外提供一个可以调用的地方或者方法
- pvc:从字面上来看就很容易明白，就是pv的client端，只需要关系调用pvc，具体怎么实现的我不关心，
- 简单的理解就是：pv可以理解成是一个生产者，pvc可以理解为消费者
- 先简单理解下吧，跟着小编做到最后就明白了
实现流程：
搭建部署nfs服务器
nfs不陌生了吧，可以想象一下，专门有台nfs服务器，来存放代码数据，然后创建的每一个pod都挂载到这个nfs上面，无论pod是怎么变，变化后pod也会自动的连接到咱的这个Nfs上面来，是不是也是一种持久性存储了呢，当然除了nfs存储之外还有好多的fs存储的，这里小编也就简单的用nfs来进行演示了
安装部署nfs-server服务器
如果是生产环境的话，都是内网环境，可是不是直接使用yum安装的，可以找一台自己的虚拟机，部署一个和自己生产环境一样的虚机，利用yum只下载不安装的方法，把所需要的rpm拿到手，然后可以拷贝到内网环境中，利用rpm或者yum localinstall来进行安装，如果大家有什么不懂得可以看下我的另一个博客，网址在下面
yum不安装只下载rpm包（含centos6,centos7)
利用yum将rpm下载下来
首先我们找一台可以上网得服务器，可以是自己得vmware虚拟机哈，因为咱们也不敢啥，就是yum下载一个包而已
WARNING! The remote SSH server rejected X11 forwarding request.
Last login: Fri Dec 2 10:03:26 2022
[root@localhost ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 00:0c:29:b2:7a:10 brd ff:ff:ff:ff:ff:ff
inet 192.168.3.129/24 brd 192.168.3.255 scope global noprefixroute dynamic ens33
valid_lft 1674sec preferred_lft 1674sec
inet6 fe80::4606:242f:9dcc:30ca/64 scope link noprefixroute
valid_lft forever preferred_lft forever
[root@localhost ~]# ping www.baidu.com
PING www.a.shifen.com (220.181.38.149) 56(84) bytes of data.
64 bytes from 220.181.38.149 (220.181.38.149): icmp_seq=1 ttl=128 time=77.5 ms
64 bytes from 220.181.38.149 (220.181.38.149): icmp_seq=2 ttl=128 time=59.5 ms
64 bytes from 220.181.38.149 (220.181.38.149): icmp_seq=3 ttl=128 time=18.8 ms
创建下载目录
[root@localhost ~]# mkdir /nfs
[root@localhost ~]# ll /nfs
total 0
安装yum下载插件
[root@localhost ~]# yum install yum-plugin-downloadonly ###少写了一个-y得参数，-y就是一路确认得意思，下面就不用使用y来进行确认了
Loaded plugins: fastestmirror
Determining fastest mirrors
* base: mirrors.bfsu.edu.cn
* extras: mirrors.bfsu.edu.cn
* updates: mirrors.huaweicloud.com
base | 3.6 kB 00:00:00
extras | 2.9 kB 00:00:00
updates | 2.9 kB 00:00:00
(1/4): base/7/x86_64/group_gz | 153 kB 00:00:00
(2/4): extras/7/x86_64/primary_db | 249 kB 00:00:01
(3/4): base/7/x86_64/primary_db | 6.1 MB 00:00:06
(4/4): updates/7/x86_64/primary_db | 18 MB 00:00:10
Resolving Dependencies
--> Running transaction check
---> Package yum.noarch 0:3.4.3-158.el7.centos will be updated
---> Package yum.noarch 0:3.4.3-168.el7.centos will be an update
--> Finished Dependency Resolution
Dependencies Resolved
========================================================================================================================================================================
Package Arch Version Repository Size
========================================================================================================================================================================
Updating:
yum noarch 3.4.3-168.el7.centos base 1.2 M
Transaction Summary
========================================================================================================================================================================
Upgrade 1 Package
Total download size: 1.2 M
Is this ok [y/d/N]: y
Downloading packages:
Delta RPMs disabled because /usr/bin/applydeltarpm not installed.
warning: /var/cache/yum/x86_64/7/base/packages/yum-3.4.3-168.el7.centos.noarch.rpm: Header V3 RSA/SHA256 Signature, key ID f4a80eb5: NOKEY.0 B/s | 634 kB --:--:-- ETA
Public key for yum-3.4.3-168.el7.centos.noarch.rpm is not installed
yum-3.4.3-168.el7.centos.noarch.rpm | 1.2 MB 00:00:00
Retrieving key from file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
Importing GPG key 0xF4A80EB5:
Userid : "CentOS-7 Key (CentOS 7 Official Signing Key) <security@centos.org>"
Fingerprint: 6341 ab27 53d7 8a78 a7c2 7bb1 24c6 a8a7 f4a8 0eb5
Package : centos-release-7-5.1804.el7.centos.x86_64 (@anaconda)
From : /etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
Is this ok [y/N]: y
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Updating : yum-3.4.3-168.el7.centos.noarch 1/2
Cleanup : yum-3.4.3-158.el7.centos.noarch 2/2
Verifying : yum-3.4.3-168.el7.centos.noarch 1/2
Verifying : yum-3.4.3-158.el7.centos.noarch 2/2
Updated:
yum.noarch 0:3.4.3-168.el7.centos
Complete!
进行rpm包得下载
[root@localhost ~]# yum install --downloadonly --downloaddir=/nfs ntp-utils -y ###进行只下载rpm不安装
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
* base: mirrors.bfsu.edu.cn
* extras: mirrors.bfsu.edu.cn
* updates: mirrors.huaweicloud.com
No package ntp-utils available.
Error: Nothing to do
[root@localhost ~]# yum install --downloadonly --downloaddir=/nfs nfs-utils -y
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
* base: mirrors.bfsu.edu.cn
* extras: mirrors.bfsu.edu.cn
* updates: mirrors.huaweicloud.com
Resolving Dependencies
--> Running transaction check
---> Package nfs-utils.x86_64 1:1.3.0-0.68.el7.2 will be installed
--> Processing Dependency: libtirpc >= 0.2.4-0.7 for package: 1:nfs-utils-1.3.0-0.68.el7.2.x86_64
--> Processing Dependency: gssproxy >= 0.7.0-3 for package: 1:nfs-utils-1.3.0-0.68.el7.2.x86_64
--> Processing Dependency: rpcbind for package: 1:nfs-utils-1.3.0-0.68.el7.2.x86_64
--> Processing Dependency: quota for package: 1:nfs-utils-1.3.0-0.68.el7.2.x86_64
--> Processing Dependency: libnfsidmap for package: 1:nfs-utils-1.3.0-0.68.el7.2.x86_64
--> Processing Dependency: libevent for package: 1:nfs-utils-1.3.0-0.68.el7.2.x86_64
--> Processing Dependency: keyutils for package: 1:nfs-utils-1.3.0-0.68.el7.2.x86_64
--> Processing Dependency: libtirpc.so.1()(64bit) for package: 1:nfs-utils-1.3.0-0.68.el7.2.x86_64
--> Processing Dependency: libnfsidmap.so.0()(64bit) for package: 1:nfs-utils-1.3.0-0.68.el7.2.x86_64
--> Processing Dependency: libevent-2.0.so.5()(64bit) for package: 1:nfs-utils-1.3.0-0.68.el7.2.x86_64
--> Running transaction check
---> Package gssproxy.x86_64 0:0.7.0-30.el7_9 will be installed
--> Processing Dependency: libini_config >= 1.3.1-31 for package: gssproxy-0.7.0-30.el7_9.x86_64
--> Processing Dependency: libverto-module-base for package: gssproxy-0.7.0-30.el7_9.x86_64
--> Processing Dependency: libref_array.so.1(REF_ARRAY_0.1.1)(64bit) for package: gssproxy-0.7.0-30.el7_9.x86_64
--> Processing Dependency: libini_config.so.3(INI_CONFIG_1.2.0)(64bit) for package: gssproxy-0.7.0-30.el7_9.x86_64
--> Processing Dependency: libini_config.so.3(INI_CONFIG_1.1.0)(64bit) for package: gssproxy-0.7.0-30.el7_9.x86_64
--> Processing Dependency: libref_array.so.1()(64bit) for package: gssproxy-0.7.0-30.el7_9.x86_64
--> Processing Dependency: libini_config.so.3()(64bit) for package: gssproxy-0.7.0-30.el7_9.x86_64
--> Processing Dependency: libcollection.so.2()(64bit) for package: gssproxy-0.7.0-30.el7_9.x86_64
--> Processing Dependency: libbasicobjects.so.0()(64bit) for package: gssproxy-0.7.0-30.el7_9.x86_64
---> Package keyutils.x86_64 0:1.5.8-3.el7 will be installed
---> Package libevent.x86_64 0:2.0.21-4.el7 will be installed
---> Package libnfsidmap.x86_64 0:0.25-19.el7 will be installed
---> Package libtirpc.x86_64 0:0.2.4-0.16.el7 will be installed
---> Package quota.x86_64 1:4.01-19.el7 will be installed
--> Processing Dependency: quota-nls = 1:4.01-19.el7 for package: 1:quota-4.01-19.el7.x86_64
--> Processing Dependency: tcp_wrappers for package: 1:quota-4.01-19.el7.x86_64
---> Package rpcbind.x86_64 0:0.2.0-49.el7 will be installed
--> Running transaction check
---> Package libbasicobjects.x86_64 0:0.1.1-32.el7 will be installed
---> Package libcollection.x86_64 0:0.7.0-32.el7 will be installed
---> Package libini_config.x86_64 0:1.3.1-32.el7 will be installed
--> Processing Dependency: libpath_utils.so.1(PATH_UTILS_0.2.1)(64bit) for package: libini_config-1.3.1-32.el7.x86_64
--> Processing Dependency: libpath_utils.so.1()(64bit) for package: libini_config-1.3.1-32.el7.x86_64
---> Package libref_array.x86_64 0:0.1.5-32.el7 will be installed
---> Package libverto-libevent.x86_64 0:0.2.5-4.el7 will be installed
---> Package quota-nls.noarch 1:4.01-19.el7 will be installed
---> Package tcp_wrappers.x86_64 0:7.6-77.el7 will be installed
--> Running transaction check
---> Package libpath_utils.x86_64 0:0.2.1-32.el7 will be installed
--> Finished Dependency Resolution
Dependencies Resolved
========================================================================================================================================================================
Package Arch Version Repository Size
========================================================================================================================================================================
Installing:
nfs-utils x86_64 1:1.3.0-0.68.el7.2 updates 413 k
Installing for dependencies:
gssproxy x86_64 0.7.0-30.el7_9 updates 111 k
keyutils x86_64 1.5.8-3.el7 base 54 k
libbasicobjects x86_64 0.1.1-32.el7 base 26 k
libcollection x86_64 0.7.0-32.el7 base 42 k
libevent x86_64 2.0.21-4.el7 base 214 k
libini_config x86_64 1.3.1-32.el7 base 64 k
libnfsidmap x86_64 0.25-19.el7 base 50 k
libpath_utils x86_64 0.2.1-32.el7 base 28 k
libref_array x86_64 0.1.5-32.el7 base 27 k
libtirpc x86_64 0.2.4-0.16.el7 base 89 k
libverto-libevent x86_64 0.2.5-4.el7 base 8.9 k
quota x86_64 1:4.01-19.el7 base 179 k
quota-nls noarch 1:4.01-19.el7 base 90 k
rpcbind x86_64 0.2.0-49.el7 base 60 k
tcp_wrappers x86_64 7.6-77.el7 base 78 k
Transaction Summary
========================================================================================================================================================================
Install 1 Package (+15 Dependent packages)
Total download size: 1.5 M
Installed size: 4.3 M
Background downloading packages, then exiting:
(1/16): gssproxy-0.7.0-30.el7_9.x86_64.rpm | 111 kB 00:00:00
(2/16): keyutils-1.5.8-3.el7.x86_64.rpm | 54 kB 00:00:00
(3/16): libnfsidmap-0.25-19.el7.x86_64.rpm | 50 kB 00:00:00
(4/16): libpath_utils-0.2.1-32.el7.x86_64.rpm | 28 kB 00:00:00
(5/16): libref_array-0.1.5-32.el7.x86_64.rpm | 27 kB 00:00:00
(6/16): libtirpc-0.2.4-0.16.el7.x86_64.rpm | 89 kB 00:00:00
(7/16): libbasicobjects-0.1.1-32.el7.x86_64.rpm | 26 kB 00:00:00
(8/16): libverto-libevent-0.2.5-4.el7.x86_64.rpm | 8.9 kB 00:00:00
(9/16): libcollection-0.7.0-32.el7.x86_64.rpm | 42 kB 00:00:00
(10/16): libevent-2.0.21-4.el7.x86_64.rpm | 214 kB 00:00:00
(11/16): quota-4.01-19.el7.x86_64.rpm | 179 kB 00:00:00
(12/16): quota-nls-4.01-19.el7.noarch.rpm | 90 kB 00:00:00
(13/16): tcp_wrappers-7.6-77.el7.x86_64.rpm | 78 kB 00:00:00
(14/16): libini_config-1.3.1-32.el7.x86_64.rpm | 64 kB 00:00:00
(15/16): rpcbind-0.2.0-49.el7.x86_64.rpm | 60 kB 00:00:00
(16/16): nfs-utils-1.3.0-0.68.el7.2.x86_64.rpm | 413 kB 00:00:00
------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total 1.0 MB/s | 1.5 MB 00:00:01
exiting because "Download Only" specified
[root@localhost ~]#
将下载好得Nfs打包
[root@localhost ~]# cd /nfs ##到刚才得目录里面，可以看到下载好得rpm包
[root@localhost nfs]# ls ##下面得就是安装nfs得时候需要得所有得rpm包
gssproxy-0.7.0-30.el7_9.x86_64.rpm libevent-2.0.21-4.el7.x86_64.rpm libref_array-0.1.5-32.el7.x86_64.rpm quota-4.01-19.el7.x86_64.rpm
keyutils-1.5.8-3.el7.x86_64.rpm libini_config-1.3.1-32.el7.x86_64.rpm libtirpc-0.2.4-0.16.el7.x86_64.rpm quota-nls-4.01-19.el7.noarch.rpm
libbasicobjects-0.1.1-32.el7.x86_64.rpm libnfsidmap-0.25-19.el7.x86_64.rpm libverto-libevent-0.2.5-4.el7.x86_64.rpm rpcbind-0.2.0-49.el7.x86_64.rpm
libcollection-0.7.0-32.el7.x86_64.rpm libpath_utils-0.2.1-32.el7.x86_64.rpm nfs-utils-1.3.0-0.68.el7.2.x86_64.rpm tcp_wrappers-7.6-77.el7.x86_64.rpm
导入内网环境进行安装
导出得办法非常多，大家可以自由发挥
导出成功后
导入内网环境
进行安装nfs-server服务
[root@nfs-a nfs]# rpm -ivh *.rpm --nodeps --force
Preparing... ################################# [100%]
Updating / installing...
1:libtirpc-0.2.4-0.16.el7 ################################# [ 6%]
2:rpcbind-0.2.0-49.el7 ################################# [ 13%]
3:libref_array-0.1.5-32.el7 ################################# [ 19%]
4:libevent-2.0.21-4.el7 ################################# [ 25%]
5:libcollection-0.7.0-32.el7 ################################# [ 31%]
6:libbasicobjects-0.1.1-32.el7 ################################# [ 38%]
7:libverto-libevent-0.2.5-4.el7 ################################# [ 44%]
8:tcp_wrappers-7.6-77.el7 ################################# [ 50%]
9:quota-nls-1:4.01-19.el7 ################################# [ 56%]
10:quota-1:4.01-19.el7 ################################# [ 63%]
11:libpath_utils-0.2.1-32.el7 ################################# [ 69%]
12:libini_config-1.3.1-32.el7 ################################# [ 75%]
13:gssproxy-0.7.0-30.el7_9 ################################# [ 81%]
14:libnfsidmap-0.25-19.el7 ################################# [ 88%]
15:keyutils-1.5.8-3.el7 ################################# [ 94%]
16:nfs-utils-1:1.3.0-0.68.el7.2 ################################# [100%]
创建挂载目录
~··
[root@nfs-a nfs]# mkdir -p /data/nfs-a ###我们将用nfs-a这个目录来当作挂载目录
[root@nfs-a nfs]# ll /data/nfs-a/
total 0
···
设置挂载路径
[root@nfs-a nfs]# vi /etc/exports
[root@nfs-a nfs]# cat /etc/exports
/data/nfs-a *(rw,no_root_squash)
启动nfs
[root@nfs-a nfs]# systemctl start nfs
[root@nfs-a nfs]# ps -ef | grep nfs
root 1421 2 0 10:37 ? 00:00:00 [nfsd4_callbacks]
root 1427 2 0 10:37 ? 00:00:00 [nfsd]
root 1428 2 0 10:37 ? 00:00:00 [nfsd]
root 1429 2 0 10:37 ? 00:00:00 [nfsd]
root 1430 2 0 10:37 ? 00:00:00 [nfsd]
root 1431 2 0 10:37 ? 00:00:00 [nfsd]
root 1432 2 0 10:37 ? 00:00:00 [nfsd]
root 1433 2 0 10:37 ? 00:00:00 [nfsd]
root 1434 2 0 10:37 ? 00:00:00 [nfsd]
root 1445 1140 0 10:37 pts/0 00:00:00 grep --color=auto nfs
注意
因为咱们使用的是nfs服务，我们还需要部署一个client端，也就是说我们得每个node节点都将会是我们得nfs-client端，也就是说我们每个node得上面都是需要安装上面得操作步骤进行操作得
原因：因为pod会降落在每一个节点上面，特殊指定得除外得，也就是说有pod产生得节点并且还需要挂载nfs上得node节点上面，需要安装下，请知晓
nfs-server操作
[root@nfs-a nfs]# ls
gssproxy-0.7.0-30.el7_9.x86_64.rpm libnfsidmap-0.25-19.el7.x86_64.rpm quota-4.01-19.el7.x86_64.rpm
keyutils-1.5.8-3.el7.x86_64.rpm libpath_utils-0.2.1-32.el7.x86_64.rpm quota-nls-4.01-19.el7.noarch.rpm
libbasicobjects-0.1.1-32.el7.x86_64.rpm libref_array-0.1.5-32.el7.x86_64.rpm rpcbind-0.2.0-49.el7.x86_64.rpm
libcollection-0.7.0-32.el7.x86_64.rpm libtirpc-0.2.4-0.16.el7.x86_64.rpm tcp_wrappers-7.6-77.el7.x86_64.rpm
libevent-2.0.21-4.el7.x86_64.rpm libverto-libevent-0.2.5-4.el7.x86_64.rpm
libini_config-1.3.1-32.el7.x86_64.rpm nfs-utils-1.3.0-0.68.el7.2.x86_64.rpm
[root@nfs-a nfs]# cd ..
[root@nfs-a /]# ls
bin boot data dev etc home lib lib64 media mnt nfs nfs.zip opt proc root run sbin srv sys tmp usr var
[root@nfs-a /]# scp -r nfs root@10.245.4.3:/
The authenticity of host '10.245.4.3 (10.245.4.3)' can't be established.
ECDSA key fingerprint is SHA256:y/V4bFrdzFrKs18Wix5RF8uFRnYKYlRZ0nIqvkU37ns.
ECDSA key fingerprint is MD5:77:f4:d6:f5:09:47:b2:0f:fd:e7:33:36:9f:09:ca:20.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '10.245.4.3' (ECDSA) to the list of known hosts.
root@10.245.4.3's password:
gssproxy-0.7.0-30.el7_9.x86_64.rpm 100% 111KB 1.0MB/s 00:00
keyutils-1.5.8-3.el7.x86_64.rpm 100% 54KB 5.1MB/s 00:00
libnfsidmap-0.25-19.el7.x86_64.rpm 100% 50KB 15.0MB/s 00:00
libpath_utils-0.2.1-32.el7.x86_64.rpm 100% 28KB 13.0MB/s 00:00
libref_array-0.1.5-32.el7.x86_64.rpm 100% 27KB 16.4MB/s 00:00
libtirpc-0.2.4-0.16.el7.x86_64.rpm 100% 89KB 21.5MB/s 00:00
libbasicobjects-0.1.1-32.el7.x86_64.rpm 100% 26KB 1.2MB/s 00:00
libverto-libevent-0.2.5-4.el7.x86_64.rpm 100% 9108 5.9MB/s 00:00
libcollection-0.7.0-32.el7.x86_64.rpm 100% 42KB 18.5MB/s 00:00
libevent-2.0.21-4.el7.x86_64.rpm 100% 214KB 35.8MB/s 00:00
quota-4.01-19.el7.x86_64.rpm 100% 179KB 35.6MB/s 00:00
quota-nls-4.01-19.el7.noarch.rpm 100% 90KB 24.5MB/s 00:00
tcp_wrappers-7.6-77.el7.x86_64.rpm 100% 78KB 20.4MB/s 00:00
libini_config-1.3.1-32.el7.x86_64.rpm 100% 64KB 26.1MB/s 00:00
rpcbind-0.2.0-49.el7.x86_64.rpm 100% 60KB 21.7MB/s 00:00
nfs-utils-1.3.0-0.68.el7.2.x86_64.rpm 100% 413KB 37.5MB/s 00:00
[root@nfs-a /]# scp -r nfs root@10.245.4.4:/
The authenticity of host '10.245.4.4 (10.245.4.4)' can't be established.
ECDSA key fingerprint is SHA256:OXpfsq3hE07ex51FOrnJXjqirNMX18u8RCOcCAtuKvM.
ECDSA key fingerprint is MD5:38:c8:af:90:4a:bf:61:c9:26:32:25:c3:bb:47:42:6e.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '10.245.4.4' (ECDSA) to the list of known hosts.
root@10.245.4.4's password:
gssproxy-0.7.0-30.el7_9.x86_64.rpm 100% 111KB 4.8MB/s 00:00
keyutils-1.5.8-3.el7.x86_64.rpm 100% 54KB 726.9KB/s 00:00
libnfsidmap-0.25-19.el7.x86_64.rpm 100% 50KB 1.8MB/s 00:00
libpath_utils-0.2.1-32.el7.x86_64.rpm 100% 28KB 1.1MB/s 00:00
libref_array-0.1.5-32.el7.x86_64.rpm 100% 27KB 11.4MB/s 00:00
libtirpc-0.2.4-0.16.el7.x86_64.rpm 100% 89KB 20.9MB/s 00:00
libbasicobjects-0.1.1-32.el7.x86_64.rpm 100% 26KB 10.3MB/s 00:00
libverto-libevent-0.2.5-4.el7.x86_64.rpm 100% 9108 5.9MB/s 00:00
libcollection-0.7.0-32.el7.x86_64.rpm 100% 42KB 16.8MB/s 00:00
libevent-2.0.21-4.el7.x86_64.rpm 100% 214KB 10.2MB/s 00:00
quota-4.01-19.el7.x86_64.rpm 100% 179KB 22.3MB/s 00:00
quota-nls-4.01-19.el7.noarch.rpm 100% 90KB 25.8MB/s 00:00
tcp_wrappers-7.6-77.el7.x86_64.rpm 100% 78KB 26.1MB/s 00:00
libini_config-1.3.1-32.el7.x86_64.rpm 100% 64KB 22.2MB/s 00:00
rpcbind-0.2.0-49.el7.x86_64.rpm 100% 60KB 29.6MB/s 00:00
nfs-utils-1.3.0-0.68.el7.2.x86_64.rpm 100% 413KB 44.1MB/s 00:00
[root@nfs-a /]#
各个node上面操作
以Node2为列子
[root@k8s-node2 /]# cd /nfs/
[root@k8s-node2 nfs]# ls
gssproxy-0.7.0-30.el7_9.x86_64.rpm libnfsidmap-0.25-19.el7.x86_64.rpm quota-4.01-19.el7.x86_64.rpm
keyutils-1.5.8-3.el7.x86_64.rpm libpath_utils-0.2.1-32.el7.x86_64.rpm quota-nls-4.01-19.el7.noarch.rpm
libbasicobjects-0.1.1-32.el7.x86_64.rpm libref_array-0.1.5-32.el7.x86_64.rpm rpcbind-0.2.0-49.el7.x86_64.rpm
libcollection-0.7.0-32.el7.x86_64.rpm libtirpc-0.2.4-0.16.el7.x86_64.rpm tcp_wrappers-7.6-77.el7.x86_64.rpm
libevent-2.0.21-4.el7.x86_64.rpm libverto-libevent-0.2.5-4.el7.x86_64.rpm
libini_config-1.3.1-32.el7.x86_64.rpm nfs-utils-1.3.0-0.68.el7.2.x86_64.rpm
[root@k8s-node2 nfs]# rpm -ivh *.rpm --nodeps --force
warning: gssproxy-0.7.0-30.el7_9.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID f4a80eb5: NOKEY
Preparing... ################################# [100%]
Updating / installing...
1:libtirpc-0.2.4-0.16.el7 ################################# [ 6%]
2:rpcbind-0.2.0-49.el7 ################################# [ 13%]
3:libref_array-0.1.5-32.el7 ################################# [ 19%]
4:libevent-2.0.21-4.el7 ################################# [ 25%]
5:libcollection-0.7.0-32.el7 ################################# [ 31%]
6:libbasicobjects-0.1.1-32.el7 ################################# [ 38%]
7:libverto-libevent-0.2.5-4.el7 ################################# [ 44%]
8:tcp_wrappers-7.6-77.el7 ################################# [ 50%]
9:quota-nls-1:4.01-19.el7 ################################# [ 56%]
10:quota-1:4.01-19.el7 ################################# [ 63%]
11:libpath_utils-0.2.1-32.el7 ################################# [ 69%]
12:libini_config-1.3.1-32.el7 ################################# [ 75%]
13:gssproxy-0.7.0-30.el7_9 ################################# [ 81%]
14:libnfsidmap-0.25-19.el7 ################################# [ 88%]
15:keyutils-1.5.8-3.el7 ################################# [ 94%]
16:nfs-utils-1:1.3.0-0.68.el7.2 ################################# [100%]
部署完需要注意两点
1、部署完之后nfs，nfs-server上需要修改配置和启动nfs，nfs-client上面不需要启动
2、可以在各个Node上面手动尝试下挂载，确保可以挂载上nfs
3、挂载不上多半可能是网络问题，需要继续排查，比如防火墙是不是将nfs和Node节点隔离了
如果不安装nfs-client的话，会出现以下错误，请知晓
---- ------ ---- ---- -------
Normal Scheduled 113s default-scheduler Successfully assigned default/nginx-dep1-548564f56f-86sgh to k8s-master1
Warning FailedMount 49s (x8 over 113s) kubelet MountVolume.SetUp failed for volume "wwwroot" : mount failed: exit status 32
Mounting command: mount
Mounting arguments: -t nfs 10.245.4.5:/data/nfs-a /var/lib/kubelet/pods/59ff676c-8caf-435c-bb89-15e668434bbf/volumes/kubernetes.io~nfs/wwwroot
Output: mount: wrong fs type, bad option, bad superblock on 10.245.4.5:/data/nfs-a,
missing codepage or helper program, or other error
(for several filesystems (e.g. nfs, cifs) you might
need a /sbin/mount.<type> helper program)
In some cases useful info is found in syslog - try
dmesg | tail or so.
在k8s里面部署应用使用nfs来进行持久化网络存储
在k8s-master上面操作
[root@k8s-master1 ~]# mkdir yaml ##创建一个专门来放置yaml文件的目录
[root@k8s-master1 ~]# cd yaml/
[root@k8s-master1 yaml]# mkdir nfs-yaml ##专门用来放置nfs相关的yaml文件目录
[root@k8s-master1 yaml]# cd nfs-yaml/
编写nfs-yaml文件
[root@k8s-master1 nfs-yaml]# vi nfs-nginx.yaml
[root@k8s-master1 nfs-yaml]# cat nfs-nginx.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-dep1
spec:
replicas: 1
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx
volumeMounts:
- name: wwwroot
mountPath: /usr/share/nginx/html
ports:
- containerPort: 80
volumes:
- name: wwwroot
nfs:
server: 10.245.4.5
path: /data/nfs-a
拉取nginx镜像
小编突然发现因为小编的这个是内网环境没有办法pull这个nginx镜像，那怎么办呢，当然这个不用着急，小编有的是办法，老规矩老套路，开始
利用刚才捞取rpm包的机器，我们捞取下nginx的镜像并放到各个节点上面去
因为捞取rpm包是一个外网的机器，我们可以捞取下面Nginx的镜像并打包上传到我们其他的机器上面，具体部署docker的步骤，小编不在演示了哈，大家可以参照下我的另一个博客，网址如下
kubernetes v1.20项目之二进制部署安装docker ce
[root@localhost ~]# ls
anaconda-ks.cfg docker-19.03.9.tgz ##docker二进制包
[root@localhost ~]# tar zxvf docker-19.03.9.tgz ##解压
docker/
docker/docker-init
docker/runc
docker/docker
docker/docker-proxy
docker/containerd
docker/ctr
docker/dockerd
docker/containerd-shim
[root@localhost ~]# mv docker/* /usr/bin ###主要为了有dockler命令
[root@localhost ~]# cat > /usr/lib/systemd/system/docker.service << EOF ##3创建systemctl启动管理文件
> [Unit]
> Description=Docker Application Container Engine
> Documentation=https://docs.docker.com
> After=network-online.target firewalld.service
> Wants=network-online.target
>
> [Service]
> Type=notify
> ExecStart=/usr/bin/dockerd
> ExecReload=/bin/kill -s HUP $MAINPID
> LimitNOFILE=infinity
> LimitNPROC=infinity
> LimitCORE=infinity
> TimeoutStartSec=0
> Delegate=yes
> KillMode=process
> Restart=on-failure
> StartLimitBurst=3
> StartLimitInterval=60s
>
> [Install]
> WantedBy=multi-user.target
> EOF
[root@localhost ~]# mkdir /etc/docker
[root@localhost ~]# cat > /etc/docker/daemon.json << EOF ####配置加速器
> {
> "registry-mirrors": ["https://b9pmyelo.mirror.aliyuncs.com"]
> }
> EOF
[root@localhost ~]# systemctl start docker ###启动
[root@localhost ~]# systemctl status docker ###查看状态
● docker.service - Docker Application Container Engine
Loaded: loaded (/usr/lib/systemd/system/docker.service; disabled; vendor preset: disabled)
Active: active (running) since Fri 2022-12-02 11:25:26 EST; 5s ago
Docs: https://docs.docker.com
Main PID: 1279 (dockerd)
Tasks: 17
Memory: 26.3M
CGroup: /system.slice/docker.service
├─1279 /usr/bin/dockerd
└─1285 containerd --config /var/run/docker/containerd/containerd.toml --log-level info
Dec 02 11:25:25 localhost dockerd[1279]: time="2022-12-02T11:25:25.857898899-05:00" level=info msg="scheme \"unix\" not registered, fallback to default sch...odule=grpc
Dec 02 11:25:25 localhost dockerd[1279]: time="2022-12-02T11:25:25.857915848-05:00" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///var...odule=grpc
Dec 02 11:25:25 localhost dockerd[1279]: time="2022-12-02T11:25:25.857929409-05:00" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
Dec 02 11:25:25 localhost dockerd[1279]: time="2022-12-02T11:25:25.933897599-05:00" level=info msg="Loading containers: start."
Dec 02 11:25:26 localhost dockerd[1279]: time="2022-12-02T11:25:26.126566346-05:00" level=info msg="Default bridge (docker0) is assigned with an IP address...P address"
Dec 02 11:25:26 localhost dockerd[1279]: time="2022-12-02T11:25:26.237376041-05:00" level=info msg="Loading containers: done."
Dec 02 11:25:26 localhost dockerd[1279]: time="2022-12-02T11:25:26.324158398-05:00" level=info msg="Docker daemon" commit=9d988398e7 graphdriver(s)=overlay...on=19.03.9
Dec 02 11:25:26 localhost dockerd[1279]: time="2022-12-02T11:25:26.324278562-05:00" level=info msg="Daemon has completed initialization"
Dec 02 11:25:26 localhost dockerd[1279]: time="2022-12-02T11:25:26.388081116-05:00" level=info msg="API listen on /var/run/docker.sock"
Dec 02 11:25:26 localhost systemd[1]: Started Docker Application Container Engine.
Hint: Some lines were ellipsized, use -l to show in full.
[root@localhost ~]#
pull 下nginx的镜像
[root@localhost ~]# docker images ###查看当前镜像，无Nginx
REPOSITORY TAG IMAGE ID CREATED SIZE
[root@localhost ~]# docker pull nginx ###从镜像仓库拉取Nngix镜像
Using default tag: latest
latest: Pulling from library/nginx
a2abf6c4d29d: Pull complete
a9edb18cadd1: Pull complete
589b7251471a: Pull complete
186b1aaa4aa6: Pull complete
b4df32aa5a72: Pull complete
a0bcbecc962e: Pull complete
Digest: sha256:0d17b565c37bcbd895e9d92315a05c1c3c9a29f762b011a10c54a66cd53c9b31
Status: Downloaded newer image for nginx:latest
docker.io/library/nginx:latest
[root@localhost ~]# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
nginx latest 605c77e624dd 11 months ago 141MB ###有了Nginx镜像
[root@localhost ~]# docker save nginx > nginx.tar ###将镜像打包，然后到k8s的各个node节点上面进行load
[root@localhost ~]# ls
anaconda-ks.cfg docker docker-19.03.9.tgz nginx.tar ###nginx.tar包就是我们打包好的镜像
各个Node上面操作
已node2为列，所有pod可以生成的no的节点上面都要load一下，谁都不知道pod会落到那个节点上面去，当然除了指定Node外哈
[root@k8s-node2 ~]# cd docker
[root@k8s-node2 docker]# ls
nginx.tar
[root@k8s-node2 docker]# docker load < nginx.tar
2edcec3590a4: Loading layer [==================================================>] 83.86MB/83.86MB
e379e8aedd4d: Loading layer [==================================================>] 62MB/62MB
b8d6e692a25e: Loading layer [==================================================>] 3.072kB/3.072kB
f1db227348d0: Loading layer [==================================================>] 4.096kB/4.096kB
32ce5f6a5106: Loading layer [==================================================>] 3.584kB/3.584kB
d874fd2bc83b: Loading layer [==================================================>] 7.168kB/7.168kB
Loaded image: nginx:latest
[root@k8s-node2 docker]# docker images | grep nginx ###可以看到节点上面已经有了Nginx这个镜像了
nginx latest 605c77e624dd 11 months ago 141MB
部署pod应用
因为上面我们刚刚在本地创建了Images镜像，所以yaml文件里面我们需要指定让他从本地进行拉取镜像，所以需要修改下yaml文件
[root@k8s-master1 nfs-yaml]# cd ~/yaml/nfs-yaml/
[root@k8s-master1 nfs-yaml]# ls
nfs-nginx.yaml
[root@k8s-master1 nfs-yaml]# cat nfs-nginx.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-dep1
spec:
replicas: 1
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx
imagePullPolicy: IfNotPresent
volumeMounts:
- name: wwwroot
mountPath: /usr/share/nginx/html
ports:
- containerPort: 80
volumes:
- name: wwwroot
nfs:
server: 10.245.4.5
path: /data/nfs-a
[root@k8s-master1 nfs-yaml]# kubectl apply -f nfs-nginx.yaml
deployment.apps/nginx-dep1 created
[root@k8s-master1 nfs-yaml]# kubectl get pod
NAME READY STATUS RESTARTS AGE
nginx-dep1-85b795ff5-njqf2 1/1 Running 0 7s
##如果长时间没有创建出来需要describe来进行查看具体问题出现在哪里
进入到pod 里面查看我们挂载的目录
[root@k8s-master1 nfs-yaml]# kubectl exec -it nginx-dep1-85b795ff5-njqf2 bash ###进入pod里面
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
root@nginx-dep1-85b795ff5-njqf2:/# ls /usr/share/nginx/html ###可以看到当前目录为空，因为咱的nfs里面文件也是空的
root@nginx-dep1-85b795ff5-njqf2:/#
nfs里面创建index.html测试文件
nfs-server服务器上面操作
[root@nfs-a /]# cd /data/nfs-a/
[root@nfs-a nfs-a]# ls
[root@nfs-a nfs-a]# echo "ni hao ya" > index.html
[root@nfs-a nfs-a]# ls
index.html
返回pod进行查看
master上操作
[root@k8s-master1 nfs-yaml]# kubectl get pod
NAME READY STATUS RESTARTS AGE
nginx-dep1-85b795ff5-njqf2 1/1 Running 0 7s
[root@k8s-master1 nfs-yaml]# kubectl exec -it nginx-dep1-85b795ff5-njqf2 bash
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
root@nginx-dep1-85b795ff5-njqf2:/# ls /usr/share/nginx/html
root@nginx-dep1-85b795ff5-njqf2:/# ls /usr/share/nginx/html
index.html
root@nginx-dep1-85b795ff5-njqf2:/# cat /usr/share/nginx/html/index.html
ni hao ya
将端口暴露出去
[root@k8s-master1 nfs-yaml]# kubectl expose deployment nginx-dep1 --port=80 --target-port=80 --type=NodePort
service/nginx-dep1 exposed
[root@k8s-master1 nfs-yaml]# kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.0.0.1 <none> 443/TCP 3d13h
nginx-dep1 NodePort 10.0.0.211 <none> 80:32142/TCP 14s
浏览器测试访问
可以看到我们基本上数据持久也就做好了，当然了有人会问那这个和pv和pvc有什么关系，当然是有关系的，大家不妨可以想想一下，如果把咱们的nfs的ip地址和挂载目录写在yaml文件里面，第一是不是很陋，第二也不安全呀，如果说yaml文件到不怀好意人的手里面，是不是对于我们的数据有很大的隐患
所以接下来我们将用pv和pvc组合来进行挂载，达到安全有效的数据持久
创建pvc
编写pvc的yaml文件
[root@k8s-master1 ~]# mkdir ~/yaml/pvc
[root@k8s-master1 ~]# cd yaml/pvc/
[root@k8s-master1 pvc]# vi pvc.yaml
[root@k8s-master1 pvc]# cat pvc.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-dep1
spec:
replicas: 3
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx
imagePullPolicy: IfNotPresent
volumeMounts:
- name: wwwroot
mountPath: /usr/share/nginx/html
ports:
- containerPort: 80
volumes:
- name: wwwroot
persistentVolumeClaim:
claimName: my-pvc
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: my-pvc
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 5Gi
[root@k8s-master1 pvc]# cd ../nfs-yaml/
[root@k8s-master1 nfs-yaml]# ls
nfs-nginx.yaml
[root@k8s-master1 nfs-yaml]# kubectl delete -f nfs-nginx.yaml ###删除我们上面创建的pod
deployment.apps "nginx-dep1" deleted
[root@k8s-master1 nfs-yaml]#
[root@k8s-master1 nfs-yaml]#
[root@k8s-master1 nfs-yaml]#
[root@k8s-master1 nfs-yaml]#
[root@k8s-master1 nfs-yaml]# cd ../pvc/
[root@k8s-master1 pvc]# ls
pvc.yaml
创建pvc应用
[root@k8s-master1 pvc]# kubectl apply -f pvc.yaml
deployment.apps/nginx-dep1 created
persistentvolumeclaim/my-pvc created
[root@k8s-master1 pvc]# kubectl get pod
NAME READY STATUS RESTARTS AGE
nginx-dep1-7b8c77b66b-bt6dh 0/1 Pending 0 10s
nginx-dep1-7b8c77b66b-g9c8h 0/1 Pending 0 10s
nginx-dep1-7b8c77b66b-mnb5k 0/1 Pending 0 10s
[root@k8s-master1 pvc]# kubectl describe pod nginx-dep1-7b8c77b66b-bt6dh ##查看为什么会pending
Name: nginx-dep1-7b8c77b66b-bt6dh
Namespace: default
Priority: 0
Node: <none>
Labels: app=nginx
pod-template-hash=7b8c77b66b
Annotations: <none>
Status: Pending
IP:
IPs: <none>
Controlled By: ReplicaSet/nginx-dep1-7b8c77b66b
Containers:
nginx:
Image: nginx
Port: 80/TCP
Host Port: 0/TCP
Environment: <none>
Mounts:
/usr/share/nginx/html from wwwroot (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-vcffm (ro)
Conditions:
Type Status
PodScheduled False
Volumes:
wwwroot:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: my-pvc
ReadOnly: false
kube-api-access-vcffm:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 34s default-scheduler 0/4 nodes are available: 4 pod has unbound immediate PersistentVolumeClaims.
### 这是因为还没找到pv，所以这里才会报这个警告,pod状态才会pending
[root@k8s-master1 pvc]#
创建pv
编写pv的yaml文件
[root@k8s-master1 pvc]# mkdir ../pv
[root@k8s-master1 pvc]# cd ../pv
[root@k8s-master1 pv]# vi pv.yaml
[root@k8s-master1 pv]# cat pv.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
name: my-pv
spec:
capacity:
storage: 5Gi
accessModes:
- ReadWriteMany
nfs:
path: /data/nfs-a
server: 10.245.4.5
创建Pv应用
[root@k8s-master1 pv]# kubectl apply -f pv.yaml
persistentvolume/my-pv created
[root@k8s-master1 pv]# kubectl get pv,pvc
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
persistentvolume/my-pv 5Gi RWX Retain Available 8s
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
persistentvolumeclaim/my-pvc Pending 5m32s ####pv还没创建好，需要等待一段时间
[root@k8s-master1 pv]# kubectl get pv,pvc
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
persistentvolume/my-pv 5Gi RWX Retain Available 15s ####pv还没创建好，需要等待一段时间
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
persistentvolumeclaim/my-pvc Pending 5m39s ####pv还没创建好，需要等待一段时间
[root@k8s-master1 pv]# kubectl get pv,pvc
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
persistentvolume/my-pv 5Gi RWX Retain Bound default/my-pvc 17s ####已经绑定到了pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
persistentvolumeclaim/my-pvc Bound my-pv 5Gi RWX 5m41s
再次查看我们的pod
[root@k8s-master1 pv]# kubectl get pod
NAME READY STATUS RESTARTS AGE
nginx-dep1-7b8c77b66b-bt6dh 1/1 Running 0 8m20s
nginx-dep1-7b8c77b66b-g9c8h 1/1 Running 0 8m20s
nginx-dep1-7b8c77b66b-mnb5k 1/1 Running 0 8m20s
暴露端口进行测试
[root@k8s-master1 pv]# kubectl expose deployment nginx-dep1 --port=80 --target-port=80 --type=NodePort
Error from server (AlreadyExists): services "nginx-dep1" already exists ###因为上一次的svc还存在，接下来我们删除之前的svc
[root@k8s-master1 pv]# kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.0.0.1 <none> 443/TCP 3d14h
nginx-dep1 NodePort 10.0.0.211 <none> 80:32142/TCP 50m
[root@k8s-master1 pv]# kubectl delete svc nginx-dep1
service "nginx-dep1" deleted
[root@k8s-master1 pv]# kubectl expose deployment nginx-dep1 --port=80 --target-port=80 --type=NodePort ###重新进行暴露端口
service/nginx-dep1 exposed
[root@k8s-master1 pv]# kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.0.0.1 <none> 443/TCP 3d14h
nginx-dep1 NodePort 10.0.0.50 <none> 80:30306/TCP 6s
结束语
努力是很痛苦的一件事情，但是很快乐，就像小编现在半夜3点了还在写稿，谢谢大家
![[附源码]Python计算机毕业设计SSM居民个人健康服务平台（程序+LW)](https://img-blog.csdnimg.cn/c7bb75a9b7974eda81efbf9f895faa17.png)
![[附源码]计算机毕业设计在线影院系统Springboot程序](https://img-blog.csdnimg.cn/d7382327a98a48d49068c649ea5bdb77.png)
![[附源码]计算机毕业设计JAVA校园求职与招聘系统](https://img-blog.csdnimg.cn/dead6759e4024606b7e2251afc49f73c.png)
