1. Fastdfs HA原理
- Fastdfs引入Tracker以支持文件操作的负载均衡调度
- Fastdfs引入基于group分组的storage以支持类似raid10模式的高可靠高性能的存储
- Fastdfs引入fastdfs-nginx-module 可以重定向文件连接到文件上传时的源服务器取文件,避免客户端由于复制延迟导致的文件无法访问错误
2.集群规划
| 节点名称 | 节点角色 | 节点路径 | Group |
| Tracker1 | Tracker | /data/opt/fastdfs/tracker /data/opt/fastdfs/tracker/logs | |
| Tracker2 | Tracker | /data/opt/fastdfs/tracker /data/opt/fastdfs/tracker/logs | |
| Nginx主 | Storage集群的Nginx代理 | /data/opt/nginx /data/opt/nginx/log | |
| Nginx备 | Storage集群的Nginx代理 | /data/opt/nginx /data/opt/nginx/log | |
| Storage1+nginx (注:nginx需预编译fastdfs-nginx-module) | Storage | /data/opt/fastdfs/storage /data/opt/fastdfs/storage/logs /data/opt/fastdfs/nginx /data/opt/fastdfs/nginx /logs | group1 |
| Storage2+nginx (注:nginx需预编译fastdfs-nginx-module) | Storage | /data/opt/fastdfs/storage /data/opt/fastdfs/storage/logs /data/opt/fastdfs/nginx /data/opt/fastdfs/nginx /logs | group1 |
| Storage3+nginx (注:nginx需预编译fastdfs-nginx-module) | Storage | /data/opt/fastdfs/storage /data/opt/fastdfs/storage/logs /data/opt/fastdfs/nginx /data/opt/fastdfs/nginx /logs | group2 |
| Storage4+nginx (注:nginx需预编译fastdfs-nginx-module) | Storage | /data/opt/fastdfs/storage /data/opt/fastdfs/storage/logs /data/opt/fastdfs/nginx /data/opt/fastdfs/nginx /logs | group2 |
3.软件版本
操作系统：CentOS Linux release 7.9.2009 (Core)
Fastdfs版本：6.0.6
Fastdfs-nginx-module版本：1.22
Libfastcommon版本:1.0.43
Nginx版本：1.16.0
Keepalived版本: 1.3.5
4.Fastdfs集群搭建
4.1.安装软件准备
a.编译fastdfs
访问https://github.com/happyfish100/fastdfs，下载fastdfs-6.06.zip
然后执行如下命令
| mkdir –p /data/opt/fastdfs unzip fastdfs-6.06.zip cd /data/opt/fastdfs/fastdfs-6.06 ./make.sh ./make.sh install |
成功标识为如下二进制文件编译成功、/usr/lib64/libfdfsclient.so编译成功以及/etc/fdfs生成client.conf.sample、storage.conf.sample、storage_ids.conf.sample、tracker.conf.sample
b.编译libfastcommon
访问https://github.com/happyfish100/libfastcommon，下载libfastcommon-1.0.43.zip
然后执行如下命令
| mkdir –p /data/opt/fastdfs/libfastcommon unzip libfastcommon-1.0.43.zip cd /data/opt/fastdfs/libfastcommon/libfastcommon-1.0.43 ./make.sh ./make.sh install |
成功标识：/usr/lib64/libfastcommon.so编译成功
c.编译fastdfs-nginx-module
访问https://github.com/happyfish100/fastdfs-nginx-module/releases/tag/V1.22，下载fastdfs-nginx-module-1.22.zip
然后执行如下命令
| mkdir –p /data/opt/fastdfs/fastdfs-nginx-module unzip fastdfs-nginx-module-1.22.zip #二次编译nginx模块 ./configure --add-module=/data/opt/fastdfs/fastdfs-nginx-module/fastdfs-nginx-module-1.22/src # 追加之前编译的模块 make |
成功标识: 会在objs目录下会生成nginx文件
4.2.Tracker集群配置
注: 需在每个Tracker节点执行以下步骤
操作步骤如下
| cd /etc/fdfs mv tracker.conf.sample tracker.conf mkdir -p /data/opt/fastdfs/tracker |
修改配置
| # the base path to store data and log files base_path = /data/opt/fastdfs/tracker |
Tracker启动和停止
| 启动: fdfs_trackerd /etc/fdfs/tracker.conf start 停止: fdfs_trackerd /etc/fdfs/tracker.conf stop |
4.3.Storage集群配置
注: 需在每个Storage节点执行以下步骤，另外group名称需根据实际storage节点所属group修改
a.安装storage
操作步骤如下
| cd /etc/fdfs mv storage.conf.sample storager.conf mkdir -p /data/opt/fastdfs/storage |
修改配置
| # the name of the group this storage server belongs to # # comment or remove this item for fetching from tracker server, # in this case, use_storage_id must set to true in tracker.conf, # and storage_ids.conf must be configured correctly. group_name = group1 # the base path to store data and log files # NOTE: the binlog files maybe are large, make sure # the base path has enough disk space, # eg. the disk free space should > 50GB base_path = /data/opt/fastdfs/storage # store path (disk or mount point) count, default value is 1 store_path_count = 1 # store_path#, based on 0, to configure the store paths to store files # if store_path0 not exists, it's value is base_path (NOT recommended) # the paths must be exist. # # IMPORTANT NOTE: # the store paths' order is very important, don't mess up!!! # the base_path should be independent (different) of the store paths store_path0 = /data/opt/fastdfs/storage/files # tracker_server can ocur more than once for multi tracker servers. # the value format of tracker_server is "HOST:PORT", # the HOST can be hostname or ip address, # and the HOST can be dual IPs or hostnames seperated by comma, # the dual IPS must be an inner (intranet) IP and an outer (extranet) IP, # or two different types of inner (intranet) IPs. # for example: 192.168.2.100,122.244.141.46:22122 # another eg.: 192.168.1.10,172.17.4.21:22122 tracker_server = tracker1:22122 tracker_server = tracker2:22122 |
Storage启动和停止
| 启动: fdfs_storaged /etc/fdfs/storage.conf start 停止: fdfs_storaged /etc/fdfs/storage.conf stop |
b.安装编译了fastdfs-nginx-module的nginx
操作步骤如下
| cd /data/opt/fastdfs/fastdfs-6.06-package/fastdfs-nginx-module-1.22/src cp mod_fastdfs.conf /etc/fdfs cd /data/opt/fastdfs/fastdfs-6.06/conf cp http.conf /etc/fdfs cp mime.types /etc/fdfs |
修改mod_fastdfs.conf配置
| # the base path to store log files base_path=/data/opt/fastdfs/nginx-module # FastDFS tracker_server can ocur more than once, and tracker_server format is # "host:port", host can be hostname or ip address # valid only when load_fdfs_parameters_from_tracker is true tracker_server = tracker1:22122 tracker_server = tracker2:22122 # the group name of the local storage server group_name=group1 # if the url / uri including the group name # set to false when uri like /M00/00/00/xxx # set to true when uri like ${group_name}/M00/00/00/xxx, such as group1/M00/xxx # default value is false url_have_group_name = true # path(disk or mount point) count, default value is 1 # must same as storage.conf store_path_count=1 # store_path#, based 0, if store_path0 not exists, it's value is base_path # the paths must be exist # must same as storage.conf store_path0=/data/opt/fastdfs/storage/files |
修改nginx.conf配置
| server { listen 8199; #拦截请求路径中包含 /group[1-9]/M0[0-9] 的请求 location ~ /group1/M0[0-9] { ngx_fastdfs_module; } } |
nginx启动和停止
| 启动: ./nginx 停止: kill -9 nginx进程 |
4.4.Storage代理集群配置
a.部署keepalived
修改keepalived.conf,注意根据角色修改标红配置项
| vrrp_script check_nginx { script "/etc/keepalived/check_ngx.sh" interval 2 } # VIP1 vrrp_instance VI_1 { state MASTER interface eth0 virtual_router_id 51 #让master 和backup在同一个虚拟路由里，id 号必须相同 priority 100 #优先级,谁的优先级高谁就是master advert_int 5 #心跳间隔时间 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { vip } track_script { check_nginx } } |
b.部署nginx
修改nginx.conf
| upstream group1 { server storage1; server storage2; } upstream group2 { server storage3; server storage4; } server { listen 8095; location ~ /group1/M0[0-9] { proxy_pass http://group1; } location ~ /group2/M0[0-9] { proxy_pass http://group2; } } |
nginx启动和停止
| 启动: ./nginx 停止: kill -9 nginx进程 |
4.5.Fastdfs HA集群展示
| [2023-06-29 09:44:34] DEBUG - base_path=/data/opt/fastdfs/storage, connect_timeout=5, network_timeout=60, tracker_server_count=2, anti_steal_token=0, anti_steal_secret_key length=0, use_connection_pool=1, g_connection_pool_max_idle_time=3600s, use_storage_id=0, storage server id count: 0 server_count=2, server_index=0 tracker server is 172.30.8.228:22122 group count: 2 Group 1: group name = group1 disk total space = 50,164 MB disk free space = 10,929 MB trunk free space = 0 MB storage server count = 2 active server count = 2 storage server port = 23000 storage HTTP port = 8888 store path count = 1 subdir count per path = 256 current write server index = 1 current trunk file id = 0 Storage 1: id = 172.30.8.228 ip_addr = 172.30.8.228 ACTIVE http domain = version = 6.06 join time = 2023-06-28 14:10:28 up time = 2023-06-28 17:18:11 total storage = 50,164 MB free storage = 10,929 MB upload priority = 10 store_path_count = 1 subdir_count_per_path = 256 storage_port = 23000 storage_http_port = 8888 current_write_path = 0 source storage id = 172.31.97.59 if_trunk_server = 0 connection.alloc_count = 256 connection.current_count = 1 connection.max_count = 3 total_upload_count = 137 success_upload_count = 137 total_append_count = 0 success_append_count = 0 total_modify_count = 0 success_modify_count = 0 total_truncate_count = 0 success_truncate_count = 0 total_set_meta_count = 123 success_set_meta_count = 123 total_delete_count = 0 success_delete_count = 0 total_download_count = 0 success_download_count = 0 total_get_meta_count = 0 success_get_meta_count = 0 total_create_link_count = 0 success_create_link_count = 0 total_delete_link_count = 0 success_delete_link_count = 0 total_upload_bytes = 3678 success_upload_bytes = 3678 total_append_bytes = 0 success_append_bytes = 0 total_modify_bytes = 0 success_modify_bytes = 0 stotal_download_bytes = 0 success_download_bytes = 0 total_sync_in_bytes = 8202 success_sync_in_bytes = 8202 total_sync_out_bytes = 0 success_sync_out_bytes = 0 total_file_open_count = 377 success_file_open_count = 377 total_file_read_count = 0 success_file_read_count = 0 total_file_write_count = 377 success_file_write_count = 377 last_heart_beat_time = 2023-06-29 09:44:13 last_source_update = 2023-06-29 09:41:35 last_sync_update = 2023-06-29 09:41:36 last_synced_timestamp = 2023-06-29 09:41:35 (0s delay) Storage 2: id = 172.31.97.59 ip_addr = 172.31.97.59 ACTIVE http domain = version = 6.06 join time = 2023-06-24 22:43:03 up time = 2023-06-28 17:22:53 total storage = 900,707 MB free storage = 251,526 MB upload priority = 10 store_path_count = 1 subdir_count_per_path = 256 storage_port = 23000 storage_http_port = 8888 current_write_path = 0 source storage id = if_trunk_server = 0 connection.alloc_count = 256 connection.current_count = 1 connection.max_count = 3 total_upload_count = 127 success_upload_count = 127 total_append_count = 0 success_append_count = 0 total_modify_count = 0 success_modify_count = 0 total_truncate_count = 0 success_truncate_count = 0 total_set_meta_count = 113 success_set_meta_count = 113 total_delete_count = 0 success_delete_count = 0 total_download_count = 0 success_download_count = 0 total_get_meta_count = 0 success_get_meta_count = 0 total_create_link_count = 0 success_create_link_count = 0 total_delete_link_count = 0 success_delete_link_count = 0 total_upload_bytes = 3090 success_upload_bytes = 3090 total_append_bytes = 0 success_append_bytes = 0 total_modify_bytes = 0 success_modify_bytes = 0 stotal_download_bytes = 0 success_download_bytes = 0 total_sync_in_bytes = 9110 success_sync_in_bytes = 9110 total_sync_out_bytes = 0 success_sync_out_bytes = 0 total_file_open_count = 387 success_file_open_count = 387 total_file_read_count = 0 success_file_read_count = 0 total_file_write_count = 387 success_file_write_count = 387 last_heart_beat_time = 2023-06-29 09:44:24 last_source_update = 2023-06-29 09:41:35 last_sync_update = 2023-06-29 09:41:35 last_synced_timestamp = 2023-06-29 09:41:35 (0s delay) Group 2: group name = group2 disk total space = 50,164 MB disk free space = 17,517 MB trunk free space = 0 MB storage server count = 2 active server count = 2 storage server port = 23000 storage HTTP port = 8888 store path count = 1 subdir count per path = 256 current write server index = 0 current trunk file id = 0 Storage 1: id = 172.30.8.97 ip_addr = 172.30.8.97 ACTIVE http domain = version = 6.06 join time = 2023-06-24 22:57:52 up time = 2023-06-28 17:25:20 total storage = 50,164 MB free storage = 17,517 MB upload priority = 10 store_path_count = 1 subdir_count_per_path = 256 storage_port = 23000 storage_http_port = 8888 current_write_path = 0 source storage id = 172.30.9.39 if_trunk_server = 0 connection.alloc_count = 256 connection.current_count = 1 connection.max_count = 3 total_upload_count = 198 success_upload_count = 198 total_append_count = 0 success_append_count = 0 total_modify_count = 0 success_modify_count = 0 total_truncate_count = 0 success_truncate_count = 0 total_set_meta_count = 185 success_set_meta_count = 185 total_delete_count = 0 success_delete_count = 0 total_download_count = 0 success_download_count = 0 total_get_meta_count = 0 success_get_meta_count = 0 total_create_link_count = 0 success_create_link_count = 0 total_delete_link_count = 0 success_delete_link_count = 0 total_upload_bytes = 4294059528 success_upload_bytes = 4294059528 total_append_bytes = 0 success_append_bytes = 0 total_modify_bytes = 0 success_modify_bytes = 0 stotal_download_bytes = 0 success_download_bytes = 0 total_sync_in_bytes = 6796083680 success_sync_in_bytes = 6796083680 total_sync_out_bytes = 0 success_sync_out_bytes = 0 total_file_open_count = 549 success_file_open_count = 549 total_file_read_count = 0 success_file_read_count = 0 total_file_write_count = 42853 success_file_write_count = 42853 last_heart_beat_time = 2023-06-29 09:44:21 last_source_update = 2023-06-29 09:41:35 last_sync_update = 2023-06-29 09:41:36 last_synced_timestamp = 2023-06-29 09:41:35 (0s delay) Storage 2: id = 172.30.9.39 ip_addr = 172.30.9.39 ACTIVE http domain = version = 6.06 join time = 2023-06-24 22:45:25 up time = 2023-06-28 17:25:53 total storage = 50,164 MB free storage = 30,213 MB upload priority = 10 store_path_count = 1 subdir_count_per_path = 256 storage_port = 23000 storage_http_port = 8888 current_write_path = 0 source storage id = if_trunk_server = 0 connection.alloc_count = 256 connection.current_count = 1 connection.max_count = 3 total_upload_count = 182 success_upload_count = 182 total_append_count = 0 success_append_count = 0 total_modify_count = 0 success_modify_count = 0 total_truncate_count = 0 success_truncate_count = 0 total_set_meta_count = 169 success_set_meta_count = 169 total_delete_count = 0 success_delete_count = 0 total_download_count = 0 success_download_count = 0 total_get_meta_count = 0 success_get_meta_count = 0 total_create_link_count = 0 success_create_link_count = 0 total_delete_link_count = 0 success_delete_link_count = 0 total_upload_bytes = 6796075484 success_upload_bytes = 6796075484 total_append_bytes = 0 success_append_bytes = 0 total_modify_bytes = 0 success_modify_bytes = 0 stotal_download_bytes = 0 success_download_bytes = 0 total_sync_in_bytes = 4294068338 success_sync_in_bytes = 4294068338 total_sync_out_bytes = 0 success_sync_out_bytes = 0 total_file_open_count = 565 success_file_open_count = 565 total_file_read_count = 0 success_file_read_count = 0 total_file_write_count = 42869 success_file_write_count = 42869 last_heart_beat_time = 2023-06-29 09:44:23 last_source_update = 2023-06-29 09:41:35 last_sync_update = 2023-06-29 09:41:35 last_synced_timestamp = 2023-06-29 09:41:35 (0s delay) |
4.6.Fastdfs HA集群功能测试
4.6.1 测试Tracker支持集群
关闭一个tracker节点，客户端仍能上传
4.6.2 测试Tracker集群支持负载调度
客户端循环上传同一个文件，会自动调度到不同的group，java客户端测试结果如下
| ["group2","M00/00/01/rB4JJ2Sc4U-AJri5AAAAQvkVImY390.txt"] ["group1","M00/00/01/rB4I5GSc4U-AcNZSAAAAQvkVImY461.txt"] ["group2","M00/00/01/rB4IYWSc4U-ARJOMAAAAQvkVImY114.txt"] ["group1","M00/00/01/rB9hO2Sc4U-AEOyUAAAAQvkVImY037.txt"] ["group2","M00/00/01/rB4JJ2Sc4U-AfavaAAAAQvkVImY542.txt"] ["group1","M00/00/01/rB4I5GSc4U-AOFOaAAAAQvkVImY966.txt"] ["group2","M00/00/01/rB4IYWSc4U-AWnrGAAAAQvkVImY960.txt"] ["group1","M00/00/01/rB9hO2Sc4U-AbYPwAAAAQvkVImY168.txt"] ["group2","M00/00/01/rB4JJ2Sc4U-AHBc3AAAAQvkVImY951.txt"] ["group1","M00/00/01/rB4I5GSc4U-AAEP5AAAAQvkVImY312.txt"] |
4.6.3 测试Storage集群支持raid1
客户端上传文件，查看是否能自动同步，如下为来自group2的storage1和storage2的文件列表结果。
Storage1文件列表
Storage2文件列表
4.6.4 测试Storage代理集群支持raid10访问
即nginx支持根据不同group路径，访问支持raid1的group1和group2
4.6.5 测试fast-nginx-module优先访问源节点
步骤一
上传一个新文件，此时通过两storage节点的nginx日志，只有源storage节点的nginx日志，会看到调用日志
步骤二
停掉同一个group某一个storage节点，上传一个新文件，此时通过备份的storage节点的nginx日志，会看到调用日志
5.Fastdfs客户端推荐
通过GitHub - happyfish100/fastdfs-client-java: FastDFS java client SDK地址，下载源代码，进行mvn编译，最终通过如下方式引入到项目中，客户端是可以支持多个tracker地址集群调用使用。
| <dependency> |
6.常见问题
6.1 如何选择storage
新上传文件如何选择storage，默认是0
| # which storage server to upload file # 0: round robin (default) # 1: the first server order by ip address # 2: the first server order by priority (the minimal) |
新下载文件如何选择storage，默认是0
| # which storage server to download file # 0: round robin (default) # 1: the source storage server which the current file uploaded to |
6.2 如何选择group
新上传文件如何选择group，默认是0
| # the method for selecting group to upload files # 0: round robin # 1: specify group # 2: load balance, select the max free space group to upload file |
6.3 如何选择storage path
新上传文件如何选择storage path，默认是0
| # which path (means disk or mount point) of the storage server to upload file # 0: round robin # 2: load balance, select the max free space path to upload file |
6.4 保留存储空间阈值
| # reserved storage space for system or other applications. # if the free(available) space of any stoarge server in # a group <= reserved_storage_space, no file can be uploaded to this group. # bytes unit can be one of follows: ### G or g for gigabyte(GB) ### M or m for megabyte(MB) ### K or k for kilobyte(KB) ### no unit for byte(B) ### XX.XX% as ratio such as: reserved_storage_space = 10% reserved_storage_space = 10% |
6.5 文件索引信息说明
| group1 group名称 M01 storage名称
file 文件名称 最终文件访问地址示例为: http://vip/group1/M01/00/00/rB9hO2SQGwaAEQv2ZUT0AKIa460992.tar |
