FastDFS的HA架构多Group多Storage多Tracker结合SpringBoot
在开始之前，这篇文章并不是初级入门的文章，在该文章发布之前已经有两篇文章讲述了FastDFS和SpringBoot相关的知识，请参阅：
《Java高级架构之FastDFS分布式文件集群》：https://blog.51cto.com/xvjunjie/2377669
《SpringBoot集成FastDFS+Nginx整合基于Token的防盗链》：https://blog.51cto.com/xvjunjie/2381361如果需要在FastDFS中实现HTTPS功能，请参阅：
《Vue、SprinBoot开发运维的一些坑和知识集锦》：https://blog.51cto.com/xvjunjie/2440933在下图中，任何一个节点都支持水平横向无限扩展。当在某一个Group中添加新的Storage服务器的时候，新加入该Group的服务器会自动同步所有数据。当新添加Group的时候，只需要修改处于前端的Nginx服务器即可。要实现主备自动切换还可以加入KeepAlived中间件。
FastDFS服务器通过HTTP提供服务，但是HTTP的性能太弱，所以在V4.05之后的版本中删除了内置的HTTP服务。FastDFS的Group之间的Storage需要复制数据，如果请求的数据正在复制，由于不是源服务器，那么fastdfs-nginx-module将进行重定向源服务器。fastdfs-nginx-module支持配置多个组，一个服务器可以配置多个Group不同的Storage。同一个Group的Storage必须端口相同。Nginx会为FastDFS提供负载均衡和数据缓存的功能。FastDFS集群的总容量等于全部Group容量之和。一个Group的容量等于最小的Storage服务器容量。组一般用于隔离不同的数据。FastDFS分布式文件集群HA架构图
集群中具体机器配置
yum updatemkdir /sourcecd /sourceyum install -y gcc gcc-c++ make cmake wget libeventwget https://github.com/happyfish200/libfastcommon/archive/V1.0.35.tar.gzwget https://github.com/happyfish200/fastdfs/archive/V5.10.tar.gztar -zxvf V1.0.35.tar.gztar -zxvf V5.10.tar.gzcd libfastcommon-1.0.35./make.sh./make.sh installln -s /usr/lib64/libfastcommon.so /usr/local/lib/libfastcommon.socd ../cd fastdfs-5.10/./make.sh./make.sh installcd ../cp /etc/fdfs/tracker.conf.sample /etc/fdfs/tracker.confcp /etc/fdfs/storage.conf.sample /etc/fdfs/storage.confcp /etc/fdfs/client.conf.sample /etc/fdfs/client.confmkdir -p /data/fdfs/trackermkdir -p /data/fdfs/storagemkdir -p /data/fdfs/clientmkdir -p /data/fdfs/disks/volume0mkdir -p /data/fdfs/disks/volume1mkdir -p /data/fdfs/disks/volume2mkdir -p /data/fdfs/disks/volume3mkdir -p /data/fdfs/disks/volume4mkdir -p /data/fdfs/disks/volume5（2）为Storage部署fastdfs-nginx-module
部署步骤
# 添加用户useradd nginx -s /sbin/nologin -M# 安装Nginx需要的pcre（Perl兼容正则表达式）库，允许Nginx使用rewrite模块提供URL重写功能。yum install pcre pcre-devel perl-ExtUtils-Embed -y# 安装openssl-devel，允许Nginx提供HTTPS服务。yum install openssl-devel -y# 下载软件包cd /sourcewget http://nginx.org/download/nginx-1.16.1.tar.gzwget http://nchc.dl.sourceforge.net/project/fastdfs/FastDFS%20Nginx%20Module%20Source%20Code/fastdfs-nginx-module_v1.16.tar.gz# 解压软件包tar xvf fastdfs-nginx-module_v1.16.tar.gztar -xvf nginx-1.16.1.tar.gz# 创建必要的软连接ln -s /usr/include/fastdfs/ /usr/local/include/fastdfsln -s /usr/include/fastcommon/ /usr/local/include/fastcommoncp fastdfs-5.10/conf/http.conf /etc/fdfs/cp fastdfs-5.10/conf/mime.types /etc/fdfs/cp fastdfs-nginx-module/src/mod_fastdfs.conf /etc/fdfs/mkdir -p /data/fdfs/fastdfs-nginx-module# 编译安装cd nginx-1.16.1./configure --user=nginx --group=nginx --prefix=/application/nginx-1.16.1 --with-http_ssl_module --with-http_gzip_static_module --with-poll_module --with-file-aio --with-http_realip_module --with-http_addition_module --with-http_random_index_module --with-pcre --with-http_stub_status_module --with-stream --add-module=/source/fastdfs-nginx-module/src/makemake installln -s /application/nginx-1.16.1/ /application/nginx# 修改权限并创建systemd控制单元chown -R nginx:nginx /application/nginx*touch /usr/lib/systemd/system/nginx.service
Nginx配置文件
user nginx;worker_processes 8;error_log logs/error.log;pid logs/nginx.pid;events { worker_connections 1024;}http { include mime.types; default_type application/octet-stream; log_format main '$remote_addr - $remote_user [$time_local] "$request" ' '$status $body_bytes_sent "$http_referer" ' '"$http_user_agent" "$http_x_forwarded_for"'; sendfile on; keepalive_timeout 65; gzip on; server { listen 80; # 配置为主机域名 server_name fastdfs-storage1-group1; access_log logs/host.access.log main; location ~/group[0-9] { ngx_fastdfs_module; } }}
配置mod_fastdfs.conf文件
base_path=/data/fdfs/fastdfs-nginx-moduletracker_server=192.168.80.21:22122tracker_server=192.168.80.22:22122tracker_server=192.168.80.23:22122# 配置所属组的名称group_name=group1url_have_group_name = truestore_path_count=6store_path0=/data/fdfs/disks/volume0store_path2=/data/fdfs/disks/volume1store_path3=/data/fdfs/disks/volume2store_path4=/data/fdfs/disks/volume3store_path5=/data/fdfs/disks/volume4store_path6=/data/fdfs/disks/volume5
Nginx SystemD单元脚本
[Unit]Description=nginxDocumentation=http://nginx.org/en/docs/After=network.target[Service]Type=forkingPIDFile=/application/nginx/logs/nginx.pidExecStartPre=/application/nginx/sbin/nginx -t -c /application/nginx/conf/nginx.confExecStart=/application/nginx/sbin/nginx -c /application/nginx/conf/nginx.confExecReload=/application/nginx/sbin/nginx -s reloadExecStop=/application/nginx/sbin/nginx -s stopRestart=on-abortPrivateTmp=true[Install]WantedBy=multi-user.target（3）配置Tracker服务器集群
# 配置tracker的IP地址或者主机域名，前提要进行配置静态hosts解析或者配置DNSbind_addr=192.168.80.21# tracker的数据和日志存储目录base_path=/data/fdfs/tracker（4）配置Storage服务器集群
# 配置该Storage服务属于哪一个Groupgroup_name=group1# 配置绑定的IP地址或者主机域名bind_addr=192.168.80.12# 配置日志存储目录base_path=/data/fdfs/storage# 配置数据存储目录的个数store_path_count=6# 配置数据存储目录store_path0=/data/fdfs/disks/volume0store_path2=/data/fdfs/disks/volume1store_path3=/data/fdfs/disks/volume2store_path4=/data/fdfs/disks/volume3store_path5=/data/fdfs/disks/volume4store_path6=/data/fdfs/disks/volume5# 配置跟踪服务器的IP地址或者主机域名tracker_server=192.168.80.21:22122tracker_server=192.168.80.22:22122tracker_server=192.168.80.23:22122
（5）在NGINX节点上配置NGINX进行负载均衡在全部主机上执行启动服务并设置为开机自启动
NGINX配置文件
user nginx;worker_processes 8;error_log logs/error.log;pid logs/nginx.pid;events { worker_connections 1024;}http { include mime.types; default_type application/octet-stream; log_format main '$remote_addr - $remote_user [$time_local] "$request" ' '$status $body_bytes_sent "$http_referer" ' '"$http_user_agent" "$http_x_forwarded_for"'; sendfile on; keepalive_timeout 65; gzip on; # Group1的Storage节点 upstream g1_pool { server 192.168.80.11 weight=1; server 192.168.80.12 weight=1; server 192.168.80.13 weight=1; } # Group2的Storage节点 upstream g2_pool { server 192.168.80.14 weight=1; server 192.168.80.15 weight=1; server 192.168.80.16 weight=1; } # Group3的Storage节点 upstream g3_pool { server 192.168.80.17 weight=1; server 192.168.80.18 weight=1; server 192.168.80.19 weight=1; } server { listen 80; server_name localhost; access_log logs/host.access.log main; location ~/group1 { proxy_pass http://g1_pool; } location ~/group2 { proxy_pass http://g2_pool; } location ~/group3 { proxy_pass http://g3_pool; } }}
启动所有服务
systemctl daemon-reloadsystemctl start nginxsystemctl enable nginx# 如果是跟踪服务器执行这两行systemctl restart fdfs_trackerdsystemctl enable fdfs_trackerd# 如果是存储服务器执行这两行systemctl restart fdfs_storagedsystemctl enable fdfs_storaged（6）使用FastDFS自带的Client上传图片进行测试
在任意一台服务器配置/etc/fdfs/client.conf
base_path=/data/fdfs/clienttracker_server=192.168.80.21:22122tracker_server=192.168.80.22:22122tracker_server=192.168.80.23:22122
在这台服务器进行上传文件测试
可以看到经过多次上传测试，多Group+多Tracker+多Storager生效了。
group2/M00/00/00/wKhQDl2y9aOAcngTACdpr-L7emo254.png[root@fastdfs-storage1-group1 source]# /usr/bin/fdfs_upload_file /etc/fdfs/client.conf /source/girl.png group2/M00/00/00/wKhQD12y9aiARzwQACdpr-L7emo948.png[root@fastdfs-storage1-group1 source]# /usr/bin/fdfs_upload_file /etc/fdfs/client.conf /source/girl.png group2/M00/00/00/wKhQEF2y9ayAZBcBACdpr-L7emo474.png[root@fastdfs-storage1-group1 source]# /usr/bin/fdfs_upload_file /etc/fdfs/client.conf /source/girl.png group3/M00/00/00/wKhQEV2y9a6Af_mrACdpr-L7emo390.png[root@fastdfs-storage1-group1 source]# /usr/bin/fdfs_upload_file /etc/fdfs/client.conf /source/girl.png group3/M00/00/00/wKhQEl2y9a-Ae1cWACdpr-L7emo570.png[root@fastdfs-storage1-group1 source]# /usr/bin/fdfs_upload_file /etc/fdfs/client.conf /source/girl.png group3/M00/00/00/wKhQE12y9bGAbWemACdpr-L7emo654.png[root@fastdfs-storage1-group1 source]# /usr/bin/fdfs_upload_file /etc/fdfs/client.conf /source/girl.png group3/M01/00/00/wKhQEV2y9bKADzEIACdpr-L7emo251.png[root@fastdfs-storage1-group1 source]# /usr/bin/fdfs_upload_file /etc/fdfs/client.conf /source/girl.png group3/M01/00/00/wKhQEl2y9bOAHT16ACdpr-L7emo211.png[root@fastdfs-storage1-group1 source]# /usr/bin/fdfs_upload_file /etc/fdfs/client.conf /source/girl.png group3/M01/00/00/wKhQE12y9bSAEwBUACdpr-L7emo190.png[root@fastdfs-storage1-group1 source]# /usr/bin/fdfs_upload_file /etc/fdfs/client.conf /source/girl.png group3/M02/00/00/wKhQEV2y9bWAc4CSACdpr-L7emo991.png[root@fastdfs-storage1-group1 source]# /usr/bin/fdfs_upload_file /etc/fdfs/client.conf /source/girl.png group3/M02/00/00/wKhQEl2y9baAJmqNACdpr-L7emo880.png[root@fastdfs-storage1-group1 source]# /usr/bin/fdfs_upload_file /etc/fdfs/client.conf /source/girl.png group3/M02/00/00/wKhQE12y9biAbE8IACdpr-L7emo426.png[root@fastdfs-storage1-group1 source]# /usr/bin/fdfs_upload_file /etc/fdfs/client.conf /source/girl.png group3/M03/00/00/wKhQEV2y9bmAGh6pACdpr-L7emo466.png[root@fastdfs-storage1-group1 source]# /usr/bin/fdfs_upload_file /etc/fdfs/client.conf /source/girl.png group3/M03/00/00/wKhQEl2y9bmATKQjACdpr-L7emo349.png[root@fastdfs-storage1-group1 source]# /usr/bin/fdfs_upload_file /etc/fdfs/client.conf /source/girl.png group3/M03/00/00/wKhQE12y9bmAXVOfACdpr-L7emo921.png[root@fastdfs-storage1-group1 source]# /usr/bin/fdfs_upload_file /etc/fdfs/client.conf /source/girl.png group3/M04/00/00/wKhQEV2y9bmAV9sqACdpr-L7emo420.png[root@fastdfs-storage1-group1 source]# /usr/bin/fdfs_upload_file /etc/fdfs/client.conf /source/girl.png group3/M04/00/00/wKhQEl2y9bqAOg2NACdpr-L7emo216.png[root@fastdfs-storage1-group1 source]# /usr/bin/fdfs_upload_file /etc/fdfs/client.conf /source/girl.png group3/M04/00/00/wKhQE12y9bqAcZKEACdpr-L7emo382.png[root@fastdfs-storage1-group1 source]# /usr/bin/fdfs_upload_file /etc/fdfs/client.conf /source/girl.png group3/M05/00/00/wKhQEV2y9bqAJVS9ACdpr-L7emo910.png[root@fastdfs-storage1-group1 source]# /usr/bin/fdfs_upload_file /etc/fdfs/client.conf /source/girl.png group3/M05/00/00/wKhQEl2y9buATzq2ACdpr-L7emo523.png[root@fastdfs-storage1-group1 source]# /usr/bin/fdfs_upload_file /etc/fdfs/client.conf /source/girl.png group3/M05/00/00/wKhQE12y9buALTHRACdpr-L7emo483.png[root@fastdfs-storage1-group1 source]# /usr/bin/fdfs_upload_file /etc/fdfs/client.conf /source/girl.png group3/M00/00/00/wKhQEV2y9buAJO-PACdpr-L7emo079.png[root@fastdfs-storage1-group1 source]# /usr/bin/fdfs_upload_file /etc/fdfs/client.conf /source/girl.png group3/M00/00/00/wKhQEl2y9buAP1NpACdpr-L7emo443.png[root@fastdfs-storage1-group1 source]# /usr/bin/fdfs_upload_file /etc/fdfs/client.conf /source/girl.png ^[[Agroup3/M00/00/00/wKhQE12y9byAdfYxACdpr-L7emo025.png[root@fastdfs-storage1-group1 source]# /usr/bin/fdfs_upload_file /etc/fdfs/client.conf /source/girl.png group3/M01/00/00/wKhQEV2y9byAJB2OACdpr-L7emo747.png[root@fastdfs-storage1-group1 source]# /usr/bin/fdfs_upload_file /etc/fdfs/client.conf /source/girl.png group3/M01/00/00/wKhQEl2y9byAIye6ACdpr-L7emo456.png[root@fastdfs-storage1-group1 source]# /usr/bin/fdfs_upload_file /etc/fdfs/client.conf /source/girl.png group3/M01/00/00/wKhQE12y9byADddiACdpr-L7emo585.png[root@fastdfs-storage1-group1 source]# /usr/bin/fdfs_upload_file /etc/fdfs/client.conf /source/girl.png ^[[Agroup3/M02/00/00/wKhQEV2y9b2AFuifACdpr-L7emo318.png[root@fastdfs-storage1-group1 source]# /usr/bin/fdfs_upload_file /etc/fdfs/client.conf /source/girl.png group3/M02/00/00/wKhQEl2y9b2AQ3prACdpr-L7emo501.png[root@fastdfs-storage1-group1 source]# /usr/bin/fdfs_upload_file /etc/fdfs/client.conf /source/girl.png group3/M02/00/00/wKhQE12y9b2AbG-UACdpr-L7emo027.png[root@fastdfs-storage1-group1 source]# /usr/bin/fdfs_upload_file /etc/fdfs/client.conf /source/girl.png group1/M00/00/00/wKhQC12y9b6AZ866ACdpr-L7emo376.png[root@fastdfs-storage1-group1 source]# /usr/bin/fdfs_upload_file /etc/fdfs/client.conf /source/girl.png group1/M00/00/00/wKhQDF2y9b6ABGW6ACdpr-L7emo312.png[root@fastdfs-storage1-group1 source]# /usr/bin/fdfs_upload_file /etc/fdfs/client.conf /source/girl.png group1/M00/00/00/wKhQDV2y9b6AFEd_ACdpr-L7emo639.png[root@fastdfs-storage1-group1 source]# /usr/bin/fdfs_upload_file /etc/fdfs/client.conf /source/girl.png group1/M01/00/00/wKhQC12y9b-AU07YACdpr-L7emo467.png[root@fastdfs-storage1-group1 source]# /usr/bin/fdfs_upload_file /etc/fdfs/client.conf /source/girl.png group1/M01/00/00/wKhQDF2y9b-ALDNrACdpr-L7emo209.png[root@fastdfs-storage1-group1 source]# /usr/bin/fdfs_upload_file /etc/fdfs/client.conf /source/girl.png group1/M01/00/00/wKhQDV2y9b-AAsKGACdpr-L7emo483.png[root@fastdfs-storage1-group1 source]# /usr/bin/fdfs_upload_file /etc/fdfs/client.conf /source/girl.png group1/M02/00/00/wKhQC12y9b-AfzkvACdpr-L7emo440.png[root@fastdfs-storage1-group1 source]# /usr/bin/fdfs_upload_file /etc/fdfs/client.conf /source/girl.png group1/M02/00/00/wKhQDF2y9cCATq8TACdpr-L7emo504.png[root@fastdfs-storage1-group1 source]# /usr/bin/fdfs_upload_file /etc/fdfs/client.conf /source/girl.png group1/M02/00/00/wKhQDV2y9cCAUPEnACdpr-L7emo555.png[root@fastdfs-storage1-group1 source]# /usr/bin/fdfs_upload_file /etc/fdfs/client.conf /source/girl.png group1/M03/00/00/wKhQC12y9cCABjj9ACdpr-L7emo576.png[root@fastdfs-storage1-group1 source]# /usr/bin/fdfs_upload_file /etc/fdfs/client.conf /source/girl.png group1/M03/00/00/wKhQDF2y9cGAHNLqACdpr-L7emo735.png[root@fastdfs-storage1-group1 source]# /usr/bin/fdfs_upload_file /etc/fdfs/client.conf /source/girl.png（7）集成到SpringBoot工程进行测试
在SpringBoot中使用FastDFS的时候需要集成fastdfs-spring-boot-starter：
项目主页GitHub地址：https://github.com/bluemiaomiao/fastdfs-spring-boot-starter
项目主页码云地址：https://gitee.com/bluemiaomiao/fastdfs-spring-boot-starter克隆项目到本地
git clone https://github.com/bluemiaomiao/fastdfs-spring-boot-starter.gitcd fastdfs-spring-boot-starter
使用Maven编译打包并安装到本地
mvn clean installmvn source:jar installmvn javadoc:jar install
在POM文件中添加依赖
<dependency> <groupId>com.bluemiaomiao</groupId> <artifactId>fastdfs-spring-boot-starter</artifactId> <version>1.0-SNAPSHOT</version></dependency>
在主配置类上添加注解 (
@EnableFastdfsClient)
@EnableFastdfsClient@SpringBootApplicationpublic class DemoApplication { @Autowired private FastdfsClientService fastdfsClientService; public static void main(String[] args) { SpringApplication.run(DemoApplication.class, args); }}
添加配置条目(application.properties)
fastdfs.nginx-servers=192.168.80.50:80,192.168.80.51:80fastdfs.tracker-servers=192.168.80.21:22122,192.168.80.22:22122,192.168.80.23:22122fastdfs.http-secret-key=2scPwMPctXhbLVOYB0jyuyQzytOofmFCBIYe65n56PPYVWrn# 在本项目中没有使用Token功能fastdfs.http-anti-steal-token=falsefastdfs.http-tracker-http-port=8080fastdfs.network-timeout=30fastdfs.connect-timeout=5fastdfs.connection-pool-max-idle=18fastdfs.connection-pool-min-idle=2fastdfs.connection-pool-max-total=18fastdfs.charset=UTF-8
或者添加配置条目(application.yml)
fastdfs: charset: UTF-8 connect-timeout: 5 http-secret-key: 2scPwMPctXhbLVOYB0jyuyQzytOofmFCBIYe65n56PPYVWrn network-timeout: 30 http-anti-steal-token: false http-tracker-http-port: 8080 connection-pool-max-idle: 20 connection-pool-max-total: 20 connection-pool-min-idle: 2 nginx-servers: 192.168.80.50:80,192.168.80.51:80 tracker-servers: 192.168.80.21:22122,192.168.80.22:22122,192.168.80.23:22122
测试主要代码
@RestController@RequestMapping("/file")public class FileController { @Autowired private FastdfsClientService fastdfsClientService; @PostMapping("/upload") public String[] upload(@RequestParam("file") MultipartFile file) { String[] remoteInfo = null; try { remoteInfo = fastdfsClientService.upload("group1", file.getBytes(), "png", null); } catch (Exception e) { e.printStackTrace(); } return remoteInfo; } @GetMapping("/download") public String download(@RequestParam("group") String group, @RequestParam("file_id") String fileId) { String url = ""; try { url = fastdfsClientService.autoDownloadWithoutToken(group, fileId, UUID.randomUUID().toString()); } catch (Exception e) { e.printStackTrace(); } return url; }}
项目测试代码：https://github.com/bluemiaomiao/fastdfs-cluster-test.git
国内码云：https://gitee.com/bluemiaomiao/fastdfs-cluster-test.git声明：本站所有文章资源内容，如无特殊说明或标注，均为采集网络资源。如若本站内容侵犯了原著者的合法权益，可联系本站删除。
