moosefs服务

MooseFS查看官方文档MooseFS — Distributed Storage五台虚拟机server1充当masterserver2、server3充当chunkserverserver4充当client初始配置添加GPG密钥curl https://repository.moosefs.com/RPM-GPG-KEY-MooseFS /etc/pki/rpm-gpg/RPM-GPG-KEY-MooseFS添加仓库文件curl http://repository.moosefs.com/MooseFS-4-el9.repo /etc/yum.repos.d/MooseFS.repo源路径[rootserver1 ~]# cd /etc/yum.repos.d[rootserver1 yum.repos.d]# cat MooseFS.repo安装 MooseFS 组件yum install moosefs-master moosefs-cgi moosefs-cgiserv moosefs-cli # For Master Serversyum install moosefs-chunkserver # For Chunkserversyum install moosefs-metalogger # For Metaloggersyum install moosefs-client # For ClientsDNS解析[rootserver1 ~]# vim /etc/hosts192.168.164.181 server1 mfsmaster192.168.164.182 server2192.168.164.183 server3192.168.164.184 server4连通性测试[rootserver1 ~]# ping -c 3 mfsmasterserver1性能模式master但是对于虚拟机没用[rootserver1 ~]# yum install -y tuned启动服务[rootserver1 ~]# systemctl enable --now moosefs-master.service[rootserver1 ~]# systemctl enable --now moosefs-gui.service[rootserver1 ~]# netstat -antlp浏览器访问192.168.164.181:9425server2、server3配置文件[rootserver2 ~]# cd /etc/mfs[rootserver2 mfs]# ls[rootserver2 mfs]# vim mfschunkserver.cfg 默认即可关闭server2、server3各添加一块硬盘20Gserver2[rootserver2 ~]# fdisk -l[rootserver2 ~]# fdisk /dev/sdb 输入n、回车、回车、回车、回车、p、w[rootserver2 ~]# mkfs.xfs /dev/sdb1[rootserver2 ~]# mkdir /mnt/ssd01[rootserver2 ~]# blkid/dev/sdb1: UUID388dfe42-9842-4c76-91a1-73b569d4f5fb TYPExfs PARTUUID34b0a0c7-01[rootserver2 ~]# vim /etc/fstab[rootserver2 ~]# systemctl daemon-reload[rootserver2 ~]# mount -a[rootserver2 ~]# df[rootserver2 ~]# chown mfs.mfs /mnt/ssd01[rootserver2 ~]# ll -d /mnt/ssd01[rootserver2 ~]# systemctl enable --now moosefs-chunkserver配置[rootserver2 ~]# vim /etc/mfs/mfshdd.cfg/mnt/ssd01[rootserver2 ~]# systemctl reload moosefs-chunkserverMooseFS Chunkserver 的数据存储目录[rootserver2 ~]# cd /mnt/ssd01[rootserver2 ssd01]# ls[rootserver2 ssd01]# ls | wc -lserver3[rootserver3 ~]# vim /etc/mfs/mfshdd.cfg/mnt/ssd01[rootserver3 ~]# fdisk -l[rootserver3 ~]# fdisk /dev/sdb[rootserver3 ~]# mkfs.xfs /dev/sdb1[rootserver3 ~]# mkdir /mnt/ssd01[rootserver3 ~]# blkid[rootserver3 ~]# vim /etc/fstab[rootserver3 ~]# systemctl daemon-reload[rootserver3 ~]# mount -a[rootserver3 ~]# df[rootserver3 ~]# chown mfs.mfs /mnt/ssd01[rootserver3 ~]# ll -d /mnt/ssd01[rootserver3 ~]# systemctl enable --now moosefs-chunkserver浏览器访问192.168.164.181:9425Disks中Disks,select选择 All disksserver4存储类管理/mnt/mfs挂载成功[rootserver4 ~]# cd /etc/mfs[rootserver4 mfs]# vim mfsmount.cfg[rootserver4 mfs]# mkdir /mnt/mfs[rootserver4 mfs]# mfsmount[rootserver4 mfs]# df[rootserver4 mfs]# cd /mnt/mfs[rootserver4 mfs]# mfslistsclass[rootserver4 mfs]# mfslistsclass -l[rootserver4 mfs]# mkdir -p /mnt/mfs/dir1[rootserver4 mfs]# cd /mnt/mfs/dir1[rootserver4 dir1]# mfsscadmin create -K 1* hot_1cpstorage class make hot_1cp: ok[rootserver4 dir1]# cp /etc/passwd .[rootserver4 dir1]# ls查看这个文件 chunk 块详细信息[rootserver4 dir1]# mfsfileinfo passwd[rootserver4 dir1]# mfssclass get .[rootserver4 dir1]# mfssclass set hot_1cp -r .[rootserver4 dir1]# mfsfileinfo passwd[rootserver4 dir1]# mfssclass get passwd[rootserver4 dir1]# mfssclass get .[rootserver4 dir1]# mfscheckfile passwd[rootserver4 dir1]# mkdir -p /mnt/mfs/dir2[rootserver4 dir1]# cd /mnt/mfs/dir2[rootserver4 dir2]# cp /etc/fstab .[rootserver4 dir2]# mfsfileinfo fstab[rootserver4 dir2]# cd ..[rootserver4 mfs]# cd dir1[rootserver4 dir1]# mfsfileinfo passwdserver2[rootserver2 ~]# vim /etc/mfs/mfschunkserver.cfgLABELS A[rootserver2 ~]# systemctl reload moosefs-chunkserver.serviceserver3[rootserver3 ~]# vim /etc/mfs/mfschunkserver.cfgLABELS B[rootserver3 ~]# systemctl reload moosefs-chunkserver.serviceserver4配合 chunkserver 节点标签使用带标签约束的存储类[rootserver4 mfs]# mfsscadmin create -K A 1CP_A[rootserver4 mfs]# mfssclass set -r 1CP_A dir1[rootserver4 mfs]# mfssclass get dir1[rootserver4 mfs]# cd dir1[rootserver4 dir1]# mfsfileinfo passwd[rootserver4 dir1]# cd ..[rootserver4 mfs]# cd dir2[rootserver4 dir2]# mfsfileinfo fstabserver5解析[rootserver5 ~]# curl https://repository.moosefs.com/RPM-GPG-KEY-MooseFS /etc/pki/rpm-gpg/RPM-GPG-KEY-Mo oseFS[rootserver5 ~]# curl http://repository.moosefs.com/MooseFS-4-el9.repo /etc/yum.repos.d/MooseFS.repo[rootserver5 ~]# yum install -y moosefs-chunkserver[rootserver5 ~]# mkdir /mnt/hdd01[rootserver5 ~]# chown mfs.mfs /mnt/hdd01[rootserver5 ~]# vim /etc/mfs/mfshdd.cfg/mnt/hdd01[rootserver5 ~]# systemctl enable --now moosefs-chunkserver[rootserver5 ~]# vim /etc/mfs/mfschunkserver.cfgLABELS C[rootserver5 ~]# systemctl reload moosefs-chunkserver.serviceserver4[rootserver4 mfs]# mfssclass set -r 2CP_AC .[rootserver4 mfs]# mfssclass set -r 2CP_AC .[rootserver4 mfs]# mfsfileinfo fstabserver2[rootserver2 ~]# vim /etc/mfs/mfschunkserver.cfgLABELS A S[rootserver2 ~]# systemctl reload moosefs-chunkserver.serviceserver3[rootserver3 ~]# vim /etc/mfs/mfschunkserver.cfgLABELS B S[rootserver3 ~]# systemctl reload moosefs-chunkserver.serviceserver5[rootserver5 ~]# vim /etc/mfs/mfschunkserver.cfgLABELS C H[rootserver5 ~]# systemctl reload moosefs-chunkserver.serviceserver4[rootserver4 mfs]# cd /mnt/mfs[rootserver4 mfs]# mkdir dir3[rootserver4 mfs]# mfssclass set -r 3CP dir3[rootserver4 mfs]# cd dir3[rootserver4 dir3]# cp /etc/hosts .[rootserver4 dir3]# mfsfileinfo hosts[rootserver4 dir3]# mfscreatesclass -K S,H 2CP_SH[rootserver4 dir3]# cd ..[rootserver4 mfs]# mfsfileinfo dir2/fstab[rootserver4 mfs]# mfssetsclass -r 2CP_SH dir2[rootserver4 mfs]# mfsfileinfo dir2/fstab[rootserver4 mfs]# cd dir2[rootserver4 dir2]# cp /etc/hostname .[rootserver4 dir2]# mfsfileinfo hostnameMFS 元数据挂载回收站 trash 文件删除与恢复rm 删除的文件不会立刻消失存放在 trash 回收站必须‑m 元数据挂载才可以访问回收站[rootserver4 dir2]# cd /mnt[rootserver4 mnt]# mkdir meta[rootserver4 mnt]# mfsmount -m /mnt/meta进入回收站目录里面存放所有被 rm 删除、还没超时彻底销毁的文件[rootserver4 mnt]# cd meta[rootserver4 meta]# cd trash查看目录的回收站保留时间[rootserver4 trash]# mfsgettrashretention /mnt/mfs/dir1[rootserver4 trash]# cd /mnt/mfs/dir1[rootserver4 dir1]# cp /etc/hosts .[rootserver4 dir1]# rm -f hosts[rootserver4 dir1]# cd /mnt/meta/trash[rootserver4 trash]# find -name *hosts*[rootserver4 trash]# mv ./00E/0000000E|dir1|hosts undel/ 单引号包含文件名[rootserver4 trash]# cd /mnt/mfs/dir1[rootserver4 dir1]# ls浏览器访问 192.168.164.181:9425/metrics标签存放安装[rootserver4 ~]# mfsmount[rootserver4 ~]# cd /mnt/mfs[rootserver4 mfs]# mfscreatesclass -K A,B 2CP_AB[rootserver4 mfs]# mfssetsclass -r 2CP_AB dir2/[rootserver4 mfs]# mfssetsclass -r 2CP_AB dir3/[rootserver4 mfs]# mfsfileinfo dir2/hostname[rootserver4 mfs]# mfsfileinfo dir3/hosts下线server5[rootserver5 ~]# systemctl disable --now moosefs-chunkserver.serviceserver4[rootserver4 ~]# umount /mnt/mfs停止server2、server3上chunkserver服务server1[rootserver1 mfs]# systemctl disable --now moosefs-master.service[rootserver1 mfs]# ls metadata.mfs启动必要文件开启高可用[rootserver1 mfs]# cd[rootserver1 ~]# cd /etc/yum.repos.d/[rootserver1 yum.repos.d]# vim rocky-addons.repoenabled1[rootserver1 yum.repos.d]# yum install -y pacemaker corosync pcs fence-agentsserver5[rootserver5 ~]# vim /etc/yum.repos.d/rocky-addons.repoenabled1[rootserver5 ~]# yum install -y pacemaker corosync pcs fence-agents[rootserver1 ~]# systemctl enable --now pcsd[rootserver5 ~]# systemctl enable --now pcsd免密[rootserver1 ~]# ssh-keygen[rootserver1 ~]# ssh-copy-id server5[rootserver1 ~]# echo westos | passwd --stdin hacluster[rootserver1 ~]# ssh server5 echo westos | passwd --stdin hacluster主机认证[rootserver1 ~]# pcs host auth server1 server5 -u hacluster -p westosserver1: Authorizedserver5: Authorized[rootserver1 ~]# pcs cluster setup mycluster server1 server5 不指定地址注意解析[rootserver1 ~]# pcs cluster start --allserver5: Starting Cluster...server1: Starting Cluster...[rootserver1 ~]# pcs cluster enable --allserver1: Cluster Enabledserver5: Cluster Enabled[rootserver1 ~]# pcs property set stonith-enabledfalse[rootserver1 ~]# pcs property set no-quorum-policyignore[rootserver1 ~]# pcs statusVIP[rootserver1 ~]# pcs resource create VIP ocf:heartbeat:IPaddr2 ip192.168.164.200 cidr_netmask24 op monitor interval30s[rootserver1 ~]# pcs statusserver5上 watch pcs status解析同步所有节点[rootserver1 ~]# vim /etc/hosts192.168.164.181 server1192.168.164.200 mfsmaster[rootserver1 ~]# scp /etc/hosts server2:/etc/hosts[rootserver1 ~]# scp /etc/hosts server3:/etc/hosts[rootserver1 ~]# scp /etc/hosts server4:/etc/hosts[rootserver1 ~]# scp /etc/hosts server5:/etc/hosts做存储server1上[rootserver1 ~]# yum list iscsi-*[rootserver1 ~]# yum install -y iscsi-initiator-utils.x86_64[rootserver5 ~]# yum install -y iscsi-initiator-utils.x86_64在discovery之前进行否则产生缓存[rootserver1 ~]# cd /etc/iscsi[rootserver1 ~]# vim initiatorname.iscsi[rootserver1 iscsi]# vim initiatorname.iscsiInitiatorNameiqn.2026-08.com.example:clientiqn.2026-08.com.example:client[rootserver1 iscsi]# scp initiatorname.iscsi server5:/etc/iscsi[rootserver1 iscsi]# iscsiadm -m discovery -t st -p 192.168.164.184[rootserver5 ~]# iscsiadm -m discovery -t st -p 192.168.164.184如果有错误可以通过重启清除缓存systemctl restart isccsid[rootserver5 ~]# iscsiadm -m node -l[rootserver5 ~]# fdisk -lserver4加磁盘做存储[rootserver4 ~]# yum install -y targetcli[rootserver4 ~]# systemctl enable --now target[rootserver4 ~]# targetcli/ ls/ /backstores/block create my_disk /dev/sdb/ /iscsi create iqn.2026-08.com.example:mfsdata/ /iscsi/iqn.2026-08.com.example:mfsdata/tpg1/luns create /backstores/block/my_disk/ /iscsi/iqn.2026-08.com.example:mfsdata/tpg1/acls create iqn.2026-08.com.example:client/ exit存储管理[rootserver1 ~]# fdisk /dev/sdb 输入n、p、回车、回车、回车、w[rootserver1 ~]# mkfs.xfs /dev/sdb1[rootserver1 ~]# mount /dev/sdb1 /mnt/[rootserver1 ~]# df[rootserver1 ~]# chown mfs.mfs /mnt/[rootserver1 ~]# cd /var/lib/mfs/[rootserver1 mfs]# ls[rootserver1 mfs]# cp -p * /mnt/[rootserver1 mfs]# cd /mnt/[rootserver1 mnt]# ll[rootserver1 mnt]# cd[rootserver1 ~]# umount /mnt[rootserver1 ~]# mount /dev/sdb1 /var/lib/mfs/[rootserver1 ~]# df[rootserver1 ~]# systemctl start moosefs-master.service[rootserver1 ~]# systemctl status moosefs-master.service[rootserver1 ~]# cd /var/lib/mfs/[rootserver1 mfs]# ls[rootserver1 mfs]# systemctl stop moosefs-master.service 只允许单节点写入[rootserver1 mfs]# ll 开机需要metadata.mfs[rootserver1 mfs]# cd[rootserver1 ~]# umount /var/lib/mfsserver5[rootserver5 ~]# yum list moosefs*[rootserver5 ~]# yum install -y moosefs-cli moosefs-master moosefs-gui[rootserver5 ~]# fdisk -l[rootserver5 ~]# partprobe[rootserver5 ~]# ll /dev/sdb1[rootserver5 ~]# mount /dev/sdb1 /var/lib/mfs/[rootserver5 ~]# df[rootserver5 ~]# systemctl start moosefs-master.service[rootserver5 ~]# systemctl status moosefs-master.service[rootserver5 ~]# systemctl is-enabled moosefs-master.service[rootserver5 ~]# umount /var/lib/mfs加存储[rootserver1 ~]# pcs resource describe ocf:heartbeat:Filesystem 设备、挂载点、文件系统[rootserver1 ~]# pcs resource create mfsdata ocf:heartbeat:Filesystem device/dev/sdb1 directory/var/lib/mfs fstypexfs op monitor interval1minserver5上[rootserver5 ~]# watch pcs status[rootserver1 ~]# pcs resource create mfsmaster systemd:moosefs-master op monitor interval30s组约束group[rootserver1 ~]# pcs resource group add mfscluster VIP mfsdata mfsmasterserver2、server3上systemctl start moosefs-chunkserver.service访问https://192.168.164.181:9425server4上[rootserver4 ~]# mfsmount[rootserver4 ~]# cd /mnt/mfs[rootserver4 mfs]# cd dir2/[rootserver4 dir2]# mfsfileinfo hostname资源都在server1上命令pcs node standby停用如果不切换使用pcs resource cleanup清理资源即可pcs node unstandby重新启用但是注意资源不会回切注意关机操作1.master停止所有集群资源停止集群 pcs cluster stop --all pcs cluster disable --all2.手动登出业务 iSCSI 会话master)iscsiadm -m node -T iqn.2026-08.com.example:mfsdata --logout3.双节点停止集群服务pcs cluster stop4.再执行关机poweroff故障转移server5iscsiadm -m node -T iqn.2026-08.com.example:mfsdata --loginserver1迁移资源组pcs resource move mfscluster server5server1立刻登出iscsiiscsiadm -m node -T iqn.2026-08.com.example:mfsdata --logoutpcs statusSBD服务首先确保集群状态,确认master给 server4 虚拟机新增一块磁盘lsblk在 server4 的 targetcli 中配置 SBD 共享盘targetcli创建 SBD 专用的 block 存储对象/backstores/block create sbd_disk /dev/sdc将 SBD 盘添加到现有 iSCSI Target 中作为第二个 LUN/iscsi/iqn.2026-08.com.example:mfsdata/tpg1/luns create /backstores/block/sbd_disk验证 ACL 自动映射ls /iscsi/iqn.2026-08.com.example:mfsdata/tpg1/acls/iqn.2026-08.com.example:client/saveconfigexitserver1、server5 端扫描并确认新磁盘iscsiadm -m session --rescanlsblk获取磁盘稳定 by-id 路径部署 SBD 时使用避免漂移ls -l /dev/disk/by-id/ | grep scsi 确认双节点上新盘的名称完全一致SBD 防脑裂部署双节点安装 SBD 相关软件包yum install -y sbd fence-agents-sbd双节点配置软件看门狗虚拟机必做加载软件看门狗内核模块 modprobe softdog设置开机自动加载 echo softdog /etc/modules-load.d/softdog.conf验证看门狗设备生成 ls -l /dev/watchdog初始化 SBD 磁盘并分配节点槽位仅在master执行即可sbd -d /dev/sdc createsbd -d /dev/sdc allocate server1sbd -d /dev/sdc allocate server5sbd -d /dev/sdc list输出server1、server2两个节点状态均为clear双节点配置 SBD 主配置文件vim /etc/sysconfig/sbdSBD_DEVICE/dev/sdcSBD_OPTS-Wcat /etc/sysconfig/sbd启动 SBD 服务,rocky9server1、server5 全部执行systemctl enable sbdsystemctl restart corosync pacemakersystemctl status sbd开启 STONITH 隔离仅在master执行即可pcs stonith create sbd-stonith fence_sbd pcmk_host_listserver1 server5pcs property set stonith-enabledtrue查看状态主动隔离测试在server1上执行隔离server5节点pcs stonith fence server1server1 终端弹出强制重启日志SSH连接断开系统硬重启。server5上执行pcs status可看到mfscluster资源组完整漂移到server5全部资源Started。server1重启完成后自动重新加入集群恢复 Online 状态查看SBD磁盘槽位心跳状态 sbd -d SBD磁盘路径 list查看隔离操作历史 pcs stonith history查看SBD服务实时日志 journalctl -u sbd -f