初探OCFS2-配置服务

2016/11/28

目的：研究如何使用iscsi存储。一、认识据zstack的童鞋们介绍：OCFS2是主流对接SAN云存储的方式存储的选择，通常有以下方式：1、NAS1）NFS简单；自建有可靠性问题，商业的NFS产品更靠谱；2）分布式文件系统，以GlusterFS为例；易懂；建议使用glusterfs的接口来访问，但也支持通过nfs来访问；2、IPSAN（iscsi）1）使用iscsiadm发现一下target即可2）研究ovirt是如何使用iscsi存储的看到一个讨论ovirt通过lvm来使用iscsi的方式http://users.ovirt.narkive.com/XzD01akX/ovirt-users-shared-storage-with-iscsi按照字面意思，我的理解（未实验）是：a、每个host挂载iscsi存储b、通过lvm划分出vmdiskc、在运行了vm的host上通过lvchange来激活指定的vmdisklvchange-ayvol_group/logical_vol3）通过集群文件系统（clusterfilesystem:OCFS2,GFS,etc.）来控制读写iscsi存储这就是本文的初衷，接下来展开记录OCFS2是如何使用的。二、配置1、准备安装oracleuek内核（因为新版本的ocfs2内核模块被uek整合了）[root@n149~]#wgethttp://public-yum.oracle.com/public-yum-ol6.repo-O/etc/yum.repos.d/public-yum-ol6.repo[root@n149~]#rpm--importhttp://public-yum.oracle.com/RPM-GPG-KEY-oracle-ol6[root@n149~]#yuminstallyum-plugin-downloadonly-y[root@n149~]#mkdir/tmp/ocfs2&&cd/tmp/ocfs2/[root@n149ocfs2]#yuminstall--downloadonly--downloaddir=/tmp/ocfs2/kernel-uekocfs2-tools-y2、安装uek[root@n149ocfs2]#yumlocalinstallkernel-uek*使用新的内核，编辑grub调整启动顺序：[root@n149ocfs2]#vim/etc/grub.conf重启系统后查看：[root@n149ocfs2]#uname-r2.6.39-400.290.1.el6uek.x86_64查看ocfs2模块是否加载：[root@n149ocfs2]#modprobeocfs2[root@n149ocfs2]#lsmod|grepococfs210944390ocfs2_nodemanager2232531ocfs2configfs293082ocfs2_nodemanagerocfs2_stackglue118961ocfs2jbd2931142ocfs2,ext43、安装ocfs2-tools[root@n149ocfs2]#yumlocalinstallocfs2-tools*-y4、配置ocfs2服务[root@n149~]#vim/etc/hosts10.50.200.149n149.ocfs2.test10.50.200.150n150.ocfs2.test[root@n149~]#serviceo2cbloadcheckingdebugfs...Mountingconfigfsfilesystemat/sys/kernel/config:OKLoadingstackplugin"o2cb":OKLoadingfilesystem"ocfs2_dlmfs":OKCreatingdirectory'/dlm':OKMountingocfs2_dlmfsfilesystemat/dlm:OK[root@n149~]#serviceo2cbonlinecheckingdebugfs...Settingclusterstack"o2cb":OKCheckingO2CBclusterconfiguration:Failed符合预期，继续配置集群：[root@n149~]#o2cb_ctl-C-nprdcluster-tcluster-aname=prdcluster[root@n149~]#o2cb_ctl-C-nn149.ocfs2.test-tnode-anumber=0-aip_address=10.50.200.149-aip_port=11111-acluster=prdcluster[root@n149~]#o2cb_ctl-C-nn150.ocfs2.test-tnode-anumber=1-aip_address=10.50.200.150-aip_port=11111-acluster=prdcluster[root@n149~]#cat/etc/ocfs2/cluster.confnode:name=n149.ocfs2.testcluster=prdclusternumber=0ip_address=10.50.200.149ip_port=11111node:name=n150.ocfs2.testcluster=prdclusternumber=1ip_address=10.50.200.150ip_port=11111cluster:name=prdclusterheartbeat_mode=localnode_count=2[root@n149~]#serviceo2cbconfigure[root@n149~]#serviceo2cbonline上述操作，在每个节点上执行。其他可选操作：[root@n149~]#serviceo2cbstatusDriverfor"configfs":LoadedFilesystem"configfs":MountedStackgluedriver:LoadedStackplugin"o2cb":LoadedDriverfor"ocfs2_dlmfs":LoadedFilesystem"ocfs2_dlmfs":MountedCheckingO2CBcluster"prdcluster":OnlineHeartbeatdeadthreshold:31Networkidletimeout:30000Networkkeepalivedelay:2000Networkreconnectdelay:2000Heartbeatmode:LocalCheckingO2CBheartbeat:NotactiveDebugfilesystemat/sys/kernel/debug:mounted另外：确认防火墙放行了对应的端口，本例是tcpport111115、配置iscsi存储[root@n149ocfs2]#iscsiadm-mdiscovery-tst-p10.50.200.41[root@n149ocfs2]#iscsiadm-mnode-odelete-Tiqn.1984-05.com.dell:powervault.md3800i.600a098000a64d8b0000000057a152ea-p192.168.131.101[root@n149ocfs2]#iscsiadm-mnode-odelete-Tiqn.1984-05.com.dell:powervault.md3800i.600a098000a64d8b0000000057a152ea-p192.168.131.102[root@n149ocfs2]#iscsiadm-mnode-Tiqn.1984-05.com.dell:powervault.md3800i.600a098000a64d8b0000000057a152ea--login[root@n149ocfs2]#yuminstalldevice-mapper-multipath-y[root@n149ocfs2]#modprobedm-multipath[root@n149ocfs2]#mpathconf[root@n149ocfs2]#cat/etc/multipath.confdefaults{path_grouping_policymultibuspath_checkerreadsector0rr_min_io_rq10max_fds8192rr_weightprioritiesfailbackimmediateno_path_retryfailuser_friendly_namesyes}multipaths{multipath{wwid3600a098000a64d8b000002a4582305f8aliastest_block_a}}[root@n149ocfs2]#servicemultipathdstart[root@n149ocfs2]#chkconfigmultipathdon[root@n149ocfs2]#multipath-lltest_block_a(3600a098000a64d8b000002a4582305f8)dm-0DELL,MD38xxisize=100Gfeatures='0'hwhandler='0'wp=rw`-+-policy='round-robin0'prio=1status=active|-4:0:0:0sda8:0activereadyrunning`-5:0:0:0sdb8:16activereadyrunning6、使用iscsi存储挂载参数请参考：manmkfs.ocfs2[root@n149~]#mkfs.ocfs2-b4K-C64K-Ltestblocka-Tdatafiles-F/dev/mapper/test_block_amkfs.ocfs21.8.6Clusterstack:classico2cbOverwritingexistingocfs2partition.WARNING:Clustercheckdisabled.Proceed(y/N):yFilesystemTypeofdatafilesLabel:testblockaFeatures:sparseextended-slotmapbackup-superunwritteninline-datastrict-journal-superxattrindexed-dirsrefcountdiscontig-bgBlocksize:4096(12bits)Clustersize:65536(16bits)Volumesize:107374182400(1638400clusters)(26214400blocks)Clustergroups:51(tailcovers25600clusters,restcover32256clusters)Extentallocatorsize:41943040(10groups)Journalsize:33554432Nodeslots:8Creatingbitmaps:doneInitializingsuperblock:doneWritingsystemfiles:doneWritingsuperblock:doneWritingbackupsuperblock:4block(s)FormattingJournals:doneGrowingextentallocator:doneFormattingslotmap:doneFormattingquotafiles:doneWritinglost+found:donemkfs.ocfs2successful[root@n149~]#cat<<_EOF>>/etc/fstab/dev/mapper/test_block_a/mnt/tbaocfs2defaults00_EOF[root@n149~]#mkdir/mnt/tba[root@n149~]#mount-a[root@n149~]#df-h/mnt/tbaFilesystemSizeUsedAvailUse%Mountedon/dev/mapper/test_block_a100G586M100G1%/mnt/tba在另一个节点上执行：[root@n150~]#cat<<_EOF>>/etc/fstab/dev/mapper/test_block_a/mnt/tbaocfs2defaults00_EOF[root@n150~]#mkdir/mnt/tba[root@n150~]#mount-a[root@n150~]#df-h/mnt/tbaFilesystemSizeUsedAvailUse%Mountedon/dev/mapper/test_block_a100G586M100G1%/mnt/tba查看对应的日志：Nov1613:46:01n149kernel:o2hb:HeartbeatmodesettolocalNov1613:46:01n149o2hbmonitor:StartingNov1613:47:42n149kernel:OCFS21.8.0Nov1613:47:43n149kernel:o2dlm:Joiningdomain55663D9EA9B0466D80626459E2B6AE9B(1)1nodesNov1613:47:43n149kernel:ocfs2:Mountingdevice(251,0)on(node1,slot0)withordereddatamode.Nov1613:48:16n149kernel:o2net:Connectiontonoden150.ocfs2.test(num0)at10.50.200.150:11111shutdown,state7Nov1613:48:18n149kernel:o2net:Connectedtonoden150.ocfs2.test(num0)at10.50.200.150:11111Nov1613:48:22n149kernel:o2dlm:Node0joinsdomain55663D9EA9B0466D80626459E2B6AE9B(01)2nodesNov1613:45:43n150kernel:o2hb:HeartbeatmodesettolocalNov1613:45:43n150o2hbmonitor:StartingNov1613:48:18n150kernel:o2net:Acceptedconnectionfromnoden149.ocfs2.test(num1)at10.50.200.149:11111Nov1613:48:22n150kernel:OCFS21.8.0Nov1613:48:22n150kernel:o2dlm:Joiningdomain55663D9EA9B0466D80626459E2B6AE9B(01)2nodesNov1613:48:22n150kernel:ocfs2:Mountingdevice(251,0)on(node0,slot1)withordereddatamode.2个节点的heartbeat的状态也都变成了Active[root@n149~]#serviceo2cbstatusDriverfor"configfs":LoadedFilesystem"configfs":MountedStackgluedriver:LoadedStackplugin"o2cb":LoadedDriverfor"ocfs2_dlmfs":LoadedFilesystem"ocfs2_dlmfs":MountedCheckingO2CBcluster"prdcluster":OnlineHeartbeatdeadthreshold:31Networkidletimeout:30000Networkkeepalivedelay:2000Networkreconnectdelay:2000Heartbeatmode:LocalCheckingO2CBheartbeat:ActiveDebugfilesystemat/sys/kernel/debug:mounted7、测试文件读写【2个节点同时写一个文件】[root@n149~]#foriin`seq19`;doecho"`hostname`->[`date`]">>/mnt/tba/1.log;sleep1s;done[root@n150~]#foriin`seq19`;doecho"`hostname`->[`date`]">>/mnt/tba/1.log;sleep1s;done[root@n149~]#cat/mnt/tba/1.logn149.ocfs2.test->[WedNov1613:55:20CST2016]n150.ocfs2.test->[WedNov1613:55:21CST2016]n149.ocfs2.test->[WedNov1613:55:21CST2016]n150.ocfs2.test->[WedNov1613:55:22CST2016]n149.ocfs2.test->[WedNov1613:55:22CST2016]n150.ocfs2.test->[WedNov1613:55:23CST2016]n149.ocfs2.test->[WedNov1613:55:23CST2016]n150.ocfs2.test->[WedNov1613:55:24CST2016]n149.ocfs2.test->[WedNov1613:55:24CST2016]n150.ocfs2.test->[WedNov1613:55:25CST2016]n149.ocfs2.test->[WedNov1613:55:25CST2016]n150.ocfs2.test->[WedNov1613:55:26CST2016]n149.ocfs2.test->[WedNov1613:55:26CST2016]n150.ocfs2.test->[WedNov1613:55:27CST2016]n149.ocfs2.test->[WedNov1613:55:27CST2016]n150.ocfs2.test->[WedNov1613:55:28CST2016]n149.ocfs2.test->[WedNov1613:55:28CST2016]n150.ocfs2.test->[WedNov1613:55:29CST2016]结论：符合预期。【一个节点写，另一个节点读】[root@n149~]#foriin`seq19`;doecho"`hostname`->[`date`]">>/mnt/tba/1.log;sleep1s;done[root@n150~]#tail-f/mnt/tba/1.logn149.ocfs2.test->[WedNov1613:57:42CST2016]n149.ocfs2.test->[WedNov1613:57:43CST2016]n149.ocfs2.test->[WedNov1613:57:44CST2016]n149.ocfs2.test->[WedNov1613:57:45CST2016]n149.ocfs2.test->[WedNov1613:57:46CST2016]n149.ocfs2.test->[WedNov1613:57:47CST2016]n149.ocfs2.test->[WedNov1613:57:48CST2016]n149.ocfs2.test->[WedNov1613:57:49CST2016]n149.ocfs2.test->[WedNov1613:57:50CST2016]^C结论：符合预期。三、问题解释1、cluster处于Offline的状态[root@n150~]#mount-amount.ocfs2:Internallogicfailurewhiletryingtojointhegroup[root@n150~]#serviceo2cbstatusCheckingO2CBcluster"prdcluster":Offline则请检查各节点的域名是否能正常解析，ocfs2要用到hostname来提供服务。有一种情况是：上述节点的hostname为：n149.test.comn150.test.com而配置ocfs2是在hosts中写死了：10.50.200.149n149.ocfs2.test10.50.200.150n150.ocfs2.test而实际上，ocfs2使用的却是hostname，因而节点之间的连接异常。2、heartbeat处于Notactive状态挂载卷后heartbeat才会启动，因此，先操作后再观察。ZYXW、参考1、[ovirt-users]sharedstoragewithiscsihttp://users.ovirt.narkive.com/XzD01akX/ovirt-users-shared-storage-with-iscsi2、Linux:OracleunbreakablekernelandOCFS2onCentOS/RHEL6http://www.itechlounge.net/2014/08/linux-oracle-unbreakable-kernel-and-ocfs2-on-centosrhel6/3、OracleLinux6:CreateanOCFS2ClusterandFilesystemhttps://www.itfromallangles.com/2013/03/oracle-linux-6-create-an-ocfs2-cluster-and-filesystem/4、InstallingAndConfiguringAnOCFS2ClusteredFileSystemhttp://www.shapeblue.com/installing-and-configuring-an-ocfs2-clustered-file-system/5、解决o2cbstatus时状态始终为offline(o2cb_ctl:InternallogicfailurefailedwhileaddingRAC1)http://www.blogjava.net/KingKong/archive/2011/06/06/351826.html6、O2CBheartbeat:Notactivehttps://oss.oracle.com/pipermail/ocfs2-users/2009-April/003477.html