Oracle 11gR2 RAC添加新节点错误之--IP子网掩码

系统环境：

操作系统：RedHat EL5

Cluster： Oracle GI(Grid Infrastructure)

Oracle： Oracle 11.2.0.1.0

如图所示：RAC 系统架构

对于Oracle 11G构建RAC首先需要构建GI(Grid Infrastructure)的架构

案例分析：

在Oracle 11gR2 RAC在添加新节点时，添加grid，在node3上执行root.sh时，出现以下错误：

[root@lxh4 ~]# /u01/11.2.0/grid/root.sh

RunningOracle11groot.shscript...Thefollowingenvironmentvariablesaresetas:ORACLE_OWNER=gridORACLE_HOME=/u01/11.2.0/gridEnterthefullpathnameofthelocalbindirectory:[/usr/local/bin]:Copyingdbhometo/usr/local/bin...Copyingoraenvto/usr/local/bin...Copyingcoraenvto/usr/local/bin...Creating/etc/oratabfile...Entrieswillbeaddedtothe/etc/oratabfileasneededbyDatabaseConfigurationAssistantwhenadatabaseiscreatedFinishedrunninggenericpartofroot.shscript.Nowproduct-specificrootactionswillbeperformed.2014-07-1015:00:36:Parsingthehostname2014-07-1015:00:36:Checkingforsuperuserprivileges2014-07-1015:00:36:UserhassuperuserprivilegesUsingconfigurationparameterfile:/u01/11.2.0/grid/crs/install/crsconfig_paramsCreatingtracedirectoryLOCALADDMODECreatingOCRkeysforuser'root',privgrp'root'..Operationsuccessful.AddingdaemontoinittabCRS-4123:OracleHighAvailabilityServiceshasbeenstarted.ohasdisstartingCRS-4402:TheCSSdaemonwasstartedinexclusivemodebutfoundanactiveCSSdaemononnodelxh2,number1,andisterminatingAnactiveclusterwasfoundduringexclusivestartup,restartingtojointheclusterCRS-2672:Attemptingtostart'ora.mdnsd'on'lxh4'CRS-2676:Startof'ora.mdnsd'on'lxh4'succeededCRS-2672:Attemptingtostart'ora.gipcd'on'lxh4'CRS-2676:Startof'ora.gipcd'on'lxh4'succeededCRS-2672:Attemptingtostart'ora.gpnpd'on'lxh4'CRS-2676:Startof'ora.gpnpd'on'lxh4'succeededCRS-2672:Attemptingtostart'ora.cssdmonitor'on'lxh4'CRS-2676:Startof'ora.cssdmonitor'on'lxh4'succeededCRS-2672:Attemptingtostart'ora.cssd'on'lxh4'CRS-2672:Attemptingtostart'ora.diskmon'on'lxh4'CRS-2676:Startof'ora.diskmon'on'lxh4'succeededCRS-2676:Startof'ora.cssd'on'lxh4'succeededCRS-2672:Attemptingtostart'ora.ctssd'on'lxh4'CRS-2676:Startof'ora.ctssd'on'lxh4'succeededCRS-2672:Attemptingtostart'ora.drivers.acfs'on'lxh4'CRS-2676:Startof'ora.drivers.acfs'on'lxh4'succeededCRS-2672:Attemptingtostart'ora.asm'on'lxh4'CRS-2676:Startof'ora.asm'on'lxh4'succeededCRS-2672:Attemptingtostart'ora.crsd'on'lxh4'CRS-2676:Startof'ora.crsd'on'lxh4'succeededCRS-2672:Attemptingtostart'ora.evmd'on'lxh4'CRS-2676:Startof'ora.evmd'on'lxh4'succeededTimedoutwaitingfortheCRSstacktostart.

CRSD服务启动失败！

查看日志：

[root@lxh4 crsd]# more crsdOUT.log

2014-07-10 15:03:56

Changingdirectoryto/u01/11.2.0/grid/log/lxh4/crsd2014-07-1015:03:56CRSDREBOOTCRSDexiting:CouldnotinitOCR,code:442014-07-1015:03:58Changingdirectoryto/u01/11.2.0/grid/log/lxh4/crsd2014-07-1015:03:58CRSDREBOOTCRSDexiting:CouldnotinitOCR,code:442014-07-1015:04:00Changingdirectoryto/u01/11.2.0/grid/log/lxh4/crsd2014-07-1015:04:00CRSDREBOOTCRSDexiting:CouldnotinitOCR,code:442014-07-1015:04:02Changingdirectoryto/u01/11.2.0/grid/log/lxh4/crsd2014-07-1015:04:02CRSDREBOOTCRSDexiting:CouldnotinitOCR,code:442014-07-1015:04:04Changingdirectoryto/u01/11.2.0/grid/log/lxh4/crsd2014-07-1015:04:04CRSDREBOOTCRSDexiting:CouldnotinitOCR,code:442014-07-1015:04:06Changingdirectoryto/u01/11.2.0/grid/log/lxh4/crsd2014-07-1015:04:06CRSDREBOOTCRSDexiting:CouldnotinitOCR,code:442014-07-1015:04:08Changingdirectoryto/u01/11.2.0/grid/log/lxh4/crsd2014-07-1015:04:08CRSDREBOOTCRSDexiting:CouldnotinitOCR,code:442014-07-1015:04:10Changingdirectoryto/u01/11.2.0/grid/log/lxh4/crsd2014-07-1015:04:10CRSDREBOOTCRSDexiting:CouldnotinitOCR,code:442014-07-1015:04:12Changingdirectoryto/u01/11.2.0/grid/log/lxh4/crsd2014-07-1015:04:12CRSDREBOOTCRSDexiting:CouldnotinitOCR,code:442014-07-1015:04:14Changingdirectoryto/u01/11.2.0/grid/log/lxh4/crsd2014-07-1015:04:14CRSDREBOOTCRSDexiting:CouldnotinitOCR,code:442014-07-1015:04:16Changingdirectoryto/u01/11.2.0/grid/log/lxh4/crsd2014-07-1015:04:16CRSDREBOOTCRSDexiting:CouldnotinitOCR,code:44

[root@lxh4 crsd]# tail crsd.log

2014-07-1015:04:17.954:[GPnP][3046512336]clsgpnp_Init:[atclsgpnp0.c:837]GPnPclientpid=12069,tl=3,f=02014-07-1015:04:17.966:[OCRAPI][3046512336]clsu_get_private_ip_addresses:noipaddressesfound.2014-07-1015:04:17.966:[GIPCXCPT][3046512336]gipcShutdownF:skippingshutdown,count2,from[clsinet.c:1732],retgipcretSuccess(0)2014-07-1015:04:17.968:[GIPCXCPT][3046512336]gipcShutdownF:skippingshutdown,count1,from[clsgpnp0.c:1021],retgipcretSuccess(0)[OCRAPI][3046512336]a_init_clsss:failedtocallclsu_get_private_ip_addr(7)2014-07-1015:04:17.970:[OCRAPI][3046512336]a_init:13!:Clusterwareinitunsuccessful:[44]2014-07-1015:04:17.970:[CRSOCR][3046512336]OCRcontextinitfailure.Error:PROC-44:ErrorinnetworkaddressandinterfaceoperationsNetworkaddressandinterfaceoperationserror[7]2014-07-1015:04:17.970:[CRSD][3046512336][PANIC]CRSDexiting:CouldnotinitOCR,code:442014-07-1015:04:17.970:[CRSD][3046512336]Done.

一个重要的错误信息“2014-07-10 15:04:17.966: [ OCRAPI][3046512336]clsu_get_private_ip_addresses: no ip addresses found.”看来造成crsd启动失败的原因和private network interface 有关系

检查网络配置：

[root@lxh4 crsd]# ifconfig eth2

eth2 Link encap:Ethernet HWaddr 08:00:27:D5:85:37

inet addr:10.10.10.13 Bcast:10.255.255.255 Mask:255.0.0.0

inet6 addr: fe80::a00:27ff:fed5:8537/64 Scope:Link

UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

RX packets:2213 errors:0 dropped:0 overruns:0 frame:0

TX packets:11568 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:1000

RX bytes:281234 (274.6 KiB) TX bytes:4006863 (3.8 MiB)

[grid@lxh2 ~]$ /sbin/ifconfig eth2

eth2 Link encap:Ethernet HWaddr 08:00:27:AE:93:9A

inet addr:10.10.10.11 Bcast:10.10.10.255 Mask:255.255.255.0

inet6 addr: fe80::a00:27ff:feae:939a/64 Scope:Link

UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

RX packets:157629 errors:0 dropped:0 overruns:0 frame:0

TX packets:140367 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:1000

RX bytes:68428231 (65.2 MiB) TX bytes:48684593 (46.4 MiB)

node3的private network interface的netmask 为255.255.255.0，和其他node的netmask（255.255.255.0）不一致！

解决方法：

1、修改node3的private network interface的netmask为255.255.255.0

2、清除crs的配置信息，重新执行root.sh

[root@lxh4 install]# perl rootcrs.pl -deconfig -force

2014-07-1015:12:13:Parsingthehostname2014-07-1015:12:13:Checkingforsuperuserprivileges2014-07-1015:12:13:UserhassuperuserprivilegesUsingconfigurationparameterfile:./crsconfig_paramsPRCR-1035:FailedtolookupCRSresourceora.cluster_vip.typefor1PRCR-1068:FailedtoqueryresourcesCannotcommunicatewithcrsdPRCR-1070:Failedtocheckifresourceora.gsdisregisteredCannotcommunicatewithcrsdPRCR-1070:Failedtocheckifresourceora.onsisregisteredCannotcommunicatewithcrsdPRCR-1070:Failedtocheckifresourceora.eonsisregisteredCannotcommunicatewithcrsdACFS-9200:SupportedCRS-4535:CannotcommunicatewithClusterReadyServicesCRS-4000:CommandStopfailed,orcompletedwitherrors.CRS-2791:StartingshutdownofOracleHighAvailabilityServices-managedresourceson'lxh4'CRS-2673:Attemptingtostop'ora.cssdmonitor'on'lxh4'CRS-2673:Attemptingtostop'ora.ctssd'on'lxh4'CRS-2673:Attemptingtostop'ora.evmd'on'lxh4'CRS-2673:Attemptingtostop'ora.asm'on'lxh4'CRS-2673:Attemptingtostop'ora.mdnsd'on'lxh4'CRS-2673:Attemptingtostop'ora.drivers.acfs'on'lxh4'CRS-2677:Stopof'ora.cssdmonitor'on'lxh4'succeededCRS-2677:Stopof'ora.evmd'on'lxh4'succeededCRS-2677:Stopof'ora.mdnsd'on'lxh4'succeededCRS-2677:Stopof'ora.ctssd'on'lxh4'succeededCRS-2677:Stopof'ora.asm'on'lxh4'succeededCRS-2673:Attemptingtostop'ora.cssd'on'lxh4'CRS-2677:Stopof'ora.cssd'on'lxh4'succeededCRS-2673:Attemptingtostop'ora.gpnpd'on'lxh4'CRS-2673:Attemptingtostop'ora.diskmon'on'lxh4'CRS-2677:Stopof'ora.diskmon'on'lxh4'succeededCRS-2677:Stopof'ora.gpnpd'on'lxh4'succeededCRS-2673:Attemptingtostop'ora.gipcd'on'lxh4'CRS-2677:Stopof'ora.drivers.acfs'on'lxh4'succeededCRS-2677:Stopof'ora.gipcd'on'lxh4'succeededCRS-2793:ShutdownofOracleHighAvailabilityServices-managedresourceson'lxh4'hascompletedCRS-4133:OracleHighAvailabilityServiceshasbeenstopped.error:packagecvuqdiskisnotinstalledSuccessfullydeconfiguredOracleclusterwarestackonthisnode

[root@lxh4 crsd]# /u01/11.2.0/grid/root.sh

RunningOracle11groot.shscript...Thefollowingenvironmentvariablesaresetas:ORACLE_OWNER=gridORACLE_HOME=/u01/11.2.0/gridEnterthefullpathnameofthelocalbindirectory:[/usr/local/bin]:Thefile"dbhome"alreadyexistsin/usr/local/bin.Overwriteit?(y/n)[n]:yCopyingdbhometo/usr/local/bin...Thefile"oraenv"alreadyexistsin/usr/local/bin.Overwriteit?(y/n)[n]:yCopyingoraenvto/usr/local/bin...Thefile"coraenv"alreadyexistsin/usr/local/bin.Overwriteit?(y/n)[n]:yCopyingcoraenvto/usr/local/bin...Entrieswillbeaddedtothe/etc/oratabfileasneededbyDatabaseConfigurationAssistantwhenadatabaseiscreatedFinishedrunninggenericpartofroot.shscript.Nowproduct-specificrootactionswillbeperformed.2014-07-1015:17:53:Parsingthehostname2014-07-1015:17:53:Checkingforsuperuserprivileges2014-07-1015:17:53:UserhassuperuserprivilegesUsingconfigurationparameterfile:/u01/11.2.0/grid/crs/install/crsconfig_paramsLOCALADDMODECreatingOCRkeysforuser'root',privgrp'root'..Operationsuccessful.AddingdaemontoinittabCRS-4123:OracleHighAvailabilityServiceshasbeenstarted.ohasdisstartingCRS-4402:TheCSSdaemonwasstartedinexclusivemodebutfoundanactiveCSSdaemononnodelxh2,number1,andisterminatingAnactiveclusterwasfoundduringexclusivestartup,restartingtojointheclusterCRS-2672:Attemptingtostart'ora.mdnsd'on'lxh4'CRS-2676:Startof'ora.mdnsd'on'lxh4'succeededCRS-2672:Attemptingtostart'ora.gipcd'on'lxh4'CRS-2676:Startof'ora.gipcd'on'lxh4'succeededCRS-2672:Attemptingtostart'ora.gpnpd'on'lxh4'CRS-2676:Startof'ora.gpnpd'on'lxh4'succeededCRS-2672:Attemptingtostart'ora.cssdmonitor'on'lxh4'CRS-2676:Startof'ora.cssdmonitor'on'lxh4'succeededCRS-2672:Attemptingtostart'ora.cssd'on'lxh4'CRS-2672:Attemptingtostart'ora.diskmon'on'lxh4'CRS-2676:Startof'ora.diskmon'on'lxh4'succeededCRS-2676:Startof'ora.cssd'on'lxh4'succeededCRS-2672:Attemptingtostart'ora.ctssd'on'lxh4'CRS-2676:Startof'ora.ctssd'on'lxh4'succeededCRS-2672:Attemptingtostart'ora.drivers.acfs'on'lxh4'CRS-2676:Startof'ora.drivers.acfs'on'lxh4'succeededCRS-2672:Attemptingtostart'ora.asm'on'lxh4'CRS-2676:Startof'ora.asm'on'lxh4'succeededCRS-2672:Attemptingtostart'ora.crsd'on'lxh4'CRS-2676:Startof'ora.crsd'on'lxh4'succeededCRS-2672:Attemptingtostart'ora.evmd'on'lxh4'CRS-2676:Startof'ora.evmd'on'lxh4'succeededclscfg:EXISTINGconfigurationversion5detected.clscfg:version5is11gRelease2.SuccessfullyaccumulatednecessaryOCRkeys.CreatingOCRkeysforuser'root',privgrp'root'..Operationsuccessful.lxh42014/07/1015:20:10/u01/11.2.0/grid/cdata/lxh4/backup_20140710_152010.olrPreparingpackagesforinstallation...cvuqdisk-1.0.7-1ConfigureOracleGridInfrastructureforaCluster...succeededUpdatinginventorypropertiesforclusterwareStartingOracleUniversalInstaller...Checkingswapspace:mustbegreaterthan500MB.Actual2047MBPassedTheinventorypointerislocatedat/etc/oraInst.locTheinventoryislocatedat/u01/app/oraInventory'UpdateNodeList'wassuccessful.

脚本执行成功！

验证：

[root@lxh4 crsd]# crsctl check crs

CRS-4638:OracleHighAvailabilityServicesisonlineCRS-4537:ClusterReadyServicesisonlineCRS-4529:ClusterSynchronizationServicesisonlineCRS-4533:EventManagerisonline

[root@lxh4 crsd]# crs_stat -t

NameTypeTargetStateHost------------------------------------------------------------ora.DG1.dgora....up.typeONLINEONLINElxh2ora.DG2.dgora....up.typeONLINEONLINElxh2ora....ER.lsnrora....er.typeONLINEONLINElxh3ora....N1.lsnrora....er.typeONLINEONLINElxh2ora....VOTE.dgora....up.typeONLINEONLINElxh2ora.RCY.dgora....up.typeONLINEONLINElxh2ora.asmora.asm.typeONLINEONLINElxh2ora.eonsora.eons.typeONLINEONLINElxh2ora.gsdora.gsd.typeONLINEONLINElxh2ora.lxh.dbora....se.typeOFFLINEOFFLINEora....taf.svcora....ce.typeOFFLINEOFFLINEora....SM1.asmapplicationONLINEONLINElxh2ora....H1.lsnrapplicationONLINEOFFLINEora.lxh2.gsdapplicationONLINEONLINElxh2ora.lxh2.onsapplicationONLINEOFFLINEora.lxh2.vipora....t1.typeONLINEONLINElxh3ora....SM2.asmapplicationONLINEONLINElxh3ora....H2.lsnrapplicationONLINEONLINElxh3ora.lxh3.gsdapplicationONLINEONLINElxh3ora.lxh3.onsapplicationONLINEOFFLINEora.lxh3.vipora....t1.typeONLINEONLINElxh3ora....SM3.asmapplicationONLINEONLINElxh4ora....H3.lsnrapplicationONLINEONLINElxh4ora.lxh4.gsdapplicationONLINEONLINElxh4ora.lxh4.onsapplicationONLINEONLINElxh4ora.lxh4.vipora....t1.typeONLINEONLINElxh4ora....networkora....rk.typeONLINEONLINElxh2ora.oc4jora.oc4j.typeONLINEONLINElxh2ora.onsora.ons.typeONLINEONLINElxh4ora....ry.acfsora....fs.typeONLINEONLINElxh2ora.scan1.vipora....ip.typeONLINEONLINElxh2

@至此，问题解决完毕！