这篇文章主要介绍“怎么解决12cRAC打季度补丁时遇到ORA-12547: TNS:lost contact错误”，在日常操作中，相信很多人在怎么解决12cRAC打季度补丁时遇到ORA-12547: TNS:lost contact错误问题上存在疑惑，小编查阅了各式资料，整理出简单好用的操作方法，希望对大家解答”怎么解决12cRAC打季度补丁时遇到ORA-12547: TNS:lost contact错误”的疑惑有所帮助！接下来，请跟着小编一起来学习吧！

一、环境说明：

操作系统：RHEL7.4

数据库版本：12.2.0.1

已打补丁：201910月的季度补丁

由于保密缘故，相关命名与IP已做变更

二、问题描述

在对GI打季度补丁时，节点1运行opatchauto apply后，CRS正常关闭，但是到自动拉起CRS时遇到ORA-12547: TNS:lost contact报错了，导致打补丁失败。

详细信息如下：

CRS-4123:StartingOracleHighAvailabilityServices-managedresourcesCRS-2672:Attemptingtostart'ora.mdnsd'on'db1'CRS-2672:Attemptingtostart'ora.evmd'on'db1'CRS-2676:Startof'ora.mdnsd'on'db1'succeededCRS-2676:Startof'ora.evmd'on'db1'succeededCRS-2672:Attemptingtostart'ora.gpnpd'on'db1'CRS-2676:Startof'ora.gpnpd'on'db1'succeededCRS-2672:Attemptingtostart'ora.gipcd'on'db1'CRS-2676:Startof'ora.gipcd'on'db1'succeededCRS-2672:Attemptingtostart'ora.cssdmonitor'on'db1'CRS-2676:Startof'ora.cssdmonitor'on'db1'succeededCRS-2672:Attemptingtostart'ora.cssd'on'db1'CRS-2672:Attemptingtostart'ora.diskmon'on'db1'CRS-2676:Startof'ora.diskmon'on'db1'succeededCRS-2676:Startof'ora.cssd'on'db1'succeededCRS-2672:Attemptingtostart'ora.cluster_interconnect.haip'on'db1'CRS-2672:Attemptingtostart'ora.ctssd'on'db1'CRS-2676:Startof'ora.ctssd'on'db1'succeededCRS-2676:Startof'ora.cluster_interconnect.haip'on'db1'succeededCRS-2672:Attemptingtostart'ora.asm'on'db1'CRS-2676:Startof'ora.asm'on'db1'succeededCRS-2672:Attemptingtostart'ora.storage'on'db1'ORA-12547:TNS:lostcontactORA-12547:TNS:lostcontactORA-15077:couldnotlocateASMinstanceservingarequireddiskgroupCRS-2883:Resource'ora.storage'failedduringClusterwarestackstart.CRS-4406:OracleHighAvailabilityServicessynchronousstartfailed.CRS-4000:CommandStartfailed,orcompletedwitherrors.2020/06/1016:46:12CLSRSC-117:FailedtostartOracleClusterwarestackAfterfixingthecauseoffailureRunopatchautoresume]OPATCHAUTO-68061:Theorchestrationenginefailed.OPATCHAUTO-68061:Theorchestrationenginefailedwithreturncode1OPATCHAUTO-68061:Checkthelogformoredetails.OPatchAutofailed.OPatchautosessioncompletedatWedJun1016:46:132020Timetakentocompletethesession17minutes,25secondsopatchautofailedwitherrorcode42

调用crsctl check crs查看集群服务，发现Cluster Ready Services、Event Manager异常。查看/u01/app/grid/diag/crs/db1/crs/trace/alert.log日志，发现如下信息：

2020-06-1016:36:08.150[ORAROOTAGENT(679524)]CRS-5019:AllOCRlocationsareonASMdiskgroups[OCR],andnoneofthesediskgroupsaremounted.Detailsareat"(:CLSN00140:)"in"/u01/app/grid/diag/crs/db1/crs/trace/ohasd_orarootagent_root.trc".2020-06-1016:45:58.112[ORAROOTAGENT(679524)]CRS-5818:Abortedcommand'start'forresource'ora.storage'.Detailsat(:CRSAGF00113:){0:5:3}in/u01/app/grid/diag/crs/db1/crs/trace/ohasd_orarootagent_root.trc.2020-06-1016:46:00.427[ORAROOTAGENT(679524)]CRS-5017:Theresourceaction"ora.storagestart"encounteredthefollowingerror:2020-06-1016:46:00.427+Storageagentstartactionaborted.Fordetailsreferto"(:CLSN00107:)"in"/u01/app/grid/diag/crs/db1/crs/trace/ohasd_orarootagent_root.trc".2020-06-1016:46:00.429[OHASD(679429)]CRS-2757:Command'Start'timedoutwaitingforresponsefromtheresource'ora.storage'.Detailsat(:CRSPE00221:){0:5:3}in/u01/app/grid/diag/crs/db1/crs/trace/ohasd.trc.2020-06-1016:47:12.689[OSYSMOND(684959)]CRS-8500:OracleClusterwareOSYSMONDprocessisstartingwithoperatingsystemprocessID684959

/u01/app/grid/diag/crs/db1/crs/trace/ohasd_orarootagent_root.trc日志如下：

2020-06-0916:34:08.611:CLSDYNAM:3440035584:[ora.storage]{0:1:251}[check]StorageAgent::parsekgforetcodesretcode=0,kgfoCheckMount(OCR),flag42020-06-0916:34:08.611:CLSDYNAM:3440035584:[ora.storage]{0:1:251}[check]StorageAgent::checkkgforeturncode02020-06-0916:34:11.243:CLSDYNAM:3429529344:[ora.crf]{0:5:3}[check]Checkreturn=0,statedetail=NULL2020-06-0916:34:15.926:CLSDYNAM:2525263616:[ora.ctssd]{0:5:3}[check]translateReturnCodes,return=0,statedetail=OBSERVERCheckcbdata[0x7f2369c293d0]:mode[0xee]offset[0ms].2020-06-0916:34:19.107:CLSDYNAM:3425326848:[ora.drivers.acfs]{0:0:2}[check]ArgValue=check2020-06-0916:34:19.107:CLSDYNAM:3425326848:[ora.drivers.acfs]{0:0:2}[check]Utils::getOracleHomeAttribgetEnvVaroracle_home:/u01/app/12.2.0/grid2020-06-0916:34:19.107:CLSDYNAM:3425326848:[ora.drivers.acfs]{0:0:2}[check]Utils::getOracleHomeAttriboracle_home:/u01/app/12.2.0/grid2020-06-0916:34:19.107:CLSDYNAM:3425326848:[ora.drivers.acfs]{0:0:2}[check]Utils::execCmd1USR_ORA_ENV:oracleHome:/u01/app/12.2.0/gridCrsHome:/u01/app/12.2.0/grid2020-06-0916:34:19.107:CLSDYNAM:3425326848:[ora.drivers.acfs]{0:0:2}[check]AddingEnvironmentVariablesORA_DAEMON_LOGGING_LEVELS=GIPCLIB=2,GIPC=2,GIPCGEN=2,GIPCTRAC=2,GIPCWAIT=2,GIPCXCPT=2,GIPCOSD=2,GIPCBASE=2,GIPCCLSA=2,GIPCCLSC=2,GIPCEXMP=2,GIPCGMOD=2,GIPCHEAD=2,GIPCMUX=2,GIPCNET=2,GIPCNULL=2,GIPCPKT=2,GIPCSMEM=2,GIPCHAUP=2,GIPCHALO=2,GIPCHTHR=2,GIPCHGEN=2,GIPCHLCK=2,GIPCHDEM=2,GIPCHWRK=2,GIPCTLS=2,GIPCHGNS=22020-06-0916:34:19.107:CLSDYNAM:3425326848:[ora.drivers.acfs]{0:0:2}[check]AddingEnvironmentVariablesORA_DAEMON_TRACE_FILE_OPTIONS=filesize=26214400,numsegments=102020-06-0916:34:19.107:CLSDYNAM:3425326848:[ora.drivers.acfs]{0:0:2}[check]AddingEnvironmentVariables_ORA_AGENT_ACTION=TRUE2020-06-0916:34:19.107:CLSDYNAM:3425326848:[ora.drivers.acfs]{0:0:2}[check]AddingEnvironmentVariables__IS_HASD_AGENT=2020-06-0916:34:19.107:CLSDYNAM:3425326848:[ora.drivers.acfs]{0:0:2}[check]Utils:execCmdaction=3flags=6ohome=(null)cmdname=acfsload.2020-06-0916:34:19.108:CLSDYNAM:3425326848:[ora.drivers.acfs]{0:0:2}[check]Utils::getOracleHomeAttribgetEnvVaroracle_home:/u01/app/12.2.0/grid2020-06-0916:34:19.108:CLSDYNAM:3425326848:[ora.drivers.acfs]{0:0:2}[check]Utils::getOracleHomeAttriboracle_home:/u01/app/12.2.0/grid2020-06-0916:34:19.219:CLSDYNAM:3425326848:[ora.drivers.acfs]{0:0:2}[check]execCmdret=02020-06-0916:34:20.911:CLSDYNAM:3427428096:[ora.driver.afd]{0:0:2}[check]Utils:execCmdaction=3flags=38ohome=(null)cmdname=afddriverstate.2020-06-0916:34:20.911:CLSDYNAM:3427428096:[ora.driver.afd]{0:0:2}[check]Utils::getOracleHomeAttribgetEnvVaroracle_home:/u01/app/12.2.0/grid2020-06-0916:34:20.911:CLSDYNAM:3427428096:[ora.driver.afd]{0:0:2}[check]Utils::getOracleHomeAttriboracle_home:/u01/app/12.2.0/grid2020-06-0916:34:21.121:CLSDYNAM:3427428096:[ora.driver.afd]{0:0:2}[check]execCmdret=02020-06-0916:34:22.763:USRTHRD:2548033280:{0:5:3}Check:0-12020-06-0916:34:26.337:AGFW:3435833088:{0:0:2}Agentreceivedthemessage:AGENT_HB[Engine]ID12293:20865052020-06-0916:34:32.281:CLSDMC:3425326848:command0failedwithstatus168432652020-06-0916:34:32.282:CLSDYNAM:3425326848:[ora.crsd]{0:1:52}[check]DaemonAgent::checkreturned02020-06-0916:34:32.282:CLSDYNAM:3425326848:[ora.crsd]{0:1:52}[check]DaemonAgent::checkcheckErrorCode=16843265,pestate=512,perole=65536,pemode=12020-06-0916:34:38.611:CLSDYNAM:3427428096:[ora.storage]{0:1:251}[check]StorageAgent::checkNODEROLE_HUBgetOCRdetails2020-06-0916:34:38.686:CLSDYNAM:3427428096:[ora.storage]{0:1:251}[check]StorageAgent::parsekgforetcodesretcode=0,kgfoCheckMount(OCR),flag22020-06-0916:34:38.686:CLSDYNAM:3427428096:[ora.storage]{0:1:251}[check]StorageAgent::checkkgforeturncode02020-06-0916:34:41.239:CLSDYNAM:3444238080:[ora.crf]{0:5:3}[check]Checkreturn=0,statedetail=NULL2020-06-0916:34:45.921:CLSDYNAM:3444238080:[ora.ctssd]{0:5:3}[check]translateReturnCodes,return=0,statedetail=OBSERVERCheckcbdata[0x7f23baa99360]:mode[0xee]offset[0ms].2020-06-0916:34:52.763:USRTHRD:2548033280:{0:5:3}Check:0-12020-06-0916:34:56.341:AGFW:3435833088:{0:0:2}Agentreceivedthemessage:AGENT_HB[Engine]ID12293:20865222020-06-0916:35:02.285:CLSDMC:2525263616:command0failedwithstatus168432652020-06-0916:35:02.285:CLSDYNAM:2525263616:[ora.crsd]{0:1:52}[check]DaemonAgent::checkreturned0

三、故障分析

从告警日志中可知，CRS进程未能识别到OCR磁盘有挂载导致CRS服务启动失败。

尝试手工启动ASM实例，CRS启动成功：

sqlplus/assysasmstartup

关闭CRS后再调用crsctl start crs，ASM实例一样未能自动拉起，只能通过startup的方式。

查看/u01/app/grid/diag/crs/db1/crs/trace/crsd.trc日志，发现如下信息：

2020-06-1016:48:24.867*:kgfn.c@1370:kgfnFindLocalNode:foundnomembers2020-06-1016:48:24.867:OCRRAW:633798784:kgfnFindLocalNode:notok2020-06-1016:48:24.867*:kgfn.c@1422:kgfnFindLocalNode:notok2020-06-1016:48:24.867:OCRRAW:633798784:kgfnTgtInit:localnodenotfound,freekgfnpds2020-06-1016:48:24.867*:kgfn.c@2208:kgfnTgtInit:notfound2020-06-1016:48:24.867:OCRRAW:633798784:kgfnGetBeqDatafailedinittarget;inst=(null)flags=0x60002020-06-1016:48:24.867*:kgfn.c@5791:kgfnGetBeqData:kgfnTgtInitfailed,inst=NULLflags=0x60002020-06-1016:48:24.867:CLSNS:633798784:clsns_SetTraceLevel:tracelevelsetto1.2020-06-1016:48:24.900:OCRRAW:633798784:8154Error4queryinglengthofattrASM_DISCOVERY_ADDRESS2020-06-1016:48:24.904:OCRRAW:633798784:8154Error4queryinglengthofattrASM_STATIC_DISCOVERY_ADDRESS2020-06-1016:48:24.940:OCRRAW:633798784:7872Error4openingdomrootin0x5fb60d02020-06-1016:48:24.958:OCRRAW:633798784:kgfnConnect2:kgfnGetBeqDatafailed2020-06-1016:48:24.958*:kgfn.c@5012:kgfnConnect2:kgfnGetBeqDatafailed2020-06-1016:48:24.992:OCRRAW:633798784:kgfnConnect2Int:cstr=(DESCRIPTION=(TCP_USER_TIMEOUT=1)(TRANSPORT_CONNECT_TIMEOUT=60)(EXPIRE_TIME=1)(ADDRESS_LIST=(LOAD_BALANCE=ON)(ADDRESS=(PROTOCOL=tcp)(HOST=10.10.10.43)(PORT=1526)))(CONNECT_DATA=(SERVICE_NAME=+ASM)))2020-06-1016:48:24.992*:kgfn.c@6788:kgfnConnect2Int:cstr=(DESCRIPTION=(TCP_USER_TIMEOUT=1)(TRANSPORT_CONNECT_TIMEOUT=60)(EXPIRE_TIME=1)(ADDRESS_LIST=(LOAD_BALANCE=ON)(ADDRESS=(PROTOCOL=tcp)(HOST=10.10.10.43)(PORT=1526)))(CONNECT_DATA=(SERVICE_NAME=+ASM)))2020-06-1016:48:24.993:OCRRAW:633798784:kgfnConnect2Int:ServerAttach2020-06-1016:48:24.993*:kgfn.c@6811:kgfnConnect2Int:OCIServerAttachfailedFailedtoconnecttoASM100(nil)02020-06-1016:48:25.994:OCRRAW:633798784:kgfnRecordErr12547OCIerror:ORA-12547:TNS:lostcontact2020-06-1016:48:25.994*:kgfn.c@1740:kgfnRecordErrPriv:12547error=ORA-12547:TNS:lostcontact2020-06-1016:48:25.995:OCRRAW:633798784:kgfnConnect2:failedtoconnect2020-06-1016:48:25.995*:kgfn.c@5333:kgfnConnect2:failedtoconnect2020-06-1016:48:25.995:OCRRAW:633798784:kgfnConnect2Retry:failedtoconnectconnectafter2attempts,331selapsed2020-06-1016:48:25.995:OCRRAW:633798784:kgfo_kge2sloserrorstackatkgfoAl06:ORA-12547:TNS:lostcontact

ORA-12547: TNS:lost contact的报错出现在该日志文件中，而该报错是要访问私网IP10.10.10.43 1526端口失败所致。

四、问题原因

1526端口是运行在ASMNET1LSNR_ASM监听上的。在12cRAC中，从FLEXASM开始，crsd.bin将使用ASMNET1LSNR_ASM监听进行远程连接。即会连接10.10.10.43的1526端口。

那么为何连接节点2的1526端口会失败呢。

查看节点2的sqlnet.ora配置，发现配置了白名单，但是没有增加节点1的私网IP。

五、解决办法

1)Modifythefilesqlnet.oravi$GRID_HOME/network/admin/sqlnet.ora2)AddtheIPwhichareusedforPrivateinterconnect.e.g.TCP.VALIDNODE_CHECKING=YESTCP.INVITED_NODES=(node1.localhost,node2.localhost,node1-priv.localhost,node2-priv.localhost,node1-vip.localhost,node2-vip.localhost,applicationserverVIPS)3)RestarttheCRSinissuenode.

到此，关于“怎么解决12cRAC打季度补丁时遇到ORA-12547: TNS:lost contact错误”的学习就结束了，希望能够解决大家的疑惑。理论与实践的搭配能更好的帮助大家学习，快去试试吧！若想继续学习更多相关知识，请继续关注亿速云网站，小编会继续努力为大家带来更多实用的文章！