Oracle12C集群打补丁失败转而使用手工打补丁报CRS-6706处理过程

2021-01-29 15:43:01

Oracle12c 12.2.0.1 opatch auto 打集群补丁失败，报错信息如下

Following active excutables are used by opatch process:
/oracle/12.2/lib/libclntsh.so.12.1
/oracle/12.2/lib/libsqlplus.so
Prerequisite check "CheckActiveFileAndExecutables" failed

如下查询没有任何输出
fuser /oracle/12.2/lib/libclntsh.so.12.1
fuser /oracle/12.2/lib/libsqlplus.so

我们尝试使用手工方式打集群和数据库补丁
opatch apply <patch_home>/patch-number -oh <GRID_HOME>
opatch apply <patch_home>/patch-number -oh <ORACLE_HOME>
成功！
然后尝试重启集群，报错如下

CRS-6706: Oracle Clusterware Release patch level('2197668489') does not match software opatch level('0').Oracle Clusterware cannot be started
CRS-4000: Command Start failed,or completed with errors.
这个错误比较明显是patch level不匹配，于是我们找到如下文章
CRS-6706: Oracle Clusterware Release patch level ('nnn') does not match Software patch level ('mmm') (Doc ID 1639285.1)
vThis note lists causes/solutions for the following issue:

Oracle 12c Grid Infrastructure fails to start after patching:

CRS-6706: Oracle Clusterware Release patch level ('xxxxxxxx') does not match Software patch level ('xxxxxxxxx'). Oracle Clusterware cannot be started

This can be encountered - for example - in cases where the post patch application instruction to run 'rootcrs.pl -postpatch' is not followed.

由于其他节点使用opatchauto打补丁，集群操作打补丁都自动完成，无需执行rootcrs.pl -postpatch操作，而此时，我们改为手工打补丁，所以忽略了这个步骤。

问题解决：
[root@pc156 /grid/12.2]#/grid/12.2/crs/install/rootcrs.sh -prepatch

Using configuration parameter file: /grid/12.2/crs/install/crsconfig_params
The log of current session can be found at:
/grid/base/crsdata/pc156/crsconfig/crspatch_pc156_2021-01-29_02-51-04PM.log

[root@pc156 /grid/12.2]#/grid/12.2/crs/install/rootcrs.pl -postpatch

Can't locate Env.pm in @INC (@INC contains: /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib64/perl5 /usr/share/perl5 . /grid/12.2/crs/install /grid/12.2/crs/install/../../perl/lib) at /grid/12.2/crs/install/crsinstall.pm line 296.
BEGIN failed--compilation aborted at /grid/12.2/crs/install/crsinstall.pm line 296.
Compilation failed in require at /grid/12.2/crs/install/rootcrs.pl line 165.
BEGIN failed--compilation aborted at /grid/12.2/crs/install/rootcrs.pl line 165.
这里rootcrs.pl -postpatch操作失败，提示也比较明显缺少Env.pm，下面安装组件

[root@n-pc-sr850-308 ~]#yum -y install perl-Env
Loaded plugins: ulninfo
Resolving Dependencies
--> Running transaction check
---> Package perl-Env.noarch 0:1.04-2.el7 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

============================================================================================================================
Package Arch Version Repository Size
============================================================================================================================
Installing:
perl-Env noarch 1.04-2.el7 ol74 15 k

Transaction Summary
============================================================================================================================
Install 1 Package

Total download size: 15 k
Installed size: 27 k
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Installing : perl-Env-1.04-2.el7.noarch 1/1
Verifying : perl-Env-1.04-2.el7.noarch 1/1

Installed:
perl-Env.noarch 0:1.04-2.el7

Complete!
继续执行[root@pc156 /grid/12.2]#/grid/12.2/crs/install/rootcrs.pl -postpatch
[root@pc156 /grid/12.2]#/grid/12.2/crs/install/rootcrs.pl -postpatch
Using configuration parameter file: /grid/12.2/crs/install/crsconfig_params
The log of current session can be found at:
/grid/base/crsdata/pc156/crsconfig/crspatch_pc156_2021-01-29_02-57-17PM.log
2021/01/29 14:57:22 CLSRSC-4015: Performing install or upgrade action for Oracle Trace File Analyzer (TFA) Collector.
2021/01/29 14:58:35 CLSRSC-4003: Successfully patched Oracle Trace File Analyzer (TFA) Collector.
2021/01/29 14:58:37 CLSRSC-329: Replacing Clusterware entries in file 'oracle-ohasd.service'
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.evmd' on 'pc156'
CRS-2672: Attempting to start 'ora.mdnsd' on 'pc156'
CRS-2676: Start of 'ora.mdnsd' on 'pc156' succeeded
CRS-2676: Start of 'ora.evmd' on 'pc156' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'pc156'
CRS-2676: Start of 'ora.gpnpd' on 'pc156' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'pc156'
CRS-2676: Start of 'ora.gipcd' on 'pc156' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'pc156'
CRS-2676: Start of 'ora.cssdmonitor' on 'pc156' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'pc156'
CRS-2672: Attempting to start 'ora.diskmon' on 'pc156'
CRS-2676: Start of 'ora.diskmon' on 'pc156' succeeded
CRS-2676: Start of 'ora.cssd' on 'pc156' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'pc156'
CRS-2672: Attempting to start 'ora.ctssd' on 'pc156'
CRS-2676: Start of 'ora.ctssd' on 'pc156' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'pc156' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'pc156'
CRS-2676: Start of 'ora.asm' on 'pc156' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'pc156'
CRS-2676: Start of 'ora.storage' on 'pc156' succeeded
CRS-2672: Attempting to start 'ora.crf' on 'pc156'
CRS-2676: Start of 'ora.crf' on 'pc156' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'pc156'
CRS-2676: Start of 'ora.crsd' on 'pc156' succeeded
CRS-6023: Starting Oracle Cluster Ready Services-managed resources
CRS-6017: Processing resource auto-start for servers: pc156
CRS-2673: Attempting to stop 'ora.pc156.vip' on 'n-pc-sr850-322'
CRS-2672: Attempting to start 'ora.ons' on 'pc156'
CRS-2672: Attempting to start 'ora.chad' on 'pc156'
CRS-2677: Stop of 'ora.pc156.vip' on 'n-pc-sr850-322' succeeded
CRS-2672: Attempting to start 'ora.pc156.vip' on 'pc156'
CRS-2676: Start of 'ora.chad' on 'pc156' succeeded
CRS-2676: Start of 'ora.pc156.vip' on 'pc156' succeeded
CRS-2672: Attempting to start 'ora.LISTENER.lsnr' on 'pc156'
CRS-2676: Start of 'ora.ons' on 'pc156' succeeded
CRS-2676: Start of 'ora.LISTENER.lsnr' on 'pc156' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'pc156'
CRS-2676: Start of 'ora.asm' on 'pc156' succeeded
CRS-6016: Resource auto-start has completed for server pc156
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
CRS-4123: Oracle High Availability Services has been started.
Oracle Clusterware active version on the cluster is [12.2.0.1.0]. The cluster upgrade state is [NORMAL]. The cluster active patch level is [2197668489].

集群启动成功！