绑定完请刷新页面
取消
刷新
登录 | 注册

分享好友

×
取消 复制
ASM的normal模式的某个失败组磁盘失效如何mount磁盘
2021-09-08 16:26:49

********ASM磁盘组normal冗余模式下失败组故障如何mount磁盘组*********
SQL> select failgroup ,name,path,mount_status, state,total_mb,free_mb from v$asm_disk where group_number=3 order by 1,2

FAILGROUP NAME PATH MOUNT_STATUS STATE TOTAL_MB FREE_MB
-------------------- ---------- ---------- --------------------- ---------- ---------- ----------
DATA1_FG1 DISK1 /dev/sde1 CACHED NORMAL 1023 941
DATA1_FG1 DISK2 /dev/sdf1 CACHED NORMAL 1023 941
DATA1_FG2 DISK3 /dev/sdg1 CACHED NORMAL 1023 945
DATA1_FG2 DISK4 /dev/sdh1 CACHED NORMAL 1023 937

ASMCMD> lsdg
State Type Rebal Sector Logical_Sector Block AU Total_MB Free_MB Req_mir_free_MB Usable_file_MB Offline_disks Voting_files Name
MOUNTED EXTERN N 512 512 4096 4194304 22524 19608 0 19608 0 N DATA/
MOUNTED NORMAL N 512 512 4096 1048576 4092 3764 2046 859 0 N DATA1/
MOUNTED EXTERN N 512 512 4096 4194304 12284 11936 0 11936 0 Y OCR/

摘掉磁盘/dev/sde1，/dev/sdf1



现在我们将失败组DATA1_FG1的所有磁盘摘掉，模拟失败组DATA1_FG1的所有磁盘故障,我们在虚拟机的介质管理器中
释放共享盘和节点的关联关系。


重启集群后数据库无法启动
数据库日志
2021-09-08T12:19:04.335639+08:00
WARNING: failed to register AMB1 with ASM instance
2021-09-08T12:19:04.355078+08:00
Errors in file /oracle/db/diag/rdbms/prod/prod1/trace/prod1_amb1_17786.trc:
ORA-12514: TNS:listener does not currently know of service requested in connect descriptor
ORA-12514: TNS:listener does not currently know of service requested in connect descriptor
WARNING: AMB1 exiting with error
2021-09-08T12:19:04.678312+08:00
Errors in file /oracle/db/diag/rdbms/prod/prod1/trace/prod1_dbw0_17500.trc:
ORA-01157: cannot identify/lock data file 2 - see DBWR trace file
ORA-01110: data file 2: '+DATA1/PROD/DATAFILE/data1.256.1082626777'
ORA-17503: ksfdopn:2 Failed to open file +DATA1/PROD/DATAFILE/data1.256.1082626777
ORA-15001: diskgroup "DATA1" does not exist or is not mounted
ORA-15001: diskgroup "DATA1" does not exist or is not mounted
ORA-15064: communication failure with ASM instance
ORA-12514: TNS:listener does not currently know of service requested in connect descriptor
ORA-12514: TNS:listener does not currently know of service requested in connect descriptor
2021-09-08T12:19:04.939328+08:00
Ping without log force is disabled:
not an Exadata system.
2021-09-08T12:19:05.064240+08:00
Errors in file /oracle/db/diag/rdbms/prod/prod1/trace/prod1_ora_17595.trc:
ORA-01157: cannot identify/lock data file 2 - see DBWR trace file
ORA-01110: data file 2: '+DATA1/PROD/DATAFILE/data1.256.1082626777'
2021-09-08T12:19:05.139709+08:00
Errors in file /oracle/db/diag/rdbms/prod/prod1/trace/prod1_ora_17595.trc:
ORA-01157: cannot identify/lock data file 2 - see DBWR trace file
ORA-01110: data file 2: '+DATA1/PROD/DATAFILE/data1.256.1082626777'
ORA-1157 signalled during: ALTER DATABASE OPEN /* db agent *//* {1:61491:494} */...
2021-09-08T12:19:08.784593+08:00
License high water mark = 1
2021-09-08T12:19:08.788726+08:00
USER(prelim) (ospid: 17822): terminating the instance
2021-09-08T12:19:10.028522+08:00
Instance terminated by USER(prelim), pid = 17822


ASM日志
SQL> ALTER DISKGROUP DATA1 MOUNT /* asm agent *//* {1:61491:1164} */
2021-09-08T15:36:30.870964+08:00
NOTE: cache registered group DATA1 1/0x54EE6E2C
2021-09-08T15:36:32.177546+08:00
NOTE: cache began mount (first) of group DATA1 1/0x54EE6E2C
2021-09-08T15:36:33.828527+08:00
NOTE: Assigning number (1,3) to disk (/dev/sdf1)
NOTE: Assigning number (1,2) to disk (/dev/sde1)
2021-09-08T15:36:33.846342+08:00
NOTE: cache dismounting (clean) group 1/0x54EE6E2C (DATA1)
NOTE: messaging CKPT to quiesce pins Unix process pid: 16021, image: oracle@rac1 (TNS V1-V3)
NOTE: dbwr not being msg'd to dismount
NOTE: LGWR not being messaged to dismount
NOTE: cache dismounted group 1/0x54EE6E2C (DATA1)
NOTE: cache ending mount (fail) of group DATA1 number=1 incarn=0x54ee6e2c
NOTE: cache deleting context for group DATA1 1/0x54ee6e2c
2021-09-08T15:36:33.858625+08:00
GMON dismounting group 1 at 26 for pid 43, osid 16021
2021-09-08T15:36:33.859948+08:00
NOTE: Disk DISK3 in mode 0x8 marked for de-assignment
NOTE: Disk DISK4 in mode 0x8 marked for de-assignment
ERROR: diskgroup DATA1 was not mounted
ORA-15032: not all alterations performed
ORA-15017: diskgroup "DATA1" cannot be mounted
ORA-15040: diskgroup is incomplete

看ORA-15040的含义
[grid@rac1 ~]$ oerr ora 15040
15040, 00000, "diskgroup is incomplete"
// *Cause: Some of the disks comprising a diskgroup were not present.
// *Action: Check the hardware to ensure that all disks are functional. Also
// check that the setting of the ASM_DISKSTRING initialization
// parameter has not changed. Alternatively, for normal or high
// redundancy diskgroups, use MOUNT FORCE to offline missing disks
// as part of mounting the diskgroup.
这个解释跟我们的测试一致，我们强制mount磁盘组，使用一个失败组看看是否可以拉起来。
SQL> alter diskgroup data1 mount;
alter diskgroup data1 mount
*
ERROR at line 1:
ORA-15032: not all alterations performed
ORA-15017: diskgroup "DATA1" cannot be mounted
ORA-15040: diskgroup is incomplete


SQL> alter diskgroup data1 mount force;

Diskgroup altered.

查询数据字典
SQL> select failgroup ,name,path,mount_status, state,total_mb,free_mb from v$asm_disk where group_number=1;

FAILGROUP NAME PATH MOUNT_STATUS STATE TOTAL_MB FREE_MB
-------------------- -------------------- -------------------- --------------------- -------------------- ---------- ----------
DATA1_FG1 _DROPPED_0000_DATA1 MISSING FORCING 1023 920
DATA1_FG1 _DROPPED_0001_DATA1 MISSING FORCING 1023 920
DATA1_FG2 DISK4 /dev/sdf1 CACHED NORMAL 1023 921
DATA1_FG2 DISK3 /dev/sde1 CACHED NORMAL 1023 919

此时Oracle做了清理，missing的磁盘被强制drop,此时我们可以正常打开数据库


此时看看ASM告警日志
SQL> alter diskgroup data1 mount force
2021-09-08T15:56:17.397002+08:00
NOTE: cache registered group DATA1 1/0x59CE6E32
NOTE: cache began mount (first) of group DATA1 1/0x59CE6E32
2021-09-08T15:56:18.769488+08:00
NOTE: Assigning number (1,3) to disk (/dev/sdf1)
NOTE: Assigning number (1,2) to disk (/dev/sde1)
2021-09-08T15:56:18.860629+08:00
WARNING: PST disk 0 not found (grp 1)
cluster guid (dd16d1353f274f80ff1bc81e95eeee15) generated for PST Hbeat for instance 1
2021-09-08T15:56:25.135046+08:00
NOTE: GMON heartbeating for grp 1 (DATA1) <<<<<此时的磁盘组号变为1，之前为3.
GMON querying group 1 at 31 for pid 38, osid 13046
2021-09-08T15:56:25.177225+08:00
NOTE: Assigning number (1,0) to disk ()
NOTE: Assigning number (1,1) to disk ()
2021-09-08T15:56:25.409934+08:00
GMON querying group 1 at 32 for pid 38, osid 13046
2021-09-08T15:56:25.514984+08:00
NOTE: process _user13046_+asm1 (13046) initiating offline of disk 0.4043218713 () with mask 0x7e in group 1 (DATA1) without client assisting
NOTE: process _user13046_+asm1 (13046) initiating offline of disk 1.4043218715 () with mask 0x7e in group 1 (DATA1) without client assisting
NOTE: checking PST: grp = 1
2021-09-08T15:56:25.613721+08:00
GMON checking disk modes for group 1 at 33 for pid 38, osid 13046
2021-09-08T15:56:25.623716+08:00
NOTE: checking PST for grp 1 done.
2021-09-08T15:56:25.624516+08:00
NOTE: sending set offline flag message (1279641324) to 2 disk(s) in group 1
2021-09-08T15:56:25.778963+08:00
WARNING: Disk 0 () in group 1 mode 0x7f is now being offlined <<<<<<磁盘Disk 0 Disk 1被强制offline
WARNING: Disk 1 () in group 1 mode 0x7f is now being offlined
2021-09-08T15:56:25.790569+08:00
INFO: Instance #2 could not find disk 2 in group 1
NOTE: initiating PST update: grp 1 (DATA1), dsk = 0/0xf0fe9f19, mask = 0x6a, op = clear mandatory <<<<< 更新PST表，强制清除两个磁盘
NOTE: initiating PST update: grp 1 (DATA1), dsk = 1/0xf0fe9f1b, mask = 0x6a, op = clear mandatory
2021-09-08T15:56:25.853348+08:00
GMON updating disk modes for group 1 at 34 for pid 38, osid 13046
2021-09-08T15:56:25.967438+08:00
NOTE: PST update grp = 1 completed successfully
NOTE: initiating PST update: grp 1 (DATA1), dsk = 0/0xf0fe9f19, mask = 0x7e, op = clear mandatory
NOTE: initiating PST update: grp 1 (DATA1), dsk = 1/0xf0fe9f1b, mask = 0x7e, op = clear mandatory
2021-09-08T15:56:25.973853+08:00
GMON updating disk modes for group 1 at 35 for pid 38, osid 13046
2021-09-08T15:56:26.056930+08:00
NOTE: cache closing disk 0 of grp 1: (not open)
2021-09-08T15:56:26.057638+08:00
NOTE: cache closing disk 1 of grp 1: (not open)
2021-09-08T15:56:26.098801+08:00
NOTE: cache closing disk 0 of grp 1: (not open)
2021-09-08T15:56:26.105284+08:00
NOTE: cache closing disk 1 of grp 1: (not open) <<<<<<关闭disk0 disk1
2021-09-08T15:56:26.123581+08:00
NOTE: PST update grp = 1 completed successfully
NOTE: cache is mounting group DATA1 created on 2021/09/07 09:22:59 <<<<<开始挂盘
NOTE: cache opening disk 2 of grp 1: DISK3 path:/dev/sde1
NOTE: cache opening disk 3 of grp 1: DISK4 path:/dev/sdf1
NOTE: 09/08/21 15:56:25 DATA1.F1X0 found on disk 3 au 2 fcn 0.0 datfmt 1
2021-09-08T15:56:26.129497+08:00
NOTE: cache mounting (first) normal redundancy group 1/0x59CE6E32 (DATA1)
2021-09-08T15:56:26.139602+08:00
* allocate domain 1, valid ? 0
kjbdomatt send to inst 2
2021-09-08T15:56:26.343997+08:00
NOTE: attached to recovery domain 1
2021-09-08T15:56:26.574101+08:00
validate pdb 1, flags x4, valid 0, pdb flags x204
* validated domain 1, flags = 0x200
NOTE: cache recovered group 1 to fcn 0.795
NOTE: redo buffer size is 256 blocks (1056768 bytes)
2021-09-08T15:56:26.704574+08:00
NOTE: LGWR attempting to mount thread 1 for diskgroup 1 (DATA1)
NOTE: LGWR found thread 1 closed at ABA 2.41 lock domain=0 inc#=0 instnum=1
NOTE: LGWR mounted thread 1 for diskgroup 1 (DATA1)
2021-09-08T15:56:26.811591+08:00
NOTE: LGWR opened thread 1 (DATA1) at fcn 0.795 ABA 3.42 lock domain=1 inc#=8 instnum=1 gx.incarn=1506700850 mntstmp=2021/09/08 15:56:26.786000
2021-09-08T15:56:26.818432+08:00
NOTE: cache mounting group 1/0x59CE6E32 (DATA1) succeeded
NOTE: cache ending mount (success) of group DATA1 number=1 incarn=0x59ce6e32
2021-09-08T15:56:27.515738+08:00
NOTE: cache closing disk 0 of grp 1: (not open) DISK1
2021-09-08T15:56:27.516472+08:00
NOTE: cache closing disk 1 of grp 1: (not open) DISK2
2021-09-08T15:56:27.527287+08:00
NOTE: Instance updated compatible.asm to 11.2.0.2.0 for grp 1 (DATA1).
2021-09-08T15:56:27.532187+08:00
NOTE: cache closing disk 0 of grp 1: (not open) DISK1
2021-09-08T15:56:27.533661+08:00
NOTE: cache closing disk 1 of grp 1: (not open) DISK2
2021-09-08T15:56:27.550394+08:00
NOTE: Instance updated compatible.asm to 11.2.0.2.0 for grp 1 (DATA1).
2021-09-08T15:56:27.646690+08:00
NOTE: Instance updated compatible.rdbms to 10.1.0.0.0 for grp 1 (DATA1).
2021-09-08T15:56:27.662473+08:00
NOTE: Instance updated compatible.rdbms to 10.1.0.0.0 for grp 1 (DATA1).
2021-09-08T15:56:28.002565+08:00
SUCCESS: diskgroup DATA1 was mounted
2021-09-08T15:56:28.562265+08:00
SUCCESS: alter diskgroup data1 mount force <<<<<mount force成功
2021-09-08T15:56:28.948585+08:00
ASM Health Checker found 1 new failures
2021-09-08T15:56:29.006000+08:00
NOTE: diskgroup resource ora.DATA1.dg is online <<<<<集群磁盘组data1资源online
2021-09-08T15:56:29.026891+08:00
ASM Health Checker found 1 new failures
2021-09-08T15:56:31.995419+08:00
NOTE: cache closing disk 0 of grp 1: (not open) DISK1
2021-09-08T15:56:31.996518+08:00
NOTE: cache closing disk 1 of grp 1: (not open) DISK2
2021-09-08T15:56:32.489531+08:00
NOTE: enlarging ACD to 2 threads for group 1/0x59ce6e32 (DATA1)
2021-09-08T15:56:34.107677+08:00
SUCCESS: ACD enlarged for group 1/0x59ce6e32 (DATA1)


下面我们将数据库打开
SQL> alter database open;

Database altered.

检查业务表
SQL> connect lin/oracle
Connected.

SQL> col table_name for a20
SQL> set line 120
SQL> select * from cat;

TABLE_NAME TABLE_TYPE
-------------------- ----------------------
TEST TABLE

SQL> select object_name from test where object_id=100;

OBJECT_NAME
--------------------
SEQ$
这里业务数据没有丢失。

上一篇：【2021/9/9】ITPUB新闻早报
下一篇：19 个 K8S 日常故障处理集锦！
分享好友

分享这个小栈给你的朋友们，一起进步吧。

Oracle运维新鲜事-技术与管理各占半边天
创建时间：2020-08-04 11:34:57
本技术栈旨在分享技术心得，运维趣事，故障处理经验，调优案例，故障处理涉及集群，DG，OGG，大家生产中遇到的问题基本都会囊括了，我会发布生产库遇到的故障，希望在交流中互助互益，共同提高，也希望大家讨论，如果您有生产中遇到的集群问题，也可以在这里提出来，一起讨论，现实中也帮助不少同学解决了生产库的故障。
展开
订阅须知

• 所有用户可根据关注领域订阅专区或所有专区

• 付费订阅：虚拟交易，一经交易不退款；若特殊情况，可3日内客服咨询

• 专区发布评论属默认订阅所评论专区（除付费小栈外）

栈主、嘉宾

查看更多
  • Abraham林老师
    栈主
  • 小雨滴
    嘉宾
  • hawkliu
    嘉宾
  • u_97a59a25246404
    嘉宾

小栈成员

查看更多
  • 栈栈
  • dapan
  • 小菜鸟___
  • hwayw
戳我，来吐槽～

北京盛拓优讯信息技术有限公司. 版权所有  京ICP备16024965号-8  北京市公安局海淀分局网监中心备案编号：11010802021510  未成年人举报专区

违法不良信息举报电话:18101376593   广播电视节目制作经营许可证(京) 字第1234号 中国互联网协会会员