Tuesday, March 10, 2015

Fast Mirror Resync in Real Example (new feature since Oracle 11.1)

SQL> select g.group_number,g.name dgname,d.path, d.repair_timer
  2  from v$asm_disk d,v$asm_diskgroup g
  3  where d.group_number=g.group_number;

GROUP_NUMBER DGNAME                         PATH                           REPAIR_TIMER
------------ ------------------------------ ------------------------------ ------------
           1 DATA                           /dev/rdsk/c2t0d0s6                        0
           1 DATA                           /dev/rdsk/c2t1d0s6                        0
           1 DATA                           /dev/rdsk/c2t2d0s6                        0

SQL> select name,value from v$asm_attribute where group_number=1
  2   and name in ('disk_repair_time','compatible.asm','compatible.rdbms');

NAME                                     VALUE
---------------------------------------- ----------------------------------------
disk_repair_time                         3.6h
compatible.asm                           12.1.0.0.0
compatible.rdbms                         10.1.0.0.0

SQL> alter diskgroup data set attribute 'compatible.rdbms'='11.2';

Diskgroup altered.

SQL> select g.group_number,g.name dgname,d.path, d.repair_timer
  2  from v$asm_disk d,v$asm_diskgroup g
  3  where d.group_number=g.group_number;

GROUP_NUMBER DGNAME                         PATH                           REPAIR_TIMER
------------ ------------------------------ ------------------------------ ------------
           1 DATA                           /dev/rdsk/c2t0d0s6                        0
           1 DATA                           /dev/rdsk/c2t1d0s6                        0
           1 DATA                           /dev/rdsk/c2t2d0s6                        0


2015-03-10 23:23:04.823000 +08:00
SQL> alter diskgroup data set attribute 'compatible.rdbms'='11.2'
NOTE: Advancing RDBMS compatibility to 11.2.0.0.0 for grp 1
GMON querying group 1 at 9 for pid 7, osid 1782
SUCCESS: Advanced compatible.rdbms to 11.2.0.0.0 for grp 1
SUCCESS: alter diskgroup data set attribute 'compatible.rdbms'='11.2'
2015-03-10 23:24:17.988000 +08:00
Warning: VKTM detected a time drift.


SQL> !chmod 000 /dev/rdsk/c2t2d0s6


SQL> select g.group_number,g.name dgname,d.path, d.repair_timer from v$asm_disk d,v$asm_diskgroup g where d.group_number=g.group_number;

GROUP_NUMBER DGNAME                         PATH                           REPAIR_TIMER
------------ ------------------------------ ------------------------------ ------------
           1 DATA                                                                 12960
           1 DATA                           /dev/rdsk/c2t0d0s6                        0
           1 DATA                           /dev/rdsk/c2t1d0s6                        0

SQL> select g.group_number,g.name dgname,d.name, d.repair_timer from v$asm_disk d,v$asm_diskgroup g where d.group_number=g.group_number;

GROUP_NUMBER DGNAME                         NAME                                     REPAIR_TIMER
------------ ------------------------------ ---------------------------------------- ------------
           1 DATA                           DATA_0002                                       12960
           1 DATA                           DATA_0000                                           0
           1 DATA                           DATA_0001                                           0
  


Tue Mar 10 23:35:53 2015
WARNING: Read Failed. group:1 disk:2 AU:0 offset:4096 size:4096
path:Unknown disk
         incarnation:0xf0f0a79d synchronous result:'I/O error'
         subsys:Unknown library krq:0xffff80ffbc948ed8 bufp:0x9c844000 osderr1:0x0 osderr2:0x0
         IO elapsed time: 0 usec Time waited on I/O: 0 usec
WARNING: cache failed reading from group=1(DATA) dsk=2 blk=1 count=1 from disk=2 (DATA_0002) mirror=0 kfkist=0x20 status=0x02 osderr=0x0 file=kfc.c line=12668
Tue Mar 10 23:35:53 2015
Errors in file /u01/app/oracle/diag/asm/+asm/+ASM/trace/+ASM_ora_3468.trc:
ORA-15025: could not open disk "/dev/rdsk/c2t2d0s6"
ORA-27041: unable to open file
Solaris-AMD64 Error: 13: Permission denied
Additional information: 3
ORA-15080: synchronous I/O operation failed to read block 0 of disk 2 in disk group DATA
WARNING: Read Failed. group:1 disk:2 AU:11 offset:4096 size:4096
path:Unknown disk
         incarnation:0xf0f0a79d synchronous result:'I/O error'
         subsys:Unknown library krq:0xffff80ffbc948ed8 bufp:0x9c844000 osderr1:0x0 osderr2:0x0
         IO elapsed time: 0 usec Time waited on I/O: 0 usec
WARNING: cache failed reading from group=1(DATA) dsk=2 blk=1 count=1 from disk=2 (DATA_0002) mirror=1 kfkist=0x20 status=0x02 osderr=0x0 file=kfc.c line=12668
ERROR: cache failed to read group=1(DATA) dsk=2 blk=1 from disk(s): 2(DATA_0002) 2(DATA_0002)
ORA-15080: synchronous I/O operation failed to read block 0 of disk 2 in disk group DATA
ORA-15080: synchronous I/O operation failed to read block 0 of disk 2 in disk group DATA

NOTE: cache initiating offline of disk 2 group DATA
NOTE: process _user3468_+asm (3468) initiating offline of disk 2.4042303389 (DATA_0002) with mask 0x7e in group 1 (DATA) with client assisting
NOTE: checking PST: grp = 1
Tue Mar 10 23:36:00 2015
GMON checking disk modes for group 1 at 10 for pid 27, osid 3468
Tue Mar 10 23:36:00 2015
NOTE: checking PST for grp 1 done.
NOTE: initiating PST update: grp 1 (DATA), dsk = 2/0xf0f0a79d, mask = 0x6a, op = clear
Tue Mar 10 23:36:00 2015
GMON updating disk modes for group 1 at 11 for pid 27, osid 3468
WARNING: GMON has insufficient disks to maintain consensus for group 1. Minimum required is 2: updating 2 PST copies from a total of 3.
NOTE: group DATA: updated PST location: disk 0000 (PST copy 0)
NOTE: group DATA: updated PST location: disk 0001 (PST copy 1)
Tue Mar 10 23:36:01 2015
NOTE: PST update grp = 1 completed successfully
NOTE: ospid 3468 initiating cluster wide offline of disk 2 in group 1
Tue Mar 10 23:36:01 2015
NOTE: disk 2 (DATA_0002) in group 1 (DATA) is locally offline for writes
Tue Mar 10 23:36:01 2015
SUCCESS: extent 1 of file 1 group 1 repaired by relocating to a different AU on the same disk or the disk is offline
Tue Mar 10 23:36:02 2015
NOTE: process _b000_+asm (3487) initiating offline of disk 2.4042303389 (DATA_0002) with mask 0x7e in group 1 (DATA) without client assisting
Tue Mar 10 23:36:02 2015
NOTE: sending set offline flag message (3705766140) to 1 disk(s) in group 1
Tue Mar 10 23:36:02 2015
WARNING: Disk 2 (DATA_0002) in group 1 mode 0x1 is now being offlined
Tue Mar 10 23:36:02 2015
NOTE: initiating PST update: grp 1 (DATA), dsk = 2/0xf0f0a79d, mask = 0x6a, op = clear
Tue Mar 10 23:36:02 2015
GMON updating disk modes for group 1 at 12 for pid 28, osid 3487
Tue Mar 10 23:36:02 2015
NOTE: PST update grp = 1 completed successfully
NOTE: initiating PST update: grp 1 (DATA), dsk = 2/0xf0f0a79d, mask = 0x7e, op = clear
Tue Mar 10 23:36:02 2015
GMON updating disk modes for group 1 at 13 for pid 28, osid 3487
NOTE: group DATA: updated PST location: disk 0000 (PST copy 0)
NOTE: group DATA: updated PST location: disk 0001 (PST copy 1)
Tue Mar 10 23:36:03 2015
NOTE: cache closing disk 2 of grp 1: DATA_0002
Tue Mar 10 23:36:03 2015
NOTE: PST update grp = 1 completed successfully
Tue Mar 10 23:36:03 2015
NOTE: cache closing disk 2 of grp 1: (not open) DATA_0002
Tue Mar 10 23:36:42 2015
WARNING: Started Drop Disk Timeout for Disk 2 (DATA_0002) in group 1 with a value 12960
Tue Mar 10 23:37:51 2015
NOTE: cache closing disk 2 of grp 1: (not open) DATA_0002
Tue Mar 10 23:37:51 2015
NOTE: cache closing disk 2 of grp 1: (not open) DATA_0002

SQL> !chmod 660 /dev/rdsk/c2t2d0s6

SQL> alter diskgroup data online disk 'DATA_0002';

Diskgroup altered.

SQL> select g.group_number,g.name dgname,d.name, d.repair_timer from v$asm_disk d,v$asm_diskgroup g where d.group_number=g.group_number;

GROUP_NUMBER DGNAME                         NAME                           REPAIR_TIMER
------------ ------------------------------ ------------------------------ ------------
           1 DATA                           DATA_0000                                 0
           1 DATA                           DATA_0001                                 0
           1 DATA                           DATA_0002                             12408

SQL> col path for a40
SQL> select g.group_number,g.name dgname,d.path, d.repair_timer from v$asm_disk d,v$asm_diskgroup g where d.group_number=g.group_number;

GROUP_NUMBER DGNAME                         PATH                                     REPAIR_TIMER
------------ ------------------------------ ---------------------------------------- ------------
           1 DATA                           /dev/rdsk/c2t0d0s6                                  0
           1 DATA                           /dev/rdsk/c2t1d0s6                                  0
           1 DATA                           /dev/rdsk/c2t2d0s6                                  0

SQL> select g.group_number,g.name dgname,d.name, d.repair_timer from v$asm_disk d,v$asm_diskgroup g where d.group_number=g.group_number;

GROUP_NUMBER DGNAME                         NAME                           REPAIR_TIMER
------------ ------------------------------ ------------------------------ ------------
           1 DATA                           DATA_0000                                 0
           1 DATA                           DATA_0001                                 0
           1 DATA                           DATA_0002                                 0


SQL> alter diskgroup data online disk 'DATA_0002'
Tue Mar 10 23:48:54 2015
NOTE: cache closing disk 2 of grp 1: (not open) DATA_0002
Tue Mar 10 23:48:54 2015
NOTE: client +ASM:+ASM:ASM dismounting group 1 (DATA)
Tue Mar 10 23:48:54 2015
NOTE: initiating resync of disk group 1 disks
DATA_0002 (2)

NOTE: process _user3694_+asm (3694) initiating offline of disk 2.4042303389 (DATA_0002) with mask 0x7e in group 1 (DATA) without client assisting
Tue Mar 10 23:48:54 2015
NOTE: sending set offline flag message (2531577311) to 1 disk(s) in group 1
Tue Mar 10 23:48:54 2015
WARNING: Disk 2 (DATA_0002) in group 1 mode 0x1 is now being offlined
Tue Mar 10 23:48:54 2015
NOTE: initiating PST update: grp 1 (DATA), dsk = 2/0xf0f0a79d, mask = 0x6a, op = clear
Tue Mar 10 23:48:54 2015
GMON updating disk modes for group 1 at 21 for pid 7, osid 3694
Tue Mar 10 23:48:54 2015
NOTE: cache closing disk 2 of grp 1: (not open) DATA_0002
Tue Mar 10 23:48:54 2015
NOTE: PST update grp = 1 completed successfully
NOTE: initiating PST update: grp 1 (DATA), dsk = 2/0xf0f0a79d, mask = 0x7e, op = clear
Tue Mar 10 23:48:54 2015
GMON updating disk modes for group 1 at 22 for pid 7, osid 3694
Tue Mar 10 23:48:54 2015
NOTE: cache closing disk 2 of grp 1: (not open) DATA_0002
Tue Mar 10 23:48:54 2015
NOTE: PST update grp = 1 completed successfully
NOTE: requesting all-instance membership refresh for group=1
NOTE: initiating PST update: grp 1 (DATA), dsk = 2/0x0, mask = 0x11, op = assign
Tue Mar 10 23:48:54 2015
GMON updating disk modes for group 1 at 23 for pid 7, osid 3694
Tue Mar 10 23:48:54 2015
NOTE: cache closing disk 2 of grp 1: (not open) DATA_0002
NOTE: group DATA: updated PST location: disk 0000 (PST copy 0)
NOTE: group DATA: updated PST location: disk 0001 (PST copy 1)
Tue Mar 10 23:48:54 2015
NOTE: PST update grp = 1 completed successfully
NOTE: requesting all-instance disk validation for group=1
Tue Mar 10 23:48:54 2015
NOTE: disk validation pending for 1 disk in group 1/0xb6705766 (DATA)
NOTE: Found /dev/rdsk/c2t2d0s6 for disk DATA_0002
NOTE: completed disk validation for 1/0xb6705766 (DATA)
Tue Mar 10 23:48:56 2015
NOTE: discarding redo for group 1 disk 2
NOTE: initiating PST update: grp 1 (DATA), dsk = 2/0x0, mask = 0x19, op = assign
Tue Mar 10 23:48:56 2015
GMON updating disk modes for group 1 at 24 for pid 7, osid 3694
NOTE: group DATA: updated PST location: disk 0000 (PST copy 0)
NOTE: group DATA: updated PST location: disk 0001 (PST copy 1)
NOTE: group DATA: updated PST location: disk 0002 (PST copy 2)
Tue Mar 10 23:48:56 2015
NOTE: PST update grp = 1 completed successfully
Tue Mar 10 23:48:56 2015
NOTE: membership refresh pending for group 1/0xb6705766 (DATA)
Tue Mar 10 23:48:56 2015
GMON querying group 1 at 25 for pid 16, osid 1718
NOTE: cache opening disk 2 of grp 1: DATA_0002 path:/dev/rdsk/c2t2d0s6
Tue Mar 10 23:48:56 2015
SUCCESS: refreshed membership for 1/0xb6705766 (DATA)
Tue Mar 10 23:48:56 2015
NOTE: initiating PST update: grp 1 (DATA), dsk = 2/0x0, mask = 0x5d, op = assign
Tue Mar 10 23:48:56 2015
GMON updating disk modes for group 1 at 26 for pid 7, osid 3694
Tue Mar 10 23:48:56 2015
NOTE: PST update grp = 1 completed successfully
NOTE: initiating PST update: grp 1 (DATA), dsk = 2/0x0, mask = 0x7d, op = assign
Tue Mar 10 23:48:57 2015
GMON updating disk modes for group 1 at 27 for pid 7, osid 3694
Tue Mar 10 23:48:57 2015
NOTE: PST update grp = 1 completed successfully
Tue Mar 10 23:48:57 2015
SUCCESS: alter diskgroup data online disk 'DATA_0002'
Tue Mar 10 23:49:00 2015
NOTE: Attempting voting file refresh on diskgroup DATA
Tue Mar 10 23:49:00 2015
NOTE: starting rebalance of group 1/0xb6705766 (DATA) at power 1
Starting background process ARB0
Tue Mar 10 23:49:00 2015
ARB0 started with pid=24, OS id=3699
NOTE: assigning ARB0 to group 1/0xb6705766 (DATA) with 1 parallel I/O
Tue Mar 10 23:49:01 2015
NOTE: header on disk 2 advanced to format #2 using fcn 0.0
NOTE: F1X0 on disk 2 (fmt 2) relocated at fcn 0.3058: AU 0 -> AU 10
NOTE: header on disk 0 advanced to format #2 using fcn 0.730
NOTE: F1B1 fcn on disk 0 synced at fcn 0.3058
NOTE: header on disk 1 advanced to format #2 using fcn 0.730
NOTE: F1B1 fcn on disk 1 synced at fcn 0.3058
Tue Mar 10 23:49:53 2015
NOTE: initiating PST update: grp 1 (DATA), dsk = 2/0x0, mask = 0x7f, op = assign
Tue Mar 10 23:49:53 2015
GMON updating disk modes for group 1 at 28 for pid 24, osid 3699
Tue Mar 10 23:49:53 2015
NOTE: PST update grp = 1 completed successfully
NOTE: reset timers for disk: 2
NOTE: completed online of disk group 1 disks
DATA_0002 (2)

Tue Mar 10 23:49:53 2015
NOTE: stopping process ARB0
Tue Mar 10 23:49:53 2015
NOTE: requesting all-instance membership refresh for group=1
Tue Mar 10 23:49:53 2015
SUCCESS: rebalance completed for group 1/0xb6705766 (DATA)
NOTE: membership refresh pending for group 1/0xb6705766 (DATA)
Tue Mar 10 23:49:53 2015
GMON querying group 1 at 29 for pid 16, osid 1718
Tue Mar 10 23:49:53 2015
SUCCESS: refreshed membership for 1/0xb6705766 (DATA)
NOTE: Attempting voting file refresh on diskgroup DATA