Friday, December 18, 2009

ORA-16766: Redo Apply unexpectedly offline in Dataguard environment

Synptoms:


DG 2009-12-18-20:32:19 0 2 0 Executing SQL [ALTER DATABASE RECOVER MANAGED STANDBY DATABASE THROUGH ALL SWITCHOVER DISCONNECT USING CURRENT LOGFILE]
DG 2009-12-18-20:32:25 0 2 0 SQL [ALTER DATABASE RECOVER MANAGED STANDBY DATABASE THROUGH ALL SWITCHOVER DISCONNECT USING CURRENT LOGFILE] Executed
successfully
DG 2009-12-18-20:32:46 0 2 705961914 rfm_inst_phase_dispatch 16 END phase processing
DG 2009-12-18-20:32:46 0 2 705961914 DMON: CTL_ENABLE operation completed
DG 2009-12-18-20:32:56 0 2 0 DMON: HEALTH CHECK ERROR: ORA-16766: Redo Apply unexpectedly offline
DG 2009-12-18-20:32:56 0 2 705959745 Operation CTL_GET_STATUS cancelled during phase 1, error = ORA-16766
DG 2009-12-18-20:33:56 0 2 0 DMON: HEALTH CHECK ERROR: ORA-16766: Redo Apply unexpectedly offline
DG 2009-12-18-20:33:56 0 2 705959748 Operation CTL_GET_STATUS cancelled during phase 1, error = ORA-16766
DG 2009-12-18-20:34:56 0 2 0 DMON: HEALTH CHECK ERROR: ORA-16766: Redo Apply unexpectedly offline


Root Cause:

One datafile was not been restored in the "restore database" command, because of "restore database" command skips readonly tablespaces by default.


Fri Dec 18 20:32:24 2009
Errors in file /u01/app/oracle/admin/hkdrt/bdump/hkdrt1_dbw0_29459.trc:
ORA-01157: cannot identify/lock data file 6 - see DBWR trace file
ORA-01110: data file 6: '+DATADG/hkdrt/datafile/wk_data_1.465.704715067'
ORA-17503: ksfdopn:2 Failed to open file +DATADG/hkdrt/datafile/wk_data_1.465.704715067
ORA-15012: ASM file '+DATADG/hkdrt/datafile/wk_data_1.465.704715067' does not exist
ORA-17503: ksfdopn:2 Failed to open file +DATADG/hkdrt/tempfile/temp.458.704725981
ORA-15012: ASM file '+DATADG/hkdrt/tempfile/temp.458.704725981' does not exist
Fri Dec 18 20:32:24 2009
MRP0: Background Media Recovery terminated with error 1110
Fri Dec 18 20:32:24 2009
Errors in file /u01/app/oracle/admin/hkdrt/bdump/hkdrt1_mrp0_30074.trc:
ORA-01110: data file 6: '+DATADG/hkdrt/datafile/wk_data_1.465.704715067'
ORA-01157: cannot identify/lock data file 6 - see DBWR trace file
ORA-01110: data file 6: '+DATADG/hkdrt/datafile/wk_data_1.465.704715067'
Fri Dec 18 20:32:24 2009
Managed Standby Recovery not using Real Time Apply