APPLIES TO: Oracle Database - Enterprise Edition - Version 11.2.0.1 to 11.2.0.1 [Release 11.2] Oracle Database - Enterprise Edition - Version 12.1.0.1 to 12.1.0.1 [Release 12.1] Oracle Database - Standard Edition - Version 11.2.0.1 and later Oracle Database - Enterprise Edition - Version 11.2.0.2 to 11.2.0.2 [Release 11.2] Information in this document applies to any platform.
SYMPTOMS After Power Fail Alter database open fails with
ORA-00600: internal error code, arguments: [kcratr_nab_less_than_odr] # The Database can't open at this Point. In the corresponding Tracefile we can find this Error Callstack:
dbkedDefDump(): Starting incident default dumps (flags=0x2, level=3, mask=0x0) ----- Current SQL Statement for this session (sql_id=1h50ks4ncswfn) ----- ALTER DATABASE OPEN
----- Call Stack Trace ----- ksedst1 <- ksedst <- dbkedDefDump <- ksedmp <- dbgexPhaseII <- dbgexProcessError <- dbgePostErrorKGE <- kgeasnmierr <- kcratr_odr_check <- kcratr <- kctrec <- kcvcrv <- kcfopd <- adbdrv <- opiexe <- opiosq0 <- kpoal8 <- opiodr <- ttcpip <- opitsk <- opiino <- opiodr <- opidrv <- sou2o <- opimai_real <- ssthrdmain <- main <- start Power failure CAUSE There was a power failure causing logical corruption in controlfile This Problem is caused by Storage Problem of the Database Files. The Subsystem (eg. SAN) crashed while the Database was open. The Database then crashed because the Database Files were not accessible anymore. This caused a lost Write into the Online RedoLogs and/or causing logical corruption in controlfile so Instance Recovery is not possible and raising the ORA-600. Option a: Do cancel based reocvery, and apply 'current online redolog' manually ------------
SQL>Startup mount ;
SQL>Show parameter control_files
Query 1 ------------
SQL> select a.member, a.group#, b.status from v$logfile a ,v$log b where a.group#=b.group# and b.status='CURRENT' ;
Note down the name of the redo log
SQL> Shutdown abort ;
Take a OS Level backup of the controlfile (This is to ensure we have a backup of current state of controlfile)
SQL> Startup mount ; SQL> recover database using backup controlfile until cancel ;
Enter location of redo log shown as current in Query 1 when prompted for recovery
Hit Enter
SQL> Alter database open resetlogs ; Option b: Recreate the controlfile using the Controlfile recreation script -----------
NOTE: If you are using RMAN backup without a catalog, you will need to catalog the backups after the controlfile is recreated. For tape backups, capture your backup information before recreating the controlfile as this will be needed.
With database in mount stage
$ rman target / rman> spool log to '/tmp/rman.log'; rman> list backup ; rman> exit
# Keep this log handy
Go to sqlplus
SQL> Show parameter control_files
Keep this location handy.
SQL> oradebug setmypid SQL> Alter session set tracefile_identifier='controlfilerecreate' ; SQL> Alter database backup controlfile to trace noresetlogs; SQL> oradebug tracefile_name ; --> This command will give the path and name of the trace file
Go to this location ,Open this trace file and select the controlfile recreation script with NOResetlogs option
SQL> Shutdown immediate;
Rename the existing controlfile to <originalname>_old ---> This is Important as we need to have a backup of existing controlfile since we plan to recreate it
SQL> Startup nomount
Now run the Controlfile recreation script with NO Resetlogs option.
SQL> Alter database open ;
For database version 10g and above
Once database is opened you can recatalog the rman backup information present in the list /tmp/rman.log using
Rman> Catalog start with '<location of backupiece>' ;
If backups are on tape, and you are not using a catalog, backups can be cataloged using information in: How to Catalog Tape Backup Pieces (Doc ID 550082.1) Once the database has been opened using the Option a or Option b, it is recommended to take a hot backup of the database. The same steps are applicable to RAC if all instance are down with same error. If both Options (a|b) fails, or you do not have the full set of files, then you have to restore and recover the Database from a recent Backup. REFERENCES NOTE:550082.1 - HOW TO CATALOG TAPE BACKUP PIECES |