RAC 实例不能启动 ORA-1589 signalled during ALTER DATABASE OPEN

今天启动RAC做实验，发现RAC实例怎么启动不了。

[oracle@rac2 bin]$ crs_stat -t

NameTypeTargetStateHost

------------------------------------------------------------

ora.orcl.dbapplicationONLINEOFFLINE

ora....oltp.cs applicationONLINEOFFLINE

ora....cl1.srv applicationONLINEOFFLINE

ora....cl2.srv applicationONLINEOFFLINE

ora....l1.inst applicationONLINEOFFLINE

ora....l2.inst applicationONLINEOFFLINE

ora....SM1.asm applicationONLINEONLINErac1

ora....C1.lsnr applicationONLINEONLINErac1

ora.rac1.gsdapplicationONLINEONLINErac1

ora.rac1.onsapplicationONLINEONLINErac1

ora.rac1.vipapplicationONLINEONLINErac1

ora....SM2.asm applicationONLINEONLINErac2

ora....C2.lsnr applicationONLINEONLINErac2

ora.rac2.gsdapplicationONLINEONLINErac2

ora.rac2.onsapplicationONLINEONLINErac2

ora.rac2.vipapplicationONLINEONLINErac2

看了一下ASM的磁盘组，都挂载成功。

[oracle@rac2 bin]$export ORACLE_SID=+ASM2

[oracle@rac2 bin]$sqlplus / as sysdba;

SQL*Plus: Release 10.2.0.1.0 - Production on Tue Sep 28 04:33:24 2010

Copyright (c) 1982, 2005, Oracle.All rights reserved.

Connected to:

Oracle Database 10g Enterprise Edition Release 10.2.0.1.0 - Production

With the Partitioning, Real Application Clusters, OLAP and Data Mining options

SQL>select name,state from v$asm_diskgroup;

NAMESTATE

------------------------------ -----------

DATAMOUNTED

FLASH_RECOVERY_AREAMOUNTED

SQL>

之前遇到过这种情况情况，所以第一时间就查看了这个ASM磁盘组的信息。

ASM没有问题，只能查看数据库的alert log了。

[oracle@rac1 bdump]$ pwd

/u01/app/oracle/admin/orcl/bdump

[oracle@rac1 bdump]$ ls

alert_orcl1.logorcl1_diag_22153.trcorcl1_lms0_22169.trc

cdmp_20100917173112orcl1_lgwr_22186.trcorcl1_smon_22190.trc

orcl1_arc0_24180.trcorcl1_lmd0_22167.trc

orcl1_arc1_24182.trcorcl1_lmon_22165.trc

[oracle@rac1 bdump]$tail -100 alert_orcl1.log

Tue Sep 28 04:05:20 2010

starting up 1 dispatcher(s) for network address '(ADDRESS=(PARTIAL=YES)(PROTOCOL=TCP))'...

MMON started with pid=15, OS id=5650

Tue Sep 28 04:05:20 2010

starting up 1 shared server(s) ...

Tue Sep 28 04:05:20 2010

lmon registered with NM - instance id 1 (internal mem no 0)

Tue Sep 28 04:05:21 2010

Reconfiguration started (old inc 0, new inc 2)

List of nodes:

0

Global Resource Directory frozen

* allocate domain 0, invalid = TRUE

Communication channels reestablished

Master broadcasted resource hash value bitmaps

Non-local Process blocks cleaned out

Tue Sep 28 04:05:21 2010

LMS 0: 0 GCS shadows cancelled, 0 closed

Set master node info

Submitted all remote-enqueue requests

Dwn-cvts replayed, VALBLKs dubious

All grantable enqueues granted

Post SMON to start 1st pass IR

Tue Sep 28 04:05:21 2010

LMS 0: 0 GCS shadows traversed, 0 replayed

Tue Sep 28 04:05:21 2010

Submitted all GCS remote-cache requests

Post SMON to start 1st pass IR

Fix write in gcs resources

Reconfiguration complete

LCK0 started with pid=19, OS id=5683

Tue Sep 28 04:05:22 2010

ALTER DATABASEMOUNT

Tue Sep 28 04:05:22 2010

This instance was first to mount

Tue Sep 28 04:05:22 2010

Starting background process ASMB

ASMB started with pid=21, OS id=5706

Starting background process RBAL

RBAL started with pid=22, OS id=5710

Loaded ASM Library - Generic Linux, version 2.0.4 (KABI_V2) library for asmlib interface

Tue Sep 28 04:05:26 2010

SUCCESS: diskgroup DATA was mounted

SUCCESS: diskgroup FLASH_RECOVERY_AREA was mounted

Tue Sep 28 04:05:34 2010

Setting recovery target incarnation to 3

Tue Sep 28 04:05:35 2010

Successful mount of redo thread 1, with mount id 1258660482

Tue Sep 28 04:05:35 2010

Database mounted in Shared Mode (CLUSTER_DATABASE=TRUE)

Completed: ALTER DATABASEMOUNT

Tue Sep 28 04:05:35 2010

ALTER DATABASE OPEN

This instance was first to open

ORA-1589 signalled during: ALTER DATABASE OPEN...

Tue Sep 28 04:05:39 2010

Shutting down instance (abort)

License high water mark = 1

Instance terminated by USER, pid = 6080

从日志里我们发现了一个很重要的信息：

ALTER DATABASE OPEN

This instance was first to open

ORA-1589 signalled during: ALTER DATABASE OPEN...

Tue Sep 28 04:05:39 2010

Shutting down instance (abort)

查看了一下错误代码：

ORA-01589:	must use RESETLOGS or NORESETLOGS option for database open
Cause:	Either incomplete or backup control file recovery has been performed. After these types of recovery you must specify either the RESETLOGS option or the NORESETLOGS option to open your database.
Action:	Specify the appropriate option.

提示我们必须使用resetlogs模式打开数据。开到这才明白，昨天做RMAN恢复后直接把数据库关了，没有用resetlogs打开。杯具中..

手工启动数据库至mount状态，在用resetlogs打开：

[oracle@rac1 bdump]$export ORACLE_SID=orcl1

[oracle@rac1 bdump]$sqlplus /nolog

SQL*Plus: Release 10.2.0.1.0 - Production on Tue Sep 28 04:17:47 2010

Copyright (c) 1982, 2005, Oracle.All rights reserved.

SQL>conn / as sysdba;

Connected to an idle instance.

SQL>startup mount;

ORACLE instance started.

Total System Global Area180355072 bytes

Fixed Size1218388 bytes

Variable Size109054124 bytes

Database Buffers67108864 bytes

Redo Buffers2973696 bytes

Database mounted.

SQL>alter database open resetlogs;

Database altered.

SQL>

在另一个节点，直接正常启动就可以了。

[oracle@rac2 bin]$export ORACLE_SID=orcl2

[oracle@rac2 bin]$sqlplus /nolog

SQL*Plus: Release 10.2.0.1.0 - Production on Tue Sep 28 04:22:50 2010

Copyright (c) 1982, 2005, Oracle.All rights reserved.

SQL>conn / as sysdba

Connected to an idle instance.

SQL>startup mount;

ORACLE instance started.

Total System Global Area180355072 bytes

Fixed Size1218388 bytes

Variable Size88082604 bytes

Database Buffers88080384 bytes

Redo Buffers2973696 bytes

Database mounted.

SQL>alter database open;

Database altered.

SQL>

查看一下RAC进程的状态：

[oracle@rac2 bin]$ crs_stat -t

NameTypeTargetStateHost

------------------------------------------------------------

ora.orcl.dbapplicationONLINEONLINErac2

ora....oltp.cs applicationONLINEONLINErac2

ora....cl1.srv applicationONLINEONLINErac1

ora....cl2.srv applicationONLINEONLINErac2

ora....l1.inst applicationONLINEONLINErac1

ora....l2.inst applicationONLINEONLINErac2

ora....SM1.asm applicationONLINEONLINErac1

ora....C1.lsnr applicationONLINEONLINErac1

ora.rac1.gsdapplicationONLINEONLINErac1

ora.rac1.onsapplicationONLINEONLINErac1

ora.rac1.vipapplicationONLINEONLINErac1

ora....SM2.asm applicationONLINEONLINErac2

ora....C2.lsnr applicationONLINEONLINErac2

ora.rac2.gsdapplicationONLINEONLINErac2

ora.rac2.onsapplicationONLINEONLINErac2

ora.rac2.vipapplicationONLINEONLINErac2

一切正常，从这个小错误中明白，遇到错误，要养成看alert log的习惯。不管是单实例还是RAC实例，看log永远是第一条。还有就是要心细，昨天偷了懒，为了补这个懒，花的时间就比昨天多了很多。杯具啊.

------------------------------------------------------------------------------