We have a lost voting disks issue yesterday, this occurs due to incorrect aliases or the device mapper persistent names have been changed after storage upgradation/migration.
CRS was not up and showing the error below Obviously the CSSD , cssd.log
So, we have lost the voting and OCR disks.
Here is the situational tasks. In normal condition (10g)
1) We know that OCR is backed up automatically.
2) Deconfigure and configure the CRS (long procedure)
From 11gr2 onwards, Before pen down the procedure some thing you must know is
1) Voting disk also will be backed up automatically to ocr backup. (Many of us does not know this)
2) Clusterware can be startup in exclusive mode i.e no crs (as you know crsd starts later along with other cluster resources)
3) OCR and voting disks are part of ASM, means you can recreate the diskgroups without CRS by manually starting the ASM and assign that diskgroup to CRS as OCR/Voting Disk
Okay, now see how this works, the situational tasks
Assuming you have not started the cluster stack and only hasd is running.
1) Start CRS in exclusive mode in any of the node.
root@rac2 ~]# crsctl start crs -excl -nocrs
CRS-4123: Oracle High Availability Services has been started.
CRS-2672: Attempting to start ‘ora.gipcd’ on ‘rac2′
CRS-2672: Attempting to start ‘ora.mdnsd’ on ‘rac2′
CRS-2676: Start of ‘ora.gipcd’ on ‘rac2′ succeeded
CRS-2676: Start of ‘ora.mdnsd’ on ‘rac2′ succeeded
CRS-2672: Attempting to start ‘ora.gpnpd’ on ‘rac2′
CRS-2676: Start of ‘ora.gpnpd’ on ‘rac2′ succeeded
CRS-2672: Attempting to start ‘ora.cssdmonitor’ on ‘rac2′
CRS-2676: Start of ‘ora.cssdmonitor’ on ‘rac2′ succeeded
CRS-2672: Attempting to start ‘ora.cssd’ on ‘rac2′
CRS-2679: Attempting to clean ‘ora.diskmon’ on ‘rac2′
CRS-2681: Clean of ‘ora.diskmon’ on ‘rac2′ succeeded
CRS-2672: Attempting to start ‘ora.diskmon’ on ‘rac2′
CRS-2676: Start of ‘ora.diskmon’ on ‘rac2′ succeeded
CRS-2676: Start of ‘ora.cssd’ on ‘rac2′ succeeded
CRS-2672: Attempting to start ‘ora.ctssd’ on ‘rac2′
CRS-2672: Attempting to start ‘ora.drivers.acfs’ on ‘rac2′
CRS-2676: Start of ‘ora.drivers.acfs’ on ‘rac2′ succeeded
CRS-2676: Start of ‘ora.ctssd’ on ‘rac2′ succeeded
CRS-2672: Attempting to start ‘ora.asm’ on ‘rac2′
CRS-2676: Start of ‘ora.asm’ on ‘rac2′ succeeded
CRS-2672: Attempting to start ‘ora.crsd’ on ‘rac2′
CRS-2676: Start of ‘ora.crsd’ on ‘rac2′ succeeded
2) Add the new disks to asm diskgroup
oracle@rac2 ~]$ sqlplus / as sysasm
SQL*Plus: Release 11.2.0.1.0 Production on Wed Jun 01 17:23:56 2013
Copyright (c) 1982, 2009, Oracle. All rights reserved.Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 – 64bit Production
With the Real Application Clusters and Automatic Storage Management optionsSQL> create diskgroup DATA external redundancy disk ‘ORCL:DATA’ attribute ‘COMPATIBLE.ASM’ = ‘11.2’;
Diskgroup Created.
3) Identify the latest backup
Identify the latest OCR backup.
root@oracle ~]# /u01/app/11.2.0/grid/bin/ocrconfig -showbackuporacle 2013/06/01 09:00:30 /u01/app/11.2.0/grid/cdata/oracle/backup00.ocr
oracle 2013/06/06 05:37:29 /u01/app/11.2.0/grid/cdata/oracle/backup01.ocr
oracle 2013/06/06 01:37:27 /u01/app/11.2.0/grid/cdata/oracle/backup02.ocr
oracle 2013/05/31 01:37:21 /u01/app/11.2.0/grid/cdata/oracle/day.ocr
oracle 2013/05/24 13:37:19 /u01/app/11.2.0/grid/cdata/oracle/week.ocr
4) Restore the OCR from automatic backup
[root@rac2 ~]# ocrconfig -restore /u01/app/11.2.0/grid/cdata/oracle/backup00.ocr
5) Start the CRS in exclusive mode
crsctl start res ora.crsd -initCRS-2672: Attempting to start ‘ora.crsd’ on ‘rac2′
CRS-2676: Start of ‘ora.crsd’ on ‘rac2′ succeeded
6) Replace the voting disk from automatic backup
[root@rac2 ~]# crsctl replace votedisk +DATA
Successful addition of voting disk ahshuehki8489020msjsjsj
Successfully replaced voting disk group with +DATA.
CRS-4266: Voting file(s) successfully replaced
7) Stop the crs and restart
$CRS_HOME/bin/crsctl stop crs -f$CRS_HOME/bin/crsctl start crs
8) Start the crs in other nodes
$CRS_HOME/bin/crsctl start crs
9) Verify the cluster
# $GRID_HOME/bin/crsctl check cluster -allrac1:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
rac2:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
To note, just in case,
If your SPFILE of ASM is in ocr/votingdisk diskgroup, you need to create temporary parameters for asm and then start asm and then finally add this spfile to Datagroup.
<pre>
Sample Parameters
*.asm_power_limit=1
*.diagnostic_dest=’/u01/app/oragrid’
*.instance_type=’asm’
*.large_pool_size=12M
*.remote_login_passwordfile=’EXCLUSIVE’SQL> create spfile=’+CRS’ from pfile=’/tmp/asm_pfile.ora';
Hope this helps..
No comments:
Post a Comment