??11.2 RAC??OCR?Votedisk??ASM Diskgroup?????
- by Liu Maclean(???)
????????Oracle Allstarts??????????ocr?votedisk?ASM diskgroup??11gR2 RAC cluster?????????,????«?11gR2 RAC???ASM DISK Path????»??????,??????CRS??????11.2??ASM???????, ????????????”crsctl start crs -excl -nocrs “;
?????????,??ASM????ocr?????votedisk?????,??11.2????ocr?votedisk???ASM?,?ASM???????ocr?votedisk,?????ocr?votedisk????????cluter??????;???????????CRS????,?????diskgroup??????????,??????????????????
??:?????????????????ASM LUN DISK,???OCR?????,????????4??????????,???????$GI_HOME,?????????;????votedisk??
????: ??dd????ocr?votedisk??diskgroup header,??diskgroup corruption:
1. ??votedisk? ocr??
[root@vrh1 ~]# crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE a853d6204bbc4feabfd8c73d4c3b3001 (/dev/asm-diskh) [SYSTEMDG]
2. ONLINE a5b37704c3574f0fbf21d1d9f58c4a6b (/dev/asm-diskg) [SYSTEMDG]
3. ONLINE 36e5c51ff0294fc3bf2a042266650331 (/dev/asm-diski) [SYSTEMDG]
4. ONLINE af337d1512824fe4bf6ad45283517aaa (/dev/asm-diskj) [SYSTEMDG]
5. ONLINE 3c4a349e2e304ff6bf64b2b1c9d9cf5d (/dev/asm-diskk) [SYSTEMDG]
Located 5 voting disk(s).
su - grid
[grid@vrh1 ~]$ ocrconfig -showbackup
PROT-26: Oracle Cluster Registry backup locations were retrieved from a local copy
vrh1 2012/08/09 01:59:56 /g01/11.2.0/maclean/grid/cdata/vrh-cluster/backup00.ocr
vrh1 2012/08/08 21:59:56 /g01/11.2.0/maclean/grid/cdata/vrh-cluster/backup01.ocr
vrh1 2012/08/08 17:59:55 /g01/11.2.0/maclean/grid/cdata/vrh-cluster/backup02.ocr
vrh1 2012/08/08 05:59:54 /g01/11.2.0/grid/cdata/vrh-cluster/day.ocr
vrh1 2012/08/08 05:59:54 /g01/11.2.0/grid/cdata/vrh-cluster/week.ocr
PROT-25: Manual backups for the Oracle Cluster Registry are not available
2. ??????????clusterware ,OHASD
crsctl stop has -f
3. GetAsmDH.sh ==> GetAsmDH.sh?ASM disk header?????
????????,????????asm header
[grid@vrh1 ~]$ ./GetAsmDH.sh
############################################
1) Collecting Information About the Disks:
############################################
SQL*Plus: Release 11.2.0.3.0 Production on Thu Aug 9 03:28:13 2012
Copyright (c) 1982, 2011, Oracle. All rights reserved.
SQL> Connected.
SQL> SQL> SQL> SQL> SQL> SQL> SQL>
1 0 /dev/asm-diske
1 1 /dev/asm-diskd
2 0 /dev/asm-diskb
2 1 /dev/asm-diskc
2 2 /dev/asm-diskf
3 0 /dev/asm-diskh
3 1 /dev/asm-diskg
3 2 /dev/asm-diski
3 3 /dev/asm-diskj
3 4 /dev/asm-diskk
SQL> SQL> Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options
-rw-r--r-- 1 grid oinstall 1048 Aug 9 03:28 /tmp/HC/asmdisks.lst
############################################
2) Generating asm_diskh.sh script.
############################################
-rwx------ 1 grid oinstall 666 Aug 9 03:28 /tmp/HC/asm_diskh.sh
############################################
3) Executing asm_diskh.sh script to
generate dd dumps.
############################################
-rw-r--r-- 1 grid oinstall 1048576 Aug 9 03:28 /tmp/HC/dsk_1_0.dd
-rw-r--r-- 1 grid oinstall 1048576 Aug 9 03:28 /tmp/HC/dsk_1_1.dd
-rw-r--r-- 1 grid oinstall 1048576 Aug 9 03:28 /tmp/HC/dsk_2_0.dd
-rw-r--r-- 1 grid oinstall 1048576 Aug 9 03:28 /tmp/HC/dsk_2_1.dd
-rw-r--r-- 1 grid oinstall 1048576 Aug 9 03:28 /tmp/HC/dsk_2_2.dd
-rw-r--r-- 1 grid oinstall 1048576 Aug 9 03:28 /tmp/HC/dsk_3_0.dd
-rw-r--r-- 1 grid oinstall 1048576 Aug 9 03:28 /tmp/HC/dsk_3_1.dd
-rw-r--r-- 1 grid oinstall 1048576 Aug 9 03:28 /tmp/HC/dsk_3_2.dd
-rw-r--r-- 1 grid oinstall 1048576 Aug 9 03:28 /tmp/HC/dsk_3_3.dd
-rw-r--r-- 1 grid oinstall 1048576 Aug 9 03:28 /tmp/HC/dsk_3_4.dd
############################################
4) Compressing dd dumps in the next format:
(asm_dd_header_all_.tar)
############################################
/tmp/HC/dsk_1_0.dd
/tmp/HC/dsk_1_1.dd
/tmp/HC/dsk_2_0.dd
/tmp/HC/dsk_2_1.dd
/tmp/HC/dsk_2_2.dd
/tmp/HC/dsk_3_0.dd
/tmp/HC/dsk_3_1.dd
/tmp/HC/dsk_3_2.dd
/tmp/HC/dsk_3_3.dd
/tmp/HC/dsk_3_4.dd
./GetAsmDH.sh: line 81: compress: command not found
ls: /tmp/HC/*.Z: No such file or directory
[grid@vrh1 ~]$
4. ??dd ?? ??ocr?votedisk??diskgroup
[root@vrh1 ~]# dd if=/dev/zero of=/dev/asm-diskh bs=1024k count=1
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00423853 seconds, 247 MB/s
[root@vrh1 ~]# dd if=/dev/zero of=/dev/asm-diskg bs=1024k count=1
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0045179 seconds, 232 MB/s
[root@vrh1 ~]# dd if=/dev/zero of=/dev/asm-diski bs=1024k count=1
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00469976 seconds, 223 MB/s
[root@vrh1 ~]# dd if=/dev/zero of=/dev/asm-diskj bs=1024k count=1
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00344262 seconds, 305 MB/s
[root@vrh1 ~]# dd if=/dev/zero of=/dev/asm-diskk bs=1024k count=1
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0053518 seconds, 196 MB/s
5. ????????????HAS
[root@vrh1 ~]# crsctl start has
CRS-4123: Oracle High Availability Services has been started.
????ocr?votedisk??diskgroup??,??CSS???????,???????:
alertvrh1.log
[cssd(5162)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /g01/11.2.0/grid/log/vrh1/cssd/ocssd.log
2012-08-09 03:35:41.207
[cssd(5162)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /g01/11.2.0/grid/log/vrh1/cssd/ocssd.log
2012-08-09 03:35:56.240
[cssd(5162)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /g01/11.2.0/grid/log/vrh1/cssd/ocssd.log
2012-08-09 03:36:11.284
[cssd(5162)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /g01/11.2.0/grid/log/vrh1/cssd/ocssd.log
2012-08-09 03:36:26.305
[cssd(5162)]CRS-1714:Unable to discover any voting files, retrying discovery in 15 seconds; Details at (:CSSNM00070:) in /g01/11.2.0/grid/log/vrh1/cssd/ocssd.log
2012-08-09 03:36:41.328
ocssd.log
2012-08-09 03:40:26.662: [ CSSD][1078700352]clssnmReadDiscoveryProfile: voting file discovery string(/dev/asm*)
2012-08-09 03:40:26.662: [ CSSD][1078700352]clssnmvDDiscThread: using discovery string /dev/asm* for initial discovery
2012-08-09 03:40:26.662: [ SKGFD][1078700352]Discovery with str:/dev/asm*:
2012-08-09 03:40:26.662: [ SKGFD][1078700352]UFS discovery with :/dev/asm*:
2012-08-09 03:40:26.665: [ SKGFD][1078700352]Fetching UFS disk :/dev/asm-diskf:
2012-08-09 03:40:26.665: [ SKGFD][1078700352]Fetching UFS disk :/dev/asm-diskb:
2012-08-09 03:40:26.665: [ SKGFD][1078700352]Fetching UFS disk :/dev/asm-diskj:
2012-08-09 03:40:26.665: [ SKGFD][1078700352]Fetching UFS disk :/dev/asm-diskh:
2012-08-09 03:40:26.665: [ SKGFD][1078700352]Fetching UFS disk :/dev/asm-diskc:
2012-08-09 03:40:26.665: [ SKGFD][1078700352]Fetching UFS disk :/dev/asm-diskd:
2012-08-09 03:40:26.665: [ SKGFD][1078700352]Fetching UFS disk :/dev/asm-diske:
2012-08-09 03:40:26.665: [ SKGFD][1078700352]Fetching UFS disk :/dev/asm-diskg:
2012-08-09 03:40:26.665: [ SKGFD][1078700352]Fetching UFS disk :/dev/asm-diski:
2012-08-09 03:40:26.665: [ SKGFD][1078700352]Fetching UFS disk :/dev/asm-diskk:
2012-08-09 03:40:26.665: [ SKGFD][1078700352]OSS discovery with :/dev/asm*:
2012-08-09 03:40:26.665: [ SKGFD][1078700352]Handle 0xdf22a0 from lib :UFS:: for disk :/dev/asm-diskf:
2012-08-09 03:40:26.665: [ SKGFD][1078700352]Handle 0xf412a0 from lib :UFS:: for disk :/dev/asm-diskb:
2012-08-09 03:40:26.666: [ SKGFD][1078700352]Handle 0xf3a680 from lib :UFS:: for disk :/dev/asm-diskj:
2012-08-09 03:40:26.666: [ SKGFD][1078700352]Handle 0xf93da0 from lib :UFS:: for disk :/dev/asm-diskh:
2012-08-09 03:40:26.667: [ CSSD][1078700352]clssnmvDiskVerify: Successful discovery of 0 disks
2012-08-09 03:40:26.667: [ CSSD][1078700352]clssnmCompleteInitVFDiscovery: Completing initial voting file discovery
2012-08-09 03:40:26.667: [ CSSD][1078700352]clssnmvFindInitialConfigs: No voting files found
2012-08-09 03:40:26.667: [ CSSD][1078700352](:CSSNM00070:)clssnmCompleteInitVFDiscovery: Voting file not found. Retrying discovery in 15 seconds
?????ocr?votedisk??diskgroup?????:
1. ?-excl -nocrs ????cluster,??????ASM?? ????CRS
[root@vrh1 vrh1]# crsctl start crs -excl -nocrs
CRS-4123: Oracle High Availability Services has been started.
CRS-2672: Attempting to start 'ora.mdnsd' on 'vrh1'
CRS-2676: Start of 'ora.mdnsd' on 'vrh1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'vrh1'
CRS-2676: Start of 'ora.gpnpd' on 'vrh1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'vrh1'
CRS-2672: Attempting to start 'ora.gipcd' on 'vrh1'
CRS-2676: Start of 'ora.cssdmonitor' on 'vrh1' succeeded
CRS-2676: Start of 'ora.gipcd' on 'vrh1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'vrh1'
CRS-2672: Attempting to start 'ora.diskmon' on 'vrh1'
CRS-2676: Start of 'ora.diskmon' on 'vrh1' succeeded
CRS-2676: Start of 'ora.cssd' on 'vrh1' succeeded
CRS-2679: Attempting to clean 'ora.cluster_interconnect.haip' on 'vrh1'
CRS-2672: Attempting to start 'ora.ctssd' on 'vrh1'
CRS-2681: Clean of 'ora.cluster_interconnect.haip' on 'vrh1' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'vrh1'
CRS-2676: Start of 'ora.ctssd' on 'vrh1' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'vrh1' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'vrh1'
CRS-2676: Start of 'ora.asm' on 'vrh1' succeeded
2.???ocr?votedisk??diskgroup,??compatible.asm???11.2:
[root@vrh1 vrh1]# su - grid
[grid@vrh1 ~]$ sqlplus / as sysasm
SQL*Plus: Release 11.2.0.3.0 Production on Thu Aug 9 04:16:58 2012
Copyright (c) 1982, 2011, Oracle. All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Real Application Clusters and Automatic Storage Management options
SQL> create diskgroup systemdg high redundancy disk '/dev/asm-diskh','/dev/asm-diskg','/dev/asm-diski','/dev/asm-diskj','/dev/asm-diskk'
ATTRIBUTE 'compatible.rdbms' = '11.2', 'compatible.asm' = '11.2';
3.?ocr backup???ocr??ocrcheck??:
[root@vrh1 ~]# ocrconfig -restore /g01/11.2.0/grid/cdata/vrh-cluster/backup00.ocr
[root@vrh1 ~]# ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 3
Total space (kbytes) : 262120
Used space (kbytes) : 3180
Available space (kbytes) : 258940
ID : 1238458014
Device/File Name : +systemdg
Device/File integrity check succeeded
Device/File not configured
Device/File not configured
Device/File not configured
Device/File not configured
Cluster registry integrity check succeeded
Logical corruption check succeeded
4. ????votedisk ,??????????:
[grid@vrh1 ~]$ crsctl replace votedisk +SYSTEMDG
CRS-4602: Failed 27 to add voting file 2e4e0fe285924f86bf5473d00dcc0388.
CRS-4602: Failed 27 to add voting file 4fa54bb0cc5c4fafbf1a9be5479bf389.
CRS-4602: Failed 27 to add voting file a109ead9ea4e4f28bfe233188623616a.
CRS-4602: Failed 27 to add voting file 042c9fbd71b54f5abfcd3ab3408f3cf3.
CRS-4602: Failed 27 to add voting file 7b5a8cd24f954fafbf835ad78615763f.
Failed to replace voting disk group with +SYSTEMDG.
CRS-4000: Command Replace failed, or completed with errors.
????????ASM???,???ASM:
SQL> alter system set asm_diskstring='/dev/asm*';
System altered.
SQL> create spfile from memory;
File created.
SQL> startup force mount;
ORA-32004: obsolete or deprecated parameter(s) specified for ASM instance
ASM instance started
Total System Global Area 283930624 bytes
Fixed Size 2227664 bytes
Variable Size 256537136 bytes
ASM Cache 25165824 bytes
ASM diskgroups mounted
SQL> show parameter spfile
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
spfile string /g01/11.2.0/grid/dbs/spfile+AS
M1.ora
[grid@vrh1 trace]$ crsctl replace votedisk +SYSTEMDG
CRS-4256: Updating the profile
Successful addition of voting disk 85edc0e82d274f78bfc58cdc73b8c68a.
Successful addition of voting disk 201ffffc8ba44faabfe2efec2aa75840.
Successful addition of voting disk 6f2a25c589964faabf6980f7c5f621ce.
Successful addition of voting disk 93eb315648454f25bf3717df1a2c73d5.
Successful addition of voting disk 3737240678964f88bfbfbd31d8b3829f.
Successfully replaced voting disk group with +SYSTEMDG.
CRS-4256: Updating the profile
CRS-4266: Voting file(s) successfully replaced
5. ??has??,??cluster????:
[root@vrh1 ~]# crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
[root@vrh1 ~]# crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 85edc0e82d274f78bfc58cdc73b8c68a (/dev/asm-diskh) [SYSTEMDG]
2. ONLINE 201ffffc8ba44faabfe2efec2aa75840 (/dev/asm-diskg) [SYSTEMDG]
3. ONLINE 6f2a25c589964faabf6980f7c5f621ce (/dev/asm-diski) [SYSTEMDG]
4. ONLINE 93eb315648454f25bf3717df1a2c73d5 (/dev/asm-diskj) [SYSTEMDG]
5. ONLINE 3737240678964f88bfbfbd31d8b3829f (/dev/asm-diskk) [SYSTEMDG]
Located 5 voting disk(s).
[root@vrh1 ~]# crsctl stat res -t
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.BACKUPDG.dg
ONLINE ONLINE vrh1
ora.DATA.dg
ONLINE ONLINE vrh1
ora.LISTENER.lsnr
ONLINE ONLINE vrh1
ora.LSN_MACLEAN.lsnr
ONLINE ONLINE vrh1
ora.SYSTEMDG.dg
ONLINE ONLINE vrh1
ora.asm
ONLINE ONLINE vrh1 Started
ora.gsd
OFFLINE OFFLINE vrh1
ora.net1.network
ONLINE ONLINE vrh1
ora.ons
ONLINE ONLINE vrh1
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr http://www.askmaclean.com
1 ONLINE ONLINE vrh1
ora.cvu
1 OFFLINE OFFLINE
ora.oc4j
1 OFFLINE OFFLINE
ora.scan1.vip
1 ONLINE ONLINE vrh1
ora.vprod.db
1 ONLINE OFFLINE
2 ONLINE OFFLINE
ora.vrh1.vip
1 ONLINE ONLINE vrh1
ora.vrh2.vip
1 ONLINE INTERMEDIATE vrh1 FAILED OVER
