Oracle 11gR2 + ASM + spfile (eng)

---

Oracle 11gR2 have many new futures in Grid Computing area. One of them is Oracle Local Repository (OLR), this repository designed to store information and profiles for local resources, resources that dedicated to particular node. It improves the performance of accessing local resources profile information, redundancy and manageability. In Grid Infrastructure RAC configuration there is usual one global shared OCR and OLR’s on each node. In Oracle Restart environment there is only OLR repository. In 11g R2 there is also new feature, Grid Plug and Play (GPNP), as the name implies, it helps to automate and simplify some of the aspects of grid administration. GPNP maintains profile, it is XML file that stores the configuration information of some components maintained by GPNP, for example vips and interconnect information is stored here.

But in this post, I wont to discuss another new feature of 11g R2, it is the possibility to store the server parameter file (spfile) on the ASM disks, also for ASM instance itself! And there is some reasonable questions arise:

1. How process can read the spfile from ASM disks, if they are not mounted yet
2. How process knows where to get spfile if there is no init.ora file in $ORACLE_HOME/dbs with spfile=[path to spfile] parameter
3. What about asm_diskstring parameter?

Third point is important for ASM instance because, before reading spfile from ASM disks, we need to identify them, to identify them we need spfile to get asm_diskstring parameter!

To make clear all this points I will use system calls tracing tools, to trace oracle processes. We can use tools like Dtrace, strace, truss and so on, it depends on platform, I will use strace because I am on Linux.

My environment is Oracle Grid Infrastructure 11g R2 (11.2.0.1.0) in Oracle Restart mode. I simulated ASM disks using loop devices, /dev/loop*, I have 5 disks

bash> /etc/init.d/oracleasm listdisks
DISK1
DISK2
DISK3
DISK4
DISK5

SQL> conn / as sysasm
Connected.
SQL> col path format a15

SQL> select a.name,a.state,b.name,b.path from v$asm_diskgroup a, v$asm_disk b where a.group_number=b.group_number order by b.name;

NAME                           STATE       NAME                           PATH
------------------------------ ----------- ------------------------------ ---------------
DGROUP1                        MOUNTED     DISK1                          ORCL:DISK1
DGROUP1                        MOUNTED     DISK2                          ORCL:DISK2
DGROUP2                        MOUNTED     DISK3                          ORCL:DISK3
DGROUP2                        MOUNTED     DISK4                          ORCL:DISK4
DGROUP2                        MOUNTED     DISK5                          ORCL:DISK5

As seen from the listing, I have two disk groups DGROUP1 and DGROUP2. My spfile is located on DGROUP1

SQL> show parameter spfile;

NAME                                 TYPE        VALUE
------------------------------------ ----------- ----------------------------------------------------
spfile                               string      +DGROUP1/asm/asmparameterfile/registry.253.740659373

Lets make some tests

SQL> conn / as sysasm
Connected.

SQL> startup
ASM instance started

Total System Global Area  284565504 bytes
Fixed Size                  1336036 bytes
Variable Size             258063644 bytes
ASM Cache                  25165824 bytes
ASM diskgroups mounted
SQL>
SQL> show parameter power

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
asm_power_limit                      integer     1
SQL> alter system set asm_power_limit=3;

System altered.

SQL> show parameter power

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
asm_power_limit                      integer     3

SQL>--We set asm_power_limit to 3 in spfile

SQL> shutdown immediate;
ASM diskgroups dismounted
ASM instance shutdown

SQL> !crs_stat -t
Name           Type           Target    State     Host
------------------------------------------------------------
ora.DGROUP1.dg ora....up.type OFFLINE   OFFLINE
ora.DGROUP2.dg ora....up.type OFFLINE   OFFLINE
ora....ER.lsnr ora....er.type ONLINE    ONLINE    testdb03
ora.asm        ora.asm.type   OFFLINE   OFFLINE
ora.cssd       ora.cssd.type  ONLINE    ONLINE    testdb03
ora.diskmon    ora....on.type ONLINE    ONLINE    testdb03

SQL>-- Lets start instance in nomount mode, it will not mount the diskgroups

SQL> startup nomount;
ASM instance started

Total System Global Area  284565504 bytes
Fixed Size                  1336036 bytes
Variable Size             258063644 bytes
ASM Cache                  25165824 bytes

SQL> show parameter spfile;

NAME                                 TYPE        VALUE
------------------------------------ ----------- ----------------------------------------------------
spfile                               string      +DGROUP1/asm/asmparameterfile/registry.253.740659373

SQL> show parameter power

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
asm_power_limit                      integer     3

SQL> select * from v$spparameter;
select * from v$spparameter
              *
ERROR at line 1:
ORA-15001: diskgroup "DGROUP1" does not exist or is not mounted

SQL> alter system set asm_power_limit=10;
alter system set asm_power_limit=10
*
ERROR at line 1:
ORA-32000: write to SPFILE requested but SPFILE is not modifiable

SQL> !asmcmd
ASMCMD> ls
ASMCMD> exit

SQL> !crs_stat -t
Name           Type           Target    State     Host
------------------------------------------------------------
ora.DGROUP1.dg ora....up.type OFFLINE   OFFLINE
ora.DGROUP2.dg ora....up.type OFFLINE   OFFLINE
ora....ER.lsnr ora....er.type ONLINE    ONLINE    testdb03
ora.asm        ora.asm.type   ONLINE    ONLINE    testdb03
ora.cssd       ora.cssd.type  ONLINE    ONLINE    testdb03
ora.diskmon    ora....on.type ONLINE    ONLINE    testdb03

So, what we see, we were able to start ASM instance without mounting the disks, our process picked up correct spfile. The command SHOW PARAMETER reads the parameter values from the memory, but if we try to read directly from spfile (select from v$spparameter) or write to it (alter system set) we will get errors, because diskgroups not mounted yet. It means that our process was read spfile directly from ASM disks. The design about from which disk to read it made based on information from disk header. I use ASMlib, and my asm_diskstring parameter is equal to default value null, ASM instance by default scans the /dev/oracleasm/disks/* on Linux, that is why process found my disks properly. But what if we use HP-UX and our disks is multipathing disks in /dev/rdisk/*, ASM will not scan them by default, we need to use asm_diskstring parameter, how our process will read asm_diskstring parameter before accessing spfile?

Lets start sqlplus, after we will connect as sysasm it will initialize server process, this process will do all job, that is why I will trace this server process

bash> sqlplus /nolog
SQL> conn / as sysasm
Connected.
SQL> !ps -aefH
.
.
oracle   15505  4255  0 13:26 pts/1    00:00:00           sqlplus
oracle   15507 15505  0 13:26 ?        00:00:00             oracle+ASM (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
.
.
SQL>

Our server process number is 15507, now I will use strace to trace it

bash> strace -o userproc.out -p 15507

start the instance with nomount option, OCSSD must be running, or instance will not start

SQL> startup nomount;

Total System Global Area  284565504 bytes
Fixed Size                  1336036 bytes
Variable Size             258063644 bytes
ASM Cache                  25165824 bytes

SQL>

Now we can analyze the contents of the userproc.out, I will not list all contents, just useful parts of it

First row is

read(9, "001\6\3\212\6\376\377\377\377\1\376\377\377\377\376\377\377\377"..., 8208) = 49

it is our “startup nomount” command transmitted to server process from sqlplus by socket

connect(6, {sa_family=AF_FILE, path="/var/tmp/.oracle/sOCSSD_LL_testhost_"...}, 110) = 0
open("/u01/oracle/product/grid11g/auth/css/testdb03/A6857753/84b0b692", O_WRONLY|O_CREAT|O_EXCL, 0644) = 13
write(13, "\nS\336[", 4)                = 4
close(13)

from this lines we can see that our process connects to socket file of OCSSD /var/tmp/.oracle/sOCSSD_LL_testhost_ to communicate with it and authenticates itself. That is why it impossible to start instance without OCSSD, if process could not connect to this socket, it would fail. Then it establishes communication with OHASD (Oracle High Availability Services Deamon) through the socket file /var/tmp/.oracle/sCRSD_UI_SOCKET, after exchange messages with OHASD it will get information about the location of spfile and asm_diskstring parameter

access("/var/tmp/.oracle/sCRSD_UI_SOCKET", F_OK) = 0
connect(14, {sa_family=AF_FILE, path="/var/tmp/.oracle/sCRSD_UI_SOCKET"...}, 110) = 0
open("/u01/oracle/product/grid11g/auth/ohasd/testdb03/A4972914/5140f6df", O_WRONLY|O_CREAT|O_EXCL, 0644) = 15
write(15, "\20\7\3519", 4)              = 4
close(15)
write(14, "4\2\2\1\1\1\1\3\1T\235\376\t"..., 52) = 52
write(14, "8\3\2\1\1\1\1\4\1T\235\376\t"..., 56) = 56
read(14, "8\3\2\1\1\1\1\3\1T\235\376\t"..., 32768) = 56
write(14, "\212\1PC\v\2\2\5\1T\235\376\t"..., 394) = 394
read(14, "\366\nPC\v\2\2\4\1T\235\376\t"..., 32768) = 2806
write(14, "0\2\20\1\1T\235\376\t"..., 48) = 48

write(3, "kggpnpSIHAGetItem 1 = +dgroup1/ASM/asmparameterfile/registry.253.740659373"..., 75) = 7
write(3, "kfspFileNameGet name=+dgroup1/ASSM/asmparameterfile/registry.253.740659373"..., 78) = 78

write(3, "kggpnpSIHAGetItem 2 =  ", 23) = 23
write(3, "kgfspb_shallow_discover dscstr=\"\""..., 33) = 33

As we can see our process got spfile location (+dgroup1/ASM/asmparameterfile/registry.253.740659373) and asm_diskstring=”” which is null by default from OHASD. Now, lets see where OHASD itself gets this information. To identify that I traced the OHASD process, and I found this lines

open("/etc/oracle/olr.loc", O_RDONLY) = 4
read(4, "olrconfig_loc=/u01/oracle/produc"..., 4096) = 108
read(4, "", 4096)                       = 0
close(4)                                = 0
open("/u01/oracle/product/grid11g/cdata/localhost/testhost.olr", O_RDONLY|O_SYNC|O_LARGEFILE) = 4
pread64(4, "\1\202VD\31\4\3OCR\226\361nA"..., 4096, 102400) = 4096
pread64(4, "\1\202\300I#\4\3OCR\226\361nA"..., 4096, 143360) = 4096
pread64(4, "\1\202VD\31\4\3OCR\226\361nA"..., 4096, 102400) = 4096
pread64(4, "\1\202hq\33\4\3OCR\226\361nA"..., 4096, 110592) = 4096
pread64(4, "\1\202\271\311#\4\20\3OCR\226\361nA"..., 4096, 4337664) = 4096
pread64(4, "\1\202\276\262$\4\20\3OCR\226\361nA"..., 4096, 4341760) = 4096
pread64(4, "\1\202VD\31\4\3OCR\226\361nA"..., 4096, 102400) = 4096
pread64(4, "\1\202hq\33\4\3OCR\226\361nA"..., 4096, 110592) = 4096
pread64(4, "\1\202\271\311#\4\20\3OCR\226\361nA"..., 4096, 4337664) = 4096
pread64(4, "\1\202\276\262$\4\20\3OCR\226\361nA"..., 4096, 4341760) = 4096
pread64(4, "\1\202\236\363%\4\2\3OCR\226\361nA"..., 4096, 4345856) = 4096
pread64(4, "\1\202\334\n&\4\2\3OCR\226\361nA"..., 4096, 4349952) = 4096
pread64(4, "\1\202\325\357-\4\2\3OCR\226\361nA"..., 4096, 4378624) = 4096
pread64(4, "\1\202VD\31\4\3OCR\226\361nA"..., 4096, 102400) = 4096
pread64(4, "\1\202hq\33\4\3OCR\226\361nA"..., 4096, 110592) = 4096
pread64(4, "\1\202\271\311#\4\20\3OCR\226\361nA"..., 4096, 4337664) = 4096
pread64(4, "\1\202\276\262$\4\20\3OCR\226\361nA"..., 4096, 4341760) = 4096
pread64(4, "\1\202\325\357-\4\2\3OCR\226\361nA"..., 4096, 4378624) = 4096

As one would expect OHASD reads this information from OLR, the path to it, it gets from /etc/oracle/olr.loc. I want to note that it is true for Oracle Restart mode, in Oracle RAC environment information is stored in GPNP profile and there is GPNPD process that maintains and manages this profile information. In Oracle Restart, as we can see OHASD process executes this role, and because there is no PGNP profile it stores information in OLR file. So, what next? Our process starts to scan all disks to identify ASM headers, after identifying them, it identifies which disks belongs to which diskgroups by information from header. There is many other metadata in the ASM diskheader that it reads, including pointers to spfile and votig disk file, it is kfdhdb.spfile, kfdhdb.spfflg (first block and number of blocks) and kfdhdb.vfstart, kfdhdb.vfend (begin block and end block). It is possible to read disk header using kfed utility from $ORACLE_HOME/bin directory. For example, lets read disk header of the /dev/loop1 which corresponds to ORCL:DISK1, spfile is on this disk.

shell> kfed read /dev/loop1 | more
kfbh.type:                            1 ; 0x002: KFBTYP_DISKHEAD --Indicates that this is ASM disk header
.
kfdhdb.grptyp:                        2 ; 0x026: KFDGTP_NORMAL --Indicates mirroring level, in my case it is NORMAL
kfdhdb.hdrsts:                        3 ; 0x027: KFDHDR_MEMBER --Indicates that disk is the member disk of diskgroup
.
kfdhdb.dskname:                   DISK1 ; 0x028: length=5 --Disk name
kfdhdb.grpname:                 DGROUP1 ; 0x048: length=7 --Disk group name, to which this disk belongs
kfdhdb.fgname:                    DISK1 ; 0x068: length=5 --To which failure group this disk belongs
.
kfdhdb.secsize:                     512 ; 0x0b8: 0x0200 --Disk sector size
kfdhdb.blksize:                    4096 ; 0x0ba: 0x1000 --Disk block size
kfdhdb.ausize:                  1048576 ; 0x0bc: 0x00100000 --Allocation Unit size, by default 1M
.
kfdhdb.vfstart:                       0 ; 0x0ec: 0x00000000 --Begin block address of voting disk file
kfdhdb.vfend:                         0 ; 0x0f0: 0x00000000 --End block address of voting disk file
kfdhdb.spfile:                       59 ; 0x0f4: 0x0000003b --Begin block address of spfile
kfdhdb.spfflg:                        1 ; 0x0f8: 0x00000001 --Number of blocks containing spfile

Now, lets see what will be next steps of our process

open("/opt/oracle/extapi/32/asm/orcl/1/libasm.so", O_RDONLY) = 17
read(17, "\177ELF\1\1\1\3\3\1000\v004"..., 512) = 512
close(17)                               = 0
open("/dev/oracleasm/.query_version", O_RDWR|O_LARGEFILE) = 17
write(17, "MSA\2\1\20", 16) = 16
read(17, "MSA\2\1\20", 16) = 16
close(17)                               = 0
open("/dev/oracleasm/.get_iid", O_RDWR|O_LARGEFILE) = 17
write(17, "MSA\2\2\30", 24) = 24
read(17, "MSA\2\2\30\3", 24) = 24
close(17)                               = 0
open("/dev/oracleasm/.check_iid", O_RDWR|O_LARGEFILE) = 17
write(17, "MSA\2\3\30\3", 24) = 24
read(17, "MSA\2\3\30\3", 24) = 24
close(17)                               = 0
open("/dev/oracleasm/iid/0000000000000003", O_RDWR|O_CREAT|O_LARGEFILE, 0770) = 17

open("/dev/oracleasm/disks/DISK1", O_RDONLY|O_LARGEFILE) = 18
read(17, "MSA\2\5 \22T/v\17\200-%\310", 32) = 32
close(18)                               = 0
read(17, "MSA\2\7P@\364\311\20\320\301\225\277"..., 80) = 80
open("/dev/oracleasm/disks/DISK2", O_RDONLY|O_LARGEFILE) = 18
read(17, "MSA\2\5 \22T/v\17\200+%\310", 32) = 32
close(18)                               = 0
read(17, "MSA\2\7P@\364\311\20\320\301\225\277"..., 80) = 80
open("/dev/oracleasm/disks/DISK3", O_RDONLY|O_LARGEFILE) = 18
read(17, "MSA\2\5 \22T/v\17\200!%\310", 32) = 32
close(18)                               = 0
read(17, "MSA\2\7P@\364\311\20\320\301\225\277"..., 80) = 80
open("/dev/oracleasm/disks/DISK4", O_RDONLY|O_LARGEFILE) = 18
read(17, "MSA\2\5 \22T/v\17\200#%\310", 32) = 32
close(18)                               = 0
read(17, "MSA\2\7P@\364\311\20\320\301\225\277"..., 80) = 80
open("/dev/oracleasm/disks/DISK5", O_RDONLY|O_LARGEFILE) = 18
read(17, "MSA\2\5 \22T/v\17\200)%\310", 32) = 32
close(18)                               = 0

read(17, "MSA\2\7P@\364\311\20\320\301\225\277"..., 80) = 80
read(17, "MSA\2\7P@\364\311\20"..., 80) = 80
read(17, "MSA\2\7P@\364\311\20"..., 80) = 80
read(17, "MSA\2\7P@\364\311\20"..., 80) = 80
read(17, "MSA\2\7P@\364\311\20"..., 80) = 80
read(17, "MSA\2\7P@\364\311\20"..., 80) = 80
.
.

open("/u01/oracle/diag/asm/+asm/+ASM/trace/alert_+ASM.log", O_WRONLY|O_CREAT|O_APPEND|O_LARGEFILE, 0660) = 16
write(16, "Tue Jan 18 17:45:13 2011\n", 25) = 25
write(16, "Starting ORACLE instance (normal"..., 33) = 33
write(16, "\n", 1)                      = 1
close(16)
.
.

It opens /opt/oracle/extapi/32/asm/orcl/1/libasm.so library, reads it. Then it opens special files /dev/oracleasm/.query_version, /dev/oracleasm/.get_iid and /dev/oracleasm/.check_iid, this files is interfaces to ASM device manager. First one is used to get managers version, second one is used to get identifier of the ASM disk device manager instance (not ASM instance) and third for verifying of this instance identifier. In our case ASM device managers instance identifier is 0000000000000003. Then, our process opens /dev/oracleasm/iid/0000000000000003, in other words it establishes connection to the ASM device manager instance. Process will use this interface to read and write to ASM disks. Then it checks all disks from DISK1 to DISK5. After it gets all information about disks, i.e. groups, mirroring level, AU size, failure group and so on it starts reading ASM disk through established interface. I think in this step it reads the spfile. After it gets initialization parameters it starts to allocate memory structures and starting necessary background processes.

Now lets see what is stored in OLR regarding ora.asm resource

shell> cd $ORACLE_HOME/cdata/localhost
shell> strings testhost.olr  | grep dgroup1/ASM/asmparameterfile/registry | sed 's/~/\n/g'
DEFAULT_TEMPLATE=PROPERTY(RESOURCE_CLASS=asm) ELEMENT(INSTANCE_NAME= %GEN_USR_ORA_INST_NAME%)
DEGREE=1
DESCRIPTION=Oracle ASM resource
ENABLED=1
GEN_USR_ORA_INST_NAME=+ASM
LOAD=1
LOGGING_LEVEL=1
NAME=ora.asm
NLS_LANG=
NOT_RESTARTING_TEMPLATE=
OFFLINE_CHECK_INTERVAL=0
PROFILE_CHANGE_TEMPLATE=
RESTART_ATTEMPTS=5
SCRIPT_TIMEOUT=60
SPFILE=+dgroup1/ASM/asmparameterfile/registry.253.740659373
START_DEPENDENCIES=hard(ora.cssd) weak(ora.LISTENER.lsnr)
START_TIMEOUT=900
STATE_CHANGE_TEMPLATE=
STOP_DEPENDENCIES=hard(ora.cssd)
STOP_TIMEOUT=600
TYPE=ora.asm.type
TYPE_ACL=owner:oracle:rwx,pgrp:dba:rwx,other::r--
UPTIME_THRESHOLD=1d
USR_ORA_ENV=
USR_ORA_INST_NAME=+ASM
USR_ORA_OPEN_MODE=mount
USR_ORA_OPI=false
USR_ORA_STOP_MODE=immediate
VERSION=11.2.0.1.0

We can found there information about spfile “SPFILE=+dgroup1/ASM/asmparameterfile/registry.253.740659373”, we also can see that ora.cssd is hard dependency resource, because OCSSD is responsible for synchronization between ASM and database instance. But what about asm_diskstring parameter, where is it?

shell> strings testhost.olr  | grep ASM_DISKSTRING | sed 's/~/\n/g'

nothing, it is because my asm_diskstring parameter equals to null, default value, lets change it

SQL> alter system set asm_diskstring='ORCL:DISK1,ORCL:DISK2' scope=spfile;

System altered.

check OLR file again

strings testdb03.olr  | grep ASM_DISKSTRING | sed 's/~/\n/g'
bASM_DISKSTRING
ACL=owner:oracle:rwx,pgrp:dba:rwx,other::r--
ACTION_FAILURE_TEMPLATE=
ACTION_SCRIPT=
AGENT_FILENAME=%CRS_HOME%/bin/oraagent%CRS_EXE_SUFFIX%
ALIAS_NAME=
ASM_DISKSTRING=ORCL:DISK1,ORCL:DISK2
AUTO_START=restore
BASE_TYPE=ora.local_resource.type

Now we can see that information about asm_diskstring is also stored in OLR profile. And next time, at ASM instance startup it will scan only specified disk strings. If we specify disks on which spfile not present, our instance will not start

SQL> alter system set asm_diskstring='ORCL:DISK3,ORCL:DISK4,ORCL:DISK5' scope=spfile;

System altered.

SQL> startup nomount force;
ORA-01078: failure in processing system parameters
ORA-01565: error in identifying file '+DGROUP1/asm/asmparameterfile/registry.253.740659373'
ORA-17503: ksfdopn:2 Failed to open file +DGROUP1/asm/asmparameterfile/registry.253.740659373
SQL>

You can also use the ocrdump utility to extract this entries on any platform in XML format:

shell> ocrdump -local -keyname SYSTEM.OHASD.RESOURCES.ora\!asm.CONFIG -xml -noheader
shell> more OCRDUMPFILE

SYSTEM.OHASD.RESOURCES.ora!asm.CONFIG
<value_type>ORATEXT
ASM_DISKSTRING=ORCL:DISK3,ORCL:D
ISK4,ORCL:DISK5~AUTO_START=restore~BASE_TYPE=ora.local_resource.type~CHECK_INTERVAL=1~CHECK_TIMEOUT=600~DEFAULT_TEMPLATE=PROPERTY(RESOURCE_CLASS=asm) ELEMENT(INSTANCE_NAME= %GEN_USR_ORA_INST_NAME%)~DE
GREE=1~DESCRIPTION=Oracle ASM resource~ENABLED=1~GEN_USR_ORA_INST_NAME=+ASM~LOAD=1~LOGGING_LEVEL=1~NAME=ora.asm~NLS_LANG=~NOT_RESTARTING_TEMPLATE=~OFFLINE_CHECK_INTERVAL=0~PROFILE_CHANGE_TEMPLATE=~RES
TART_ATTEMPTS=5~SCRIPT_TIMEOUT=60~SPFILE=+dgroup1/ASM/asmparameterfile/registry.253.740659373~START_DEPENDENCIES=hard(ora.cssd) weak(ora.LISTENER.lsnr)~START_TIMEOUT=900~STATE_CHANGE_TEMPLATE=~STOP_DE
PENDENCIES=hard(ora.cssd)~STOP_TIMEOUT=600~TYPE=ora.asm.type~TYPE_ACL=owner:oracle:rwx,pgrp:dba:rwx,other::r--~UPTIME_THRESHOLD=1d~USR_ORA_ENV=~USR_ORA_INST_NAME=+ASM~USR_ORA_OPEN_MODE=mount~USR_ORA_O
PI=false~USR_ORA_STOP_MODE=immediate~VERSION=11.2.0.1.0~]]>
<user_permission>PROCR_ALL_ACCESS
<group_permission>PROCR_NONE
<other_permission>PROCR_NONE
<user_name>oracle
<group_name>dba

So, draw conclusions, gentlemen!

---

(c) Aychin Gasimov, 01/2011, Azerbaijan Republic

---