Saturday, October 2, 2010

[bug?]After patchset 11.2.0.2 installation, ASM/Listener still run under old home (using Oracle Restart)

[Update - begin]
After manually correct following files to point correct grid home, problem resolved.
/etc/init.d/ohasd
/etc/init/init.ohasd


[oracle@rh5 ~]$ srvctl config listener
Name: LISTENER
Home: /u11gr2/app/oracle/product/11.2.0/grid_11202
End points: TCP:1521

The root cause is /u11gr2/app/oracle/product/11.2.0/grid_11202/rootupgrade.sh did not update above files for out-of-place patchset installation.

[Update - end]

[11gr2@rh5 ~]$ cd /u11gr2/app/oracle/diag/asm/+asm/+ASM/trace/
[11gr2@rh5 trace]$ less alert_+ASM.log

NOTE: Instance updated compatible.asm to 11.2.0.0.0 for grp 1
SUCCESS: diskgroup DATA was mounted
SUCCESS: ALTER DISKGROUP ALL MOUNT /* asm agent */
SQL> ALTER DISKGROUP ALL ENABLE VOLUME ALL /* asm agent */
SUCCESS: ALTER DISKGROUP ALL ENABLE VOLUME ALL /* asm agent */
NOTE: diskgroup resource ora.DATA.dg is online
Sat Oct 02 07:56:10 2010
ERROR: Refusing connection from client orcl:orcl because client
software version (11.2.0.2.0) is greater than current version 11.2.0.1.0
Sat Oct 02 07:56:11 2010
ERROR: Refusing connection from client orcl:orcl because client
software version (11.2.0.2.0) is greater than current version 11.2.0.1.0

[11gr2@rh5 trace]$ ls -l /u11gr2/app/oracle/product/11.2.0/grid_11202/dbs/
total 16
-rw-rw---- 1 11gr2 oinstall 1743 Oct 1 21:11 ab_+ASM.dat
-rw-rw---- 1 11gr2 oinstall 1544 Oct 1 21:11 hc_+ASM.dat
-rw-r--r-- 1 11gr2 oinstall 2851 May 15 2009 init.ora
-rw-r----- 1 11gr2 oinstall 1536 Sep 28 14:03 orapw+ASM
[11gr2@rh5 trace]$ ls -l /u11gr2/app/oracle/product/11.2.0/grid/dbs/
total 20
-rw-rw---- 1 11gr2 oinstall 939 Oct 2 07:55 ab_+ASM.dat
-rw-rw---- 1 11gr2 oinstall 1544 Oct 2 07:55 hc_+ASM.dat
-rw-r--r-- 1 11gr2 oinstall 2851 May 15 2009 init.ora
-rw-r----- 1 11gr2 oinstall 1536 Sep 27 22:30 orapw+ASM
drwx------ 2 11gr2 oinstall 4096 Sep 27 22:29 peshm_+ASM_1
[11gr2@rh5 trace]$


[11gr2@rh5 trace]$ ps -ef|grep tns
11gr2 3248 1 0 07:54 ? 00:00:00 /u11gr2/app/oracle/product/11.2.0/grid/bin/tnslsnr LISTENER -inherit
11gr2 4854 4491 0 08:15 pts/0 00:00:00 grep tns
[11gr2@rh5 trace]$
[11gr2@rh5 trace]$ crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora.DATA.dg ora....up.type ONLINE ONLINE rh5
ora....ER.lsnr ora....er.type ONLINE ONLINE rh5
ora.asm ora.asm.type ONLINE ONLINE rh5
ora.cssd ora.cssd.type ONLINE ONLINE rh5
ora.diskmon ora....on.type ONLINE ONLINE rh5
ora.evmd ora.evm.type ONLINE ONLINE rh5
ora.ons ora.ons.type OFFLINE OFFLINE
ora.orcl.db ora....se.type ONLINE OFFLINE


[11gr2@rh5 trace]$ srvctl config asm
ASM home: /u11gr2/app/oracle/product/11.2.0/grid_11202
ASM listener: LISTENER
Spfile: +DATA/asm/asmparameterfile/registry.253.730852187
ASM diskgroup discovery string: /dev/hda*
[11gr2@rh5 trace]$ srvctl config listener
Name: LISTENER
Home: /u11gr2/app/oracle/product/11.2.0/grid
End points: TCP:1521
[11gr2@rh5 trace]$


[11gr2@rh5 trace]$ srvctl modify listener -o /u11gr2/app/oracle/product/11.2.0/grid_11202
[11gr2@rh5 trace]$ srvctl config listener
Name: LISTENER
Home: /u11gr2/app/oracle/product/11.2.0/grid
End points: TCP:1521
[11gr2@rh5 trace]$ srvctl modify listener -l LISTENER -o /u11gr2/app/oracle/product/11.2.0/grid_11202
[11gr2@rh5 trace]$ srvctl config listener
Name: LISTENER
Home: /u11gr2/app/oracle/product/11.2.0/grid
End points: TCP:1521
[11gr2@rh5 trace]$



[11gr2@rh5 trace]$ srvctl modify listener -o /u11gr2/app/oracle/product/11.2.0/grid_11202
[11gr2@rh5 trace]$ srvctl config listener
Name: LISTENER
Home: /u11gr2/app/oracle/product/11.2.0/grid
End points: TCP:1521
[11gr2@rh5 trace]$ srvctl modify listener -l LISTENER -o /u11gr2/app/oracle/product/11.2.0/grid_11202
[11gr2@rh5 trace]$ srvctl config listener
Name: LISTENER
Home: /u11gr2/app/oracle/product/11.2.0/grid
End points: TCP:1521
[11gr2@rh5 trace]$ srvctl remove listener
PRCR-1025 : Resource ora.LISTENER.lsnr is still running

[11gr2@rh5 trace]$ srvctl stop listener
[11gr2@rh5 trace]$ srvctl remove listener
[11gr2@rh5 trace]$ srvctl add listener -l LISTENER -o /u11gr2/app/oracle/product/11.2.0/grid_11202 -p 1521
[11gr2@rh5 trace]$ srvctl config listener
Name: LISTENER
Home: /u11gr2/app/oracle/product/11.2.0/grid
End points: TCP:1521
[11gr2@rh5 trace]$ srvctl remove listener
[11gr2@rh5 trace]$ srvctl add listener -l LISTENER -o /u11gr2/app/oracle/product/11.2.0/grid_11202 -p 1521
[11gr2@rh5 trace]$ cat /u11gr2/app/oracle/product/11.2.0/grid_11202/network/admin/
listener.ora listener.ora.bak.rh5 samples/ shrept.lst sqlnet.ora
[11gr2@rh5 trace]$ cat /u11gr2/app/oracle/product/11.2.0/grid_11202/network/admin/listener.ora
# listener.ora Network Configuration File: /u11gr2/app/oracle/product/11.2.0/grid/network/admin/listener.ora
# Generated by Oracle configuration tools.

LISTENER =
(DESCRIPTION_LIST =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC1521))
(ADDRESS = (PROTOCOL = TCP)(HOST = rh5.lab.dbaglobe.com)(PORT = 1521))
)
)

ADR_BASE_LISTENER = /u11gr2/app/oracle

ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER=ON # line added by Agent
[11gr2@rh5 trace]$ srvctl add listener -l LISTENER -o "/u11gr2/app/oracle/product/11.2.0/grid_11202" -p 1521
[11gr2@rh5 trace]$ srvctl config listener
Name: LISTENER
Home: /u11gr2/app/oracle/product/11.2.0/grid
End points: TCP:1521



[11gr2@rh5 trace]$ /u11gr2/app/oracle/product/11.2.0/grid_11202/bin/srvctl config listener
Name: LISTENER
Home: /u11gr2/app/oracle/product/11.2.0/grid
End points: TCP:1521
[11gr2@rh5 trace]$


[11gr2@rh5 trace]$ strace srvctl modify listener -l LISTENER -o /u11gr2/app/oracle/product/11.2.0/grid_11202
execve("/u11gr2/app/oracle/product/11.2.0/grid/bin/srvctl", ["srvctl", "modify", "listener", "-l", "LISTENER", "-o", "/u11gr2/app/oracle/product/11.2."], [/* 28 vars */]) = 0

4 comments:

  1. Just noticed below file has not been updated during the patch process:

    [root@rh5 ~]# grep grid /etc/init/init.ohasd
    ORA_CRS_HOME=/u11gr2/app/oracle/product/11.2.0/grid
    case `/u11gr2/app/oracle/product/11.2.0/grid/bin/crsctl check has | $GREP 4638` in
    /u11gr2/app/oracle/product/11.2.0/grid/bin/ohasd restart &
    $SU 11gr2 -c "/u11gr2/app/oracle/product/11.2.0/grid/bin/ohasd restart &"

    The correct value should be:

    [root@rh5 ~]# grep grid /etc/init.d/init.ohasd
    ORA_CRS_HOME=/u11gr2/app/oracle/product/11.2.0/grid_11202
    case `/u11gr2/app/oracle/product/11.2.0/grid_11202/bin/crsctl check has | $GREP 4638` in
    /u11gr2/app/oracle/product/11.2.0/grid_11202/bin/ohasd restart &
    $SU 11gr2 -c "/u11gr2/app/oracle/product/11.2.0/grid_11202/bin/ohasd restart &"

    ReplyDelete
  2. Another file discoverred not upgraded under out-of-place patch installation
    ==================================

    [root@rh5 ~]# grep grid /etc/init.d/ohasd
    ORA_CRS_HOME=/u11gr2/app/oracle/product/11.2.0/grid
    ALERTLOG=/u11gr2/app/oracle/product/11.2.0/grid/log/rh5/alertrh5.log
    /u11gr2/app/oracle/product/11.2.0/grid/bin/crsctl start has &
    /u11gr2/app/oracle/product/11.2.0/grid/bin/crsctl stop has




    [root@rh5 ~]# vi /etc/init.d/ohasd
    [root@rh5 ~]# grep grid /etc/init.d/ohasd
    ORA_CRS_HOME=/u11gr2/app/oracle/product/11.2.0/grid_11202
    ALERTLOG=/u11gr2/app/oracle/product/11.2.0/grid_11202/log/rh5/alertrh5.log
    /u11gr2/app/oracle/product/11.2.0/grid_11202/bin/crsctl start has &
    /u11gr2/app/oracle/product/11.2.0/grid_11202/bin/crsctl stop has

    ReplyDelete
  3. Hello! I'm seeing the same thing as you. It has been driving me nuts. I have an SR open with Oracle about this. Do you know why these files fail to be updated?

    ReplyDelete
  4. Found it:

    Standalone GI: init.ohasd/ohasd not updated after 11201 to 11202 upgrade (Doc ID 1233183.1)

    ReplyDelete