Monday, September 7, 2009

How to: ORA-16817: unsynchronized Fast-Start Failover configuration


DG 2009-09-04-13:29:19 0 2 0 RSM0: HEALTH CHECK WARNING: ORA-16817: unsynchronized Fast-Start Failover configuration
DG 2009-09-04-13:29:19 0 2 696687836 Operation CTL_GET_STATUS continuing with warning, status = ORA-16817
DG 2009-09-04-13:29:19 0 2 696687836 Operation CTL_GET_STATUS continuing with warning, status = ORA-16817



There is no much information could be found on metalink for this error code, but from the error message, valuable information is available from Oracle Documentation:

ORA-16817: unsynchronized Fast-Start Failover configuration
Cause: The Fast-Start Failover target standby database was not synchronized with the primary database. As a result, a Fast-Start Failover could not happen automatically in case of a primary database failure.
Action: Ensure that the Fast-Start Failover target standby database is running and that the primary database can ship redo logs to it. When the standby database has received all of the redo logs from the primary database, the primary and standby databases will then be synchronized. The Data Guard configuration may then failover automatically to the standby database in the event of loss of the primary database.

This can be confirmed by checking "v$archived_log".




SQL> select SEQUENCE# ,APPLIED,STATUS from v$archived_log
2 where first_time > sysdate -4/24;


SEQUENCE# APP S
---------- --- -
16 YES A
17 NO A
17 YES A
18 NO A
18 YES A
19 YES A
20 YES A
29 NO A
24 NO A
25 NO A
26 NO A

SEQUENCE# APP S
---------- --- -
27 NO A
28 NO A
30 NO A
39 NO A
40 NO A
41 NO A
42 NO A
43 NO A
44 NO A
45 NO A
46 NO A

SEQUENCE# APP S
---------- --- -
47 NO A
48 NO A

Thursday, September 3, 2009

Possible issues during Oracle 8i installation on Solaris 9

Issue 1: JRE not found when invoking runInstaller


bash-2.05$ /oradata/stage/Disk1/runInstaller
bash-2.05$ The Java RunTime Environment was not found at bin/jre. Hence, the Oracle Universal Installer cannot be run.
Please visit http://www.javasoft.com and install JRE version 1.1.7 or higher and try again.
: No such file or directory


Solution: Use the runInstaller under Disk1/install/solaris folder



bash-2.05$ /oradata/stage/Disk1/install/solaris/runInstaller
Initializing Java Virtual Machine from /tmp/OraInstall/jre/bin/jre. Please wait...




Issue 2: Error in invoking target ntcontab.o of makefile /oradata/product/8.1.7/network/lib/ins_net_client_mk




/usr/ccs/bin/make -f ins_net_client.mk ntcontab.o ORACLE_HOME=/oradata/product/8.1.7
(if [ "assemble" = "compile" ] ; then \
/oradata/product/8.1.7/bin/gennttab > ntcontab.c ;\
cc -c ntcontab.c ;\
rm -f /oradata/product/8.1.7/lib/ntcontab.o ;\
mv ntcontab.o /oradata/product/8.1.7/lib ;\
/usr/ccs/bin/ar rv /oradata/product/8.1.7/lib/libn8.a /oradata/product/8.1.7/lib/ntcontab.o ; fi)
(if [ "assemble" = "assemble" ] ; then \
/oradata/product/8.1.7/bin/gennttab > ntcontab.s ;\
/usr/ccs/bin/as -P -o ntcontab.o ntcontab.s ;\
rm -f /oradata/product/8.1.7/lib/ntcontab.o ;\
mv ntcontab.o /oradata/product/8.1.7/lib ;\
/usr/ccs/bin/ar rv /oradata/product/8.1.7/lib/libn8.a /oradata/product/8.1.7/lib/ntcontab.o ; fi)
Error in finding run directory. Using default /opt/SUNWspro/binerror: Unable to find current directory
make: Fatal error: Command failed for target `ntcontab.o'
*** Error code 1


Solution: change the underline folder for mount point ownership and permission


bash-2.05# umount /oradata
bash-2.05# ls -ld /oradata
drwxr-x--- 2 root other 512 Aug 24 17:52 /oradata
bash-2.05# chown oracle:oinstall /oradata
bash-2.05# chmod 755 /oradata
bash-2.05# mount /oradata


Btw, following banner shows installation completed succesfully.


SQL> select * from v$version;

BANNER
----------------------------------------------------------------
Oracle8i Enterprise Edition Release 8.1.7.0.0 - Production
PL/SQL Release 8.1.7.0.0 - Production
CORE 8.1.7.0.0 Production
TNS for Solaris: Version 8.1.7.0.0 - Production
NLSRTL Version 3.4.1.0.0 - Production

Wednesday, September 2, 2009

Can not install 10gR2 Patchset 3 because of %ORACLE_HOME/bin/msvcr71.dll

Today I have installed Oracle 10gR2 patchset 3 (10.2.0.4) to 10gR2 Client, the environment is Windows 2003 R2 (SP2). Following error message occurred during copying file stage.

Error message:

Error in writing to file d:\oracle\product\10.2.0\client_1\bin\msvcr71.dll (The process can not access the file because it is being used by another process).

Root Cause: System service "Distributed Transaction Coordinator"

Solution: Stop this service before the patchset installation

Poosible affected Oracle Version: All Oracle versions on Windows platform with "Distributed Transaction Coordinator" started.

Tuesday, September 1, 2009

"project.max-shm-memory" is NOT equal to "shmsys:shminfo_shmmax"

In Oracle documentation 10g/11g Oracle Installation Guide for Solaris, these examples are a bit misleading, such that some readers think "project.max-shm-memory" in resource control facility is just a literal replacement for "shmsys:shminfo_shmmax" in /etc/system.

Actually, it does not work that way.

shmsys:shminfo_shmmax
Description :
Maximum size of system V shared memory segment that can be created. This parameter is an upper limit that is checked before the system sees if it actually has the physical resources to create the requested memory segment.
Attempts to create a shared memory section whose size is zero or whose size is larger than the specified value will fail with an EINVAL error.
Reference:
http://docs.sun.com/app/docs/doc/817-1759/6mhfh76gs?l=en&a=view&q=shmsys%3Ashminfo_s

project.max-shm-memory
Description :
Total amount of System V shared memory allowed for this project.
Reference:
http://docs.sun.com/app/docs/doc/819-2450/rmctrls-3?l=en&a=view

Conclusion:
"shmmax" specifies the limit for single shared memory segment, and "max-shm-memory" specifies the limit for the whole project.


Referenced Oracle Metalink document id 859898.1: The ORA-27102 error is generated on Solaris 10 having apparently correct settings of kernel parameters.

Monday, August 31, 2009

ORA-00704 Bootstrap process failure during the DBCA

After 11g1 Patchset 1 installation, i decided to to created a new database using DBCA with "General Purpose" template, error ORA-00740 raised around 40% of completion time.



Errors in file /u02/app/oracle/diag/rdbms/orcl11/ORCL11/trace/ORCL11_ora_25029.trc:
ORA-00704: bootstrap process failure
ORA-39700: database must be opened with UPGRADE option
Error 704 happened during db open, shutting down database
USER (ospid: 25029): terminating the instance due to error 704
Instance terminated by USER, pid = 25029
ORA-1092 signalled during: alter database "ORCL11" open resetlogs...
ORA-1092 : opiodr aborting process unknown ospid (25029_2201808)
Mon Aug 31 00:14:53 2009
ORA-1092 : opitsk aborting process



This was caused by incorrect permission during the installation. (Oracle owner should have "write" permission on these template files. (Similar issue occurs for 10.2.0.3, refer to notes 471765.1)


[oracle11g@vmxdb01 templates]$ cd $ORACLE_HOME/assistants/dbca/templates
[oracle11g@vmxdb01 templates]$ ls -lh
total 204M
-rw-r--r-- 1 oracle11g oinstall 5.2K Sep 10 2008 Data_Warehouse.dbc
-r-xr-xr-x 1 oracle11g oinstall 21M Aug 4 2007 example01.dfb
-r-xr-xr-x 1 oracle11g oinstall 1.1M Aug 4 2007 example.dmp
-rw-r--r-- 1 oracle11g oinstall 5.1K Sep 10 2008 General_Purpose.dbc
-rw-r--r-- 1 oracle11g oinstall 12K Jun 21 2007 New_Database.dbt
-r-xr-xr-x 1 oracle11g oinstall 9.3M Aug 4 2007 Seed_Database.ctl
-r-xr-xr-x 1 oracle11g oinstall 173M Aug 4 2007 Seed_Database.dfb


These write-protected files have different size comparing to these coming with 11.1.0.7 patchset.


[oracle11g@vmxdb01 templates]$ ls -lh /u02/stage/Disk1/stage/Patches/\
> oracle.rdbms.install.seeddb/11.1.0.7.0/1/DataFiles/Expanded/filegroup1/
total 186M
-rwxr-xr-x 1 oracle11g oinstall 9.3M Sep 12 2008 Seed_Database.ctl
-rwxr-xr-x 1 oracle11g oinstall 176M Sep 12 2008 Seed_Database.dfb

[oracle11g@vmxdb01 templates]$ ls -lh /u02/stage/Disk1/stage/Patches/\
> oracle.rdbms.install.seeddb.sample_schema/11.1.0.7.0/1/DataFiles/Expanded/filegroup1/
total 22M
-rwxr-xr-x 1 oracle11g oinstall 21M Sep 12 2008 example01.dfb
-rwxr-xr-x 1 oracle11g oinstall 1.2M Sep 12 2008 example.dmp


Solutions: Copy these files from patchset and re-run DBCA.



[oracle11g@vmxdb01 templates]$ chmod u+w *

[oracle11g@vmxdb01 templates]$ cp /u02/stage/Disk1/stage/Patches/\
>oracle.rdbms.install.seeddb/11.1.0.7.0/1/DataFiles/Expanded/filegroup1/* .
[oracle11g@vmxdb01 templates]$ cp /u02/stage/Disk1/stage/Patches/\
>oracle.rdbms.install.seeddb.sample_schema/11.1.0.7.0/1/DataFiles/Expanded/filegroup1/* .


[oracle11g@vmxdb01 templates]$ ls -lh
total 207M
-rw-r--r-- 1 oracle11g oinstall 5.2K Sep 10 2008 Data_Warehouse.dbc
-rwxr-xr-x 1 oracle11g oinstall 21M Aug 31 00:23 example01.dfb
-rwxr-xr-x 1 oracle11g oinstall 1.2M Aug 31 00:23 example.dmp
-rw-r--r-- 1 oracle11g oinstall 5.1K Sep 10 2008 General_Purpose.dbc
-rw-r--r-- 1 oracle11g oinstall 12K Jun 21 2007 New_Database.dbt
-rwxr-xr-x 1 oracle11g oinstall 9.3M Aug 31 00:22 Seed_Database.ctl
-rwxr-xr-x 1 oracle11g oinstall 176M Aug 31 00:22 Seed_Database.dfb

Sunday, August 30, 2009

Troubleshooting Oracle Performance - Shared Cursors

Both parent cursor and child curor are stored in the library cache during the parse phase. Key parent cursor information is the SQL text itself. And child cursor is assosicated with parent cursor and contain execution plan and execution envrionment.

Below lab shows how to identity parent and child cursor and find out why they are not sharing.

Flush shared pool and exeute 4 querys, first 3 queries are different in lower/upper cases, and extra blank space. The forth sql statement is identical to first statement.



SQL> conn donghua/donghua
Connected.

SQL> alter system flush shared_pool;

System altered.

SQL> select * from t;

no rows selected

SQL> SELECT * FROM T;

no rows selected

SQL> select * from t;

no rows selected

SQL> select * from t;

no rows selected




Now check how many parent cursor has been created.



SQL> col sql_text for a30
SQL> select sql_id, sql_text, executions, version_count from v$sqlarea
2 where lower(sql_text) like 'select * from%t%';

SQL_ID SQL_TEXT EXECUTIONS VERSION_COUNT
------------- ------------------------------ ---------- -------------
89km4qj1thh13 select * from t 2 1
6k1fc899x4ud2 SELECT * FROM T 1 1
3k2ncgn581pdv select * from t 1 1


We also need to understand that not every parent cursor has a unique sql plan. From below, they share the same sql execution plan.



SQL> select sql_id, sql_text, plan_hash_value from v$sqlarea
2 where lower(sql_text) like 'select * from%t%';

SQL_ID SQL_TEXT PLAN_HASH_VALUE
------------- ------------------------------ ---------------
89km4qj1thh13 select * from t 1601196873
6k1fc899x4ud2 SELECT * FROM T 1601196873
3k2ncgn581pdv select * from t 1601196873


Execution envrionment changing will create new child child cursor, for example, new optimizer_mode value.



SQL> alter session set optimizer_mode='FIRST_ROWS_1';

Session altered.

SQL> select * from t;

no rows selected

SQL> select sql_id, sql_text, executions, version_count from v$sqlarea
2 where lower(sql_text) like 'select * from%t%';

SQL_ID SQL_TEXT EXECUTIONS VERSION_COUNT
------------- ------------------------------ ---------- -------------
89km4qj1thh13 select * from t 3 2
6k1fc899x4ud2 SELECT * FROM T 1 1
3k2ncgn581pdv select * from t 1 1


Using synonym will not create new child cursor, as long as they refer to the same based object.



SQL> conn donghua1/donghua1@orcl
Connected.

SQL> create synonym t for donghua.t;

Synonym created.

SQL> select * from t;

no rows selected

SQL> select sql_id, sql_text, executions, version_count from v$sqlarea
2 where lower(sql_text) like 'select * from%t%';

SQL_ID SQL_TEXT EXECUTIONS VERSION_COUNT
------------- ------------------------------ ---------- -------------
89km4qj1thh13 select * from t 4 2
6k1fc899x4ud2 SELECT * FROM T 1 1
3k2ncgn581pdv select * from t 1 1


Enable sql trace will cause sql statement parse again. And new child cursor will created. (Sometimes this is the reason we can not reproceduce some performance issue related to binding variable peeking by turn on sql trace.)



SQL> conn donghua/donghua@orcl
Connected.
SQL> alter session set sql_trace=true;

Session altered.

SQL> select * from t;

no rows selected

SQL> select sql_id, sql_text, executions, version_count from v$sqlarea
2 where lower(sql_text) like 'select * from%t%';

SQL_ID SQL_TEXT EXECUTIONS VERSION_COUNT
------------- ------------------------------ ---------- -------------
89km4qj1thh13 select * from t 5 3
6k1fc899x4ud2 SELECT * FROM T 1 1
3k2ncgn581pdv select * from t 1 1


To find out why child cursor is not shared by querying v$sql_shared_cursor.



SQL> select sql_text, child_number, optimizer_mode, plan_hash_value
2 from v$sql where sql_id='89km4qj1thh13';

SQL_TEXT CHILD_NUMBER OPTIMIZER_ PLAN_HASH_VALUE
------------------------------ ------------ ---------- ---------------
select * from t 0 ALL_ROWS 1601196873
select * from t 1 FIRST_ROWS 1601196873
select * from t 2 ALL_ROWS 1601196873

QL> select child_number, child_address, stats_row_mismatch, optimizer_mode_mismatch
2 from v$sql_shared_cursor where sql_id='89km4qj1thh13';

HILD_NUMBER CHILD_AD S O
----------- -------- - -
0 2AD9B784 N N
1 2AC7B1B4 N Y
2 2ACA0C9C Y N



Possible reasons for a child cursor incompatible with anonther one listed below, (63 reasons in 11gR2).


SQL> select * from v$version;

BANNER
--------------------------------------------------------------------
Oracle Database 11g Enterprise Edition Release 11.2.0.0.1 - Beta
PL/SQL Release 11.2.0.0.1 - Beta
CORE 11.2.0.0.1 Beta
TNS for 32-bit Windows: Version 11.2.0.0.1 - Beta
NLSRTL Version 11.2.0.0.1 - Beta

SQL> desc v$sql_shared_cursor
Name Null? Type
----------------------------------------- -------- ----------------
SQL_ID VARCHAR2(13)
ADDRESS RAW(4)
CHILD_ADDRESS RAW(4)
CHILD_NUMBER NUMBER
UNBOUND_CURSOR VARCHAR2(1)
SQL_TYPE_MISMATCH VARCHAR2(1)
OPTIMIZER_MISMATCH VARCHAR2(1)
OUTLINE_MISMATCH VARCHAR2(1)
STATS_ROW_MISMATCH VARCHAR2(1)
LITERAL_MISMATCH VARCHAR2(1)
FORCE_HARD_PARSE VARCHAR2(1)
EXPLAIN_PLAN_CURSOR VARCHAR2(1)
BUFFERED_DML_MISMATCH VARCHAR2(1)
PDML_ENV_MISMATCH VARCHAR2(1)
INST_DRTLD_MISMATCH VARCHAR2(1)
SLAVE_QC_MISMATCH VARCHAR2(1)
TYPECHECK_MISMATCH VARCHAR2(1)
AUTH_CHECK_MISMATCH VARCHAR2(1)
BIND_MISMATCH VARCHAR2(1)
DESCRIBE_MISMATCH VARCHAR2(1)
LANGUAGE_MISMATCH VARCHAR2(1)
TRANSLATION_MISMATCH VARCHAR2(1)
BIND_EQUIV_FAILURE VARCHAR2(1)
INSUFF_PRIVS VARCHAR2(1)
INSUFF_PRIVS_REM VARCHAR2(1)
REMOTE_TRANS_MISMATCH VARCHAR2(1)
LOGMINER_SESSION_MISMATCH VARCHAR2(1)
INCOMP_LTRL_MISMATCH VARCHAR2(1)
OVERLAP_TIME_MISMATCH VARCHAR2(1)
EDITION_MISMATCH VARCHAR2(1)
MV_QUERY_GEN_MISMATCH VARCHAR2(1)
USER_BIND_PEEK_MISMATCH VARCHAR2(1)
TYPCHK_DEP_MISMATCH VARCHAR2(1)
NO_TRIGGER_MISMATCH VARCHAR2(1)
FLASHBACK_CURSOR VARCHAR2(1)
ANYDATA_TRANSFORMATION VARCHAR2(1)
INCOMPLETE_CURSOR VARCHAR2(1)
TOP_LEVEL_RPI_CURSOR VARCHAR2(1)
DIFFERENT_LONG_LENGTH VARCHAR2(1)
LOGICAL_STANDBY_APPLY VARCHAR2(1)
DIFF_CALL_DURN VARCHAR2(1)
BIND_UACS_DIFF VARCHAR2(1)
PLSQL_CMP_SWITCHS_DIFF VARCHAR2(1)
CURSOR_PARTS_MISMATCH VARCHAR2(1)
STB_OBJECT_MISMATCH VARCHAR2(1)
CROSSEDITION_TRIGGER_MISMATCH VARCHAR2(1)
PQ_SLAVE_MISMATCH VARCHAR2(1)
TOP_LEVEL_DDL_MISMATCH VARCHAR2(1)
MULTI_PX_MISMATCH VARCHAR2(1)
BIND_PEEKED_PQ_MISMATCH VARCHAR2(1)
MV_REWRITE_MISMATCH VARCHAR2(1)
ROLL_INVALID_MISMATCH VARCHAR2(1)
OPTIMIZER_MODE_MISMATCH VARCHAR2(1)
PX_MISMATCH VARCHAR2(1)
MV_STALEOBJ_MISMATCH VARCHAR2(1)
FLASHBACK_TABLE_MISMATCH VARCHAR2(1)
LITREP_COMP_MISMATCH VARCHAR2(1)
PLSQL_DEBUG VARCHAR2(1)
LOAD_OPTIMIZER_STATS VARCHAR2(1)
ACL_MISMATCH VARCHAR2(1)
FLASHBACK_ARCHIVE_MISMATCH VARCHAR2(1)
LOCK_USER_SCHEMA_FAILED VARCHAR2(1)
REMOTE_MAPPING_MISMATCH VARCHAR2(1)
LOAD_RUNTIME_HEAP_FAILED VARCHAR2(1)
HASH_MATCH_FAILED VARCHAR2(1)
PURGED_CURSOR VARCHAR2(1)
BIND_LENGTH_UPGRADEABLE VARCHAR2(1)

Check MySQL storage engine option


mysql> show storage engines;
+------------+---------+---------+--------------+------+------------+
| Engine | Support | Comment | Transactions | XA | Savepoints |
+------------+---------+---------+--------------+------+------------+
| InnoDB | YES | | YES | YES | YES |
| MRG_MYISAM | YES | | NO | NO | NO |
| BLACKHOLE | YES | | NO | NO | NO |
| CSV | YES | | NO | NO | NO |
| MEMORY | YES | | NO | NO | NO |
| FEDERATED | NO | | NULL | NULL | NULL |
| ARCHIVE | YES | | NO | NO | NO |
| MyISAM | DEFAULT | | NO | NO | NO |
+------------+---------+---------+--------------+------+------------+
8 rows in set (0.00 sec)