Tuesday, June 21, 2022

Automatic start observer during instance startup/reboot

Observer plays critical role during automatic failover, but sometimes routine maintenance would restart the Linux instance running observer and causes observer in stopped after instance reboot. Here are scripts to start observer automatically during Linux OS startup.

Running observer in background requires Oracle wallet setup, refer to below 2 links for more details:

/home/oracle/start_observer.sh

#!/bin/bash

export PATH

export ORACLE_BASE=/u01/app/oracle
export ORACLE_HOME=/u01/client
export LD_LIBRARY_PATH=/u01/lib
export PATH=/u01/sqlcl/bin:$ORACLE_HOME/bin:$PATH

ObserverRunning=`ps -ef|grep dgmgrl|grep -v grep|wc -l`
# echo $ObserverRunning
if [[ $ObserverRunning -eq 1 ]]
then
         echo "Observer is already running"
else
dgmgrl  <<EOD
start observer in background logfile is '/tmp/observer.log' connect identifier is rds_custom_orcl_a;
EOD
fi

/etc/rc.local

#!/bin/bash
# THIS FILE IS ADDED FOR COMPATIBILITY PURPOSES
#
# It is highly advisable to create own systemd services or udev rules
# to run scripts during boot instead of using this file.
#
# In contrast to previous versions due to parallel execution during boot
# this script will NOT be run after all other services.
#
# Please note that you must run 'chmod +x /etc/rc.d/rc.local' to ensure
# that this script will be executed during boot.

touch /var/lock/subsys/local

su - oracle -c "nohup /home/oracle/start_observer.sh &" 

Make sure change these scripts to be *executable".

chmod +x /home/oracle/start_observer.sh
chmod +x /etc/rc.local 

pyexpat.cpython-36m-x86_64-linux-gnu.so: undefined symbol: XML_SetHashSalt

Symptom:


Command line "aws cli" failed with error /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so: undefined symbol: XML_SetHashSalt.

-bash-4.2$ aws s3 ls
Traceback (most recent call last):
  File "/usr/local/bin/aws", line 27, in <module>
    sys.exit(main())
  File "/usr/local/bin/aws", line 23, in main
    return awscli.clidriver.main()
  File "/usr/local/lib/python3.6/site-packages/awscli/clidriver.py", line 69, in main
    driver = create_clidriver()
  File "/usr/local/lib/python3.6/site-packages/awscli/clidriver.py", line 79, in create_clidriver
    event_hooks=session.get_component('event_emitter'))
  File "/usr/local/lib/python3.6/site-packages/awscli/plugin.py", line 44, in load_plugins
    modules = _import_plugins(plugin_mapping)
  File "/usr/local/lib/python3.6/site-packages/awscli/plugin.py", line 61, in _import_plugins
    module = __import__(path, fromlist=[module])
  File "/usr/local/lib/python3.6/site-packages/awscli/handlers.py", line 43, in <module>
    from awscli.customizations.history import register_history_mode
  File "/usr/local/lib/python3.6/site-packages/awscli/customizations/history/__init__.py", line 28, in <module>
    from awscli.customizations.history.show import ShowCommand
  File "/usr/local/lib/python3.6/site-packages/awscli/customizations/history/show.py", line 16, in <module>
    import xml.parsers.expat
  File "/usr/lib64/python3.6/xml/parsers/expat.py", line 4, in <module>
    from pyexpat import *
ImportError: /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so: undefined symbol: XML_SetHashSalt

Troubleshooting:


Check the dependencies for pyexpat.cpython-36m-x86_64-linux-gnu.so:

-bash-4.2$ ldd /usr/lib64/python3.6/lib-dynload/pyexpat.cpython-36m-x86_64-linux-gnu.so
        linux-vdso.so.1 =>  (0x00007fffa67fc000)
        libexpat.so.1 => /lib64/libexpat.so.1 (0x00007f16058bb000)
        libpython3.6m.so.1.0 => /lib64/libpython3.6m.so.1.0 (0x00007f1605394000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f1605178000)
        libc.so.6 => /lib64/libc.so.6 (0x00007f1604daa000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00007f1604ba6000)
        libutil.so.1 => /lib64/libutil.so.1 (0x00007f16049a3000)
        libm.so.6 => /lib64/libm.so.6 (0x00007f16046a1000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f1605cf4000)

Check the suspicious file, found multiple of it in the LD_LIBRARY_PATH:

$ find / -name libexpat.so.1 2>/dev/null
/usr/lib64/libexpat.so.1
/rdsdbbin/oracle.12.1.custom.r1.EE.1/lib/libexpat.so.1

Solution:


Add /usr/lib64/ to the beginning of LD_LIBRARY_PATH:

export LD_LIBRARY_PATH=/usr/lib64/:$LD_LIBRARY_PATH