Monday, June 13, 2016

Shell script to Automate Alert log errors on daily basis (Mailing ORA- errors to DBA )

#!/bin/ksh
ORACLE_HOME=/p01/prod/db/tech_st/11.2.0.4
export ORACLE_HOME
ORACLE_SID=PROD
export ORACLE_SID
echo $ORACLE_HOME
echo $ORACLE_SID
x=$(date +%Y_%m_%d_%H%M%S)
/p01/prod/db/tech_st/11.2.0.4/bin/sqlplus -s / <<EOF
sho parameter db_name
exit;
EOF
if [[ $? -eq 0 ]]; then
#mailx -s "PROD instance is  up !!! "  xyz@gmail.com</tmp
echo "PROD instance is  up !!! "
else
mailx -s "PROD instance is not up !! " xyz@gmail.com</tmp
fi

cd /p01/prod/db/tech_st/11.2.0.4/admin/PROD_scdoraprd/diag/rdbms/prod/PROD/trace/
rm alrt_tmp.log
touch alrt_tmp.log
grep -i ORA- alert_PROD.log  >> alrt_tmp.log

mv alert_PROD.log   alert_PROD.log$x

if [[ $? -eq 0 ]]; then
mailx -s "Alert log reported errors in PROD !!! "  xyz@gmail.com<alrt_tmp.log
fi

Shell script to Automate AWR report (to mail Oracle awr performance reports daily for peak periods )

Cat daily_awr_gen.sh

#!/bin/ksh
ORACLE_HOME=/app/11.2.0/prdobi
export ORACLE_HOME
ORACLE_SID=prodobi
export ORACLE_SID

/app/11.2.0/prdobi/bin/sqlplus -s / as sysdba << EOF

create or replace function snap_values (p_snap_range  varchar2)
return number
as
l_min_snap_id number;
l_max_snap_id number;
begin
SELECT min(snap_id),max(snap_id)  into l_min_snap_id,l_max_snap_id
FROM dba_hist_snapshot
WHERE trunc(begin_interval_time )= trunc(SYSDATE )
and trunc(end_interval_time) = trunc(SYSDATE )
and (to_char(begin_interval_time,'hh24') >= 10 and to_char(begin_interval_time,'hh24') <=16)
order by begin_interval_time;
if p_snap_range='MIN'  then
return l_min_snap_id;
elsif p_snap_range='MAX' then
return l_max_snap_id;
else
return null;
end if;
exception when others then
return null;
end;


EOF
echo $dbid
echo $db_name
echo $inst_num
echo $inst_name
echo $num_days

#repnam='awrrpt_'$inst_num'_'$inst_name'.html'

rm -rf awr_report_*.*

/app/11.2.0/prdobi/bin/sqlplus -s / as sysdba << EOF
    define report_type=html
    define num_days=1
    @/home/obiee/snu/a.sql
    @?/rdbms/admin/awrrpt
EOF

if [[ $? -eq 0 ]]; then
#mailx -s "AWR report for PRODOBIEE for Today !!! " -a awr_report_today.*  giriraj_inja@cree.com</tmp
mutt -s "AWR Report on PRODOBIEE for today " xyz@gmail.com -a awr_report*.*</tmp
else
mailx -s "PRODOBIEE instance is not up !! " xyz@gmail.com</tmp
fi


cat a.sql
set feedback off
set heading off
    define report_type=html
    define num_days=1
    define report_name=awr_report_today.html

spool /home/obiee/snu/b.sql
select 'define begin_snap=' || snap_values('MIN') from dual;
select 'define end_snap=' || snap_values('MAX') from dual;
spool off
@b.sql

cat b.sql  (This gets generated from a.sql automatically)

define begin_snap=13507                                                         

define end_snap=13509             

Sunday, June 12, 2016

Recovering Standby Database from corrupt data file


Found the following errors in alert logfile of standby database
ORA-10567: Redo is inconsistent with data block (file# 11, block# 83962, file offset is 687816704 bytes) 
ORA-10564: tablespace SYSAUX 
ORA-01110: data file 11: '/p02/prod/proddata/sysaux01.dbf' 
ORA-10561: block type 'TRANSACTION MANAGED DATA BLOCK', data object# 13366 

We realized that file no 11 of sysaux tablespace was corrupted

So we checked the health status of the object on Primary for the same by querying and found that they were healthy

SELECT owner, segment_name, segment_type, tablespace_name 
FROM dba_extents 
WHERE file_id = 11 
and 83962 between block_id AND block_id + blocks - 1; 

SQL>select owner,object_name,object_type,subobject_name,status,temporary, object_id,data_object_id from dba_objects where object_id = 13366; 
Primary database:
alter tablespace sysaux begin backup;
Standby database:
Verify the archive gap on standby database with following query
SELECT SEQUENCE#, applied FROM V$ARCHIVED_LOG order BY sequence# desc;
Shut immediate 
  Primary database:
scp /p02/prod/proddata/sysaux01.dbf  prod@scdebsdbrpt01:/p02/prod/proddata/sysaux01.dbf 
alter tablespace sysaux end backup;

Standby database :

Startup nomount pfile='$ORACLE_HOME/dbs/initprod.ora';
Alter database mount standby database;
ALTER DATABASE RECOVER MANAGED STANDBY DATABASE USING CURRENT LOGFILE DISCONNECT;

SELECT database_role, open_mode FROM v$database;

Following Queries can be used to find the archive gap between Primary and standby database:

SELECT MAX(SEQUENCE#) FROM V$ARCHIVED_LOG WHERE APPLIED='YES'

SELECT SEQUENCE#, FIRST_TIME, NEXT_TIME FROM V$ARCHIVED_LOG ORDER BY SEQUENCE# desc

SELECT SEQUENCE#,applied FROM V$ARCHIVED_LOG order by SEQUENCE# desc



Check for the MRP process status which is responsible for applying the logs with below query :

select process "MRP PROCESS Status" ,status,sequence#
  from v$managed_standby where PROCESS like '%MRP%';

Check for Log seq# applied on PRIMARY vs Standby Database

select  almax "Last Seq Received from PROD", lhmax "Last Seq Applied on PRODRPT"
from (select thread# thrd, max(sequence#) almax
      from v$archived_log
      where resetlogs_change#=(select resetlogs_change# from v$database)
      group by thread#) al,
     (select thread# thrd, max(sequence#) lhmax
      from v$log_history
      where first_time=(select max(first_time) from v$log_history)
      group by thread#) lh
where al.thrd = lh.thrd;



Cloning RMAN Database in Oracle

Source Database for Clone

1)Ensure the database is in Archive mode  (for Hot backups)
2) Make rmancat TNS enteries in TNSnames.ora (RMANCAT is the catalog database service name)
3) Rman target sys/xxxxx catalog=rcat_user/xxx@rmancat
4) Register database; (While cloning for the first time you would require to do this since catalog          would not have information about the new DBID)
5) CONFIGURE channel c1 type disk format 'K: \RMAN\Backup\df_%d_%U_%p_%c';
    (Had to change the backup location from default location)
6) Backup database plus archivelog;   

Steps for the Clone or Target Database 

12) Startup nomount pfile='pfile with absolute path';  (Edit the Pfile on the targe DB ) 
13) Rman target sys/manager catalog=rcat_user/rcatpass@rmancat
14) RESTORE CONTROLFILE FROM "K:\RMAN\BACKUP\DF_CPAUPG_03R4NTIJ_1_1_1_1";
 (Will get this backup piece information from "list backup" command)
 15) Mount the database through sqlplus or RMAN
 16) RESTORE DATABASE;
17) RECOVER DATABASE;

 18) ALTER DATABASE OPEN RESETLOGS;

Recovering Standby database for log synchronization with Primary DB

SQL>  startup nomount pfile='/p01/prod/db/tech_st/11.2.0.4/dbs/initPRODRPT.ora';
ORACLE instance started.

Total System Global Area 5.1310E+10 bytes
Fixed Size                  2265224 bytes
Variable Size            5502930808 bytes
Database Buffers         4.5768E+10 bytes
Redo Buffers               36069376 bytes
SQL> Alter database mount standby database;

Database altered.

SQL>
SQL> alter database open read only;

Database altered.

SQL> alter database recover managed standby database using current logfile disconnect from session;

Database altered.

SQL>  select process "MRP PROCESS Status" ,status,sequence#
  from v$managed_standby where PROCESS like '%MRP%';
  2
MRP PROCE STATUS        SEQUENCE#
--------- ------------ ----------
MRP0      APPLYING_LOG      67465

SQL> select  almax "Last Seq Received from PROD", lhmax "Last Seq Applied on PRODRPT"
  2  from (select thread# thrd, max(sequence#) almax
  3        from v$archived_log
  4        where resetlogs_change#=(select resetlogs_change# from v$database)
  5        group by thread#) al,
  6       (select thread# thrd, max(sequence#) lhmax
  7        from v$log_history
  8        where first_time=(select max(first_time) from v$log_history)
  9        group by thread#) lh
 10  where al.thrd = lh.thrd;

Last Seq Received from PROD Last Seq Applied on PRODRPT
--------------------------- ---------------------------
                      67464                       67464

SQL> SELECT database_role, open_mode FROM v$database;

DATABASE_ROLE    OPEN_MODE
---------------- --------------------
PHYSICAL STANDBY READ ONLY WITH APPLY


Note: You could stop the listener on the standby database server before putting into recovery mode and start it once you start the recovery/log apply process.

Pinning Oracle DB/EBS Objects for better Performance

Imagine a large package (or any object) has to be loaded into the shared pool.
Large PL/SQL objects present particular challenges. 
The database has to search for free space for the object. 
If it cannot get enough contiguous space, it will free many small objects to satisfy the request.
If several large objects need to be loaded, the database has to throw out many small objects in the shared pool. 
Finding candidate objects and freeing memory is very costly.  These tasks will impact CPU resources.
To view a list of all objects that are kept in the shared pool, one can query the v$db_object_cache:

Select owner, name, type, sharable_mem from v$db_object_cache where kept='YES';
In EBS application since 80-90% objects are owned by APPS
Select owner, name, type, sharable_mem from v$db_object_cache where kept='YES' and owner='APPS';
The SQL query above will list all of the objects that are 'kept' in the shared pool using dbms_shared_pool.keep.

One approach to avoiding performance overhead and memory allocation errors is to keep large PL/SQL objects in the shared pool at startup time. 
This process is known as pinning.
This loads the objects into the shared pool and ensures that the objects are never aged out of the shared pool.
If the objects are never aged out, then that avoids problems with insufficient memory when trying to reload them.
Following Package is used to PIN Objects
execute dbms_shared_pool.keep('owner. object');
(I Have generated dynamic sql for the list of objects to be pinned in PROD)
However there is a word of caution on this :
1.Be careful not to pin too many objects into memory and use up all of your available shared buffer space or the performance of the remainder of the applications will suffer.(we could consider filtering the list a bit)
2. The database startup routine should execute the following script after opening the database:
i.e.
execute dbms_shared_pool.keep (APPS.GL_CURRENCY_API) ;
execute dbms_shared_pool.keep (APPS.HR_SECURITY) ;

3. Oracle Corporation recommends that the following packages always be pinned into the shared pool at database startup (if you have installed them into the database):
SYS.STANDARD ,SYS.DBMS_LOCK
SYS.DBMS_STANDARD  ,SYS.DBMS_PIPE
SYS.DBMS_DESCRIBE ,SYS.DBMS_OUTPUT
SYS.DBMS_UTILITY
Pinning objects into memory yields best results for OLTP users. Because long-running jobs will load these packages into the SGA at the beginning of their job and hold them there for the duration of the task, performance improvements will be minimal.
(Prior to pinning the object the object should have executed at least once to make sure it exists in the shared pool)

The $AD_TOP/sql/ADXGNPIN.sql script is provided to pin the packages in Oracle Applications.  
This script pins all Oracle Applications objects, which is generally not completely necessary or advisable.
You should modify this script based on your analyses of v$db_object_cache and x$ksmlru to identify the right objects to be pinned for your environment.


It doesn’t matter whether it’s EBS or non-EBS because all loaded/executed objects are found in the view “v$db_object_cache” irrelevant of whether it’s an EBS/NON-EBS.
How these objects are loaded as in how many times executed/Loaded & memory used varies from environment to environment and also application to application.

So it finally comes down to one thing list out the objects based on no, of execution, Loads, memory.

Below query gives this extract of the objects to be pinned based on the no of executions, loads, etc.
Select 'execute dbms_shared_pool.keep (' ||''''|| substr (owner,1,10)||'.'||substr(name,1,35)  ||''''||',' ||''''|| decode(type,'PACKAGE','P','PROCEDURE','P','FUNCTION','P','Q','TRIGGER','R') ||'''' ||');'    from v$db_object_cache  where type in ('TRIGGER','PROCEDURE','PACKAGE BODY','PACKAGE') and executions >0 order by executions desc,loads desc,sharable_mem desc


Also in my experience it is good to avoid pinning triggers because triggers can behave very weirdly and can be a show stopper for the application functionality, so avoided pinning triggers.

Cloning Oracle HOME on the same Machine

Create OS User DEVOBI under dba group on scdobi01-new

create ORACLE_BASE BASE directory structure with devobi login
/app/11.2.0/devobi/

[uatobi@uatobi]$ echo $ORACLE_HOME   (Soucre Oracle HOME)
/app/11.2.0/prdobi

[oracle@orahost ~]$ cd /app/11.2.0/prdobi
[uatobi@uatobi]$  cp -Rp $ORACLE_HOME /devobi_bin/app/11.2.0/devobi/
(Copying Oracle Home binaries from souce to destination )

export ORACLE_HOME=/devobi_bin/app/11.2.0/devobi/prdobi

./runInstaller -silent -clone ORACLE_BASE="/devobi_bin/app/11.2.0/devobi" ORACLE_HOME="/devobi_bin/app/11.2.0/devobi/prdobi" ORACLE_HOME_NAME="OraHome2"


nstallation in progress (Wednesday, June 8, 2016 9:33:53 AM GMT-05:00)
..............................................................................                                                  78% Done.
Install successful

Linking in progress (Wednesday, June 8, 2016 9:34:16 AM GMT-05:00)
Link successful

Setup in progress (Wednesday, June 8, 2016 9:35:18 AM GMT-05:00)
Setup successful

End of install phases.(Wednesday, June 8, 2016 9:35:25 AM GMT-05:00)


The following configuration scripts need to be executed as the "root" user.

1)/home/devobi/oraInventory/orainstRoot.sh
2)/devobi_bin/app/11.2.0/devobi/prdobi/root.sh

To execute the configuration scripts:
    1. Open a terminal window
    2. Log in as "root"
    3. Run the scripts


 /devobi_bin/app/11.2.0/devobi/prdobi/OPatch/opatch lsinventory
Invoking OPatch 11.2.0.1.1

Oracle Interim Patch Installer version 11.2.0.1.1
Copyright (c) 2009, Oracle Corporation.  All rights reserved.


Oracle Home       : /devobi_bin/app/11.2.0/devobi/prdobi
Central Inventory : /home/devobi/oraInventory
   from           : /etc/oraInst.loc
OPatch version    : 11.2.0.1.1
OUI version       : 11.2.0.2.0
OUI location      : /devobi_bin/app/11.2.0/devobi/prdobi/oui
Log file location : /devobi_bin/app/11.2.0/devobi/prdobi/cfgtoollogs/opatch/opatch2016-06-09_05-33-50AM.log

Patch history file: /devobi_bin/app/11.2.0/devobi/prdobi/cfgtoollogs/opatch/opatch_history.txt

Lsinventory Output file location : /devobi_bin/app/11.2.0/devobi/prdobi/cfgtoollogs/opatch/lsinv/lsinventory2016-06-09_05-33-50AM.txt

--------------------------------------------------------------------------------
Installed Top-level Products (1):

Oracle Database 11g                                                  11.2.0.2.0
There are 1 products installed in this Oracle Home.


Interim patches (2) :

Patch  13072654     : applied on Sat Oct 05 07:26:48 GMT-05:00 2013
Unique Patch ID:  15973278
   Created on 14 Feb 2013, 05:19:41 hrs PST8PDT
   Bugs fixed:
     13072654

Patch  9825461      : applied on Fri Aug 02 20:58:15 GMT-05:00 2013
Unique Patch ID:  13822089
   Created on 13 Jun 2011, 17:17:07 hrs PST8PDT
   Bugs fixed:
     9825461



--------------------------------------------------------------------------------

OPatch succeeded.

Note :This above Opatch command confirms that your inventory is created/updated with new ORACLE_HOME location and Oracle_HOME is healthy

Run the ./runInstaller  from $ORACLE_HOME/oui/bin
Also you can netca/dbca from $ORACLE_HOME/bin

these checks will confirm that all your executables are healthy and working fine