Oracle wait event enq: JI – contention and p1 p2 and p3 value anayze

Oracle wait event enq ji – contention and p1/p2/p3 value anayze

Find session that waiting enq: JI – contention and p1 p2 p3 value

SELECT username,status,program,machine,
sid,serial#,p1,p2,p3,p1text,p2text,p3text
FROM V$SESSION
WHERE TYPE <> ‘BACKGROUND’
AND STATUS = ‘ACTIVE’
AND EVENT = ‘enq: JI – contention’;

USERNAME    STATUS   PROGRAM                       MACHINE       SID    SERIAL#         P1         P2         P3 P1TEXT        P2TEXT             P3TEXT   
----------- -------- ----------------------------- ----------------- ---------- ---------- ---------- ---------- ------------- ------------------ ---------
SYS         ACTIVE   sqlplus@hstn (TNS V1-V3)      hstn           25      23109 1246298118     990549          0 name|mode     view object #      0        
SYS         ACTIVE   sqlplus@hstn (TNS V1-V3)      hstn         1044      20069 1246298118     990549          0 name|mode     view object #      0        
SYS         ACTIVE   sqlplus@hstn (TNS V1-V3)      hstn         2058      40479 1246298118     990549          0 name|mode     view object #      0        
SYS         ACTIVE   sqlplus@hstn (TNS V1-V3)      hstn         2060      49559 1246298118     990549          0 name|mode     view object #      0        
SYS         ACTIVE   sqlplus@hstn (TNS V1-V3)      hstn         3060      31231 1246298118     990549          0 name|mode     view object #      0        
SYS         ACTIVE   sqlplus@hstn (TNS V1-V3)      hstn         3069      25883 1246298118     990549          0 name|mode     view object #      0        
SYS         ACTIVE   sqlplus@hstn (TNS V1-V3)      hstn         5091       5895 1246298118     990549          0 name|mode     view object #      0        
SYS         ACTIVE   sqlplus@hstn (TNS V1-V3)      hstn         5103      44721 1246298118     990549          0 name|mode     view object #      0

Find Name and Mode from p1 value

set lines 500
var p1 number;
exec :p1:=1246298118;

select CHR (BITAND (:p1, -16777216) / 16777215)
|| CHR (BITAND (:p1, 16711680) / 65535)
“Name”,
(BITAND (:p1, 65535))”Mode”
from dual;

Name       Mode
---- ----------
JI            6

Find object from p2 value
We see this is fast refresh on commit Materialized View.
You can try not to do costly operation on that table references Materialized View or drop and recreate it to do your operation.

select * from dba_objects
where object_id=990549;

OWNER        OBJECT_NAME         SUBOBJECT_NAME    OBJECT_ID DATA_OBJECT_ID OBJECT_TYPE         CREATED    LAST_DDL_TIME TIMESTAMP           STATUS  TEMPORARY GENERATED SECONDARY  NAMESPACE EDITION_NAME                  
------------ ------------------- ---------------- ---------- -------------- ------------------- ---------- ------------- ------------------- ------- --------- --------- --------- ---------- ------------------------------
OWNERNA      MV_NAME_XXXXXX                           990549         990549 TABLE               05/10/2016 15/11/2019    2016-10-05:19:50:49 VALID   N         N         N                  1

SELECT owner,name,table_name,type,refresh_mode,status
FROM dba_snapshots
where name=’MV_NAME_XXXXXX

OWNER        NAME                TABLE_NAME         TYPE     REFRESH_MODE STATUS 
-----------  ------------------- ------------------ -------- ------------ -------
OWNERNA      MV_NAME_XXXXXX      MV_NAME_XXXXXX     FAST     COMMIT       VALID

19c db patch 19.3.0 to 19.3.2 RUR

To apply grid patch you can visit 19c grid patch site.

1-) Opatch version control.

For each Oracle RAC database home and the GI home that are being patched, run the following commands as the home owner to extract the OPatch utility.

$ unzip  -d 
$ /OPatch/opatch version

The version output of the previous command should be 12.2.0.1.17 or later.

2-) validation oracle inventory. Below command should be successful.

/OPatch/opatch lsinventory -detail -oh

3-) Run the following commands to control if there is any conflict or not.

$ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir /oracle/oracd/19cRUR/30135696/30087906
$ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir /oracle/oracd/19cRUR/30135696/29585399

4-) Apply the rdbms patch executing like that.

opatchauto apply /oracle/oracd/19cRUR/30135696 -oh /oracle/app/product/19.3.0.0/dbhome_1

[root@hostname ~]# opatchauto apply /oracle/oracd/19cRUR/30135696 -oh /oracle/app/product/19.3.0.0/dbhome_1

OPatchauto session is initiated at Fri Jan 10 14:01:28 2020

System initialization log file is /oracle/app/product/19.3.0.0/dbhome_1/cfgtoollogs/opatchautodb/systemconfig2020-01-10_02-01-34PM.log.

Session log file is /oracle/app/product/19.3.0.0/dbhome_1/cfgtoollogs/opatchauto/opatchauto2020-01-10_02-01-44PM.log

The id for this session is WDCV

Executing OPatch prereq operations to verify patch applicability on home /oracle/app/product/19.3.0.0/dbhome_1

Patch applicability verified successfully on home /oracle/app/product/19.3.0.0/dbhome_1

Verifying SQL patch applicability on home /oracle/app/product/19.3.0.0/dbhome_1

SQL patch applicability verified successfully on home /oracle/app/product/19.3.0.0/dbhome_1

Preparing to bring down database service on home /oracle/app/product/19.3.0.0/dbhome_1

Successfully prepared home /oracle/app/product/19.3.0.0/dbhome_1 to bring down database service

Bringing down database service on home /oracle/app/product/19.3.0.0/dbhome_1

Following database has been stopped and will be restarted later during the session: vgbmtst1

Database service successfully brought down on home /oracle/app/product/19.3.0.0/dbhome_1

Start applying binary patch on home /oracle/app/product/19.3.0.0/dbhome_1

Binary patch applied successfully on home /oracle/app/product/19.3.0.0/dbhome_1

Starting database service on home /oracle/app/product/19.3.0.0/dbhome_1

Database service successfully started on home /oracle/app/product/19.3.0.0/dbhome_1

Preparing home /oracle/app/product/19.3.0.0/dbhome_1 after database service restarted

No step execution required………

Trying to apply SQL patch on home /oracle/app/product/19.3.0.0/dbhome_1

SQL patch applied successfully on home /oracle/app/product/19.3.0.0/dbhome_1

OPatchAuto successful.

——————————–Summary——————————–

Patching is completed successfully. Please find the summary as follows:

Host:hella01

SIDB Home:/oracle/app/product/19.3.0.0/dbhome_1

Version:19.0.0.0.0

Summary:

==Following patches were SKIPPED:

Patch: /oracle/oracd/19cRUR/30135696/29517247

Reason: This patch is not applicable to this specified target type – “oracle_database”

Patch: /oracle/oracd/19cRUR/30135696/29401763

Reason: This patch is not applicable to this specified target type – “oracle_database”

Patch: /oracle/oracd/19cRUR/30135696/29585399

Reason: This patch is already been applied, so not going to apply again.

==Following patches were SUCCESSFULLY applied:

Patch: /oracle/oracd/19cRUR/30135696/30087906

Log: /oracle/app/product/19.3.0.0/dbhome_1/cfgtoollogs/opatchauto/core/opatch/opatch2020-01-10_14-03-20PM_1.log

OPatchauto session completed at Fri Jan 10 14:08:49 2020

Time taken to complete the session 7 minutes, 22 seconds

5-) Post Installation Steps

5.1-) Load Modified Files into the Database

[oracle@hella01 ~]$ cd $ORACLE_HOME
[oracle@hella01 dbhome_1]$ cd OPatch/
[oracle@hella01 OPatch]$ ./datapatch -verbose

SQL Patching tool version 19.3.2.0.0 Production on Fri Jan 10 14:13:27 2020
Copyright (c) 2012, 2019, Oracle. All rights reserved.

Log file for this invocation: /oracle/app/db/cfgtoollogs/sqlpatch/sqlpatch_812_2020_01_10_14_13_27/sqlpatch_invocation.log

Connecting to database…OK
Gathering database info…done
Bootstrapping registry and package to current versions…done
Determining current state…done

Current state of interim SQL patches:
No interim patches found

Current state of release update SQL patches:
Binary registry:
19.3.2.0.0 Release_Update_Revision 190829192137: Installed
SQL registry:
Applied 19.3.2.0.0 Release_Update_Revision 190829192137 successfully on 10-JAN-20 02.08.48.328593 PM

Adding patches to installation queue and performing prereq checks…done
Installation queue:
No interim patches need to be rolled back
No release update patches need to be installed
No interim patches need to be applied

SQL Patching tool complete on Fri Jan 10 14:13:51 2020

5.2-) Compile Invalid Objects

[oracle@hella01 ~]$ cd $ORACLE_HOME/rdbms/admin
[oracle@hella01 admin]$ sqlplus / as sysdba

SQL*Plus: Release 19.0.0.0.0 – Production on Sun Jan 12 16:22:10 2020
Version 19.3.2.0.0

Copyright (c) 1982, 2019, Oracle. All rights reserved.
Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 – Production
Version 19.3.2.0.0

SQL> @utlrp.sql

 

5.3-) Upgrade Catalog

$ rman catalog username/password@alias

RMAN> UPGRADE CATALOG;

RMAN> UPGRADE CATALOG;

RMAN> EXIT;

19c grid patch 19.3.0 to 19.3.2 RUR

To appliy rdbms patch visit 19c rdbms patch post.

1-) Opatch version control.

For each Oracle RAC database home and the GI home that are being patched, run the following commands as the home owner to extract the OPatch utility.

$ unzip <OPATCH-ZIP> -d <ORACLE_HOME>
$ <ORACLE_HOME>/OPatch/opatch version

The version output of the previous command should be 12.2.0.1.17 or later.

2-) validation oracle inventory. Below command should be successful.

<ORACLE_HOME>/OPatch/opatch lsinventory -detail -oh <ORACLE_HOME>
3-) Run the following commands to control if there is any conflict or not.

$ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir /grid/patch/30135696/30087906
$ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir /grid/patch/30135696/29585399
$ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir /grid/patch/30135696/29517247
$ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir /grid/patch/30135696/
$ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir /grid/patch/30135696/29401763

4-) Apply the patch executing like that.

opatchauto apply /grid/patch/30135696 -oh /grid/app/19.3.0.0/grid

[root@hostname patch]# opatchauto apply /grid/patch/30135696 -oh /grid/app/19.3.0.0/grid

OPatchauto session is initiated at Fri Dec 20 16:46:31 2019

System initialization log file is /grid/app/19.3.0.0/grid/cfgtoollogs/opatchautodb/systemconfig2019-12-20_04-46-37PM.log.

Session log file is /grid/app/19.3.0.0/grid/cfgtoollogs/opatchauto/opatchauto2019-12-20_04-46-45PM.log
The id for this session is 61QQ

Executing OPatch prereq operations to verify patch applicability on home /grid/app/19.3.0.0/grid
Patch applicability verified successfully on home /grid/app/19.3.0.0/grid
Bringing down CRS service on home /grid/app/19.3.0.0/grid
Prepatch operation log file location: /grid/app/grid/crsdata/hostname/crsconfig/hapatch_2019-12-20_04-47-13PM.log
CRS service brought down successfully on home /grid/app/19.3.0.0/grid
Start applying binary patch on home /grid/app/19.3.0.0/grid
Binary patch applied successfully on home /grid/app/19.3.0.0/grid
Starting CRS service on home /grid/app/19.3.0.0/grid
Postpatch operation log file location: /grid/app/grid/crsdata/hostname/crsconfig/hapatch_2019-12-20_04-50-40PM.log
CRS service started successfully on home /grid/app/19.3.0.0/grid

OPatchAuto successful.

——————————–Summary——————————–

Patching is completed successfully. Please find the summary as follows:

Host:hostname
SIHA Home:/grid/app/19.3.0.0/grid
Version:19.0.0.0.0
Summary:

==Following patches were SKIPPED:

Patch: /grid/patch/30135696/29585399
Reason: This patch is already been applied, so not going to apply again.

Patch: /grid/patch/30135696/29517247
Reason: This patch is already been applied, so not going to apply again.

Patch: /grid/patch/30135696/29401763
Reason: This patch is already been applied, so not going to apply again.
==Following patches were SUCCESSFULLY applied:

Patch: /grid/patch/30135696/30087906
Log: /grid/app/19.3.0.0/grid/cfgtoollogs/opatchauto/core/opatch/opatch2019-12-20_16-47-46PM_1.log

OPatchauto session completed at Fri Dec 20 16:59:58 2019
Time taken to complete the session 13 minutes, 27 seconds

oracle diskgroup mount fails ORA-15017: diskgroup “” cannot be mounted ORA-15063: ASM discovered an insufficient number of disks for diskgroup “” ( detected duplicate paths to the same disk )

When upgrading oracle grid, we had following errors.

ASM alert log

WARNING: detected duplicate paths to the same disk:
‘/dev/vx/dmp/tagmastore-usp0_0127s1’ and
/dev/raw/raw162
More trace information dumped to ‘/grid/11.2.0/grid_4/log/diag/asm/+asm/+ASM/trace/+ASM_ora_12599.trc’
WARNING: detected duplicate paths to the same disk:
‘/dev/vx/dmp/tagmastore-usp0_0117s1’ and
/dev/raw/raw210
More trace information dumped to ‘/grid/11.2.0/grid_4/log/diag/asm/+asm/+ASM/trace/+ASM_ora_12599.trc’
WARNING: detected duplicate paths to the same disk:
‘/dev/vx/dmp/tagmastore-usp0_0124s1’ and
/dev/raw/raw114
More trace information dumped to ‘/grid/11.2.0/grid_4/log/diag/asm/+asm/+ASM/trace/+ASM_ora_12599.trc’
WARNING: detected duplicate paths to the same disk:
‘/dev/vx/dmp/tagmastore-usp0_0116s1’ and
/dev/raw/raw194
More trace information dumped to ‘/grid/11.2.0/grid_4/log/diag/asm/+asm/+ASM/trace/+ASM_ora_12599.trc’
WARNING: detected duplicate paths to the same disk:
‘/dev/vx/dmp/tagmastore-usp0_0126s1’ and
/dev/raw/raw146
More trace information dumped to ‘/grid/11.2.0/grid_4/log/diag/asm/+asm/+ASM/trace/+ASM_ora_12599.trc’
WARNING: detected duplicate paths to the same disk:
‘/dev/vx/dmp/tagmastore-usp0_0115s1’ and
/dev/raw/raw178
More trace information dumped to ‘/grid/11.2.0/grid_4/log/diag/asm/+asm/+ASM/trace/+ASM_ora_12599.trc’
WARNING: detected duplicate paths to the same disk:
‘/dev/vx/dmp/tagmastore-usp0_0125s1’ and
/dev/raw/raw130

ERROR: diskgroup DATA was not mounted
ORA-15032: not all alterations performed
ORA-15017: diskgroup “DATA” cannot be mounted
ORA-15063: ASM discovered an insufficient number of disks for diskgroup “DATA”

ERROR: diskgroup DATA_OTHER was not mounted
ORA-15032: not all alterations performed
ORA-15017: diskgroup “DATA_OTHER” cannot be mounted
ORA-15063: ASM discovered an insufficient number of disks for diskgroup “DATA_OTHER”

Then we control of the ownership of the raw device. We recognized some of the disk’s owner under /dev/raw/raw* path was belong to oracle.

Solution

1-) Change the owner of the duplicate disks under the /dev/raw/raw* to the root to prevent oracle from seeing.

2-) And then control udev rules. If not exists create like that.

cat /etc/udev/rules.d/99-oracle-asmdevices.rules. This is example.

KERNEL==”sdb”,OWNER=”grid”,GROUP=”oinstall”,MODE=”0660″,ACTION==”add|change”
KERNEL==”sdc”,OWNER=”grid”,GROUP=”oinstall”,MODE=”0660″,ACTION==”add|change”
KERNEL==”sdd”,OWNER=”grid”,GROUP=”oinstall”,MODE=”0660″,ACTION==”add|change”
KERNEL==”sde”,OWNER=”grid”,GROUP=”oinstall”,MODE=”0660″,ACTION==”add|change”

Upgrade Oracle grid fails with ORA-00119: invalid specification for system parameter LOCAL_LISTENER and ORA-00132: syntax error or unresolved network name ‘LISTENER_+ASM’

When upgrading oracle grid asm 11.2.0.3.to 11.2.0.4 following errors appear in the  oraagent_grid.log.

vi /grid/11.2.0/grid_4/log/myhostname/agent/ohasd/oraagent_grid/oraagent_grid.log

CRS-5017: The resource action “ora.asm start” encountered the following error:
ORA-00119: invalid specification for system parameter LOCAL_LISTENER
ORA-00132: syntax error or unresolved network name ‘LISTENER_+ASM’
. For details refer to “(:CLSN00107:)” in “/grid/11.2.0/grid_4/log/myhostname/agent/ohasd/oraagent_grid/oraagent_grid.log”.

2019-11-28 14:15:58.380: [ AGFW][2377918208]{0:0:2} sending status msg [CRS-5017: The resource action “ora.asm start” encountered the following error:
ORA-00119: invalid specification for system parameter LOCAL_LISTENER
ORA-00132: syntax error or unresolved network name ‘LISTENER_+ASM’
. For details refer to “(:CLSN00107:)” in “/grid/11.2.0/grid_4/log/myhostname/agent/ohasd/oraagent_grid/oraagent_grid.log”.
] for start for resource: ora.asm myhostname 1

 

Solution

According to Oracle Support document we  commented out local_listener in spfile. Then proplem was solved.

$ORACLE_HOME/dbs/spfile+ASM.ora

*.asm_power_limit=1
*.diagnostic_dest=’/grid/11.2.0/grid_4′
*.instance_type=’asm’
*.large_pool_size=12M
#*.local_listener=’LISTENER_+ASM’
*.remote_login_passwordfile=’EXCLUSIVE’

Alert log shows ORA-00310: archived log contains sequence 10; sequence 9 required

When we startup database we got the error below.

ORA-01194: file 1 needs more recovery to be consistent
ORA-01110: data file 1: ‘/oradata/DBNAME/datafile/o1_mf_system_b8x7qg5m_.dbf’

Then we recovered the database but still we have the same problem.

ORA-01547: warning: RECOVER succeeded but OPEN RESETLOGS would get error below
ORA-01194: file 1 needs more recovery to be consistent

We examine the alert log and find  the error like this.

ALTER DATABASE RECOVER LOGFILE ‘/oradata/DBNAME/onlinelog/o1_mf_2_gkmjnpnm_.log’
Media Recovery Log /oradata/DBNAME/onlinelog/o1_mf_2_gkmjnpnm_.log
Fri Dec 27 14:24:49 2019
Errors with log /oradata/DBNAME/onlinelog/o1_mf_2_gkmjnpnm_.log
Errors in file /oracle/diag/rdbms/dbname/DBNAME/trace/DBNAME_pr00_3427.trc:
ORA-00310: archived log contains sequence 10; sequence 9 required
ORA-00334: archived log: ‘/oradata/DBNAME/onlinelog/o1_mf_2_gkmjnpnm_.log’

Solution

List redolog files

SQL> r
1* select * from v$logfile

GROUP# STATUS TYPE MEMBER IS_
———– ——– ——- ————————— ————————-

3 ONLINE /oradata/DBNAME/onlinelog/o1_mf_3_gkmjnpsy_.log NO

2 ONLINE /oradata/DBNAME/onlinelog/o1_mf_2_gkmjnpnm_.log NO

1 ONLINE /oradata/DBNAME/onlinelog/o1_mf_1_gkmjnpgs_.log NO

4 ONLINE /oradata/DBNAME/onlinelog/onlinelog1.log NO

 

SQL> SQL> select * from v$log;

    GROUP#    THREAD#  SEQUENCE#      BYTES  BLOCKSIZE    MEMBERS ARC STATUS           FIRST_CHANGE# FIRST_TIM NEXT_CHANGE# NEXT_TIME
---------- ---------- ---------- ---------- ---------- ---------- --- ---------------- ------------- --------- ------------ ---------
         1          1          9   52428800        512          1 NO  ACTIVE               131425515 25-DEC-19    131429147 25-DEC-19
         4          1          8  104857600        512          1 NO  INACTIVE             131388466 25-DEC-19    131425515 25-DEC-19
         3          1          7   52428800        512          1 NO  INACTIVE             131388434 25-DEC-19    131388466 25-DEC-19
         2          1         10   52428800        512          1 NO  CURRENT              131429147 25-DEC-19   2.8147E+14

Recover Database using ACTIVE and CURRENT redo logs. First use ACTIVE then use CURRENT.

SQL> recover database using backup controlfile until cancel ;
ORA-00279: change 131425515 generated at 12/25/2019 23:00:22 needed for thread 1
ORA-00289: suggestion : /oracle/product/11.2.0.4/db/dbs/arch1_9_1027939009.dbf
ORA-00280: change 131425515 for thread 1 is in sequence #9
Specify log: {=suggested | filename | AUTO | CANCEL}
/oradata/DBNAME/onlinelog/o1_mf_1_gkmjnpgs_.log
ORA-00279: change 131429147 generated at 12/25/2019 23:19:44 needed for thread 1
ORA-00289: suggestion : /oracle/product/11.2.0.4/db/dbs/arch1_10_1027939009.dbf
ORA-00280: change 131429147 for thread 1 is in sequence #10
ORA-00278: log file ‘/oradata/DBNAME/onlinelog/o1_mf_1_gkmjnpgs_.log’ no longer needed for this recovery
Specify log: {=suggested | filename | AUTO | CANCEL}
/oradata/DBNAME/onlinelog/o1_mf_2_gkmjnpnm_.log
Log applied.
Media recovery complete.

OPEN the database with resetlogs option

SQL> Alter database open resetlogs ;

Database altered.

Oracle instance startup fails ORA-01157: cannot identify/lock data file x

Oracle instance startup fails because of wrongly deleted undo datafile.  When we create new undo tablespace , by mistake we delete datafile via os rm command and then could not mount the database because controlfile is containing that datafile.

Error is here.

ORA-01157: cannot identify/lock data file 3 – see DBWR trace file
ORA-01110: data file 3: ‘/oradata/DBNAME/UNDOTBS_23122019.dbf

Solution

1-) Create missing datafile.

SQL> alter database create datafile ‘/oradata/DBNAME/UNDOTBS_23122019.dbf‘ as ‘/oradata/DBNAME/datafile/SIL.dbf‘;

Database altered.

SQL> select name from v$datafile;

NAME
--------------------------------------------------------------------------------
/oradata/DBNAME/datafile/o1_mf_system_b8x7qg5m_.dbf
/oradata/DBNAME/datafile/o1_mf_sysaux_b8x7qg87_.dbf
/oradata/DBNAME/datafile/SIL.dbf
/oradata/DBNAME/datafile/o1_mf_users_b8x7qgb3_.dbf
/oradata/DBNAME/datafile/o1_mf_tbs_idmo_bb5f70kk_.dbf
/oradata/DBNAME/datafile/o1_mf_dbaall_d_fybvbvvx_.dbf
/oradata/DBNAME/datafile/UNDOTBS_23122019.dbf
/oradata/DBNAME/datafile/undotbs1.dbf

SQL> alter database open;
alter database open
*
ERROR at line 1:
ORA-01113: file 3 needs media recovery
ORA-01110: data file 3: ‘/oradata/DBNAME/datafile/SIL.dbf

Datafile that should be dropped.

select FILE#,TS# ,name from v$datafile where file#=3;
FILE#     TS#      NAME
3         10       /oradata/DBNAME/datafile/SIL.dbf

Undo Tablespace that should be dropped.

SQL> select * from v$tablespace;

TS# NAME                           INC BIG FLA ENC
---------- ------------------------------ --- --- --- ---
0 SYSTEM                         YES NO  YES
1 SYSAUX                         YES NO  YES
10 UNDOTBS_23122019               YES NO  YES
4 USERS                          YES NO  YES
3 TEMP                           NO  NO  YES
6 TBS_IDMOM                      YES NO  YES
7 DBAALL_DATA                    YES NO  YES
11 UNDOTBS_233122019              YES NO  YES
2 UNDOTBS1                       YES NO  YES

2-) Offline drop datafile and open the database.

SQL> alter database datafile 3 offline drop;

Database altered.

SQL> alter database open;

Database altered.

3-) Drop wrong undo tablespace.

SQL> drop tablespace UNDOTBS_23122019 including contents and datafiles;

Tablespace dropped.

SQL>  select open_mode from v$database;

OPEN_MODE
——————–
READ WRITE

Reference: ORA-1157 Troubleshooting (Doc ID 184327.1)

Step by step troubleshooting ORA-04031 unable to allocate x bytes of shared memory error in oracle

Step by step analyzing ORA-04031 unable to allocate x bytes of shared memory error in oracle

 

In alert log the error is ORA-04031: unable to allocate 32 bytes of shared memory (“shared pool”,”unknown object”,”KGLH0^e481b9fe”,”kglHeapInitialize:temp”)

If you get ORA-04031 error on shared pool and can not logon your database you must restart your instance. I will try to explain why you get ORA-04031 error and show you the solitions step by step.

1-) One of the reason is fragmentation of shared pool. If your shared pool is fragmented then you can flush your shared pool or restart your instance.

alter system flush shared_pool;

2-) Sometimes Shared pool need to grow. Then you should increase your shared pool.

alter system set shared_pool_size=32G;

3.1-) Sql that coming with Literals and use same plan_hash_value and do not use bind variables

select * from my_table where my_column=’variable1′;
select * from my_table where my_column=’variable2′;
select * from my_table where my_column=’variable44′;

3.2-) Similar sql statement that coming same plan_hash_value using bind variable.

select * from my_table where my_column in (:b1,:b2);
select * from my_table where my_column in (:b1);
select * from my_table where my_column in (:b1,:b2,:b3);

  • Script to find how much memory used by similar sql or sql that coming literals.

select inst_id,plan_hash_value,round(sum(S.SHARABLE_MEM)/(1024*1024)) as Memory_MB,count(1) from gv$sql s
group by inst_id,plan_hash_value
order by 3 desc

   INST_ID PLAN_HASH_VALUE  MEMORY_MB   COUNT(1)
---------- --------------- ---------- ----------
         1               0       5222      26414
         2               0       4598      22007
         2      1316043113        208       2234
         1      1316043113        200       2151
         2      3095531958        112       2239
         1      3095531958        108       2151

NOTE: You can ignore plan_hash_value=0 but some specific case such as mine it should be considered with sql_id together.  plan_hash_value=0 may be an insert statement, plsql blok,procedure call, sql statement with dblink etc.
4-) Too many child of sql and high version count ( script to find why that sqls have high version count is at the bottom). If you want to flush/purge sql from shared pool  that has high child and version count , you can click the link.

  • Script to find sql child count.

select inst_id,sql_id,round(sum(S.SHARABLE_MEM)/(1024*1024),1) as Memory_MB,count(1) as CHILDCOUNT
from gv$sql s
group by inst_id,sql_id
order by 3 desc;

   INST_ID SQL_ID         MEMORY_MB CHILDCOUNT
---------- ------------- ---------- ----------
         1 7yj8ubju3vc57     3936,2       5725
         2 7yj8ubju3vc57     3372,4       4878
         1 7y3md7fy66cgx       41,3         31
         1 f6jp03faawzs2       38,2         43
         2 7txvv1msuyyu9       37,4         42
         2 f6jp03faawzs2       36,8         46
  • Script to find sql version count. (script to find why that sqls have high version count is at the bottom)

select
inst_id,sql_id,version_count
from
gv$sqlarea
order by version_count desc

   INST_ID SQL_ID        VERSION_COUNT
---------- ------------- -------------
         2 7yj8ubju3vc57           783
         1 7yj8ubju3vc57           608
         2 04c5k4phtxm7a           349
         2 dyjzd44h1t0r1            83
         2 6raq7sb0yp6su            79
         1 6raq7sb0yp6su            79
  • Sql plan hash value list that has high version count. This script can be used on specific case. If versioncount > count there might be a problem.

set lines 500
select * from (
select
plan_hash_value,sum(version_count) as versioncount ,count(1)
from
v$sqlarea
group by plan_hash_value
order by sum(version_count) desc
) where rownum<11
;

PLAN_HASH_VALUE VERSIONCOUNT   COUNT(1)
--------------- ------------ ----------
              0        18895       4918
      564703301         2835       2834
     3095531958         2239       2239
     1316043113         2234       2234
     1788691278          609        125
      989574962          465        465
     1404353691          457        457

Analyzing shared pool and component

  • Shared pool size by instance

select inst_id,round(sum(S.SHARABLE_MEM)/(1024*1024),1) as Memory_MB
from gv$sql s
group by inst_id
order by 2 desc;

   INST_ID  MEMORY_MB
---------- ----------
         1     9852,6
         2     7861,9
  • Shared pool component size

select *
from gv$sgastat
where pool=’shared pool’
–and name=’free memory’
and inst_id=1
order by bytes desc;

   INST_ID POOL         NAME                            BYTES
---------- ------------ -------------------------- ----------
         1 shared pool  free memory                1060714008
         1 shared pool  KGLHD                       229558520
         1 shared pool  SQLA                        221058248
         1 shared pool  KQR L PO                    191218792
         1 shared pool  ges resource                134750712
         1 shared pool  KGLH0                       110085888
         1 shared pool  ASH buffers                 104857600
         1 shared pool  ASM extent pointer array     90770352
         1 shared pool  KQR X PO                     84272960
         1 shared pool  kkslLoadParentOnLock:lite    83647224
         1 shared pool  init_heap_kfsg               74839336
         1 shared pool  NUMA pool 0                  73181032
         1 shared pool  event statistics per sess    60393984
         1 shared pool  KKSSP                        56685344
         1 shared pool  dbktb: trace buffer          49152000
         1 shared pool  PRTDS                        39276760
         1 shared pool  ges big msg buffers          31869448
         1 shared pool  KGLS                         28888312
         1 shared pool  kglhdusr                     19822008
  • Historical shared pool free memory in each instance. We can use this query to compare when shared pool grow fast.

select HS.BEGIN_INTERVAL_TIME,ss.*
from DBA_HIST_SGASTAT ss ,dba_hist_snapshot hs
where pool=’shared pool’ and name=’free memory’
and SS.SNAP_ID=HS.SNAP_ID
and SS.INSTANCE_NUMBER=HS.INSTANCE_NUMBER
and ss.instance_number=1
–and HS.BEGIN_INTERVAL_TIME between to_date(’17-09-2019 13:00:00′,’dd-mm-yyyy hh24:mi:ss’) and to_date(’17-09-2019 15:30:00′,’dd-mm-yyyy hh24:mi:ss’)
order by ss.snap_id desc;

BEGIN_INTERVAL_TIME        NAME            POOL              BYTES
-------------------------- ---------------------------- ----------
17/09/2019 15:15:52,808    free memory     shared pool  1062792200
17/09/2019 15:00:27,628    free memory     shared pool  1065576904
17/09/2019 14:46:00,338    free memory     shared pool  1070302504
17/09/2019 14:30:31,065    free memory     shared pool  1082045840
17/09/2019 14:15:11,962    free memory     shared pool  1094113208
17/09/2019 14:00:21,753    free memory     shared pool  1101234304
17/09/2019 13:45:32,422    free memory     shared pool  1110825320
17/09/2019 13:30:15,446    free memory     shared pool  1119258664
17/09/2019 13:15:45,254    free memory     shared pool  1131666016
17/09/2019 13:00:11,987    free memory     shared pool  1140481080

select HS.BEGIN_INTERVAL_TIME,ss.*
from DBA_HIST_SGASTAT ss ,dba_hist_snapshot hs
where pool=’shared pool’ and name=’free memory’
and SS.SNAP_ID=HS.SNAP_ID
and SS.INSTANCE_NUMBER=HS.INSTANCE_NUMBER
and ss.instance_number=2
–and HS.BEGIN_INTERVAL_TIME between to_date(’17-09-2019 13:00:00′,’dd-mm-yyyy hh24:mi:ss’) and to_date(’17-09-2019 15:30:00′,’dd-mm-yyyy hh24:mi:ss’)
order by ss.snap_id desc;

BEGIN_INTERVAL_TIME        NAME            POOL              BYTES
------------------------------------------------------- ----------
17/09/2019 15:15:52,735    free memory     shared pool  1006936880
17/09/2019 15:00:27,675    free memory     shared pool  1023507584
17/09/2019 14:46:00,372    free memory     shared pool  1036382744
17/09/2019 14:30:31,037    free memory     shared pool  1049451160
17/09/2019 14:15:12,002    free memory     shared pool  1065164048
17/09/2019 14:00:21,703    free memory     shared pool  1072838016
17/09/2019 13:45:32,494    free memory     shared pool  1083137664
17/09/2019 13:30:15,419    free memory     shared pool  1093108792
17/09/2019 13:15:45,217    free memory     shared pool  1105843808
17/09/2019 13:00:11,963    free memory     shared pool  1119044688

 

  • Script to find the reasons why sql has high version count.

select version_count,a.sql_id,hash_value,parsing_schema_name,reason,sql_text from (
select
address,sql_id,”
||decode(max( UNBOUND_CURSOR),’Y’, ‘ UNBOUND_CURSOR’)
||decode(max( SQL_TYPE_MISMATCH),’Y’, ‘ SQL_TYPE_MISMATCH’)
||decode(max( OPTIMIZER_MISMATCH),’Y’, ‘ OPTIMIZER_MISMATCH’)
||decode(max( OUTLINE_MISMATCH),’Y’, ‘ OUTLINE_MISMATCH’)
||decode(max( STATS_ROW_MISMATCH),’Y’, ‘ STATS_ROW_MISMATCH’)
||decode(max( LITERAL_MISMATCH),’Y’, ‘ LITERAL_MISMATCH’)
–||decode(max( SEC_DEPTH_MISMATCH),’Y’, ‘ SEC_DEPTH_MISMATCH’)
||decode(max( EXPLAIN_PLAN_CURSOR),’Y’, ‘ EXPLAIN_PLAN_CURSOR’)
||decode(max( BUFFERED_DML_MISMATCH),’Y’, ‘ BUFFERED_DML_MISMATCH’)
||decode(max( PDML_ENV_MISMATCH),’Y’, ‘ PDML_ENV_MISMATCH’)
||decode(max( INST_DRTLD_MISMATCH),’Y’, ‘ INST_DRTLD_MISMATCH’)
||decode(max( SLAVE_QC_MISMATCH),’Y’, ‘ SLAVE_QC_MISMATCH’)
||decode(max( TYPECHECK_MISMATCH),’Y’, ‘ TYPECHECK_MISMATCH’)
||decode(max( AUTH_CHECK_MISMATCH),’Y’, ‘ AUTH_CHECK_MISMATCH’)
||decode(max( BIND_MISMATCH),’Y’, ‘ BIND_MISMATCH’)
||decode(max( DESCRIBE_MISMATCH),’Y’, ‘ DESCRIBE_MISMATCH’)
||decode(max( LANGUAGE_MISMATCH),’Y’, ‘ LANGUAGE_MISMATCH’)
||decode(max( TRANSLATION_MISMATCH),’Y’, ‘ TRANSLATION_MISMATCH’)
–||decode(max( ROW_LEVEL_SEC_MISMATCH),’Y’, ‘ ROW_LEVEL_SEC_MISMATCH’)
||decode(max( INSUFF_PRIVS),’Y’, ‘ INSUFF_PRIVS’)
||decode(max( INSUFF_PRIVS_REM),’Y’, ‘ INSUFF_PRIVS_REM’)
||decode(max( REMOTE_TRANS_MISMATCH),’Y’, ‘ REMOTE_TRANS_MISMATCH’)
||decode(max( LOGMINER_SESSION_MISMATCH),’Y’, ‘ LOGMINER_SESSION_MISMATCH’)
||decode(max( INCOMP_LTRL_MISMATCH),’Y’, ‘ INCOMP_LTRL_MISMATCH’)
||decode(max( OVERLAP_TIME_MISMATCH),’Y’, ‘ OVERLAP_TIME_MISMATCH’)
–||decode(max( SQL_REDIRECT_MISMATCH),’Y’, ‘ SQL_REDIRECT_MISMATCH’)
||decode(max( MV_QUERY_GEN_MISMATCH),’Y’, ‘ MV_QUERY_GEN_MISMATCH’)
||decode(max( USER_BIND_PEEK_MISMATCH),’Y’, ‘ USER_BIND_PEEK_MISMATCH’)
||decode(max( TYPCHK_DEP_MISMATCH),’Y’, ‘ TYPCHK_DEP_MISMATCH’)
||decode(max( NO_TRIGGER_MISMATCH),’Y’, ‘ NO_TRIGGER_MISMATCH’)
||decode(max( FLASHBACK_CURSOR),’Y’, ‘ FLASHBACK_CURSOR’)
||decode(max( ANYDATA_TRANSFORMATION),’Y’, ‘ ANYDATA_TRANSFORMATION’)
–||decode(max( INCOMPLETE_CURSOR),’Y’, ‘ INCOMPLETE_CURSOR’)
||decode(max( TOP_LEVEL_RPI_CURSOR),’Y’, ‘ TOP_LEVEL_RPI_CURSOR’)
||decode(max( DIFFERENT_LONG_LENGTH),’Y’, ‘ DIFFERENT_LONG_LENGTH’)
||decode(max( LOGICAL_STANDBY_APPLY),’Y’, ‘ LOGICAL_STANDBY_APPLY’)
||decode(max( DIFF_CALL_DURN),’Y’, ‘ DIFF_CALL_DURN’)
||decode(max( BIND_UACS_DIFF),’Y’, ‘ BIND_UACS_DIFF’)
||decode(max( PLSQL_CMP_SWITCHS_DIFF),’Y’, ‘ PLSQL_CMP_SWITCHS_DIFF’)
||decode(max( CURSOR_PARTS_MISMATCH),’Y’, ‘ CURSOR_PARTS_MISMATCH’)
||decode(max( STB_OBJECT_MISMATCH),’Y’, ‘ STB_OBJECT_MISMATCH’)
–||decode(max( ROW_SHIP_MISMATCH),’Y’, ‘ ROW_SHIP_MISMATCH’)
||decode(max( PQ_SLAVE_MISMATCH),’Y’, ‘ PQ_SLAVE_MISMATCH’)
||decode(max( TOP_LEVEL_DDL_MISMATCH),’Y’, ‘ TOP_LEVEL_DDL_MISMATCH’)
||decode(max( MULTI_PX_MISMATCH),’Y’, ‘ MULTI_PX_MISMATCH’)
||decode(max( BIND_PEEKED_PQ_MISMATCH),’Y’, ‘ BIND_PEEKED_PQ_MISMATCH’)
||decode(max( MV_REWRITE_MISMATCH),’Y’, ‘ MV_REWRITE_MISMATCH’)
||decode(max( ROLL_INVALID_MISMATCH),’Y’, ‘ ROLL_INVALID_MISMATCH’)
||decode(max( OPTIMIZER_MODE_MISMATCH),’Y’, ‘ OPTIMIZER_MODE_MISMATCH’)
||decode(max( PX_MISMATCH),’Y’, ‘ PX_MISMATCH’)
||decode(max( MV_STALEOBJ_MISMATCH),’Y’, ‘ MV_STALEOBJ_MISMATCH’)
||decode(max( FLASHBACK_TABLE_MISMATCH),’Y’, ‘ FLASHBACK_TABLE_MISMATCH’)
||decode(max( LITREP_COMP_MISMATCH),’Y’, ‘ LITREP_COMP_MISMATCH’)
reason
from
v$sql_shared_cursor –where sql_id=’1s4cu90p8sdab’
group by
address,sql_id
) a join v$sqlarea using(address) where version_count>10
order by version_count desc,address
;

Identifying which sessions generate most redo in oracle

finding which sessions generate most redo in oracle

Find which statname you will use to find out. In this situation we will use statistic#=194 that is redo size .

select * from v$statname where name like '%redo size%';

STATISTIC#    NAME                                         CLASS    STAT_ID
----------    -------------------------------------------------- ----------
       194    redo size                                        2 1236385760
       196    redo size for lost write detection               2 3835726666
       197    redo size for direct writes                      2  726277760

Create base table and one minute later create another table to compare it.

create table USERNAME.redo_size_stat_1 as
select S.USERNAME,S.MACHINE,S.TYPE,SS.INST_ID,SS.SID,SS.VALUE as redoBytes
from gv$sesstat ss , gv$session s
where
SS.INST_ID=S.INST_ID
and SS.SID=S.SID
and statistic#=194
and S.TYPE not in (‘BACKGROUND’)
order by SS.VALUE desc;

exec dbms_lock.sleep(60);

create table USERNAME.redo_size_stat_2 as
select S.USERNAME,S.MACHINE,S.TYPE,SS.INST_ID,SS.SID,SS.VALUE as redoBytes
from gv$sesstat ss , gv$session s
where
SS.INST_ID=S.INST_ID
and SS.SID=S.SID
and statistic#=194
and S.TYPE not in (‘BACKGROUND’)
order by SS.VALUE desc;

Compare the tables that created one minute apart.

set lines 500
select * from (
select t1.username,t1.machine,t1.sid,t1.inst_id,round((t2.redoBytes-t1.redoBytes)/(1024*1024),2)||’ MB redo per minute’ as DIFF
from USERNAME.redo_size_stat_1 t1,
USERNAME.redo_size_stat_2 t2
where t1.sid=t2.sid
and t1.inst_id=t2.inst_id
order by t2.redoBytes-t1.redoBytes desc
) a
where rownum<11;

The following results gives us information about which sessions generates most redo in one minute.

USERNAME                       MACHINE                               SID    INST_ID   DIFF                         
------------------------------ ------------------------------ ---------- ----------   -----------------------------
USERNAME2                      DOMAIN\SERVER-03                     3963          2   12,26 MB redo per minute          
USERNAME2                      DOMAIN\SERVER-02                     6526          1   9,18  MB redo per minute           
USERNAME2                      DOMAIN\SERVER-03                     2585          1   8,76  MB redo per minute           
USERNAME2                      DOMAIN\SERVER-03                     6428          1   8,74  MB redo per minute           
USERNAME2                      DOMAIN\SERVER-01                     4389          2   7,74  MB redo per minute           
USERNAME                       DOMAIN\HOST-01                       5183          1   7,02  MB redo per minute           
USERNAME2                      DOMAIN\SERVER-01                     6545          1   6,91  MB redo per minute           
USER2                          DOMAIN\HOSTAPP08                     1182          2   6,86  MB redo per minute           
USERNAME2                      DOMAIN\SERVER-01                     5087          1   6,79  MB redo per minute           
USER2                          DOMAIN\HOSTAPP03                     2398          2   6,5   MB redo per minute

 

This query is the same as above but result gives us redo generate KB per second.

set lines 500
select * from (
select t1.username,t1.machine,t1.sid,t1.inst_id,round((t2.redoBytes-t1.redoBytes)/(1024*60),2)||’ KB redo per second’ as DIFF
from USERNAME.redo_size_stat_1 t1,
USERNAME.redo_size_stat_2 t2
where t1.sid=t2.sid
and t1.inst_id=t2.inst_id
order by t2.redoBytes-t1.redoBytes desc
) a
where rownum<11;

USERNAME                       MACHINE                               SID    INST_ID   DIFF                                                                    
------------------------------ ------------------------------ ---------- ----------   -------------------------------
USERNAME2                      DOMAIN\SERVER-03                     3963          2   209,22 KB redo per second                                                    
USERNAME2                      DOMAIN\SERVER-02                     6526          1   156,6  KB redo per second                                                     
USERNAME2                      DOMAIN\SERVER-03                     2585          1   149,56 KB redo per second                                                    
USERNAME2                      DOMAIN\SERVER-03                     6428          1   149,2  KB redo per second                                                     
USERNAME2                      DOMAIN\SERVER-01                     4389          2   132,13 KB redo per second                                                    
USERNAME                       DOMAIN\HOST-01                       5183          1   119,88 KB redo per second                                                    
USERNAME2                      DOMAIN\SERVER-01                     6545          1   118    KB redo per second                                                       
USER2                          DOMAIN\HOSTAPP08                     1182          2   117,06 KB redo per second                                                    
USERNAME2                      DOMAIN\SERVER-01                     5087          1   115,89 KB redo per second                                                    
USER2                          DOMAIN\HOSTAPP03                     2398          2   110,96 KB redo per second