Accessibility Preferences My Oracle Support Logo

Neil (Available)

Last Refreshed: Aug 11, 2020 6:20 PM NZST

(0)

Help

Give Feedback...

Installation of Oracle Clusterware 12c(12cR1 & 12cR2) on SLES 12 fails with CRS-8503 [Signal / Exception: 11] (Doc ID 2270947.1)

In this Document

Applies to:

Oracle Database - Standard Edition - Version 12.1.0.2 and later
Oracle Database - Enterprise Edition - Version 12.1.0.2 to 12.1.0.2 [Release 12.1]
Oracle Database Cloud Schema Service - Version N/A and later
Oracle Database Exadata Cloud Machine - Version N/A and later
Oracle Cloud Infrastructure - Database Service - Version N/A and later
Linux x86-64

Symptoms

Installation of Oracle Clusterware 12.2.0.1 on SLES 12 SP4 fails with below error,when running root.sh

2017-05-09 14:44:51: Starting CSS in exclusive mode
2017-05-09 14:44:51: Configured CRS Home: $GRID_HOME
2017-05-09 14:44:51: Executing cmd: $GRID_HOME/bin/crsctl start crs -excl -cssonly
2017-05-09 14:44:55: Command output:

> CRS-2672: Try to start "ora.evmd" on "<nodename1>"
> CRS-2672: Try to start "ora.mdnsd" on "<nodename1>"
> CRS-2676: Starting "ora.mdnsd" on "<nodename1>" successfully
> CRS-2676: Starting "ora.evmd" on "<nodename1>" successfully
> CRS-2672: Try to start "ora.gpnpd" on "los-oraracn1"
> CRS-2676: Starting "ora.gpnpd" on "<nodename1>" successful
> CRS-2672: Try to start "ora.cssdmonitor" on "<nodename1>"
> CRS-2672: Try to start "ora.gipcd" on "<nodename1>"
> CRS-2676: Starting "ora.cssdmonitor" on "<nodename1>" successfully
> CRS-2674: Starting "ora.gipcd" on "<nodename1>" not successful
> CRS-2679: Try to clean up "ora.gipcd" on "<nodename1>"
> CRS-2681: Clean up of "ora.gipcd" to "<nodename1>" successfully
> CRS-2673: Try to stop "ora.cssdmonitor" on "<nodename1>"
> CRS-2677: Stopping "ora.cssdmonitor" on "<nodename1>" successful
> CRS-2673: Try to stop "ora.mdnsd" on "<nodename1>"
> CRS-2677: Stopping "ora.mdnsd" on "<nodename1>" successful
> CRS-5804: Error in communication with agent process
> CRS-4000: Start command not successful or completed with errors.
>End Command output

2017-05-09 14:44:55: CRS-2672: Try to start "ora.evmd" on "<nodename1>"
2017-05-09 14:44:55: CRS-2672: Try to start "ora.mdnsd" on "<nodename1>"
2017-05-09 14:44:55: CRS-2676: Starting "ora.mdnsd" on "<nodename1>" successfully
2017-05-09 14:44:55: CRS-2676: Starting "ora.evmd" on "<nodename1>" successfully
2017-05-09 14:44:55: CRS-2672: Try to start "ora.gpnpd" on "<nodename1>"
2017-05-09 14:44:55: CRS-2676: Starting "ora.gpnpd" on "<nodename1>" successful
2017-05-09 14:44:55: CRS-2672: Try to start "ora.cssdmonitor" on "<nodename1>"
2017-05-09 14:44:55: CRS-2672: Try to start "ora.gipcd" on "<nodename1>"
2017-05-09 14:44:55: CRS-2676: Starting "ora.cssdmonitor" on "<nodename1>" successfully
2017-05-09 14:44:55: CRS-2674: Starting "ora.gipcd" on "<nodename1>" failed successfully
2017-05-09 14:44:55: CRS-2679: Try to clean up "ora.gipcd" on "<nodename1>"
2017-05-09 14:44:55: CRS-2681: Clean up "ora.gipcd" on "<nodename1>" successfully
2017-05-09 14:44:55: CRS-2673: Try to stop "ora.cssdmonitor" on "<nodename1>"
2017-05-09 14:44:55: CRS-2677: Stopping "ora.cssdmonitor" on "<nodename1>" successfully
2017-05-09 14:44:55: CRS-2673: Try to stop "ora.mdnsd" on "<nodename1>"
2017-05-09 14:44:55: CRS-2677: Stopping "ora.mdnsd" on "<nodename1>" successfully
2017-05-09 14:44:55: CRS-5804: Error in communication with agent process
2017-05-09 14:44:55: CRS-4000: Command start not successful or completed with errors.

2017-05-09 14:44:55: The exlusive mode cluster start failed, see Clusterware alert log for more information
2017-05-09 14:44:55: Executing cmd: $GRID_HOME/bin/clsecho -p has -f clsrsc -m 119
2017-05-09 14:44:55: Command output:
> CLSRSC-119: Start of the exclusive mode cluster failed

From GI Alert.log

2017-05-10 14: 19: 47.702 [CLSECHO (17912)] CLSRSC-0567: Beginning Oracle Grid Infrastructure configuration.
2017-05-10 14: 20: 35.332 [OCRCONFIG (22865)] CRS-2101: OLR has been formatted with version 4.
2017-05-10 14: 21: 17.717 [OHASD (23972)] CRS-8500: The Oracle Clusterware OHASD process starts with the operating system process ID 23972
2017-05-10 14: 21: 17.718 [OHASD (23972)] CRS-0714: Oracle Clusterware Release 12.2.0.1.0.
2017-05-10 14: 21: 17.722 [OHASD (23972)] CRS-2112: OLR service started on node los-oraracn1.
2017-05-10 14: 21: 17.730 [OHASD (23972)] CRS-1301: Oracle High Availability Service started on node los-oraracn1.
2017-05-10 14: 21: 38.953 [CLSECHO (24692)] OKA-0620: OK is not supported on this operating system version: '4.4.49-92.14-default SP2'
2017-05-10 14: 21: 57.605 [OHASD (25699)] CRS-8500: Oracle Clusterware's OHASD process starts with the operating system process ID 25699
2017-05-10 14: 21: 57.606 [OHASD (25699)] CRS-0714: Oracle Clusterware Release 12.2.0.1.0.
2017-05-10 14: 21: 57.611 [OHASD (25699)] CRS-2112: OLR service started on node los-oraracn1.
2017-05-10 14: 21: 57.618 [OHASD (25699)] CRS-1301: Oracle High Availability Service started on node los-oraracn1.
2017-05-10 14: 21: 57.837 [ORAAGENT (25806)] CRS-8500: The ORAAGENT process of the Oracle Clusterware starts with the operating system process ID 25806
2017-05-10 14: 21: 57.854 [CSSDMONITOR (25832)] CRS-8500: The Oracle Clusterware CSSDMONITOR process starts with the operating system process ID 25832
2017-05-10 14: 21: 57.854 [CSSDAGENT (25827)] CRS-8500: Oracle Clusterware's CSSDAGENT process starts with operating system process ID 25827
2017-05-10 14: 21: 57.862 [ORAROOTAGENT (25819)] CRS-8500: The ORAROOTAGENT process of Oracle Clusterware starts with the operating system process ID 25819
2017-05-10 14: 21: 57.898 [ORAROOTAGENT (25819)] CRS-8503: The ORAROOTAGENT Oracle Clusterware process with operating system process ID 25819 received a fatal error or exception code 11.
2017-05-10T14: 21: 58.064744 + 02: 00
Errors in file $TRACE/ohasd_orarootagent_root.trc (incident = 985):
CRS-8503 [Signal / Exception: 11] [Instruction Addr: 0x7f1ec2deb4a0] [Memory Addr: (nil)] [] [] [] [] [] [] [] []
Incident details in: $INCIDENT/ohasd_orarootagent_root_i985.trc

2017-05-10 14: 21: 58.424 [ORAROOTAGENT (25878)] CRS-8500: The ORAROOTAGENT process of the Oracle Clusterware starts with the operating system process ID 25878
2017-05-10 14: 21: 58.455 [ORAROOTAGENT (25878)] CRS-8503: The ORAROOTAGENT Oracle Clusterware process with operating system process ID 25878 received a fatal error or exception code 11.
2017-05-10T14: 21: 58.489246 + 02: 00
Errors in file$TRACE/ohasd_orarootagent_root.trc (incident = 993):
CRS-8503 [Signal / Exception: 11] [Instruction Addr: 0x7f91731a84a0] [Memory Addr: (nil)] [] [] [] [] [] [] [] []
Incident details in: $INCIDENT/ohasd_orarootagent_root_i993.trc

2017-05-10 14: 21: 58.839 [ORAROOTAGENT (25904)] CRS-8500: The ORAROOTAGENT process of Oracle Clusterware starts with the operating system process ID 25904
2017-05-10 14: 22: 06.902 [ORAROOTAGENT (25904)] CRS-5822: Agent '$GRID_HOME/bin/orarootagent_root' logged off from server. Details at (: CRSAGF00117 :) {0: 11: 3} in $TRACE/ohasd_orarootagent_root.trc.
[OHASD (26860)] CRS-8500: The Oracle Clusterware OHASD process starts with the operating system process ID 26860
2017-05-10 14: 22: 18.879 [OHASD (26860)] CRS-0714: Oracle Clusterware Release 12.2.0.1.0.
2017-05-10 14: 22: 18.883 [OHASD (26860)] CRS-2112: OLR service started on node los-oraracn1.
2017-05-10 14:22:18,891 [OHASD (26860)] CRS-1301: Oracle High Availability Service started on node los-oraracn1.
2017-05-10 14: 22: 19.348 [ORAAGENT (26951)] CRS-8500: The ORAAGENT process of Oracle Clusterware starts with the operating system process ID 26951
2017-05-10 14: 22: 19.363 [CSSDAGENT (26972)] CRS-8500: Oracle Clusterware's CSSDAGENT process starts with operating system process ID 26972
2017-05-10 14: 22: 19.364 [CSSDMONITOR (26977)] CRS-8500: The Oracle Clusterware CSSDMONITOR process starts with the operating system process ID 26977
2017-05-10 14: 22: 19.372 [ORAROOTAGENT (26969)] CRS-8500: The ORAROOTAGENT process of Oracle Clusterware starts with the operating system process ID 26969
[ORAROOTAGENT (26969)] CRS-8503: The Oracle clusterware process ORAROOTAGENT with operating system process ID 26969 received a fatal error or exception code 11.
2017-05-10T14: 22: 19.445790 + 02: 00
Errors in file $TRACE/ohasd_orarootagent_root.trc (incident = 1001):
CRS-8503 [__lll_unlock_elision () + 48] [Signal / Exception: 11] [Instruction Addr: 0x7f132bc6c4a0] [Memory Addr: (n

ohasd_orarootagent_root_<incident number>.trc shows the following stack trace-

Dump file $INCIDENT/ohasd_orarootagent_root_i985.trc
[TOC00000]
Jump to table of contents
Dump continued from file: $TRACE/ohasd_orarootagent_root.trc
[TOC00001]
CRS-8503 [__lll_unlock_elision()+48] [Signal/Exception: 11] [Instruction Addr: 0x7f1ec2deb4a0] [Memory Addr: (nil)] [] [] [] [] [] [] [] []

========= Dump for incident 985 (CRS 8503 [__lll_unlock_elision()+48]) ========
Starting a Diag Context default dump (level=3)
[TOC00003]
----- START Event Driven Actions Dump ----
---- END Event Driven Actions Dump ----
[TOC00003-END]
[TOC00004]
----- START DDE Actions Dump -----
Executing SYNC actions
Executing ASYNC actions
[TOC00005]
----- START DDE Action: 'dumpFrameContext' (Sync) -----
[TOC00006]
----- START Frame Context DUMP -----
Status: 0
---------------------------------------------------------------------------
[0001] (location=FILE:clsdadr.c LINE:4896 ID:dbgec*):
clsdsigSignal pointer=0x7f1ebd3921e0
---------------------------------------------------------------------------
----- END Frame Context DUMP -----
[TOC00006-END]
----- END DDE Action: 'dumpFrameContext' (SUCCESS, 0 csec) -----
[TOC00005-END]
[TOC00007]
----- START DDE Action: 'dumpDiagCtx' (Sync) -----
[TOC00008]
------- Diag Context Dump -------
----- General DiagCtx Dump -----
Serial#: 0
Status: 3
Flags: 0x21D
Usage type: 0
Diag Fixed SGA information:
Status: 2
Flags: 0x0
Incident Flag: UNSET
Heap: 0x7f1ea8001170
Flood Control information:
Initialized: NO
Enabled: NO
Diag init parameters:
Flags: 0x1
Base Flag: 0x0
Options: 0xC0820
ADR option: 1
ADR Base: '/u01/app/oracle'
Product Type: 3
Product ID: 'los-oraracn1'
Instance ID: 'crs'
Flood Control Gate: (nil)
Flood Control Ctx size: 0
ADR default lib ID: 5
UTS in mem trace enabled: TRUE
UTS default bucket size: 4294967295
UTS default bucket: 0x29fda50
Incident ID cache cbf: (nil)
Incident ID cache pop init: FALSE
UTS default trace file attr:
Flags: 0x1
Prefix: 'ohasd_orarootagent_root'
PIDStr: ''
Suffix: ''
Notify cbk: 0x7f1ec79b87fe
Size limits: 18446744073709551615
Size cbk: (nil)
DDE alive update cbk: (nil)
DDE alive start cbk: (nil)
DDE alive clear cbk: (nil)
Custom Flood Control cbk: (nil)
----- END General DiagCtx Dump -----
----- DDE Diagnostic Information Dump -----
Depth: 1
DDE flags: 0x0
Heap: 0x7f1ea8014110
Incident Context pointer in diag: 0x7f1ebd38ae60
Incident ID Cache: 0x7f1ea801dff0 (PGA)
Invocation Context #: 0
----- Invocation Context Dump -----
Address: 0x7f1ea8024110
Phase: 3
flags: 0x10E0000
Incident ID: 985
Error Descriptor: CRS-8503 [__lll_unlock_elision()+48] [Signal/Exception: 11] [Instruction Addr: 0x7f1ec2deb4a0] [Memory Addr: (nil)] [] [] [] [] [] [] [] []
Error class: 0
Problem Key # of args: 1
Number of actions: 9
----- Incident Context Dump -----
Address: 0x7f1ebd38ae60
Incident ID: 985
Problem Key: CRS 8503 [__lll_unlock_elision()+48]
Error: CRS-8503 [__lll_unlock_elision()+48] [Signal/Exception: 11] [Instruction Addr: 0x7f1ec2deb4a0] [Memory Addr: (nil)] [] [] [] [] [] [] [] []

Changes

Cause

Known issue in SLES 12.glibc in SuSE 12 makes use of a Hardware Lock Elision (HLE) available in newer Intel Processors.

This exposes Bug 25851874 causing OHASD to crash on startup.

Solution

Solution 1

=================

Download fix for bug 25851874 for Linux x86-64 from the below Link.Apply this Patch to the 12.1 or 12.2 Home before running root.sh

https://updates.oracle.com/download/25851874.html

Solution 2

================

1. Assuming root.sh has already failed, deconfigure the failed install as the ROOT user:

Note: Do NOT close out the OUI Window, we will need this to complete the installation after root.sh is successful.

# $GRID_HOME/crs/install/roothas.pl -deconfig -force

2. Modify the /etc/ld.so.conf adding /lib64/noelision as the FIRST entry. It should look similar to the following:

/lib64/noelision
/usr/local/lib64
/usr/local/lib
include /etc/ld.so.conf.d/*.conf
# /lib64, /lib, /usr/lib64 and /usr/lib gets added
# automatically by ldconfig after parsing this file.
# So, they do not need to be listed.

3. Create a link in $GI_HOME/lib for the noelision version of the libpthread library:

# ln -s /lib64/noelision/libpthread-2.19.so $GI_HOME/lib/libpthread.so.0

4. Rerun the root.sh script and complete the installation via the OUI once root.sh has successfully completed.

Solution 3

=================

1. Remove existing 12c Binary Installation Cleanly.

Instructions for removal of Binaries can be found here -

How to Deconfigure/Reconfigure(Rebuild OCR) or Deinstall Grid Infrastructure (Doc ID 1377349.1) ===> D. Grid Infrastructure Deinstall

2. Make the HLE related changes as below

• Create a file /etc/ld.so.conf.d/noelision.conf containing the line "/lib64/noelision".
• Run "ldconfig" to rebuild caches.

3.Reinstall the GI Binaries & Retry running root.sh

References

NOTE:1915729.1 - 12.1.0.2 Grid Infrastructure Oracle Clusterware Diagnostic (traces) and Alert Log Moved to ADR

NOTE:1949541.1 - ora.gipcd not starting due to wrong ownership/permission

BUG:25780860 - CLSRSC-180 FAILED TO REGISTER ORA.ONS VIA SRVCTL ON ORACLE RESTART/SLES12