Oracle Clusterware

Oracle Clusterware is the software which enables the nodes to communicate with each other, allowing them to form the cluster of nodes which behaves as a single logical server. Oracle Clusterware is run by Cluster Ready Services (CRS) consisting of two key components: Oracle Cluster Registry (OCR), which records and maintains the cluster and node membership information; voting disk, which polls for consistent heartbeat information from all the nodes when the cluster is running, and acts as a tiebreaker during communication failures.

The CRS service has four components, each handling a variety of functions: Cluster Ready Services daemon (CRSd), Oracle Cluster Synchronization Service Daemon (OCSSd), Event Volume Manager Daemon (EVMd), and Oracle Process Clusterware Daemon (OPROCd). Failure or death of the CRS daemon can cause node failure, which triggers automatic reboots of the nodes to avoid the corruption of data (due to the possible failure of communication between the nodes), also known as fencing. The CRS daemon runs as "root" (super user) on UNIX platforms and runs as a service on Windows platforms.

The following functions are provided by the Oracle Cluster Ready Services daemon (CRSd):

• CRS is installed and run from a different ORACLE_HOME known as ORA_CRS_HOME, which is independent of ORACLE_HOME.
• CRSd manages the resources like starting and stopping the services and failing-over the application resources. It spawns separate processes to manage application resources.
• CRS daemon has two modes of running. During startup and after a shutdown. During planned clusterware start it is started as ‘reboot’ mode. It is started as ‘restart’ mode after unplanned shutdown.
• In reboot mode it ‘auto’ starts all the resources under its management. In restart mode it prevails the previous state and brings back the resources to it previous state before shutdown
• Manages the Oracle Cluster Registry and stores the current known state in the Oracle Cluster Registry
• Runs as ‘root’ on Unix and ‘LocalSystem’ on windows and automatically restarts in case of failure.
• CRS requires the public interface, private interface and the Virtual IP (VIP) for the operation. All these interfaces should be up and running, and they should be able to ping each other before starting CRS Installation. Without the above network infrastructure CRS cannot be installed.

Oracle Cluster Synchronization Services daemon (OCSSd) provides basic ‘group services’ support. Group Services is a distributed group membership system that allows the applications to coordinate activities to achieve a common result. As such, it provides synchronization services between nodes, access to the node membership information, as well as enabling basic cluster services, including cluster group services and cluster locking. It can also run without integration with vendor clusterware. Failure of OCSSd causes the machine to reboot to avoid a split-brain situation. This is also required in a single instance configuration if Automatic Storage Management (ASM) is used. ASM was a new feature in Oracle 10g. OCSSd runs as the "oracle" user.

The following functions are provided by the Oracle Cluster Synchronization Services daemon (OCSSd):

• ’Group Services’ uses vendor-provided clusterware group services when the latter is available, but is also capable of working independently if that is unavailable
• ‘Lock Services’ provides the basic cluster-wide serialization locking functions, and uses a FIFO mechanism to manage locking
• 'Node Services' uses OCR to store state data, and updates the information during reconfiguration. It also manages the OCR data, which is static otherwise.

The third component in OCS is the Event Volume Management Logger daemon (EVMd). EVMd spawns a permanent child process called "evmlogger" and generates events. The EVMd child process ‘evmlogger’ spawns new children processes on demand and scans the callout directory to invoke callouts. It will restart automatically on failures and death of the EVMd process does not halt the instance. EVMd runs as the "oracle" user.

OPROCd provides the server fencing solution for the Oracle Clusterware. It is the process monitor for Oracle Clusterware and it uses the hang check timer or watchdog timer (depending on the implementation) for the cluster integrity. OPROCd is locked in the memory and runs as a real time process. This sleeps for a fixed time and runs as the "root" user. Failure of the OPROCd process causes the node to restart. OPROCd is so important that even it is being monitored by a process called OCLSOMON and causes a cluster node to reboot if OPROCd is hung.

• Oracle Real Application Clusters Administrator's Guide
• Oracle Database 10g Real Application Clusters Handbook - Oracle Press
• Using srvctl to Manage your 10g RAC Database - includes description of Oracle Clusterware components.