Mucking around with RAC and FCF

I've been exploring the use of RAC and our Fast Connection Failover (FCF) recently.

After some hiccups related to using incompatible versions, I now have it working.

*** If you are looking to test out RAC, I can thoroughly recommend downloading the Oracle Database 10gR2 RAC VMware images we have on OTN ***

*** I also recommend reading this RAC/FCF tech note put out by Oracle's very own Senor Castro ***

It's great to see it in operation, where you bring down a database instance and observe the connection pool automatically cleaning up the connections that were pointing at the instance. The load-balancing across the instances is also pretty cool to watch.

One thing I found, as I was hitching it all up, was that running the ONSTest utility (published on OTN) in receive mode was helpful to observe the ONS (Oracle Notification Service) that are being sent by the app server and the database. Particularly as an instance is taken up or down on the database service, you see the database send a notification of the action.

** HA event received -- Printing header:
Notification Type:         database/event/service
Affected Components:       null
Affected Nodes:            null
Delivery Time:             1171580840859
Generating Component:      database/rac/service
Generating Node:           raclinux1.us.oracle.com
Generating Process:        raclinux1.us.oracle.com:21131
Notification ID:           raclinux1.us.oracle.com:2113104
Notification Creation Time:1171583159
Cluster ID:                databaseClusterId
Cluster Name:              databaseClusterName
Instance ID:               databaseInstanceId
Instance Name:             databaseInstanceName
Local Only Flag:           FALSE
Body:                      [B@1546e25
** Body length = 95
** Event type = database/event/service
** About to generate Body Block **
The event body contains the following key/value pairs:
Key: VERSION = 1.0
Key: service = RACDB
Key: instance = RACDB1
Key: database = RACDB
Key: host = raclinux1
Key: status = down
Key: reason = user

The app server also sends out regular ONS notifications which should be visible.

** HA event received -- Printing header:
Notification Type:         IAS/PM/PROC_READY
Affected Components:       null
Affected Nodes:            null
Delivery Time:             1171580868171
Generating Component:      IAS/OC4J
Generating Node:           duraace
Generating Process:        1171503227656
Notification ID:           117150322765632626
Notification Creation Time:1171580868171
Cluster ID:                1
Cluster Name:              default
Instance ID:               j2ee.duraace.au.oracle.com
Instance Name:             j2ee.duraace.au.oracle.com
Local Only Flag:           FALSE
PROPERTY ==> GenNum = -1022827834
PROPERTY ==> OPMN_UID = 1934495451
PROPERTY ==> OPMFirst = false

Once you start seeing these combined notifications from the ONSTest utility, then you know you have it wired up together correctly and you're well down the track.

The only thing remaining from there is to configure the connection-pool to use the appropriate URL for RAC access and to enable fastConnectionFailover and implicitConnectionCaching. Here's an example entry for data-sources.xml that points to the RAC database.

<managed-data-source connection-pool-name="RACDB"
                    jndi-name="jdbc/RACDS"
                    name="RACDS"/>
<connection-pool name="RACDB"
                initial-limit="10"
                max-connections="30"
                min-connections="5">
<connection-factory
  factory-class="oracle.jdbc.pool.OracleDataSource"
  user="system" password="oracle"
  url="jdbc:oracle:thin:@
     (DESCRIPTION=
       (LOAD_BALANCE=ON)
       (ADDRESS=
         (PROTOCOL=TCP)
         (HOST=192.168.203.11)
         (PORT=1521))
       (ADDRESS=
         (PROTOCOL=TCP)
         (HOST=192.168.203.111)
         (PORT=1521))
       (CONNECT_DATA=
         (SERVICE_NAME= RACDB)))">
  <property name="connectionCachingEnabled" value="true"/>
     <property name="fastConnectionFailoverEnabled" value="true"/>
</connection-factory>
</connection-pool>

Once you have that all done , you can then enable JDBC logging (that needs another blog on its on right) and view the handler activity when an instance is taken down.

Below is a truncated and modified for readability sample of the log output.

OracleFailoverEventHandlerThread.handleEvent():<database/event/service>
oracle.jdbc.pool.OracleConnectionCacheManager verifyAndHandleEvent
eventBody=
<VERSION=1.0
 service=RACDB
 instance=RACDB1
 database=RACDB
 host=raclinux1
 status=down
 reason=user>
oracle.jdbc.pool.OracleImplicitConnectionCache processFailoverEvent
OracleImplicitConnectionCache.processFailoverEvent(
 eventType=256,
 instName=RACDB1,
 dbUniqName=RACDB,
 hostName=raclinux1,
 status=down,
 card=0)
oracle.jdbc.pool.OracleImplicitConnectionCache markDownLostConnections
OracleImplicitConnectionCache.markDownLostConnections(
 serviceDownEvent=true
 hostDownEvent=false,
 instName=RACDB1,
 dbUniqName=RACDB,
 hostName=raclinux1,
 status=down)
oracle.jdbc.pool.OracleImplicitConnectionCache cleanupFailoverConnections
OracleImplicitConnectionCache.cleanupFailoverConnections(
 serviceDownEvent=true
 hostDownEvent=false,
 instName=RACDB1,
 dbUniqName=RACDB,
 hostName=raclinux1,status=down)
oracle.jdbc.pool.OracleImplicitConnectionCache doForEveryCachedConnection
oracle.jdbc.pool.OracleImplicitConnectionCache performPooledConnectionTask
oracle.jdbc.pool.OracleImplicitConnectionCache closeAndRemovePooledConnection