31 May 2006

Establishing remote connections to OC4J with jconsole

I'm struggling with this task at the moment.

What I'm trying to do is to use the jconsole utility provided in JDK5 and establish a remote connection to an OC4J instance.

It works a treat when connecting using its local model -- just start OC4J with the following command:

java -Dcom.sun.management.jmxremote=true -jar oc4j.jar The JVM process running oc4j.jar will appear in the local connection tab and can be connected to.

However the remote connection model is presenting more of a challenge at the moment. Using the advanced tab, you can specify the direct JMX Service URL to connect the remote OC4J instance.

Now since OC4J uses its ORMI protocol for remote JMX connections, jconsole needs to be configured with the appropriate libraries and property settintgs to allow it to work over ORMI.

After a bit of trial and error, I captured what I think is the required command to launch jconsole and allow it to work as a remote OC4J client. The OC4J specific settings

jconsole
-J-Dcom.sun.management.jmxremote.ssl=false

-J-Dcom.sun.management.jmxremote.authenticate=true
-J-Djmx.remote.protocol.provider.pkgs=oracle.oc4j.admin.jmx.remote
-J-Djava.class.path=;%JAVA_HOME%\lib\jconsole.jar;/
%JAVA_HOME%lib\tools.jar;/
%ORACLE_HOME%\j2ee\home\lib\adminclient.jar

Using the advanced connection tab, you can specify the direct JMX Service URL to identify the target server.

To connect remotely to an OC4J process the URL is of the form

service:jmx:rmi://[oc4j-host]:[oc4j-port]



So all systems go .... except pressing the connect button results in an exception being thrown on the command line and the connection failing.

Exception in thread "JConsole.addUrl" java.lang.ClassCastException:[Ljava.lang.String;
at oracle.oc4j.admin.jmx.remote.rmi.RMIJMXConnectorImpl.getDomain(RMIJMXConnectorImpl.java:83)
at oracle.oc4j.admin.jmx.remote.JMXConnectorImpl.connect(JMXConnectorImpl.java:363)
at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:248)
at sun.tools.jconsole.ProxyClient.(ProxyClient.java:117)
at sun.tools.jconsole.ProxyClient.getProxyClient(ProxyClient.java:87)
at sun.tools.jconsole.JConsole$2.run(JConsole.java:410)

Which at least when looking at the sunny side, shows that the client side configuration being used here must be close to the mark as OC4J client is being used and the exceptions are coming from OC4J JMX classes.

On the bummer side, it doesn't work!

I wanted to then see what is being passed into our RMIJMXConnector from jconsole and compare it with what we're expecting.

I hunted around for the source code for the JDK5/jconsole but couldn't seem to find it anywhere. Has anyone seen the source for jconsole?

The Mustang (JDK6) source snapshot release looks to include it, but I'm not using Mustang here.

Anyway, I'll keep digging and see how this can be made to work.


30 May 2006

Multicast Checker

I still get email comments from time-to-time on this utility to verify the availability of multicast on a network.

So I thought I'd post the link on this blog as well.

The OC4J admin_client distribution

With the OC4J 10.1.3 release, we now provide a separate client side administration distribution:

http://download.oracle.com/otn/java/oc4j/1013/oc4j_admin_client_101300.zip

The distribution is approximately 5.5MB in side and contains the admin_client.jar utility and all the necessary libraries to perform command line administration of a remote OC4J instance.



Using this distribution, the admin_client.jar utility can connect to a remote OC4J 10.1.3 or Oracle Application Server 10.1.3 instance and execute any of its management commands.

As a simple example of using this distribution, I can use it to view the list of shared-libraries installed in the OC4J Home instance in an Oracle Application Server 10.1.3 server running at HQ in Redwood shores from my desktop PC here in Australia.
>mkdir oc4j_admin_client
>cd oc4J_admin_client
>jar xf d:\downloads\oc4j_admin_client_101300.zip
>cd j2ee
>cd home
>java -jar admin_client.jar deployer:oc4j:opmn://stadp57.us.oracle.com/home
oc4jadmin welcome1 -listSharedLibraries

Found 23 shared libraries:

Modifiable libraries:
global.libraries version 1.0
global.tag.libraries version 1.0
oracle.persistence version 1.0
oracle.expression-evaluator version 10.1.3
adf.oracle.domain version 10.1.3
adf.generic.domain version 10.1.3

Non-modifiable (system) libraries:
oracle.dms version 3.0
oracle.gdk version 10.1.0_2
oracle.jdbc version 10.1.0_2
oracle.xml version 10.1.0_2
oracle.cache version 10.1.3
oracle.sqlj version 10.1.3
oracle.http.client version 10.1.3
soap version 10.1.3
oracle.jwsdl version 10.1.3
oracle.ws.client version 10.1.3
oracle.xml.security version 10.1.3
oracle.ws.security version 10.1.3
oracle.toplink version 10.1.3
oracle.ws.reliability version 10.1.3
oracle.ws.testpage version 10.1.3
oracle.ws.core version 10.1.3
oracle.wsm version 10.1.3
Any of the commands available in admin_client.jar can be used -- deploy/undeploy/start/stop applications, bind Web modules to web-sites, install/publish/modify shared-libraries.

Using the variants of the Deployer URI available, admin_client.jar can be executed against any of the OC4J 10.1.3 models.

Oc4J Standalone

  • deployer:[oc4j-host]:[ormi-port]

  • deployer:oc4j:duraace.au.oracle.com:23791


Oracle Application Server OC4J single instance

  • deployer:oc4j:opmn://[opmn-host]:[opmn-port]/[oc4j-name]

  • deployer:oc4j:opmn://stadp57.us.oracle.com:6003/home


Oracle Application Server OC4J Group

  • deployer:oc4j:opmn://[opmn-host]:[opmn-port]/[group-name]

  • deployer:oc4j:opmn://stadp57.us.oracle.com:6003/colors

24 May 2006

A bughunt week

We're conducting a bughunt exercise this week with the 10.1.3.1 SOA product. I love 'em because it gives me the ideal excuse to examine and use the product even more than normal, but without the guilts you can sometimes get when doing so.

I'm hoping to take a close loo at the new extensions we've added in the area of group constructs, deployment and management, command line utilities to create server resources, the changes to the ASC screens and finally wire up some common Cluster Topologies from the installation and see how it comes up.

We have a lot of internal people signed up for the exercise -- so this week playing will be fun, but the following weeks of bug triage may not be quite as exciting.

22 May 2006

Locating OC4J Instances via OPMN

A good question came to me the other day.

A partner of ours wanted to be able to establish a set of connections to each of the MBeanServer instances running in each OC4J instance contained within an Oracle Application Server cluster.

For configuration simplicity, they wanted to have administrators identify one location from which the OC4J instances (and thus the MBeanServers) could be identified.

The solution was to make use of the Cluster MBeanServer which runs when an Oracle Application Server cluster topology is created and access it via the OPMN service.

For a little background, Cluster MBeanServer in Oracle Application Server provides a view of the cluster topology via JMX -- enabling the various elements within the Cluster Topology to be discovered and acted upon. It contains a few different types of MBeans which represent the elements within to the Cluster Topology.

The hierachy of MBeans within the Cluster MBeanServer can be seen using the "Cluster MBean Browser" provided by ASC.

From this MBeans, the elements of the Cluster can be identified and acted upon. For example, the J2EEServerGroup MBean acts as a MBean Proxy over the Oc4J instances residing within the Group. Using the J2EEServerGroup MBean therefore enables attributes to be retrieved or operations to be executed on a named MBean on ALL of the OC4J instances residing in the Group.

But getting back to the problem at hand, the partner wanted to be able to identify each OC4J instance in the Cluster and establish a connection to each of the individual MBeanServers running therein.

To do this we can make use of the "JVMProxy" MBean.

This JVMProxy MBean represents an active JavaVM which is running OC4J. As such, it exposes two key attributes we can use to identify the OC4J instance -- node and rmiPort -- and therefore directly connect to the MBeanServer.



Using the Groovy work I've been mucking around with lately, I prototyped the solution in 5 minutes. Here it is in Groovy:

import demo.oc4j.jmx.*;

client = new OC4JClient()
client.connect("service:jmx:rmi:///opmn://duraace/cluster","oc4jadmin","welcome1")
helper = client.helper

println "Creating MBeanServer Collection\n"
mbeanservers = new ArrayList()
for(jvmProxy in helper.getMBeanNamesFromQuery("j2eeType=JVMProxy")) {
jvm = helper.createGroovyMBean(jvmProxy)
serviceurl = "service:jmx:rmi://$jvm.node:$jvm.rmiPort"
mbeanserver = new OC4JClient()
mbeanserver.connect(serviceurl,"oc4jadmin","welcome1")
if(mbeanserver.connected) {
println " Connected to: \"$serviceurl\""
mbeanservers.add(mbeanserver)
} else {
println " Could not connect to : \"$serviceurl\""
}
}

// Work over the MBeanServers and show we are connected, then close

println "\nChecking MBeanServer Collection\n"
for(mbeanserver in mbeanservers) {
println " $mbeanserver"
println " Is connected? : $mbeanserver.connected"
println " MBean count : $mbeanserver.connection.mBeanCount"
println " ... calling close()"
mbeanserver.close()
println " Is connected? : $mbeanserver.connected"
}

Which when executed produces the following output, showing that this approach of discovering the OC4J instances from the Cluster MBeanServer is feasible:

Creating MBeanServer Collection

Connected to: "service:jmx:rmi://duraace:12403"
Connected to: "service:jmx:rmi://duraace:12407"
Connected to: "service:jmx:rmi://duraace:12406"
Connected to: "service:jmx:rmi://duraace:12402"
Connected to: "service:jmx:rmi://duraace:12404"
Connected to: "service:jmx:rmi://duraace:12401"
Connected to: "service:jmx:rmi://duraace:12405"
Connected to: "service:jmx:rmi://dreid-lap:12401"

Checking MBeanServer Collection

Client is connected to: rmi://duraace:12403 oc4jadmin
Is connected? : true
MBean count : 198
... calling close()
Is connected? : false
Client is connected to: rmi://duraace:12407 oc4jadmin
Is connected? : true
MBean count : 144
... calling close()
Is connected? : false
...

So it worked in Groovy. The next step was to translate the above into direct Java and package for execution as a JMX client which I'll post in the next installment.

I don't want to shoot all my blog bikkies at once :)

19 May 2006

Using Groovy for System Management

I've been funking around with Groovy in some of my spare time recently. My unabashed opinion is that Groovy is a wonderful scripting language since it enables me to use all the aspects of my favourite programming language in a very dynamic away.

One of my work mates, The Tugman(tm) -- made mention that Groovy has something called a GroovyMBean. Since I tend to work with the runtime and management areas of OC4J, I thought I'd check it out.

Turns out that the GroovyMBean provides a lovely little proxy over a MBean running on a server, enabling the MBean as if it was a standard object inside of Groovy. The GroovyMBean delegates all the calls made to it through to the underlying MBean it is wrappering.

Well that sounded quite interesting. But what's more interesting is the possibility that this opens up for the configuration and administration of OracleAS/OC4J 10.1.3.

How so?

Well OC4J and OracleAS 10.1.3 use MBeans for a lot of the actual server configuration and management tasks, as well as providing MBeans for all of the standard J2EE applications and artifacts. In fact, the management console in 10.1.3 (Application Server Control) uses JMX and the set of MBeans provided by OC4J to actually perform all of the operations it exposes in the GUI.

Thus armed with the GroovyMBean, it should be possible perform system and application administration and configuration tasks.

After a bit of playing around with it, it turns out it really IS possible to do.

As a simple illustation, here's an example of a script which displays the details of the JDK which is running an instance of OracleAS

client = new OC4JClient()
client.connect("localhost","23791","oc4jadmin","welcome1")

helper = client.helper

println "\n--> Access JVM MBean"
jvm = helper.createGroovyMBean("oc4j:J2EEServer=standalone,j2eeType=JVM,name=single")

println "\n--> Print selected attributes from JVM MBean"
println ""
println " JVM Vendor : $jvm.javaVendor"
println " JVM Version : $jvm.javaVersion"


Which when run shows:

--> Access JVM MBean

--> Print selected attributes from JVM MBean

JVM Vendor : Sun Microsystems Inc.
JVM Version : 1.5.0_04


This Groovy script simply establishes a connection to the MBeanServer running in the specified OracleAS instance, creates a GroovyMBean for the specified MBean and then displays some attributes from the MBean.

A close look at the script shows the use of two additional classes I developed when developing scripts.

  • OC4JClient -- this class makes establishing the connection to the MBeanServer running OracleAS/OC4J instance breeze. All you need to do is tell it where to connect to the server and a username and password to use. It will take care of all the details of creating the required JMX remote connection.

  • OC4JMBeanHelper -- this is the object type returned from client.helper property. The OC4JMBeanHelper provides a set of convenience methods for working with the set of OC4J System MBeans, for creating GroovyMBeans, etc.
While these additional classes are certainly not required, using them does reduce the amount of work required to work with the MBeans, reducing it down to simple scripting calls.

This is a simple illustation of a Groovy management script, but don't let it fool you. The exact same concepts can be applied using any of the OracleAS/OC4J MBeans to build up more powerful and wide ranging administration scripts.

As a further simple example, lets say you want to find out the running status of all the applications deployed to an OracleAS/OC4J instance. With groovy this is as simple as the below.

client = new OC4JClient()
client.connect("localhost","23791","oc4jadmin","welcome1")
helper = client.helper

// Print a header
30.times { print "----" }
print "\nName "
9.times { print "\t" }
println "State \t Startup \t"
30.times { print "----" }
println ""

// List all the application MBeans and print their state and startimes
for (String item : helper.AllApplicationMBeanNames ) {
app = helper.createGroovyMBean(item)
println "\n$app.objectName \t" + helper.decodeState(app.state) +"\t "+ new Date(app.startTime)
}
30.times { print "----" }
println ""

client.close()


Which when executed shows:
-------------------------------------------------------------------------------------------------------------
Name State Startup
-------------------------------------------------------------------------------------------------------------

oc4j:j2eeType=J2EEApplication,name=archer,J2EEServer=standalone RUNNING Fri May 19 11:19:02 CST 2006

oc4j:j2eeType=J2EEApplication,name=ascontrol,J2EEServer=standalone RUNNING Fri May 19 11:19:02 CST 2006

oc4j:j2eeType=J2EEApplication,name=default,J2EEServer=standalone RUNNING Fri May 19 11:19:01 CST 2006

oc4j:j2eeType=J2EEApplication,name=exploded,J2EEServer=standalone RUNNING Fri May 19 11:19:02 CST 2006

oc4j:j2eeType=J2EEApplication,name=system,J2EEServer=standalone RUNNING Fri May 19 11:19:00 CST 2006
-------------------------------------------------------------------------------------------------------------


Now want to see the configured users for each application?


client = new OC4JClient()
client.connect("localhost","23791","oc4jadmin","welcome1")
helper = client.helper

sysname = helper.getMBeanNameForJ2EEApplication("system")
sysapp = helper.createGroovyMBean(sysname)

println "System Users"
for(user in sysapp.users) {
println " $user.name"
}
client.close()


Which shows:
System Users
JtaAdmin
steve
oc4jadmin
anonymous
j2ee


The way I see it is that given administrators have common and/or regular tasks to perform, with type of direct scripting access to the OC4J System MBeans it provides them with the ability to capture those tasks in scripts.

Which can then be executed when needed, stored in a SCCS for posterity, shared with other users and administrators, etc.

I'll be posting a lot more about this over the coming weeks and plan to put the helper classes, some more detailed documentation and a set of starter scripts onto OTN over the next few weeks.

16 May 2006

InstanceTrackingFilter -- a bit of spit and polish

As a bit of an extension to the InstanceTrackingFilter, I added a small piece of code to its init() method so as to allow the names of the elements inserted into the HttpSession to be provided, as well as enabling the JMX notification functionality to be turned on or off.

The parameters are passed in as context-param and init-param values in the web.xml file.

The parameter names and behaviours are:
  • SEND_JMX_NOTIFICATION: [true | false] specifies whether to broadcast JMX notifications when a session failover is detected. Will also gate whether to register the NotifierMBean.

  • SESSION_IDENTIFIER: specifies the name of the attribute injected into the HttpSession which contains the InstanceIdentifier object

  • SESSION_FAILOVER: specifies the name of the attribute injected into the HttpSesison which reflects whether a session failover has occurred

When used in the web.xml file they appear as follows:
  <context-param>
<param-name>SEND_JMX_NOTIFICATION</param-name>
<param-value>true</param-value>
</context-param>

<filter>
<filter-name>InstanceTrackingFilter</filter-name>
<filter-class>sab.demo.cluster.InstanceTrackingFilter</filter-class>
<param-name>SESSION_IDENTIFIER</param-name>
<param-value>INSTANCE_IDENTIFIER</param-value>
</init-param>
<init-param>
<param-name>SESSION_FAILOVER</param-name>
<param-value>FAILOVER_OCCURRED</param-value>
</init-param>
</filter>
<filter-mapping>
<filter-name>InstanceTrackingFilter</filter-name>
<url-pattern>/*</url-pattern>
</filter-mapping>


This gives an administrator or developer the ability to control how the parameters are put into the HttpSession -- perhaps to either to avoid conflicts with other session attribute names used by the application itself or to specify a specific attribute name the application wants to use if it is reading the objects placed into the session by the InstanceTrackingFilter.

Reading these parameters is very easy to do:
    private void readConfigurationParameters() {
String filterName = _filterConfig.getFilterName();
// Note this is being read from the ServletContext -- this is to enable
// one common place to specify this setting which both the
// ServletContextListener and ServletFilter can access.
String jmxNotifications = _filterConfig.getServletContext().getInitParameter("SEND_JMX_NOTIFICATIONS");

// These parameters are being read from the filter/init-param section
// of web.xml
String sessionIdentiferParam = _filterConfig.getInitParameter("SESSION_IDENTIFIER");
String sessionFailoverParam = _filterConfig.getInitParameter("SESSION_FAILOVER");

if (jmxNotifications != null &&
jmxNotifications.equalsIgnoreCase("false")) {
ENABLE_JMX = false;
}

if(sessionFailoverParam!=null &&
!sessionFailoverParam.equalsIgnoreCase("")) {
SESSION_FAILOVER = sessionFailoverParam;
}

if(sessionIdentiferParam != null &&
sessionIdentiferParam.equalsIgnoreCase("")) {
SESSION_IDENTIFIER = sessionIdentiferParam;
}

debug("ENABLE_JMX: " + ENABLE_JMX);
debug("SESSION_FAILOVER: " + SESSION_FAILOVER);
debug("SESSION_IDENTIFIER: " + SESSION_IDENTIFIER);
}

With that in place the InstanceTrackingFilter is just about ready to use!

There's one final little piece to this which I haven't written up -- in order to dynamically register the MBean used to send out the notifications, I used a ServletContextListener. The benefit of using this over doing hte registration in the ServletFilter init method is that the MBean can be registered/unregistered once in the respective contextInitialized and contextDestroyed methods. Enabling the MBean to be created before the ServletFilter is ever used through setting the appropriate load-on-startup value for the web module. This enables subscriptions to the notifications to be done on each node easily without needing to make sure the ServletFilter has accessed.

Here's the relevant pieces of code from the ServletContextListener

    /**
* The context for the Web module is being initialized. Create and register
* the NotifierMBean if the the configuration says to do do.
* @param event
*/
public void contextInitialized(ServletContextEvent event) {
context = event.getServletContext();
readConfigurationParameters();
if (ENABLE_JMX) {
InstanceTrackingNotifier notifier = new InstanceTrackingNotifier();
try {
MBeanServer mbeanserver =
MBeanServerFactory.createMBeanServer();
ObjectName objectname = new ObjectName(NOTIFIER_MBEAN);
if (!mbeanserver.isRegistered(objectname)) {
mbeanserver.registerMBean(notifier, objectname);
}
} catch (Exception e) {
e.printStackTrace();
}
}

}

/**
* ServletContext is being destroyed, unregister the NotifierMBean.
* @param event
*/
public void contextDestroyed(ServletContextEvent event) {
context = event.getServletContext();

if(ENABLE_JMX) {
try {
MBeanServer mbeanserver =
MBeanServerFactory.createMBeanServer();
ObjectName objectname = new ObjectName(NOTIFIER_MBEAN);
mbeanserver.unregisterMBean(objectname);
} catch (Exception e) {
e.printStackTrace();
}
}

}


And that really is all folks ... :)

15 May 2006

InstanceTrackingFilter -- sending JMX notifications

Following on from my previous post, I wanted to explore how-to somehow some form of data baed on Session failovers in Application Server Control (ASC).

Being interested in JMX, one approach that seemed possible was to use the notification support available within JMX to broadcast when session failovers occur and have ASC subscribe to these notification events so that it received them.

In reality, it turned out to be quite easy to do by adding an MBean to the module containing the ServletFilter and calling into it from the ServletFilter.

A simple notification emitter

To emit JMX notifications a very basic MBean can be used.

The MBean interface is defined as:

public interface InstanceTrackingNotifierMBean {
public void notifyFailover(String sessionId,
InstanceIdentifier previous,
InstanceIdentifier current);
}

The MBean implements this interface. It also subclasses the NotificationBroadcasterSupport class provided in JMX to handle the mechanics of adding/removing listener registrations and sending out the notifications.

Here's the full MBean implementation:
public class InstanceTrackingNotifier
extends NotificationBroadcasterSupport
implements InstanceTrackingNotifierMBean {

public InstanceTrackingNotifier() {
}

private String notifications[] =
{
"Request Failover"
};

/**
* @param sessionId
* @param previous
* @param current
*/
public void notifyFailover(String sessionId,
InstanceIdentifier previous,
InstanceIdentifier current) {

sendNotification(
createNotification(notifications[0],
"Session " + sessionId +
" switched from: " + previous +
" to: " + current));
}


/**
* Creates a new notification object.
* @param desc - the message to put into the notification
* @return the notification
*/
private Notification createNotification(String operation, String desc)
{
return new Notification(operation,
this,
System.currentTimeMillis(),
desc);
}

/**
* Informs any interested party what notifications this MBean emits.
* @return the notifications this MBean emits
*/
public MBeanNotificationInfo[] getNotificationInfo()
{

MBeanNotificationInfo[] info =
{
new MBeanNotificationInfo(notifications,
"javax.management.Notification",
"Notification set for InstanceTracking")
};
return info;
}
}


This MBean exposes an operation called notifyFailover which emits a JMX notification that contains a message detailing the affected client session and which hosts are involved in the switch. It's all fired up and ready to go.

Using the MBean from the ServletFilter
Once the MBean has been developed, all that needs to be done is to make use of it from the client.

The ServletFilter init method is used to lookup the MBean and create a working proxy of it which can be used when a failover is detected, thus emitting a JMX notification.


public void init(FilterConfig filterConfig) throws ServletException {
debug("init");
_filterConfig = filterConfig;
try {
MBeanServer mbeanserver = MBeanServerFactory.createMBeanServer();
ObjectName objectname = new ObjectName(":name=InstanceTrackingNotifier");
notifier = (InstanceTrackingNotifierMBean)
MBeanServerInvocationHandler.newProxyInstance(
mbeanserver,
objectname,
InstanceTrackingNotifierMBean.class,
false);

} catch (Exception e) {
e.printStackTrace();
}
}


Once notifier proxy has been created for the MBean, the notifyFailover operation can be called from the ServletFilter when a failover is detected.


} else {
// Is not the same host so upate the InstanceIdentifier
// Log a message, put notification into session
session.setAttribute(SESSION_FAILOVER, Boolean.TRUE);
session.setAttribute(SESSION_IDENTIFIER, currentInstance);
System.out.println("SessionId: " + session.getId() +
" switched from: " + previousInstance +
" to: " + currentInstance + "");
notifier.notifyFailover(session.getId(),
previousInstance,
currentInstance);
}

Hooking it up with ASC
Once the ServletFilter module has been extended to support the emission of JMX notifications for session failover activity, ASC can register to receive the notifications.

Since the MBean is created by a deployed application, it is accessed via the "Application Defined MBean" icon associated with the specific application.

Once the MBean is visible, the notifications can be seen and subscribed to.



Once the subscription has been made, any time a Session failover is detected, ASC will receive a notification which it displays in the notification list.



So there you have it -- rudimentary Session failover detection, JMX notifications and visibility from ASC.

The InstanceTrackingFilter -- watching for session failovers

Working on a demo the other day to demonstrate application state replication and clustering, I recognized (belatedly) that every time I built a demo in this area, I was always implementing a similar pattern to expose the details of which OC4J instance was servicing the request, whether a session failover had occured, etc.

There are two ways in which I have typically used this information -- for one, a session failover detection can be logged or an event fired at some form of system level; and two it can be injected into the HttpSession itself so that the application *could* make use of it if desired.

Since my demos usually focus on the most common place where this state replication is used -- the Web tier -- a ServletFilter provided a good mechanism to implement the session tracking of client requests. The benefit of using a ServletFilter to do this is that it doesn't need to be tightly integrated into the code of the Web application itself. A ServletFilter can be packaged into a JAR file and then applied to any Web application by putting the JAR file into the WEB-INF/lib directory of the Web application and adding a few tags to the web.xml file to enable the ServletFilter.

Having worked out the loose requirements and a possible approach with which to implement it, the next task was to determine how to get the information from which decisions can be made as to the failover activity.

The InstanceIdentifier

The first thing which needed to be done was to create some identifier of the instance which is servicing the current request. A fairly simplistic but workable approach is to use a combination of the System properties the OracleAS and OC4J components set when they are running, with the use of the InetAddress class from the Java API.
  private String oracleHome = null;
private String j2eeHome = null;
private String oc4jInstance = null;
private String processId = null;
private String hostName = null;

public InstanceIdentifier() {
oracleHome = System.getProperty("oracle.home");
j2eeHome = System.getProperty("oracle.j2ee.home");
try {
hostName = InetAddress.getLocalHost().getHostName();
} catch(Exception e) {
hostName = "N/A";
}

if(System.getProperty("OPMN","false").equals("true")) {
oc4jInstance = System.getProperty("oracle.oc4j.instancename");
processId = System.getProperty("oracle.ons.indexid");
processId = processId.substring(processId.lastIndexOf(".")+1);
} else {
processId="standalone";
oc4jInstance = j2eeHome.substring(j2eeHome.lastIndexOf(File.separator)+1);
}
}

Important to note in here is the applied knowledge of what makes an OC4J instance unique. In an OracleAS environment, it's possible to run multiple OC4J instances from the same OracleAS installation -- and further, its possible to run multiple JVM processes from the same OC4J instance configuration. The InstanceIdentifier attempts to determine/construct the required set of properties to uniquely identifer the specific OC4J instance.

Detecting a Failover
Once an identifier of the current instance servicing the request has been created, its possible to then calculate if a failover has occured by comparing the current instance with the previous instance. To do this, an equals method is added to the InstanceIdentifier class.
  public boolean equals(Object obj) {
if((obj==null) || !(obj instanceof InstanceIdentifier)) {
return false;
}

InstanceIdentifier id = (InstanceIdentifier)obj;

if(id.getOracleHome().equals(this.getOracleHome()) &&
id.getProcessId().equals(this.getProcessId()) &&
id.getJ2eeHome().equals(this.getJ2eeHome()) &&
id.getOc4jInstance().equals(this.getOc4jInstance()) &&
id.getHostName().equals(this.getHostName())) {
return true;
} else {
return false;
}
}


By using a ServletFilter that gets invoked on every request to the Web application which stores an instance of the InstanceIdentifier for the previous request in the HttpSession object, its possible to determine if a failover has occured by comparing it with the InstanceIdentifier from the current request.

public void doFilter(ServletRequest request, ServletResponse response,
FilterChain chain) throws IOException,
ServletException {
/*
* The broad goal of this filter is to detect some details about the
* instance which is servicing the request and insert it into the
* HttpSession. Then on subsequent requests, compare the Instance
* with the old instance and log a note if a failover has occurred.
*/

if ((request instanceof HttpServletRequest) &&
(response instanceof HttpServletResponse)) {
HttpSession session = ((HttpServletRequest)request).getSession();

InstanceIdentifier currentInstance = new InstanceIdentifier();
InstanceIdentifier previousInstance =
(InstanceIdentifier)session.getAttribute(SESSION_IDENTIFIER);

if (previousInstance == null) {
// This is the first request so set the InstanceIdentifier
debug("First request received for: " + session.getId());
debug("Session isNew: " + session.isNew());
session.setAttribute(SESSION_IDENTIFIER, currentInstance);
session.setAttribute(SESSION_FAILOVER, Boolean.FALSE);
} else if(previousInstance.equals(currentInstance)) {
session.setAttribute(SESSION_FAILOVER, Boolean.FALSE);
} else {
// Is not the same host so probable is a failover
// Upate the InstanceIdentifier in the session
// Log a message, put notification into session

session.setAttribute(SESSION_FAILOVER, Boolean.TRUE);
session.setAttribute(SESSION_IDENTIFIER, currentInstance);

System.out.println("SessionId: " + session.getId() +
" switched from: " + previousInstance +
" to: " + currentInstance + "");
notifier.notifyFailover(session.getId(),
previousInstance,
currentInstance);
}
}
chain.doFilter(request, response);
}

Just to recap, this ServletFilter method will be invoked on each request when it is configured to run in a Web application. It compares the InstanceIdenfitier of the current request with the InstanceIdentifier from the previous request. If they are the same then its probable a failover has not occurred and the SESSION_FAILOVER is set to FALSE. If the two InstanceIdentifiers are not the same then its probable a failover has occured and the ServletFilter method will set the SESSION_FAILOVER attribute of the HttpSession to TRUE and update the SESSION_IDENTIFIER with the new InstanceIdentifier object.

It's all in the Packaging

That's the crux of the implementation. The rest of it is basically all about how to package and employ the ServletFilter.

Once the ServletFilter has been coded and compiled, it gets packaged into a JAR file. To make use of the instance comparison, the JAR file is put into the WEB-INF/lib directory of a Web application and configured to listen to requests using the WEB-INF/web.xml file.
    <filter>
<filter-name>InstanceTrackingFilter</filter-name>
<filter-class>sab.demo.cluster.InstanceTrackingFilter</filter-class>
</filter>
<filter-mapping>
<filter-name>InstanceTrackingFilter</filter-name>
<url-pattern>/*</url-pattern>
</filter-mapping>

With the ServletFilter enabled in this way, it tracks each request received by the Web application and does its 'thang.

This basic implementation will result in messages being printed to the System.out printstream when a hostname switch is detected.
steve~red~steve~1:06/05/12 01:04:14 
SessionId: 8c570a1e30d8c83b2772d4044562bf70c3813e5b1952 switched from:
stadp57.us.oracle.com:blue[1]@/scratch/sbutton/SOA_M1 to:
stadp57.us.oracle.com:red[1]@/scratch/sbutton/SOA_M1


Making use of it in an application
With the data injected into the HttpSession by the ServletFilter, an application can retrieve it and use it to make decisions. For the usual sorts of demo purposes, this usually entails simply displaying the current Instance serviving the request and whether a failover has occurred.

For example in a JSP it can be minimally used as:
    
Last request: <%=session.getAttribute("XXX_INSTANCE_IDENTIFIER")%>

Failover? <%=session.getAttribute("XXX_FAILOVER")%>


In closing and that's not all folks
That's about it as a first uyp basic exploration of this idea. We can uniquely identify an OC4J instance servicing requests and can be determine if its likely that a failover has occurred.

As a further path of exploration of this idea, I'll next detail how the InstanceTrackingFilter can utilize the notification support in JMX to broadcast notifications when session failovers occur. By exposing and broadcasting JMX notifications, any JMX compatible management client can subscribe to and receive the notifications, enabling it to provide information to administrators when session failovers occur.

09 May 2006

Springing into OC4J MBeans

I noticed a while back that Spring has a funky little mechanism by which it can export its beans as JMX MBeans.

Which pricked my attention since it meant that theoretically with OC4J 10.1.3, any Spring Beans which are configured as part of an application and exported as MBeans, could be accessed through Application Server Control using the Application MBean Browser. Giving developers the power to allow administrators to configure elements of their Spring based applications.

So I tried it out and found that it just bloody worked! Which blew me away. These standards we all operate on are a wonderful thing when they all come together.

Here's an example of a Spring configuration which defines a bean and injects some initial values for its properties:

<bean id="aussieGreetingService"
class="test.spring.beans.GreetingServiceImpl">
<property name="greeting" value="gday mate!"/>
<property name="language" value="Aussie"/>
</bean>

To make this bean configuration available as an MBean then you need to do two extra things in this configuration file.

1. Define a Spring bean which represents an MBeanServer

<bean id="greetingMbeanServer"
class="org.springframework.jmx.support.MBeanServerFactoryBean">
<property name="defaultDomain" value="GreetingService" />
</bean>

Note that this MBeanServer bean creates an MBeanServer using the specified defaultDomain -- this is injected as a property.

2. Specify which of the Spring beans should be exported as MBeans using the MBeanExport Spring bean.


<bean id="exporter"
class="org.springframework.jmx.export.MBeanExporter">
<property name="beans">
<map>
<entry key="bean:name=aussieGreetingService"
value-ref="aussieGreetingService" />
</map>
</property>
<property name="server" ref="greetingMbeanServer" />
</bean>

Note that the name of the MBeanServer in which the MBeans are to be created has to be provided -- and it points to the MBeanServer we created in Step 1.

The MBean will expose all of the relevant properties (JMX:attributes) of the JavaBean as well as the methods (jmx:operations) the JavaBean supports.

And now the cool thing -- if this configuration file is deployed as part of an application with the Spring framework libraries and the corresponding JavaBeans, then the Spring beans will be accessible as JMX MBeans using Application Server Control.

Viewing and setting JavaBean properties:



Viewing JavaBean methods:



Invoking a JavaBean method:



And there you have it -- managing Spring beans using JMX with Application Server Control.

03 May 2006

OC4J Virtual Directory Mapping

The OC4J web module supports setting a virtual directory. The virtual directory allows a web module to serve content from a physical directory outside of the actual deployed web module.

In the orion-web.xml file the tag is of this form:

<orion-web-app>
....
<virtual-directory virtual-path="/bar" real-path="D:\static_content\bar" />
</orion-web-app>

Which means that when the web module is deployed it can serve content directly from the self contained WAR file (and its sub-directories) via the URL of /foo.

But it also means that the web module can now also serve content from the directory specified as the real-path attribute via the URL of /foo/bar.

You can specify this virtual-directory setting using an embedded orion-web.xml in which it is already set OR with the OC4J 10.1.3 release you can use the funky Deployment Plan Editor to specify it at deployment time.










This can be handy for example when you have web modules which need to serve content which is being periodically generated by other services such as a report engine (or graphic designers updating HTML or images) but don't want to lump the content into the exploded directory of the web module.

02 May 2006

Another opmnctl Gem - sequential operations

Following yesterdays theme of handy but less well known opmnctl commands, here's another one I like.

If you want to start the processes OPMN is configured to manage on a sequential basis (ie one process is started and completes its startup before the next one) then you can use the command below.

$opmnctl startproc sequential=true

The usage text describes it succinctly as:

$opmnctl usage startproc

...
If the value of the sequential attribute is "true", each process or application targetted by the request will be acted upon sequentially (one at a time). The request order of affected managed-process dependencies will be honored.

01 May 2006

Hidden Gems in the OPMN opmnctl Utility

The opmnctl command has a few hidden gems. Well I'm sure they're well documented somewhere but I don't see too many people using them.

So my two picks for the day are:

1. Starting and stopping individual applications.

Yes that's right! The opmnctl utility can start and stop individual applications running on an OC4J instance under the control of OPMN.

The command form is:

$opmnctl stopproc application=FOO

where FOO is the name of the application.

This will cause the FOO application to be shutdown and the setting persisted in the OC4J j2ee/home/config/server.xml so that the application will be kept in the last requested state if/when the OC4J instance itself is restarted.

This command is particularly useful if you have wired together a few 10.1.3 instances in which the "Application Server Control" is active on each instance and you receive a warning when you use it that there are multiple instances running. It's a bit of a pain since ASC won't let you stop an ASC application, even if its not the current one. Therefore to stop an instance of ASC from running you can use this nice opmnctl facility.

Go to the $ORACLE_HOME/opmn/bin directory of the instance in which you want to stop ASC and type

$ opmnctl stopproc application=ascontrol
opmnctl: stopping opmn managed processes...

Further, if you have access to one instance of opmnctl which is mapped into the topology then you don't even need to go into the specific $ORACLE_HOME directory of the instance where you want to stop ASC. You can just add an "instance" scope to the command and point it at the instance where you want to stop ASC.

2. Live reporting

The default mode of operation of opmnctl is to report the output of its operations when all of the operations have completed.

If you are like me and like to see results as they happen, then you can give opmnctl an extra switch and it'll report status as it receives it.

You can do this by tacking on the additional switch

$opmnctl report=true

It's a little hard to visualize this, but the following comes in on a line-by-line basis as the individual activities are finishing -- not as a result blob at the very end of the restartproc operation

$opmnctl @cluster restartproc report=true
opmnctl: restarting opmn managed processes...
HTTP_Server/HTTP_Server/HTTP_Server/1,1770851305,3071: success
default_group/home/default_group/1,1770851307,27024: success
multi_group/multi/default_group/2,1770851317,27969: success
multi_group/multi/default_group/1,1770851318,27970: success
060426_ohsj2ee.stadp57.us.oracle.com: 4 of 4 processes restarted.

System Properties Using a Classloader Query

I noticed I had a post here which covers the basic way to dump out the current set of System properties.

Well there's an even easier way with OC4J(10.1.3) by using one of the pre-built classloader queries we have to do it.

The SystemProperties query will dump out a list of the current System properties. Below I show this used in conjuction with the Exit query which basically nukes the server after the queries have returned.

>java -Doc4j.start.query=SystemProperties+Exit -jar oc4j.jar

06/05/01 15:53:47 Shutting down OC4J...
System properties:

1. ajp.connection.listener.state = down
2. awt.toolkit = sun.awt.windows.WToolkit
3. com.sun.CORBA.connection.ORBSocketFactoryClass = oracle.oc4j.corba.iiop.
IIOPSSLSocketFactory
4. file.encoding = Cp1252
5. file.encoding.pkg = sun.io
6. file.separator = 7. java.awt.graphicsenv = sun.awt.Win32GraphicsEnvironment
8. java.awt.printerjob = sun.awt.windows.WPrinterJob
9. java.class.path = oc4j.jar
10. java.class.version = 49.0

... well you get the picture ...