25 June 2008

Scheduling OC4J periodic tasks using java.util.concurrent

A question on OTN last week raised the issue of how to fulfill a requirement to have a task run periodically within OC4J that did some form of directory read/processing ativity.

If take out the actual tasks that need to be performed for the moment, then what we are looking to do here is to run some form of activity on a repeated basis.

One way I thought to tackle this was to make use of the new facilities provided by the java.util.concurrent package. Particularly, this package has a ScheduledExecutorService which allows for tasks to be submitted that are configured to run on a repeated basis.

The ScheduledExecutorService has a method that enables tasks to run on a repeated basis with a fixed delay:

http://java.sun.com/j2se/1.5.0/docs/api/java/util/concurrent/ScheduledExecutorService.html#scheduleWithFixedDelay(java.lang.Runnable,%20long,%20long,%20java.util.concurrent.TimeUnit)

This method takes in a java.lang.Runnable, which encapsulates the steps that need to be performed when the periodic task runs.

It then takes in the data for how the task should be run -- initial delay, the fixed delay and the time unit.

Armed with this, we can create our task by implementing the java.lang.Runnable interface and schedule it for running using the ScheduledExecutorService.

A trivial example of a task for the purpose of testing this could look like this.
package sab.demo.susdtest;

import java.text.DateFormat;
import java.util.Date;

public class MyScheduledTask implements Runnable {

private DateFormat DF = DateFormat.getDateTimeInstance();

public MyScheduledTask() {
}

public void run() {
// Do some long winded thing to simulate "processing and
// provide a place to issue a ctrl+c and show what happens
System.out.printf("[%s] MyTask awake, doing something ....\n", DF.format(new Date()));
String ret = "";
int j=0;
for (int i=0;i<15000;i++) {
ret = ret + Integer.valueOf(i).toString();
j=i;
}
System.out.printf("[%s] MyTask finished processing [%s]....\n",
DF.format(new Date()),
j);
}
}
If we now consider that our task may be performing operations that have open handles to resources that will need to be handled carefully, the ScheduledExecutorService also has a method called shutdown() which will nicely shutdown the ScheduledExecutorService by cancelling all the pending tasks and ultimately calling interrupt on any currently executing task.

Now to wire this into OC4J so that the ScheduledExecutorService is created/started when OC4J starts and it shuts down when OC4J itself is stipped, then one readily apparent place to start is the use of the OC4JStartup and OC4JShutdown classes.

Using the OC4JStartup interface, a class can be created that performs the steps needed to create/start/schedule the task. The ScheduledExecutorService is bound into the Context provided to the class so that it can be accessed in other startup and shutdown classes that are called.
package sab.demo.susdtest;

import oracle.j2ee.server.OC4JStartup;
import java.util.Hashtable;
import java.util.concurrent.Executors;
import java.util.concurrent.ScheduledExecutorService;
import java.util.concurrent.TimeUnit;
import javax.naming.Context;

public class Startup implements OC4JStartup {
private static final int INITIAL_DELAY = 10;
private static final int DELAY = 15;

public Startup() {
}

public String preDeploy(Hashtable hashtable,
Context context) throws Exception {

// Create a ScheduledExecutorService, schedule the task we want to
// regularly run and then bind it into context so it can be accessed
// by other OC4JStartup or OC4JShutdown classes. The delay parameters
// could be provided as config parameters and extracted from the
// hashtable
ScheduledExecutorService executor = Executors.newSingleThreadScheduledExecutor();
executor.scheduleWithFixedDelay(
new MyScheduledTask(), INITIAL_DELAY, DELAY, TimeUnit.SECONDS);
context.bind("task", executor);
return null;
}

public String postDeploy(Hashtable hashtable,
Context context) throws Exception {
return null;
}
}
The OC4JShutdown class implementation class needs to extract the ScheduledExecutorService from the context, issue the shutdown call on it and wait for the service to terminate.
package sab.demo.susdtest;

import java.text.DateFormat;
import java.util.Date;
import java.util.Hashtable;
import java.util.concurrent.Executors;
import java.util.concurrent.ScheduledExecutorService;
import java.util.concurrent.TimeUnit;
import javax.naming.Context;
import oracle.j2ee.server.OC4JShutdown;

public class Shutdown implements OC4JShutdown {

private static final int MAX_DELAY = 60;

DateFormat DF = DateFormat.getDateTimeInstance();

public Shutdown() {
}

public String preUndeploy(Hashtable hashtable,
Context context) throws Exception {

// Extract the ScheduledExecutorService from the context and ask it to
// shutdown, waiting for a max of 60 seconds for it to complete its task
ScheduledExecutorService executor =
(ScheduledExecutorService)context.lookup("task");

if(executor != null ) {
System.out.printf("[%s] Calling executor service shutdown\n", now());
executor.shutdown();
// The max wait time could be obtained as a parameter via the hashtable
System.out.printf("[%s] Waiting on executor service shutdown\n", now());
executor.awaitTermination(MAX_DELAY, TimeUnit.SECONDS);
System.out.printf("[%s] Done with executor service shutdown \n", now());
} else {
System.out.printf("[%s] No executor service found in context, " +
"no shutdown performed....\n", now());
}
return null;
}

public String postUndeploy(Hashtable hashtable,
Context context) throws Exception {
return null;
}

private String now() {
return DF.format(new Date());
}
}
With those classes created, compiled and packaged into a JAR file, you then configure OC4J to make use of them by adding entries into its server.xml file.
        <init-library path="D:\java\jdev-10133\jdev\mywork\susd-test\test\deploy\susdtest.jar" />
<startup-classes>
<startup-class classname="sab.demo.susdtest.Startup" failure-is-fatal="true" >
<execution-order>0</execution-order>
</startup-class>
</startup-classes>
<shutdown-classes>
<shutdown-class classname="sab.demo.susdtest.Shutdown" >
<execution-order>0</execution-order>
</shutdown-class>
</shutdown-classes>
Once all that has been completed, it's time to give it a field test.

Here we start OC4J, wait until the task begins its execution then shutdown OC4J. Note how the container waits for the task to complete before it completes its shutdown fully.
D:\java\oc4j-10133-prod\j2ee\home>java -jar oc4j.jar
08/06/25 11:42:15 Oracle Containers for J2EE 10g (10.1.3.3.0) initialized
[25/06/2008 11:42:21] MyTask awake, doing something ....
Shutting down OC4J...
[25/06/2008 11:42:32] Calling executor service shutdown
[25/06/2008 11:42:32] Waiting on executor service shutdown
[25/06/2008 11:42:35] MyTask finished processing [14999]....
[25/06/2008 11:42:35] Done with executor service shutdown
Here we start OC4J, allow the task to begin and complete, then signal OC4J to shutdown. Since there are no running tasks, the shutdown completes almost immediately and OC4J completes its full shutdown likewise.

D:\java\oc4j-10133-prod\j2ee\home>java -jar oc4j.jar
08/06/25 11:43:16 Oracle Containers for J2EE 10g (10.1.3.3.0) initialized
08/06/25 11:43:22 [25/06/2008 11:43:22] MyTask awake, doing something ....
[25/06/2008 11:43:37] MyTask finished processing [14999]....
Shutting down OC4J...
[25/06/2008 11:43:40] Calling executor service shutdown
[25/06/2008 11:43:40] Waiting on executor service shutdown
[25/06/2008 11:43:40] Done with executor service shutdown

12 June 2008

mod_weblogic

Well it had to happen eventually -- welcome to Weblogic Server :-)

Thought I'd walk through how to configure Apache as the front end server for a Weblogic Server.

Turns out, it's as easy as the proverbial:
  1. Install WLS, create a domain, start it up
  2. Download and install Apache (I used 2.2)
  3. Copy the corresponding Apache module (for 2.2 I used mod_wl_22.so) from the WLS installation into $APACHE_HOME/modules directory
  4. Edit $APACHE_HOME/conf/httpd.conf and enable the mod_weblogic as follows:

    # Load BEA Module
    LoadModule weblogic_module modules/mod_wl_22.so

    # Config file for WebLogic Server that defines the parameters
    Include conf/mod_weblogic.conf

  5. Create the referenced mod_weblogic.conf file in which you place the configuration elements for mod_weblogic to use:

    # Basic config for mod_weblogic to point at my WLS instance

    WebLogicHost localhost
    WebLogicPort 7001
    Debug ON
    WLLogFile c:\temp\wlproxy.log
    WLTempDir c:\temp
    DebugConfigInfo On


    # Map the URLs to expose via Apache
    # Expose the console via Apache

    SetHandler weblogic-handler


  6. Test the config with Apache

    $APACHE_HOME\bin\httpd -t

  7. Start Apache
  8. Access the console via Apache (assuming all default ports)

    http://localhost:8080/console

And that's the basic set of steps -- you should be looking at the WLS console being serviced via Apache.

With the config above, I've enabled the DebugConfigInfo option -- which allows you to see a bunch of useful debug/usage information by entering a URL serviced by mod_weblogic and tacking on a request parameter of the form ?__WebLogicBridgeConfig.

05 June 2008

Classloader Loader Event Tracing

Classloader Event Tracing

If you are seeing classloader closed errors from time to time, then one really useful utility OC4J has is the ability to log the various events related to different aspects of the classloading mechanism.

So for example, lets say you are seeing errors occasionally containing this type of message: "This loader has been closed and should not be in use."

Then what you can do to help trace why this is occurring is to enable log tracing for the various loader events.

In the error message above, it should also explicitly provide the name of the loader that was being used, of the form appname.modulename[.module]:major.minor. For example myapp.webmodule.root:0.0

To observe classloading events relating to this specific loader, you configure OC4J to start as follows:

>java -Dclass.load.trace=loader:myapp.webmodule.* -Dclass.load.trace.level=all -jar oc4j.jar


That setup configures OC4J to output all the loader related classloading events, at the log level of ALL, where the loader name matches the pattern "myapp.webmodule.*".

In the output, you'll see entries such as:

ClassLoader created: myapp.webmodule.root:0.0.0 (instance 1cba87)
ClassLoader committed: myapp.webmodule.root:0.0.0 (instance 1cba87)
...
ClassLoader closing: myapp.webmodule..jsp13471351:0.0.0 (instance 14441b6)
...

That shows you where the loaders are being operated on, and confirms (one way or another) that the loader is actually being closed -- this may correlate with other actions you have performs -- say a redeployment operation.

Now to help further understand why/where the loader event occured, you can add an extra event filter. The "stack" event filter adds the associated stacktrace that caused the loader event to occur to the output -- thus for the loader closed events, you can trace back and identify the source of the loader close.

To add the stack event filter to the classloader tracing, simply concatentate it to the class.load.trace property:
>java -Dclass.load.trace=stack+loader:myapp.webmodule.* -Dclass.load.trace.level=all -jar oc4j.jar

That then causes a stacktrace to be appended to each event that is logged:

ClassLoader closing: myapp.webmodule:0.0.0 (instance 18aea9e)
at oracle.classloader.util.ClassLoadTracing$LoaderListener.loaderClosing(ClassLoadTracing.java:738)
at oracle.classloader.EventDispatcher.loaderClosing(EventDispatcher.java:232)
at oracle.classloader.PolicyClassLoader.close(PolicyClassLoader.java:1056)
at oracle.classloader.PolicyClassLoader.close(PolicyClassLoader.java:1041)
at com.evermind.server.http.HttpApplication.destroy(HttpApplication.java:5860)
at com.evermind.server.http.HttpApplication.stopCleanUp(HttpApplication.java:8437)
at com.evermind.server.http.HttpApplication.componentStop(HttpApplication.java:8383)
at com.evermind.server.Application.doStop(Application.java:2092)
at com.evermind.server.Application.stop(Application.java:2019)
at oracle.oc4j.admin.management.mbeans.J2EEStateManageableObjectBase.stop(J2EEStateManageableObjectBase.java:275)
at oracle.oc4j.admin.management.mbeans.J2EEApplication.stop(J2EEApplication.java:1542)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:36)
at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:239)
at javax.management.modelmbean.RequiredModelMBean.invokeMethod(RequiredModelMBean.java:1071)
at javax.management.modelmbean.RequiredModelMBean.invoke(RequiredModelMBean.java:953)
at oracle.oc4j.admin.jmx.server.mbeans.model.DefaultModelMBeanImpl.invoke(DefaultModelMBeanImpl.java:700)
at com.sun.jmx.mbeanserver.DynamicMetaDataImpl.invoke(DynamicMetaDataImpl.java:213)
at com.sun.jmx.mbeanserver.MetaDataImpl.invoke(MetaDataImpl.java:220)
at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:815)
at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:784)
at oracle.oc4j.admin.jmx.server.state.ApplicationStateFilterMBeanServer.invoke(ApplicationStateFilterMBeanServer.java:572)
at oracle.oc4j.admin.jmx.shared.UserMBeanServer.invoke(UserMBeanServer.java:634)
at oracle.oc4j.admin.jmx.server.state.LocalizationFilterMBeanServer.invoke(LocalizationFilterMBeanServer.java:579)
at oracle.sysman.ias.studio.jmx.spi.JMXConnectorImpl.invoke(JMXConnectorImpl.java:389)
at oracle.sysman.ias.studio.oc4j.JMXAppAdminBean.stopApp(JMXAppAdminBean.java:276)
at oracle.sysman.ias.studio.oc4j.AppHelper.doStopAppOp(AppHelper.java:776)
at oracle.sysman.ias.studio.oc4j.AppHelper.access$600(AppHelper.java:116)
at oracle.sysman.ias.studio.oc4j.AppHelper$AppControlProcessingBean.appControlOperation(AppHelper.java:1628)
at oracle.sysman.ias.studio.oc4j.AppHelper$AppControlProcessingBean.doWork(AppHelper.java:1642)
at oracle.sysman.ias.studio.util.ProcessingBean$ProcessingThread.run(ProcessingBean.java:273)
at com.evermind.util.ReleasableResourcePooledExecutor$MyWorker.run(ReleasableResourcePooledExecutor.java:298)
at java.lang.Thread.run(Thread.java:595)


Looking at the stacktrace for the loader closing event, you can reasonably surmise that the event was caused by the application being "stopped" from ascontrol (sysman).

Now that's a contrived example, I agree! But I hope it serves in some small way to demonstrate how you can use the classloader tracing mechanism to get a handle on the different events that happen to loaders and why they have occurred.

04 June 2008

FAQ for OC4J 10.1.3. Management

A few questions I see from time to time in the OTN OC4J Forum are covered in the OC4J 10.1.3 Management/Admin FAQ

http://www.oracle.com/technology/tech/java/oc4j/1013/whitepapers/OC4J-FAQ-MGT-10131.pdf

17 April 2008

Creating Users in OC4J Groups

Another relatively common question:

The ascontrol group page only allows creation/configuration of JMS and JDBC resources at the group level. Is it possible to create users across all the OC4J instances in Group from ascontrol?

Yes this can be done.

The solution is to use the Cluster MBean Browser provided in ascontrol and execute an operation on the J2EEApplication MBean via the desired J2EEServerGroup MBean.

Luckily its documented in the OC4J Security Guide, so I don't need to butcher the description any more!

http://download.oracle.com/docs/cd/B32110_01/web.1013/b28957/configxml.htm#sthref316

05 March 2008

Starting OC4J from Ant

Good question today from a colleague in Dublin, who basically wanted to know how to start an OC4J instance from Ant using the oc4j:restart.

On the surface that looks like a reasonable thing to do. The problem though is that the OC4J Ant tasks actually require a connection to be made to the server in order to execute the operation -- so if the OC4J instance is down, you can't issue a restart since you can't connect to it.

For situations where the OC4J instance is being managed by OPMN, then one possible solution to this is to call out to the opmnctl executable and ask it to start the OC4J instance.

To do this, you can make use of the ant:exec task as shown below:


<project name="opmn" basedir="." default="status">
<property name="oracle.home" value="d:/oracleas/10131/ohs_j2ee"/>
<property name="opmn.cmd" value="${oracle.home}/opmn/bin/opmnctl.exe"/>

<target name="base" description="Verify callout to opmnctl">
<exec executable="${opmn.cmd}"/>
</target>

<target name="status" description="Show the current status">
<exec executable="${opmn.cmd}">
<arg line="status"/>
</exec>
</target>

<!-- Start the oc4j.name instance-->
<target name="start-oc4j" description="Start the oc4j.name instance" depends="get-oc4j-name">
<echo message="${opmn.cmd} startproc process-type=${oc4j.name}"/>
<exec executable="${opmn.cmd}">
<arg line="startproc process-type=${oc4j.name}"/>
</exec>
<antcall target="status"/>
</target>

<target name="get-oc4j-name" unless="oc4j.name">
<input message="Specify OC4J instance: " addproperty="oc4j.name"/>
</target>
</project>


When this executes, it prompts for the OC4J instance name, calls opmnctl to execute a startproc on it and then displays the status.


>ant -f opmn.xml start-oc4j
Buildfile: opmn.xml

get-oc4j-name:
[input] Specify OC4J instance:
fred

start-oc4j:
[echo] d:/oracleas/10131/ohs_j2ee/opmn/bin/opmnctl.exe startproc process-type=fred
[exec] opmnctl: starting opmn managed processes...

status:

[exec] Processes in Instance: ohs_j2ee_changed
[exec] ---------------------------------+--------------------+---------+---------
[exec] ias-component | process-type | pid | status
[exec] ---------------------------------+--------------------+---------+---------
[exec] OC4JGroup:default_group | OC4J:foo | 2128 | Alive
[exec] OC4JGroup:default_group | OC4J:fred | 6012 | Alive
[exec] OC4JGroup:default_group | OC4J:home | N/A | Down

07 November 2007

Application Supplied System Properties

We added a feature in the 10.1.3.1 release which enables an application to supply its own set of System properties, which are added and removed to the global System property set as the application is started and stopped. The documentation for it is not extensive, but there's a how-to on OTN which walks through the use of it.

Check it out on OTN.

The System properties (name, value) are defined in the orion-application.xml descriptor provided with the application.

<orion-application>
<property name="application.tax.rate" value="10" />
<property name="application.tax.code" value="AU" />
</orion-application>
During deployment with ASC, the deployment plan editor can be used to view the set of specified System properties in the application. It can also be used to add, remove, edit the defined properties so they can be tailored to the specific deployment being performed.

05 November 2007

Run a Coherence Cache Server from JDeveloper

I was working on a Coherence based handson lab for the upcoming Open World conference last week. The lab configures Coherence to use a JPA based CacheStore to store/load cache entries into/from a database.

The labs are all based around an Ant script that runs the lab code, plus the Cache Servers. But one handy thing that could have been documented in the labs is the ability to run a Cache Server(s) directly from JDeveloper using its "Run Manager".

Here's how it works:

1. Have a project set up which has the Coherence JAR files added as a library.

2. Create a new Run Configuration for your Cache Server.

3. In the "Default Run Target" field enter:

C:\coherence-ejb3\coherence\lib\tangosol.jar!\com\tangosol\net\DefaultCacheServer.class

4. Specify any additional JVM or Coherence properties as desired.

Click save and you're done.



Now to run a Cache Server, simply select your Cache Server run target from the list and the Cache Server will start. A log window will open up showing the output from the Cache Server. The process running the Cache Server will be shown in the Process Manager window.



The other Run Configuration I found helpful was to launch the Console that comes with Coherence. This little application is a cache client, and allows you to interactively perform operations against the cache. Handy to quickly see the size of a cache or list out some elements from the cache to see that its containing the expected data objects.

To run this Coherence application, you create a new Run Configuration as above, except specifying the Default Run Target as:

C:\coherence-ejb3\coherence\lib\coherence.jar!\com\tangosol\coherence\component\application\Console.class



Since the Coherence console is an interactive application, one extra little step you need to do here is to enable "Allow Program Input" for this running process.

In the Run Configuration dialog, select the "Tool Settings" node and then check the "Allow Program Input" option.



Save the entry and execute it. You should then the Coherence Console application fire up and prompt for your command in the "Input: [ ]" field.










25 October 2007

Locating the process that has a specific network port open

Ever tried to start a process on Windows, only to be told that the network port it needs is unavailable?

TNSLSNR for 32-bit Windows: Version 10.2.0.1.0 - Production
Error listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=mypc.oracle.com)(PORT=1521)))
TNS-12542: TNS:address already in use

Bugger. Now, which process is using that port? The task manager doesn't show you any of these sorts of details.

There's two ways I know of to work out which process which already has the port open.

1. If you have cygwin or MKS or some Unix styled set of command utilities -and who using Windows commands shells wouldn't? -- then you can use the grep and ps commands it provides with the standard netstat command.

First off, use the netstat command with the -ao switch and pipe it into grep to locate the listening details. The -o switch tells netstat to output the process-id of the owner of the port.

D:\>netstat -ao | grep 1521
TCP mypc:1521 mypc:0 LISTENING 2944

Then using the ps command, search for the process-id:

D:\>ps -eaf | grep 2944
SYSTEM 2944 1488 0 11:37:44 CONIN$ 0:00 vmware-authd

This could be combined into a nice awk script I suspect.

2. Use the very handy SysInternals utilities to locate the owning process. These used to be available on the sysinternals.com website, but the site now redirects to Microsoft, so presumably they have purchased them. Good idea Microsoft.

The specific utility for this operation is TCPView:

http://www.microsoft.com/technet/sysinternals/utilities/TcpView.mspx

This comes in two flavours, GUI and cmd line and identifies each process that is using a network port. The GUI is easy to use, just fire it up, look for the port in question which then identifies the process in the process table.

I prefer using the command line utility, tcpvcon.exe.

This results in a nicely formatted text output that shows the process and network port details for all processes that have a network port in use in some manner.

D:\>tcpvcon.exe -a
...
[TCP] vmware-authd.exe
PID: 2944
State: ESTABLISHED
Local: mypc:1521
Remote: mypc:0
...

To make it easier to identify a specific process which owns a specific port, then you can ask the output to be provided in CSV format, which lists each process on a single line, and which you can then pipe into find to locate a specific port

D:\>tcpvcon.exe -c -a | find "1521"
TCP,VMNET.EXE,2944,LISTENING,mypc:1521,mypc:0

----------------
Listening to: John Butler Trio - Good Excuse

11 October 2007

OC4J 10.1.3.3 Ant Taskname Error

OC4J 10.1.3.x supplies a set of Ant tasks that enable operations such as application deployment, resource configuration and server lifecycle tasks to be performed.

The abridged history of the tasks is:
  • OC4J 10.1.3.0 introduced the basic tasks
  • OC4J 10.1.3.1 and later versions extends the set of tasks to support JDBC/JMS resource configuration options and server shutdown/restart tasks
The OC4J Configuration and Administration guide documents the Ant tasks.

However in the OC4J 10.1.3.3 release several of the tasks are incorrectly defined in the antlib.xml and therefore not available using the names as described in the documentation.

The documentation is correct and shows how the tasks should be named. However if you use these tasks as documented you will get errors. The screen capture below highlights the specific differences.



The workarounds are either:

1. Revert to using the names as defined in antlib.xml for these tasks:

  • addDataSourceConnectionPool == createJDBCConnectionPool
  • addManagedDataSource == createManagedDataSource
  • addNativeDataSource == createNativeDataSource
  • testDataSourceConnectionPool == testConnectionPool
2. Update the /j2ee/utilities/ant-oracle-classes.jar!/oracle/antlib.xml to change the names of the tasks so they reflect the correct names as specified in the documentation:

Change: taskdef name="createJDBCConnectionPool"
To: taskdef name="addDataSourceConnectionPool"

Change: taskdef name="createManagedDataSource"
To: taskdef name="addManagedDataSource"

Change: taskdef name="addManagedDataSource"
To: taskdef name="createNativeDataSource"

Change: taskdef name="testConnectionPool"
To: taskdef name="testDataSourceConnectionPool"

A bug has been filed for this and a patch will be released which fixes the issue along the lines of option 2 shown above.