OpenNMS qosdaemon

Functionality Overview

The qosdaemon provides an elementary OSS/J Qos interface for OpenNMS.

For more information on the OSS/J Qos Specification please see the OSS/J site www.ossj.org. For more details on configuration and functionality provided please see the documentation and example configuration in the $opennms_home/contrib/qosdaemon directory.

The interface can run natively in OpenNMS and use OSS/J XML messages to reflect the state of the OpenNMS internal alarm list or it can use an external ejb running in JBOSS to expose the alarm list as JVT objects. Currently only OSS/J XVT and XML AlarmEvents have been implemented. A future release will service OSS/J XML queries. The interface uses the services of an external JMS provider. We use the JbossMQ implementation of JMS however it should be fairly easy to substitute another JMS implementation.

This interface is in two parts which can be run separately as two opennms daemons;

qosd is an OSS/J server which exposes the OpenNMS alarm list as an OSS/J alarm list to other systems. The changes to the local alarm list are exposed as JMS XML messages which can be used to update an alarm list in a remote client.

qosdrx is an OSS/J event client which can listen for alarm update events from other servers. It updates the local OpenNMS alarm list based upon the received events.

During the development of the qosdaemon, OpenNMS migrated to using the Spring framework. This has been adopted extensively in the design of the qosdrx daemon. However some of the code in the qosd daemon pre-dated the introduction of spring and has been adapted to use spring rather than re-written. This has lead to a somewhat convoluted initialization process which will be tidied up in a later release. In general most classes use injection of control and spring wiring to get their dependencies and most classes are exposed as implementations of interfaces with the 'impl' suffix to their name. Where classes have been re-implemented or gone through several versions, the implxx suffix is used where xx indicates the version. This has allowed several versions of the same class to co exist and be wired in for testing etc during development. ( As an example see QosDimpl2.java )

Package Overview

OssDao

Overview

The OssDao package provides a buffer between the internals of OpenNMS and the requirements of the OSS/J code. The internal alarm definition in OpenNMS contains different fields and severity values to the OSS/J X.733 definitions. This mapping is (mostly) confined to the OnmsAlarmOssjMapper class.

The OpenNMS alarm definition has been extended to allow it to represent alarms from remote systems. Each OpenNMS alarm now contains an ApplicationDN which identifies the unique name of the remote system which generated the alarm and an OssPrimaryKey which identifies the local identifier for that alarm on the remote machine. When an alarm update event is received, the OssDao searches for an existing alarm with matching ApplicationDN and OssPrimaryKey in order to do the update. (If ApplicationDN is blank then the alarm is from the local machine).

Each OSS/J alarm contains an identifier for the managed object which issued the alarm. This is always mapped to an OpenNMS node. The Asset Register entry for nodes now has extra fields to uniquely identify the managed object system wide ( ManagedObjectInstance and ManagedObjectClass) . The OssDao provides methods to do a rapid reverse look up on the Asset register to find nodes with a unique id which is given by the concatenation of the ManagedObjectInstance and ManagedObjectClass.

Note that when the OssDao is initialized, the QoSDrx log file will record any nodes having no data or duplicate data in the ManagedObjectInstance and ManagedObjectType fields of the Asset table. If the ManagedObjectInstance and ManagedObjectType fields of nodes are empty or duplicated the OSS/J Alarm management will not work correctly.

QoSD Daemon

Overview

The QoSD Daemon realises an OSS/J QoS server interface by mediating between the OpenNMS internal alarm management model and an OSS/J implementation realized using the OSSBeans OSS/J Qos library. The OSSbeans AlarmMonitor bean can run natively in the OpenNMS spring container or in a remote the Jboss J2EE server. If it is running in the remote J2EE server then it exposes it's alarm list according to OSS/J semantics as JVT objects through a facade bean. Note that presently only a very simple query is currently supported to get the entire alarm list. The AlarmMonitor bean also sends out XVT ( XML) and JVT AlarmEvents as the alarm list changes. In the future it is intended to extend the OssBeans alarm monitor to also support full JVT and XML queries. If the AlarmMonitor bean is running natively in OpenNMS, it cannot expose the OSS/J JVT interface and only generates the JMS events however it is envisaged that this is the way most users will want to use the interface.

How it works

The qosd.properties file contains the address information for the Qosd to connect to it's JMS queues. The configuration for the QoSD daemon is picked up from the /etc/qosd.properties file using the singleton PropertiesLoader class. ( In the future this could be picked up using the spring application context in a similar manner to the qosdrx code. )

The QosD Daemon registers with the OpenNMS eventd daemon to receive events using the call;

eventIpcManager.addEventListener(this, ueiList);

The events which qosd responds to are determined by the ueiList object which is configured by the QosDConfigFactory using and the QoSDConfiguration.xml file in the OpenNMS /etc directory. Castor is used to un-marshal the XML into the ueiList according to the QoSDConfiguration.xsd file.

When an event which qosd has registered for occurs its onEvent() method is called. Qosd responds to two types of events. Firstly any events corresponding to changes in the asset register are used to force an update in the OssDao node cache. Secondly an 'alarm changed event' causes the qosd to review the local OpenNMS alarm list for changes.

The qosd depends upon the vacuumd daemon which must be configured with an automation to look for new or changed alarms in the local alarm list and throw an 'alarm changed event' if a change is detected. An example vacuumd-configuration.xml file is included in /contrib/qosdaemon/qos-example-configuration.

The vacuumd automation also implements the X733 alarm life cycle whereby OpenNMS correlates alarm raising and alarm clearing SNMP traps and removes cleared and acknowledged alarms from the alarm list.

Qosd responds to an alarm changed event by causing the OssDao to update it's cache of OnmsAlarm values with a new snapshot of the alarm list in the database, translates the OnmsAlarm values to OSS/J AlarmValues using the OnmsAlarmOssjMapper and then forwards this new list of alarms to the OSSBeans AlarmMonitor bean. The AlarmMonitor bean compares this new list with it's existing list and sends out AlarmEvents as alarms in its current list are added, deleted or changed to match the new list.

The AlarmListConnectionManager interface determines whether the remote J2EE or local spring implementation of the AlarmMonitor is used. Two implementations of the AlarmListConnectionManager are available to be wired in using spring; AlarmListConnectionManagerJ2eeImpl connects to the remote bean using RMI. AlarmListConnectionManagerSpringImpl.java implements the bean locally.

This choice is determined by the setting of the system variable -Dqosd.usej2ee={true}|{false}' in the /etc/opennms.conf file which is read when the qosd is initialized.

In a heavy alarm storm, vacuumd could trigger multiple alarm changed events such that qosd initiates multiple parallel look ups to the database which would be very costly. The OpenNMSEventHandlerThread is used to avoid this eventuality by latching any new alarm changed events while a database look up is being performed. Thus a lookup has to complete before a new one is initiated. All of the changes which occur during this time will be picked up on the next alarm retrieval.

QoSDrx Deamon

Overview

The QoSDrx daemon implements multiple OSS/J JMS event listener clients for OpenNMS. Each client connects to a different OSS/J JMS AlarmEvent Topic on a JMS provider (tested with JbossMQ but other providers could be used ) . Alarm Events are used to update the local OpenNMS alarm list such that it mirrors the alarm event list in the remote OSS/J servers. The clients are fully configurable using an XML configuration file in the /etc directory. Planned future extensions to the Daemon will allow it to also synchronize with remote servers using OSS/J XML queries.

How It Works

The QoSDrx Daemon realises an OSS/J QoS client interface using the OSSBeans OSS/J Qos library. The Library provides a collection of classes for implementing clients and servers for receiving or transmitting OSS/J messages as XML or JVT objects. Fundamental to the library is the concept of an OSSbean which is spring class which can be run as a separate thread for handling OSS/J messages and is configured using a spring application context XML file.

As with the QoSD daemon, the qosdrx.jmx classes (QoSDrx and QoSDrxMBean) are used by opennms to spawn the qosdrx daemon and initialise an OssDao if it does not already exist. The daemon passes the OpenNMS application context to the OSSBeans OSSBeanRunner class which reads the /etc/QoSDrxOssBeanRunnerSpringContext.xml file to generate an application context and spawn separate threads for each defined OssBeanAlarmEventReceiver class. Each OssBeanAlarmEventReceiver then registers to listen to it's defined JMS AlarmEventTopic and waits to receive OSS/J AlarmEvents ( both XVT and JVT events are supported ). If any receiver cannot attach or gets disconnected from its JMS Topic, it waits for a defined timeout and then tries to reconnect.

The OssBeanAlarmEventReceivers need to have AlarmEventReceiverEventHandler classes assigned which are used to perform appropriate actions on reception of each event. The QoSDrx daemon defines an AlarmEventReceiverEventHandler implementation which interacts with the OssDao to update the OpenNMS Alarm list database on each received event. ( The present QoSDrx demon AlarmEventReceiverEventHandler implementation is called QoSDrxAlarmEventReceiverEventHandlerImpl2 )

The intention of this design is to extend the OSSbean concept in the future to make each OSSbean also a JMX Mbean. This would allow finer real time monitoring and management of each receiver client. However in the present design all of the OSSbeans are spawned off the root OpenNMS JMX bean which is able to report the statistics from all of the beans using the OpenNMS MX4J console.

Alarm Life cycle

Note that the present use case only utilizes NewAlarmEvent and AlarmCleared events to update the Alarm list. Other AlarmEvents are simply logged. However the QoSDrxAlarmEventReceiverEventHandlerImpl can easily be extended to use more of the OSS/J alarm event types if a tighter coupling between client and server is required. The following process is followed by QoSDrx to process AlarmEvents;

1. When an NewAlarmEvent message is received, it's ApplicationDN and OssPrimaryKey are checked against the OpenNMS alarm list. If the alarm exists, there is an error which is logged and the NewAlarmEvent message is ignored ( the NewAlarmEvent messages should refer only to a new not an existing alarm ).

2. If the alarm is new ( i.e. Not in the AlarmList ), QoSDrx has two modes of operation based upon how it is desired to handle the managedObjectInstance and ManagedObjectType fields in the incoming messages.

   a) Alarm can be logged against a node which is named to represent the AlarmTopic. Each OssBeanAlarmEventReceiver is given a unique name in the QoSDrxOssBeanRunnerSpringContext.xml file. In this case OpenNMS should be configured with nodes named after the name of each OssBeanAlarmEventReceiver. When an alarm comes in to a given topic, the alarm is logged in OpenNMS against a node named with the same name as the OssBeanAlarmEventReceiver. This is a good configuration if you simply want to look at alarms from subordinate OSS/J servers on a per server basis but you don't want to put every managed object in the OpenNMS database corresponding to the managedObjectInstance and ManagedObjectType of all the objects being monitored remotely.

   b) Alarms can be logged against nodes having the same managedObjectInstance and ManagedObjectType fields as the incoming message. In this case the listening OpenNMS must have a database populated with all of the nodes (with the same Asset data for managedObjectInstance and ManagedObjectType) as populated in the subordinate OpenNMS's

   Note that in both the above cases, regardless of how the alarm is displayed locally, it will be forwarded through QosD with the same managedObjectInstance and ManagedObjectType as the original message. ( i.e. OpenNMS will not modify the origin of the message even though it displays it against a different node locally )

3. When an AlarmClearedEvent message is received it's ApplicationDN and OssPrimaryKey are checked against the OpenNMS alarm list. If the alarm does not exist, there is an error which is logged and the ClearedAlarmEvent message is ignored ( the ClearedAlarmEvent messages should refer to an existing alarm ).

4. If the corresponding alarm exists in the Alarm List, it will be set to cleared and acknowledged. The vacummd process will subsequently remove the cleared and acknowledged alarm from the alarm list.

   Note that the behavior of the interface could easily be extended to allow usage of separate acknowledgment messages of cleared alarms before removing them from the list. However this has not been done as it appeared to complicate the use case without much extra value. In the future we will make alternative alarm life cycle behaviour's a configurable item.