OpenNMS API 1.2.3

org.opennms.netmgt.rrd
Class QueuingRrdStrategy

java.lang.Object
  extended byorg.opennms.netmgt.rrd.QueuingRrdStrategy
All Implemented Interfaces:
RrdStrategy, java.lang.Runnable

class QueuingRrdStrategy
extends java.lang.Object
implements RrdStrategy, java.lang.Runnable

Provides queueing implementation of RrdStrategy. In order to provide a more scalable collector. We created a queuing RrdStrategy that enabled the system to amortize the high cost of opening an round robin database across multiple updates. This RrdStrategy implementation enqueues the create and update operations on a per file basis and maintains a set of threads that process enqueued work file by file. If the I/O system can keep up with the collection threads while performing only a single update per file then eventually all the data is processed and the threads sleep until there is more work to do. If the I/O system is initially slower than than the collection threads then work will enqueue here and the write threads will get behind. As this happens each file will eventually have more than a single update enqueued and therefore the number of updates pushed thru the system will increase because more then one will be output per 'open' Eventually, the I/O system and the collection system will balance out. When this happens all data will be collected but will not be output to the rrd files until the next time the file is processed by the write threads. As another performance improving strategy. The queue distinguishes between files with signficant vs insignifact updates. Files with only insignificant updates are put at the lowest priority and are only written when the highest priority updates have been written This implementation delegates all the actual writing to another RrdStrategy implementation. System properites effecting the operation: org.opennms.rrd.queuing.writethreads: (default 2) The number of rrd write threads that process the queue org.opennms.rrd.queuing.queueCreates: (default true) indicates whether rrd file creates should be queued or processed synchronously org.opennms.rrd.queuing.maxInsigUpdateSeconds: (default 0) the number of seconds over which all files with significant updates only should be promoted onto the significant less. This is to ensure they don't stay unprocessed forever. Zero means not promotion. org.opennms.rrd.queuing.modulus: (default 10000) the number of updates the get enqueued between statistics output org.opennms.rrd.queuing.category: (default "UNCATEGORIZED") the log category to place the statistics output in TODO: Promote files when ZeroUpdate operations can't be merged. This may be a collection miss which we want to push thru. It should also help with memory. TODO: Set an upper bound on enqueued operations TODO: Provide an event that will write data for a particular file... Say right before we try to graph it.


Nested Class Summary
 class QueuingRrdStrategy.CreateOperation
          This class represents an operation to create an rrd file
(package private) static class QueuingRrdStrategy.Operation
          This is the base class for an enqueueable operation
 class QueuingRrdStrategy.UpdateOperation
          Represents an update to a rrd file.
 class QueuingRrdStrategy.ZeroUpdateOperation
          Represents an update whose value is 0.
 
Field Summary
(package private) static int CREATE
           
(package private)  long createsCompleted
           
(package private)  long dequeuedItems
           
(package private)  long dequeuedOperations
           
(package private)  long enqueuedOperations
           
(package private)  long errors
           
(package private)  java.util.Map fileAssignments
           
(package private)  java.util.LinkedList filesWithInsignificantWork
           
(package private)  java.util.LinkedList filesWithSignificantWork
           
(package private)  long lastDequeued
           
(package private)  long lastDequeuedItems
           
(package private)  long lastEnqueued
           
(package private)  long lastLap
           
(package private)  long lastOpsPending
           
(package private)  long lastSignificantCompleted
           
(package private)  long lastSignificantDequeued
           
(package private)  long lastSignificantEnqueued
           
(package private)  long lastStatsTime
           
private static java.lang.String LOG4J_CATEGORY
           
(package private)  RrdStrategy m_delegate
           
private static long MAX_INSIG_UPDATE_SECONDS
           
private static long MODULUS
           
(package private)  java.util.Map pendingFileOperations
           
(package private)  long promotionCount
           
private static boolean QUEUE_CREATES
           
(package private)  java.util.Set reservedFiles
           
(package private)  long significantOpsCompleted
           
(package private)  long significantOpsDequeued
           
(package private)  long significantOpsEnqueued
           
(package private)  int threadsRunning
           
(package private)  long totalOperationsPending
           
(package private) static int UPDATE
           
(package private)  long updatesCompleted
           
(package private)  long updateStart
           
private static long WRITE_THREAD_EXIT_DELAY
           
private static long WRITE_THREAD_SLEEP_TIME
           
(package private) static int WRITE_THREADS
           
 
Constructor Summary
QueuingRrdStrategy(RrdStrategy delegate)
           
 
Method Summary
 void addOperation(QueuingRrdStrategy.Operation op)
          Add an operation to the queue.
 void closeFile(java.lang.Object rrd)
          This closes the supplied round robin database
private  void completeAssignment()
          Record that fact that the current thread has finished process operations for its current assignement
 java.lang.Object createDefinition(java.lang.String creator, java.lang.String directory, java.lang.String dsName, int step, java.lang.String dsType, int dsHeartbeat, java.lang.String dsMin, java.lang.String dsMax, java.util.List rraList)
          Create a round robin database definition from the supplied parameters.
 void createFile(java.lang.Object op)
          Creates the round robin database defined by the supplied definition.
 java.io.InputStream createGraph(java.lang.String command, java.io.File workDir)
          Creates an InputStream representing the bytes of a graph created from round robin data.
 void ensureThreadsStarted()
          Ensure that we have threads started to process the queue.
 java.lang.Double fetchLastValue(java.lang.String rrdFile, int interval)
          Fetches the last value from the round robin database with the given name.
 java.lang.String getLapTime()
           
 java.util.LinkedList getNext()
          Get the operations for the next file that should be worked on.
 java.lang.String getStats()
          Print queue statistics.
 void graphicsInitialize()
          This Initializes the graphics subsystem only.
private  boolean hasOnlyInsignificant(java.util.LinkedList pendingOps)
          Return true if and only if all the operations in the list are insignificant
 void initialize()
          Initialize the appropriate round robin system
private  void log(java.lang.String msg)
           
 QueuingRrdStrategy.Operation makeCreateOperation(java.lang.String fileName, java.lang.Object rrdDef)
           
 QueuingRrdStrategy.Operation makeUpdateOperation(java.lang.String fileName, java.lang.String update)
           
 java.lang.Object openFile(java.lang.String fileName)
          Opens the round robin database with the supplied name.
(package private)  void printLapTime(java.lang.String message)
           
 void printStats()
           
private  void processClose(java.lang.Object rrd)
          close the rrd file
private  void processPendingOperations()
          Actually process the operations be calling the underlying delegate strategy
private  void promoteAgedFiles()
          Ensure that files with insignificant changes are getting promoted if necessary
 void run()
           
private  java.lang.String selectNewAssignment()
          Return the name of the next file with available work
private  void storeAssignment(QueuingRrdStrategy.Operation op)
          We need to track which files are being processed by which threads so that we don't try to process updates for the same file on more than one thread.
private  java.util.LinkedList takeAssignment(java.lang.String newAssignment)
          register the file that the currentThread is be working on.
 void updateFile(java.lang.Object rrdFile, java.lang.String data)
          Updates the supplied round robin database with the given timestamp:value point
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Field Detail

m_delegate

RrdStrategy m_delegate

UPDATE

static final int UPDATE
See Also:
Constant Field Values

CREATE

static final int CREATE
See Also:
Constant Field Values

WRITE_THREADS

static final int WRITE_THREADS

QUEUE_CREATES

private static final boolean QUEUE_CREATES

MODULUS

private static final long MODULUS

LOG4J_CATEGORY

private static final java.lang.String LOG4J_CATEGORY

MAX_INSIG_UPDATE_SECONDS

private static final long MAX_INSIG_UPDATE_SECONDS

WRITE_THREAD_SLEEP_TIME

private static final long WRITE_THREAD_SLEEP_TIME

WRITE_THREAD_EXIT_DELAY

private static final long WRITE_THREAD_EXIT_DELAY

filesWithSignificantWork

java.util.LinkedList filesWithSignificantWork

filesWithInsignificantWork

java.util.LinkedList filesWithInsignificantWork

pendingFileOperations

java.util.Map pendingFileOperations

fileAssignments

java.util.Map fileAssignments

reservedFiles

java.util.Set reservedFiles

totalOperationsPending

long totalOperationsPending

enqueuedOperations

long enqueuedOperations

dequeuedOperations

long dequeuedOperations

significantOpsEnqueued

long significantOpsEnqueued

significantOpsDequeued

long significantOpsDequeued

significantOpsCompleted

long significantOpsCompleted

dequeuedItems

long dequeuedItems

createsCompleted

long createsCompleted

updatesCompleted

long updatesCompleted

errors

long errors

threadsRunning

int threadsRunning

updateStart

long updateStart

promotionCount

long promotionCount

lastLap

long lastLap

lastStatsTime

long lastStatsTime

lastEnqueued

long lastEnqueued

lastDequeued

long lastDequeued

lastSignificantEnqueued

long lastSignificantEnqueued

lastSignificantDequeued

long lastSignificantDequeued

lastSignificantCompleted

long lastSignificantCompleted

lastDequeuedItems

long lastDequeuedItems

lastOpsPending

long lastOpsPending
Constructor Detail

QueuingRrdStrategy

public QueuingRrdStrategy(RrdStrategy delegate)
Method Detail

makeCreateOperation

public QueuingRrdStrategy.Operation makeCreateOperation(java.lang.String fileName,
                                                        java.lang.Object rrdDef)

makeUpdateOperation

public QueuingRrdStrategy.Operation makeUpdateOperation(java.lang.String fileName,
                                                        java.lang.String update)

addOperation

public void addOperation(QueuingRrdStrategy.Operation op)
Add an operation to the queue.


ensureThreadsStarted

public void ensureThreadsStarted()
Ensure that we have threads started to process the queue.


getNext

public java.util.LinkedList getNext()
Get the operations for the next file that should be worked on.

Returns:
a linkedList of oeprations to be processed all for the same file.

storeAssignment

private void storeAssignment(QueuingRrdStrategy.Operation op)
We need to track which files are being processed by which threads so that we don't try to process updates for the same file on more than one thread.


promoteAgedFiles

private void promoteAgedFiles()
Ensure that files with insignificant changes are getting promoted if necessary


hasOnlyInsignificant

private boolean hasOnlyInsignificant(java.util.LinkedList pendingOps)
Return true if and only if all the operations in the list are insignificant


takeAssignment

private java.util.LinkedList takeAssignment(java.lang.String newAssignment)
register the file that the currentThread is be working on. This enables us to ensure that another thread doesn't try to work on operations for that file.


selectNewAssignment

private java.lang.String selectNewAssignment()
Return the name of the next file with available work


completeAssignment

private void completeAssignment()
Record that fact that the current thread has finished process operations for its current assignement


closeFile

public void closeFile(java.lang.Object rrd)
               throws java.lang.Exception
Description copied from interface: RrdStrategy
This closes the supplied round robin database

Specified by:
closeFile in interface RrdStrategy
Parameters:
rrd - an rrd object created using openFile
Throws:
java.lang.Exception - if an error occurs closing the file

createDefinition

public java.lang.Object createDefinition(java.lang.String creator,
                                         java.lang.String directory,
                                         java.lang.String dsName,
                                         int step,
                                         java.lang.String dsType,
                                         int dsHeartbeat,
                                         java.lang.String dsMin,
                                         java.lang.String dsMax,
                                         java.util.List rraList)
                                  throws java.lang.Exception
Description copied from interface: RrdStrategy
Create a round robin database definition from the supplied parameters. This definition is used in the createFile call to create the actual file.

Specified by:
createDefinition in interface RrdStrategy
Parameters:
creator - - A string representing who is creating this file for use in log msgs
directory - - The directory to create the file in
dsName - - The datasource name for use in the round robin database
step - - the step for the database
dsType - - the type for the datasource
dsHeartbeat - - the heartbeat for the datasouce
dsMin - - the minimum allowable value for the datasource
dsMax - - the maximum allowable value for the datasouce
rraList - - a List of the round robin archives to create in the database
Returns:
an object representing the definition of an round robin database
Throws:
java.lang.Exception - If an error occurs while creating the definition

createFile

public void createFile(java.lang.Object op)
                throws java.lang.Exception
Description copied from interface: RrdStrategy
Creates the round robin database defined by the supplied definition.

Specified by:
createFile in interface RrdStrategy
Parameters:
op - an round robin database definition created using the createDefinition call.
Throws:
java.lang.Exception - if an error occurs create the file

initialize

public void initialize()
                throws java.lang.Exception
Description copied from interface: RrdStrategy
Initialize the appropriate round robin system

Specified by:
initialize in interface RrdStrategy
Throws:
java.lang.Exception - if an Error occurs

graphicsInitialize

public void graphicsInitialize()
                        throws java.lang.Exception
Description copied from interface: RrdStrategy
This Initializes the graphics subsystem only. This is used by the web application to avoid the need for the JNI infrastructure in the webapp.

Specified by:
graphicsInitialize in interface RrdStrategy
Throws:
java.lang.Exception

openFile

public java.lang.Object openFile(java.lang.String fileName)
                          throws java.lang.Exception
Description copied from interface: RrdStrategy
Opens the round robin database with the supplied name. It is assumed the name refers to a round robin database appropriate for this strategy implementation

Specified by:
openFile in interface RrdStrategy
Parameters:
fileName - the name of the associated rrd file
Returns:
an open rrd reference that can by used in calls to updateFile and closeFile
Throws:
java.lang.Exception - if an error occurs opening the file

updateFile

public void updateFile(java.lang.Object rrdFile,
                       java.lang.String data)
                throws java.lang.Exception
Description copied from interface: RrdStrategy
Updates the supplied round robin database with the given timestamp:value point

Specified by:
updateFile in interface RrdStrategy
Parameters:
rrdFile - an rrd object created using openFile
data - a string of the form :
Throws:
java.lang.Exception - if an error occurs updating the file

fetchLastValue

public java.lang.Double fetchLastValue(java.lang.String rrdFile,
                                       int interval)
                                throws java.lang.NumberFormatException,
                                       RrdException
Description copied from interface: RrdStrategy
Fetches the last value from the round robin database with the given name. The interval passed in should be the interval associated with the round robin database.

Specified by:
fetchLastValue in interface RrdStrategy
Parameters:
rrdFile - a name the represents a round robin database
interval - a step interval of the round robin database
Returns:
The last value as a Double (if the last value didn't exist returns a Double.NaN)
Throws:
java.lang.NumberFormatException
RrdException

createGraph

public java.io.InputStream createGraph(java.lang.String command,
                                       java.io.File workDir)
                                throws java.io.IOException,
                                       RrdException
Description copied from interface: RrdStrategy
Creates an InputStream representing the bytes of a graph created from round robin data. It accepts an rrdtool graph command. The underlying implementation converts this command to a format appropriate for it .

Specified by:
createGraph in interface RrdStrategy
Parameters:
command - the command needed to create the graph
workDir - the directory that all referenced files are relative to
Returns:
an input stream representing the bytes of a graph image as a PNG file
Throws:
RrdException - if an RRD error occurs
java.io.IOException - if an IOError occurs

run

public void run()
Specified by:
run in interface java.lang.Runnable

processPendingOperations

private void processPendingOperations()
Actually process the operations be calling the underlying delegate strategy


processClose

private void processClose(java.lang.Object rrd)
close the rrd file


getStats

public java.lang.String getStats()
Print queue statistics.

Specified by:
getStats in interface RrdStrategy
Returns:
a non-null string representing any staticstics to be included in the logs

printStats

public void printStats()

printLapTime

void printLapTime(java.lang.String message)

log

private void log(java.lang.String msg)
Parameters:
msg -

getLapTime

public java.lang.String getLapTime()

OpenNMS API 1.2.3

Generated by eevans on May 29 2005 2015.