|
OpenNMS API 1.2.3 | |||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | |||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |
java.lang.Objectorg.opennms.netmgt.rrd.QueuingRrdStrategy
Provides queueing implementation of RrdStrategy. In order to provide a more scalable collector. We created a queuing RrdStrategy that enabled the system to amortize the high cost of opening an round robin database across multiple updates. This RrdStrategy implementation enqueues the create and update operations on a per file basis and maintains a set of threads that process enqueued work file by file. If the I/O system can keep up with the collection threads while performing only a single update per file then eventually all the data is processed and the threads sleep until there is more work to do. If the I/O system is initially slower than than the collection threads then work will enqueue here and the write threads will get behind. As this happens each file will eventually have more than a single update enqueued and therefore the number of updates pushed thru the system will increase because more then one will be output per 'open' Eventually, the I/O system and the collection system will balance out. When this happens all data will be collected but will not be output to the rrd files until the next time the file is processed by the write threads. As another performance improving strategy. The queue distinguishes between files with signficant vs insignifact updates. Files with only insignificant updates are put at the lowest priority and are only written when the highest priority updates have been written This implementation delegates all the actual writing to another RrdStrategy implementation. System properites effecting the operation: org.opennms.rrd.queuing.writethreads: (default 2) The number of rrd write threads that process the queue org.opennms.rrd.queuing.queueCreates: (default true) indicates whether rrd file creates should be queued or processed synchronously org.opennms.rrd.queuing.maxInsigUpdateSeconds: (default 0) the number of seconds over which all files with significant updates only should be promoted onto the significant less. This is to ensure they don't stay unprocessed forever. Zero means not promotion. org.opennms.rrd.queuing.modulus: (default 10000) the number of updates the get enqueued between statistics output org.opennms.rrd.queuing.category: (default "UNCATEGORIZED") the log category to place the statistics output in TODO: Promote files when ZeroUpdate operations can't be merged. This may be a collection miss which we want to push thru. It should also help with memory. TODO: Set an upper bound on enqueued operations TODO: Provide an event that will write data for a particular file... Say right before we try to graph it.
Nested Class Summary | |
class |
QueuingRrdStrategy.CreateOperation
This class represents an operation to create an rrd file |
(package private) static class |
QueuingRrdStrategy.Operation
This is the base class for an enqueueable operation |
class |
QueuingRrdStrategy.UpdateOperation
Represents an update to a rrd file. |
class |
QueuingRrdStrategy.ZeroUpdateOperation
Represents an update whose value is 0. |
Field Summary | |
(package private) static int |
CREATE
|
(package private) long |
createsCompleted
|
(package private) long |
dequeuedItems
|
(package private) long |
dequeuedOperations
|
(package private) long |
enqueuedOperations
|
(package private) long |
errors
|
(package private) java.util.Map |
fileAssignments
|
(package private) java.util.LinkedList |
filesWithInsignificantWork
|
(package private) java.util.LinkedList |
filesWithSignificantWork
|
(package private) long |
lastDequeued
|
(package private) long |
lastDequeuedItems
|
(package private) long |
lastEnqueued
|
(package private) long |
lastLap
|
(package private) long |
lastOpsPending
|
(package private) long |
lastSignificantCompleted
|
(package private) long |
lastSignificantDequeued
|
(package private) long |
lastSignificantEnqueued
|
(package private) long |
lastStatsTime
|
private static java.lang.String |
LOG4J_CATEGORY
|
(package private) RrdStrategy |
m_delegate
|
private static long |
MAX_INSIG_UPDATE_SECONDS
|
private static long |
MODULUS
|
(package private) java.util.Map |
pendingFileOperations
|
(package private) long |
promotionCount
|
private static boolean |
QUEUE_CREATES
|
(package private) java.util.Set |
reservedFiles
|
(package private) long |
significantOpsCompleted
|
(package private) long |
significantOpsDequeued
|
(package private) long |
significantOpsEnqueued
|
(package private) int |
threadsRunning
|
(package private) long |
totalOperationsPending
|
(package private) static int |
UPDATE
|
(package private) long |
updatesCompleted
|
(package private) long |
updateStart
|
private static long |
WRITE_THREAD_EXIT_DELAY
|
private static long |
WRITE_THREAD_SLEEP_TIME
|
(package private) static int |
WRITE_THREADS
|
Constructor Summary | |
QueuingRrdStrategy(RrdStrategy delegate)
|
Method Summary | |
void |
addOperation(QueuingRrdStrategy.Operation op)
Add an operation to the queue. |
void |
closeFile(java.lang.Object rrd)
This closes the supplied round robin database |
private void |
completeAssignment()
Record that fact that the current thread has finished process operations for its current assignement |
java.lang.Object |
createDefinition(java.lang.String creator,
java.lang.String directory,
java.lang.String dsName,
int step,
java.lang.String dsType,
int dsHeartbeat,
java.lang.String dsMin,
java.lang.String dsMax,
java.util.List rraList)
Create a round robin database definition from the supplied parameters. |
void |
createFile(java.lang.Object op)
Creates the round robin database defined by the supplied definition. |
java.io.InputStream |
createGraph(java.lang.String command,
java.io.File workDir)
Creates an InputStream representing the bytes of a graph created from round robin data. |
void |
ensureThreadsStarted()
Ensure that we have threads started to process the queue. |
java.lang.Double |
fetchLastValue(java.lang.String rrdFile,
int interval)
Fetches the last value from the round robin database with the given name. |
java.lang.String |
getLapTime()
|
java.util.LinkedList |
getNext()
Get the operations for the next file that should be worked on. |
java.lang.String |
getStats()
Print queue statistics. |
void |
graphicsInitialize()
This Initializes the graphics subsystem only. |
private boolean |
hasOnlyInsignificant(java.util.LinkedList pendingOps)
Return true if and only if all the operations in the list are insignificant |
void |
initialize()
Initialize the appropriate round robin system |
private void |
log(java.lang.String msg)
|
QueuingRrdStrategy.Operation |
makeCreateOperation(java.lang.String fileName,
java.lang.Object rrdDef)
|
QueuingRrdStrategy.Operation |
makeUpdateOperation(java.lang.String fileName,
java.lang.String update)
|
java.lang.Object |
openFile(java.lang.String fileName)
Opens the round robin database with the supplied name. |
(package private) void |
printLapTime(java.lang.String message)
|
void |
printStats()
|
private void |
processClose(java.lang.Object rrd)
close the rrd file |
private void |
processPendingOperations()
Actually process the operations be calling the underlying delegate strategy |
private void |
promoteAgedFiles()
Ensure that files with insignificant changes are getting promoted if necessary |
void |
run()
|
private java.lang.String |
selectNewAssignment()
Return the name of the next file with available work |
private void |
storeAssignment(QueuingRrdStrategy.Operation op)
We need to track which files are being processed by which threads so that we don't try to process updates for the same file on more than one thread. |
private java.util.LinkedList |
takeAssignment(java.lang.String newAssignment)
register the file that the currentThread is be working on. |
void |
updateFile(java.lang.Object rrdFile,
java.lang.String data)
Updates the supplied round robin database with the given timestamp:value point |
Methods inherited from class java.lang.Object |
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait |
Field Detail |
RrdStrategy m_delegate
static final int UPDATE
static final int CREATE
static final int WRITE_THREADS
private static final boolean QUEUE_CREATES
private static final long MODULUS
private static final java.lang.String LOG4J_CATEGORY
private static final long MAX_INSIG_UPDATE_SECONDS
private static final long WRITE_THREAD_SLEEP_TIME
private static final long WRITE_THREAD_EXIT_DELAY
java.util.LinkedList filesWithSignificantWork
java.util.LinkedList filesWithInsignificantWork
java.util.Map pendingFileOperations
java.util.Map fileAssignments
java.util.Set reservedFiles
long totalOperationsPending
long enqueuedOperations
long dequeuedOperations
long significantOpsEnqueued
long significantOpsDequeued
long significantOpsCompleted
long dequeuedItems
long createsCompleted
long updatesCompleted
long errors
int threadsRunning
long updateStart
long promotionCount
long lastLap
long lastStatsTime
long lastEnqueued
long lastDequeued
long lastSignificantEnqueued
long lastSignificantDequeued
long lastSignificantCompleted
long lastDequeuedItems
long lastOpsPending
Constructor Detail |
public QueuingRrdStrategy(RrdStrategy delegate)
Method Detail |
public QueuingRrdStrategy.Operation makeCreateOperation(java.lang.String fileName, java.lang.Object rrdDef)
public QueuingRrdStrategy.Operation makeUpdateOperation(java.lang.String fileName, java.lang.String update)
public void addOperation(QueuingRrdStrategy.Operation op)
public void ensureThreadsStarted()
public java.util.LinkedList getNext()
private void storeAssignment(QueuingRrdStrategy.Operation op)
private void promoteAgedFiles()
private boolean hasOnlyInsignificant(java.util.LinkedList pendingOps)
private java.util.LinkedList takeAssignment(java.lang.String newAssignment)
private java.lang.String selectNewAssignment()
private void completeAssignment()
public void closeFile(java.lang.Object rrd) throws java.lang.Exception
RrdStrategy
closeFile
in interface RrdStrategy
rrd
- an rrd object created using openFile
java.lang.Exception
- if an error occurs closing the filepublic java.lang.Object createDefinition(java.lang.String creator, java.lang.String directory, java.lang.String dsName, int step, java.lang.String dsType, int dsHeartbeat, java.lang.String dsMin, java.lang.String dsMax, java.util.List rraList) throws java.lang.Exception
RrdStrategy
createDefinition
in interface RrdStrategy
creator
- -
A string representing who is creating this file for use in log
msgsdirectory
- -
The directory to create the file indsName
- -
The datasource name for use in the round robin databasestep
- -
the step for the databasedsType
- -
the type for the datasourcedsHeartbeat
- -
the heartbeat for the datasoucedsMin
- -
the minimum allowable value for the datasourcedsMax
- -
the maximum allowable value for the datasoucerraList
- -
a List of the round robin archives to create in the database
java.lang.Exception
- If an error occurs while creating the definitionpublic void createFile(java.lang.Object op) throws java.lang.Exception
RrdStrategy
createFile
in interface RrdStrategy
op
- an round robin database definition created using the
createDefinition call.
java.lang.Exception
- if an error occurs create the filepublic void initialize() throws java.lang.Exception
RrdStrategy
initialize
in interface RrdStrategy
java.lang.Exception
- if an Error occurspublic void graphicsInitialize() throws java.lang.Exception
RrdStrategy
graphicsInitialize
in interface RrdStrategy
java.lang.Exception
public java.lang.Object openFile(java.lang.String fileName) throws java.lang.Exception
RrdStrategy
openFile
in interface RrdStrategy
fileName
- the name of the associated rrd file
java.lang.Exception
- if an error occurs opening the filepublic void updateFile(java.lang.Object rrdFile, java.lang.String data) throws java.lang.Exception
RrdStrategy
updateFile
in interface RrdStrategy
rrdFile
- an rrd object created using openFiledata
- a string of the form java.lang.Exception
- if an error occurs updating the filepublic java.lang.Double fetchLastValue(java.lang.String rrdFile, int interval) throws java.lang.NumberFormatException, RrdException
RrdStrategy
fetchLastValue
in interface RrdStrategy
rrdFile
- a name the represents a round robin databaseinterval
- a step interval of the round robin database
java.lang.NumberFormatException
RrdException
public java.io.InputStream createGraph(java.lang.String command, java.io.File workDir) throws java.io.IOException, RrdException
RrdStrategy
createGraph
in interface RrdStrategy
command
- the command needed to create the graphworkDir
- the directory that all referenced files are relative to
RrdException
- if an RRD error occurs
java.io.IOException
- if an IOError occurspublic void run()
run
in interface java.lang.Runnable
private void processPendingOperations()
private void processClose(java.lang.Object rrd)
public java.lang.String getStats()
getStats
in interface RrdStrategy
public void printStats()
void printLapTime(java.lang.String message)
private void log(java.lang.String msg)
msg
- public java.lang.String getLapTime()
|
OpenNMS API 1.2.3 | |||||||||
PREV CLASS NEXT CLASS | FRAMES NO FRAMES | |||||||||
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD |