Copyright © 2004-2020 The OpenNMS Group, Inc.

Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.1 or any later version published by the Free Software Foundation; with no Invariant Sections, with no Front-Cover Texts and with no Back-Cover Texts. A copy of the license is available at http://www.gnu.org/copyleft/fdl.html OpenNMS is the creation of numerous people and organizations, operating under the umbrella of the OpenNMS project. The source code is published under the GNU Affero GPL, version 3 or later and is Copyright © 2002-2020 The OpenNMS Group, Inc.

The current corporate sponsor of OpenNMS is The OpenNMS Group, which also owns the OpenNMS trademark.

Please report any omissions or corrections to this document by creating an issue at http://issues.opennms.org.

What’s New in OpenNMS Horizon 26

System Requirements

  • Java 8 through 11: OpenNMS Horizon 26 runs on JDK 8 through 11. We recommend the most recent version of OpenJDK 11.

  • PostgreSQL 10 or higher: Horizon 26 requires any supported version of PostgreSQL 10 or higher.

Breaking Changes

Flow Classification

In versions before Horizon 26, the Flow Classification UI accepted invalid IP address values, e.g., 10,192.1,168.1,5.1-160. When upgrading OpenNMS Horizon, existing Flow Classification Rules may be considered invalid. In this case they are silently ignored by the Flow Classification Engine.

In order to assure no invalid Flow Classification Rules exist, please run the OSGi shell command opennms-classification:list-invalid-rules to list all invalid rules. If there are any invalid rules, you must manually fix them using the Flow Classification UI. For more details, refer to issue NMS-12422.

Events and Kafka

Events forwarded to Kafka are now keyed by id rather than uei.

Karaf Shell Command Changes

All of the custom Karaf shell commands have now been updated to use a common scope of opennms. This makes it easier to find and enumerate the commands.

Mapping from previous to new commands is as follows:

Previous Command New Command

opennms-provision:detect

opennms:detect

opennms-provision:list-detectors

opennms:list-detectors

opennms-provision:show-import

opennms:show-import

opennms-provision:import-requisition

opennms:import-requisition

opennms-dns:lookup

opennms:dns-lookup

opennms-dns:reverse-lookup

opennms:dns-reverse-lookup

opennms-ping:sweep

opennms:ping-sweep

opennms-ping:ping

opennms:ping

opennms-jira:list-projects

opennms:jira-list-projects

opennms-jira:verify

opennms:jira-verify

opennms-jira:show-config

opennms:jira-show-config

opennms-jira:list-priorities

opennms:jira-list-priorities

opennms-jira:list-fields

opennms:jira-list-fields

opennms-jira:list-versions

opennms:jira-list-versions

opennms-jira:list-issue-types

opennms:jist-list-issue-types

opennms-jira:list-components

opennms:jira-list-components

opennms-scv:set

opennms:scv-set

opennms-scv:get

opennms:scv-get

opennms-datachoices:reset-system-id

opennms:datachoices-reset-system-id

opennms-datachoices:display-usage-report

opennms:datachoices-display-usage-report

opennms-datachoices:send-usage-report

opennms:datachoies-send-usage-report

opennms-poller:test

opennms:poll-test

opennms-poller:list-monitors

opennms:list-monitors

opennms-poller:poll

opennms:poll

opennms-filters:filter

opennms:filter

opennms-measurements:show-newts-samples

opennms:show-newts-samples

opennms-measurements:delete-resource

opennms:delete-measurement-resources

opennms-measurements:show-measurements

opennms:show-measurements

opennms-measurements:show-resources

opennms:show-measurement-resources

opennms-enlinkd:delete-topology

opennms:delete-topology

opennms-enlinkd:generate-topology

opennms:generate-topology

opennms-provision:list-detectors

opennms:list-detectors

opennms-collection:list-collectors

opennms:list-collectors

opennms-minion:id

opennms:minion-id

opennnms-minion:ping

*Removed in favor of opennms:health-check

opennms-dns:stress

opennms:stress-dns

opennms-kafka-producer:sync-alarms

opennms:kafka-sync-alarms

opennms-kafka-producer:push-topology-edges

opennms:kafka-push-topology-edges

opennms-kafka-producer:evaluate-filter

opennms:kafka-evaluate-filter

opennms-kafka-producer:list-alarms

opennms:kafka-list-alarms

opennms-nodecache:sync

opennms:sync-node-cache

opnennms-coordination:join-election-domain

opennms:join-election-domain

opennms-kv-blob:put

opennms:kv-put-blob

opennms-kv-blob:get

opennms:kv-get-blob

opennms-kv-blob:benchmark

opennms:kv-benchmark-blob

opennms-kv-json:get

opennms:kv-get-json

opennms-kv-json:put

opennms:kv-put-json

opennms-classification:list-rules

opennms:list-classification-rules

opennms-classification:reload-engine

opennms:reload-classification-engine

opennms-classification:list-groups

opennms:list-classification-groups

opennms-classification:list-invalid-rules

opennms:list-classification-invalid-rules

opennms-classification:classify

opennms:classify-flow

opennms-elasticsearch:send-historic-events

opennms:send-events-to-elasticsearch

opennms-events:show-event-config

opennms:show-event-config

opennms-reload:daemon

opennms:reload-daemon

opennms-events:stress

opennms:stress-events

opennms-events:send

opennms:send-event

opennms-bsm:generate-hierarchies

opennms:bsm-generate-hierarchies

opennms-bsm:delete-generated-hierarchies

opennms:bsm-delete-generated-hierarchies

opennms-bsm:render-graph

opennms:bsm-render-graph

opennms-asset-topology:regenerate

opennms:asset-topo-regenerate

opennms-asset-topology:remove

opennms:asset-topo-remove

opennms-asset-topology:list

opennms:asset-topo-list

opennms-asset-topology:create

opennms:asset-topo-create

opennms-asset-topology:regenerateall

opennms:asset-topo-regenerate-all

opennms-topo:show-history

opennms:topo-show-history

opennms-topo:delete-history

opennms:topo-delete-history

opennms-onms:listnamespaces

opennms:list-namespaces

opennms-topo:listoperations

opennms:topo-list-operations

opennms-activemq:stats

opennms:activemq-stats

opennms-activemq:purge-queue

opennms:activemq-purge-queue

opennms-sentinel:id

opennms:id

opennms-collection:list-collectors

opennms:list-collectors

opennms-collection:collect

opennms:collect

opennms-metrics:stress

opennms:stress-metrics

opennms-threshold-states:details

opennms:threshold-details

opennms-threshold-states:enumerate

opennms:threshold-enumerate

opennms-threshold-states:clear-all

opennms:threshold-clear-all

opennms-threshold-states:clear

opennms:threshold-clear

opennms-meta:test

opennms:metadata-test

opennms-rpc:stress

opennms:stress-rpc

opennms-kafka-sink:topics

opennms:kafka-sink-topics

opennms-kafka-rpc:topics

opennms:kafka-rpc-topics

opennms-snmp:fit

opennms:snmp-fit

opennms-snmp:show-config

opennms:snmp-show-config

opennms-snmp:remove-from-definition

opennms:snmp-remove-from-definition

opennms-snmp:walk

opennms:snmp-walk

snmp:local-engine-id

opennms:snmp-local-engine-id

opennms-health:metrics-list

opennms:metrics-list

opennms-health:metrics-display

opennms:metrics-display

opennms-health:check

opennms:health-check

opennms-graph:get

opennms:graph-get

opennms-graph:list

opennms:graph-list

opennms-graph:force-reload

opennms:graph-force-reload

opennms-graph:search

opennms:graph-search

New Features and Improvements

Documentation Updates

Documentation improvements have been made in a number of areas:

  • Docker

  • Grafana PDF Reports

  • Graph Service API

  • Minion

  • Telemetry/flows

  • Thresholding

  • Timeseries Integration

Graph API

A new API has been added for querying graph data, inspired by GraphML. This API consolidates a number of internal APIs for accessing topology, application, BSM, link, and other data.

Kafka and RPC

The RPC APIs for communicating between the OpenNMS core, Minion, and Sentinel have been improved, including adding support for GRPC communication.

It is now possible to configure Kafka to use a single topic for communication rather than one-per-module.

Flow data can now be configured to be enriched with node metadata before being forwarded to Kafka.

Minion and Sentinel

  • The Minion can now be configured using confd by creating a minion-config.yaml file. For details, see the README in the Minion source.

  • The Minion Docker container has a number of enhancements:

    • Minion container sizes have been halved.

    • Native ICMP support has been added to improve performance.

    • Dockerized Minion now supports ARM.

  • The Jolokia HTTP-JMX bridge has been added to the Minion and Sentinel.

Polling and Monitoring

  • The HTTP POST monitor has been updated to support injecting headers into the request.

  • The poller now does a better job of validating configs on startup/reload.

Provisioning

  • Provisiond has been enhanced to improve performance of scans.

  • Nodes provisioned through new suspect events now get assigned a foreign source.

Telemetry and BMP

  • The flow classification editor now supports CIDR notation for IP address expressions.

  • Support has been added for the BGP Monitoring Protocol (BMP).

Time-Series API

A new experimental internal API has been added for supporting persisting time-series data in a way that doesn’t carry the baggage of the existing persistence APIs which are still heavily influenced by RRDTool.

Web UI Improvements

  • The navbar notification alert now shows everywhere, not just the front page.

  • The "user" menu has been reorganized and cleaned up.

  • The requisitions UI now supports deleting a node from its node page in the editor.

Kafka Producer Metrics

  • For Interface Resources, a String attribute named __ifIndex is added to represent missing Interface ifIndex in String form.

Important Internal Changes

  • Kafka has been updated to version 2.4.0.

  • Protobuf has been updated to version 3.11.4.

  • The OpenNMS Integration API is now version 0.4.

  • Internally, events are, in most cases, immutable. For the most part, you should see no changes in the external and code places that events are interacted with. They are converted to mutable versions in places where they are often manipulated (like Drools).

Changelog

Release 26.1.2

Release 26.1.2 is the fifth release in the Horizon 26 series.

It contains a bunch of bug fixes and improvements, including docker container optimizations, Netflow changes, documentation updates, time-series enhancements, and more.

The codename for 26.1.2 is Plague.

Bug

  • AbstractSnmpValue.allBytesDisplayable() IndexOutOfBound Exception (Issue NMS-7547)

  • Unable to collect SNMP through minions on a large scale (Issue NMS-10389)

  • Update examples/opennms.conf to be JDK11-compatible (Issue NMS-12468)

  • RRD-to-Newts converter only handles AVERAGE RRAs (Issue NMS-12722)

  • Parameters with dots handled incorrectly in BMP feature config (Issue NMS-12738)

  • The ReST end-point for the Flow Exporter details is returning invalid content (Issue NMS-12740)

  • Netflow 5 records in ES do not contain value for delta_switched (Issue NMS-12750)

  • dependency commons-beanutils 1.8.3 vulnerability (Issue NMS-12757)

  • Template field 'APPLICATION TAG' has illegal size (Issue NMS-12783)

  • Kafka Producer puts all events on the same partition when using donotpersist (Issue NMS-12784)

Enhancement

  • Reduce Docker container image size (Issue NMS-12284)

  • Document how to use meta-data with thresholding (Issue NMS-12735)

  • Add documentation for TcpListener (Issue NMS-12736)

  • upgrade to latest Jetty security/bug fixes (Issue NMS-12743)

  • Run a comparison: implementation before changes and after (Issue NMS-12752)

  • Optimize Performance of InfluxDb Plugin (Issue NMS-12759)

  • Be able to ignore certificate validation on all Karaf features that push data to Elasticsearch (Issue NMS-12768)

  • Evaluate and improve opennms-cortex-tss-plugin (Issue NMS-12771)

  • Provide a test harness for time series plugins (Issue NMS-12772)

Release 26.1.1

Release 26.1.1 is the fourth release in the Horizon 26 series.

It contains a number of documentation and bug fixes including improvements to Kafka event and alarm publishing.

The codename for 26.1.1 is Hydrating.

Bug

  • SSLCertMonitor server-name parameter results in NPE (Issue NMS-12332)

  • Provisiond accepts multiple primary SNMP interfaces (Issue NMS-12605)

  • Fix warnings during documentation build (Issue NMS-12702)

  • Cleanup removed Elasticsearch REST plugin and hint to Plugin Manager (Issue NMS-12716)

  • The Alarm History feature is not working (Issue NMS-12718)

  • Events forwarded by Kafka Producer doesn’t have any parameters set (Issue NMS-12723)

  • Netflow ingress performance regression (Issue NMS-12724)

Enhancement

  • Bump Docker base dependencies in build-env and OCI artifacts (Issue NMS-12699)

  • Send trouble ticket id to kafka alarm topic (Issue NMS-12725)

Release 26.1.0

Release 26.1.0 is the third release in the Horizon 26 series.

It is an enhancement release with a number of bug fixes and improvements, including updates to telemetry, provisioning, and more.

The codename for 26.1.0 is Surgical.

Bug

  • Security vulnerability in io.netty:netty-handler < 4.1.45 (need upgrade) (Issue NMS-12541)

  • NPE in KafkaFlowForwarder (Issue NMS-12660)

  • Add more context to Response Time resources (Kafka Producer) (Issue NMS-12661)

  • BMP parse error for path attribute MP_UNREACH_NLRI (Issue NMS-12671)

  • Reloading the Pollerd daemon causes multiple nodeDown messages (Issue NMS-12681)

  • Streaming Telemetry is broken when using OpenJDK 11 and minion (Issue NMS-12688)

Enhancement

  • Document JDBCQueryMonitor "compare_string" Action (Issue NMS-9581)

  • Add opentracing support for Provisiond (Issue NMS-12374)

  • SystemExecuteMonitor fails with exit code 6 (Issue NMS-12564)

  • Add an example for SystemExecuteMonitor into the docs (Issue NMS-12568)

  • Prometheus collector (Issue NMS-12577)

  • Timeseries Plugin Influx 1.x (Issue NMS-12633)

  • Update smoketests to support various Kafka compression codecs (Issue NMS-12647)

  • Bump ES version used in Smoke Tests (Issue NMS-12648)

  • Provide written procedures on the proper way to restart (Issue NMS-12650)

  • Aggregate flow metrics w/ stream processing (Issue NMS-12656)

  • Provisiond: Add NodeScanStarted event for scheduled scans (Issue NMS-12658)

  • Flow aggregation - alternate indices based on duration of time range filter (Issue NMS-12663)

  • Flow aggregation - Identify minimal set of fields required for current queries (Issue NMS-12664)

  • Enable node enrichment for Topology providers comming from the Integration Api (Issue NMS-12674)

  • Add tooltip support to the LegacyGraphProvider (Issue NMS-12694)

  • add a telemetry adapter for the Graphite plaintext protocol (Issue NMS-12695)

Release 26.0.1

Release 26.0.1 is the second release in the Horizon 26 series.

It is an off-schedule release to fix a vulnerability in ActiveMQ and the Minion. Thanks to Florian Hauser of Code White for catching this one.

The codename for 26.0.1 is Luchador.

Bug

  • Authenticated RCE vulnerability via ActiveMQ Minion payload deserialization (Issue NMS-12673)

Release 26.0.0

Release 26.0.0 is the first release in the Horizon 26 series.

It contains a large number of bug fixes and new features, most notably initial support for handling the BGP Monitoring Protocol in Telemetryd. It also contains a few security fixes for issues reported by Johannes Moritz.

For a high-level overview of what has changed in Horizon 26, see What’s New in OpenNMS Horizon 26.

The codename for 26.0.0 is Balaclava.

Bug

  • RTC subscription events should not be persisted to DB (Issue NMS-9754)

  • Make Events immutable (avoid CMEs and fix non-deterministic behavior) (Issue NMS-10720)

  • Invalid poller filter rules can break service web UI page (Issue NMS-12400)

  • Discovery and foreignSource service detection get in the way (Issue NMS-12411)

  • SNMP Remove from definitions fails for definitions with profile label (Issue NMS-12413)

  • persisted defaultCalendarReport database reports are broken (Issue NMS-12438)

  • Filter related errors in karaf.log when using new search (Issue NMS-12502)

  • Security issue disclosures, 31 Jan 2020 (Issue NMS-12513)

  • Selecting an Icon on Topology Map breaks the map (Issue NMS-12532)

  • BMP parser is to strict for unknown elements / types (Issue NMS-12552)

  • "No future found for message" warnings in telemetryd log (Issue NMS-12565)

  • The health check script for Minion and Sentinel on Docker Images stopped working (Issue NMS-12600)

  • Open Redirect security issues (Issue NMS-12612)

  • PR’s fail circleci RPM build steps due to missing GPG setup (Issue NMS-12615)

  • XSS security issues (Issue NMS-12617)

  • sentinel-coordination-zookeeper doesn’t start due to missing dependency (Issue NMS-12619)

  • Description: Cannot create monitored-service with JSON via ReST (Issue NMS-12625)

  • Minion Docker image for develop is tagged as 27.0.0-SNAPSHOT instead of bleeding (Issue NMS-12627)

  • Push Minion OCI to DockerHub for release branches (Issue NMS-12630)

  • Restore CAP_NET_RAW capabilities in Minion when running as non-root (Issue NMS-12635)

  • Can’t change password using the user self service (Issue NMS-12636)

  • GraphService is throwing Error with an NPE Karaf startup (Issue NMS-12637)

  • Telemetryd with BMP adapter throws java.util.ConcurrentModificationException (Issue NMS-12638)

  • Confd download fails silently on Docker install (Issue NMS-12642)

  • Error parsing MP_UNREACH_NLRI attribute (Issue NMS-12643)

  • BMP Parser Bulkhead Config does not work (Issue NMS-12644)

  • Error parsing label information from BGP MP_REACH_NLRI attribute (Issue NMS-12649)

Enhancement

  • Add more information into BSM alarms (Issue NMS-9352)

  • Prefix all shell commands with "opennms" (Issue NMS-10413)

  • Shorten log message in SNMP trap event definitions (Issue NMS-10564)

  • Clarify Basic Installation scenario (Issue NMS-11812)

  • Provide DefaultFocus capabilities (Issue NMS-11979)

  • Provide a better graph service with an actual API/Model and better import/export/integration capabilities (Issue NMS-12086)

  • Rework the current navbar (Issue NMS-12128)

  • Event documentation is missing tokens (Issue NMS-12228)

  • Add a "Delete" button on the Node page of the Requisition UI (Issue NMS-12292)

  • Discovery enhancements (Issue NMS-12314)

  • Update discoveryd to support detectors (Issue NMS-12315)

  • Provide criteria for determining the "management IP" in provisiond (Issue NMS-12316)

  • Process newSuspect events in a single threaded fashion (Issue NMS-12318)

  • Provide Central Search Function (Issue NMS-12347)

  • Create BMP listener and parser for telemetryd (Issue NMS-12358)

  • Add gRPC support for IPC between Minion & OpenNMS (Issue NMS-12372)

  • Allow querying a graph using vertices in focus and a szl (Issue NMS-12376)

  • Implement Enrichment of vertices (Issue NMS-12382)

  • Develop Timeseries Integration Layer (Issue NMS-12383)

  • Allow persisting edges with references to at least one vertex which is not in the edge’s namespace (Issue NMS-12388)

  • Implement caching strategies for graphs (Issue NMS-12389)

  • Add HTTP header injection in HTTP Monitors (Issue NMS-12393)

  • Add possibility to listen for graph changes (Issue NMS-12408)

  • Create BMP Adapter for Telemetry (Issue NMS-12414)

  • Create BMP Adapter for Peer Up / Down Events (Issue NMS-12415)

  • Allow CIDR notation in our IP filter implementation (Issue NMS-12423)

  • Create BMP Adapter forwarding to OpenBMP (Issue NMS-12424)

  • Create graph definitions for BMP statistics (Issue NMS-12425)

  • Add BMP config example and documentation (Issue NMS-12426)

  • Use Router Id (and maybe AS) to associate node with exporting router’s data (Issue NMS-12436)

  • Provide basic development documentation (Issue NMS-12441)

  • Integrate the new Graph Service API with the OpenNMS Integration API (Issue NMS-12445)

  • Remove getVertexType() on GraphInfo (Issue NMS-12447)

  • Avoid rebuilding the graph view when enriching (Issue NMS-12448)

  • Expose status information when fetching a graph view (Issue NMS-12453)

  • DatacollectionFailed event definitions are located in wrong file (Issue NMS-12471)

  • dataCollectionSucceeded does event auto-clean (Issue NMS-12474)

  • Remove obsolete entry in log4j2.xml (Issue NMS-12475)

  • Make Kafka RPC topics configurable to use module in topic names (Issue NMS-12479)

  • Docker Image Improvements (Issue NMS-12481)

  • Reduce Minion docker image size (Issue NMS-12482)

  • Publish arm64 and armhf Docker images for Minion (Issue NMS-12483)

  • Use jicmp (and jicmp6) by default in Minion Docker images (Issue NMS-12484)

  • Implement GRPC Server that can route all RPC/Sink messages from OpenNMS to Minion and vice versa (Issue NMS-12486)

  • Use protobuf instead of bson for encoding/decoding Netflow payloads (Issue NMS-12521)

  • Enrich content of nodeAdded event (Issue NMS-12526)

  • Migrate config-tester wiki to the docs (Issue NMS-12527)

  • Splitting Docker documentation in Horizon, Minion and Sentinel (Issue NMS-12529)

  • Add Jolokia features to Minion & Sentinel (Issue NMS-12533)

  • Expose OnmsIpInterface, OnmsSnmpInterface, others as top-level resources in REST API (Issue NMS-12538)

  • Use ProtoBuf to transport parsed BMP messages (Issue NMS-12547)

  • Add support for per AFI/SAFI statistics (Issue NMS-12553)

  • Add basic system test for BMP processing (Issue NMS-12554)

  • Improve parsing of BGP extended communities attribute (Issue NMS-12559)

  • Populate path id and labels attributes in unicast prefix messages (OpenBMP integration) (Issue NMS-12560)

  • Async DNS resolution for Hostnames in BMP (Issue NMS-12569)

  • Add support for Local RIB (Issue NMS-12570)

  • Parse BGP Capabilities (Issue NMS-12571)

  • Refine parameter handling in Adapters (Issue NMS-12573)

  • Apply more sensible defaults to OpenBMP kafka producer (Issue NMS-12574)

  • Confd templates for Minion configuration (Issue NMS-12578)

  • Improve node cache in flow document enrichment (Issue NMS-12580)

  • Improve OIA performance when mapping alarms (Issue NMS-12581)

  • Upgrade Kafka components to 2.4.0 (Issue NMS-12582)

  • Write enriched flows to Kafka (Issue NMS-12583)

  • Create threshold documentation (Issue NMS-12588)

  • Document how to generate PDFs from dashboards using OpenNMS (Issue NMS-12599)

  • Minion should bind to 0.0.0.0 by default for SNMP traps (Issue NMS-12626)

  • Minion confd template should disable JMS when using Kafka (Issue NMS-12631)

  • Add required dependencies to use ZSTD inside Kafka to features.xml (Issue NMS-12639)

  • Set RPM compression type and level inside RPM Spec Files (Issue NMS-12640)

  • Support for more extended community types in BMP (Issue NMS-12641)