Copyright © 2004-2020 The OpenNMS Group, Inc.

Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.1 or any later version published by the Free Software Foundation; with no Invariant Sections, with no Front-Cover Texts and with no Back-Cover Texts. A copy of the license is available at http://www.gnu.org/copyleft/fdl.html OpenNMS is the creation of numerous people and organizations, operating under the umbrella of the OpenNMS project. The source code is published under the GNU Affero GPL, version 3 or later and is Copyright © 2002-2020 The OpenNMS Group, Inc.

The current corporate sponsor of OpenNMS is The OpenNMS Group, which also owns the OpenNMS trademark.

Please report any omissions or corrections to this document by creating an issue at http://issues.opennms.org.

What’s New in OpenNMS Horizon 27

System Requirements

  • Java 8 through 11: OpenNMS Horizon 27 runs on JDK 8 through 11. We recommend the most recent version of OpenJDK 11.

  • PostgreSQL 10 or higher: Horizon 27 requires any supported version of PostgreSQL from 10 up to (and including) 13.

Breaking Changes

Remote Poller Removed

The remote poller has been removed from Horizon 27. It has been replaced with new infrastructure for perspective-based monitoring that runs on Minions.

VMware Assets

VMware-related data is not longer stored in node assets. The install command will move any existing VMware asset data to the node’s metadata when you upgrade.

New Features and Improvements

Karaf Shell

A number of useful commands have been added or updated in the Karaf shell.

  • alarm-list: list current alarms (optionally filtered by UEI)

  • event-list: list events (optionally filtered by UEI)

  • ip-addr-range: generate a list of IP addresses to use in other commands

  • reload-daemon: now accepts an optional configuration file to target for reload

  • send-event: additional options supported to match the CLI version of send-event more closely

  • snmp-fit: can now optionally send a newSuspect event on success

  • snmp-get: like snmp-walk, but fetches a single MIB object and prints the result

Minion and Sentinel

  • The Minion confd configuration support now handles a bunch more options, including setting Jetty, RMI, and ssh host and port, Secure Credentials Vault configuration, and environment variables and options for the JVM. For details, see the CONFD_README file.

  • It’s now possible to override entries in a Sentinel’s system.properties with a file called custom.system.properties

Application Perspective Monitoring

Application Perspective Monitoring replaces the old Remote Poller.

It uses the Minion infrastructure to monitor a service’s availability from these different perspectives. When a service is not responsive an outage will be generated with the corresponding perspective.

A number of provisioning, polling, and web UI changes have been made to accomodate this new infrastructure.

Provisioning

  • Node scan has been updated to limit the number of concurrent RPC requests so as not to overload communication with Minions.

  • New policies have been added for creating node and interface metadata during provisioning.

Polling and Monitoring

  • WMI/SMB support has been updated to use a wider set of encryption types.

  • Data collection configuration has been added for Netscaler vServer.

Thresholding

  • It is now possible to use metadata in threshold evaluation.

RPC, Telemetry, Flows, and BMP

  • Improvements have been made for supporting aggregate flow metrics.

  • Flow DNS enrichment has been enhanced to improve performance.

  • Aggregate flow queries now support resolving hostnames if available.

  • Sink RPC APIs have been enhanced to expose metrics to Karaf for tracing and performance tuning.

  • The OpenConfig integration now supports gNMI, in addition to JTI.

Notifications

  • It is now possible to use node-, interface-, and service-level metadata in notifications.

ReST API

  • Events for applicationCreated/applicationDeleted are now sent when applications are added or removed through the ReST API.

  • A new v2 API for retrieving application status has been added.

Web UI Improvements

  • A new universal search bar has been added which allows for searching alarms, nodes, and more from the top of any page in the UI.

Documentation Updates

Documentation improvements and additions have been made in a number of areas:

  • Application Perspective Monitoring

  • Data Collection Configuration

  • Metadata

  • Mail Transport Monitor

  • Minion/Kafka RPC Tuning

  • Thresholding

  • Users Configuration, Duty Schedules, and Security Roles

…​plus TONS of smaller readability improvements.

Kafka Producer Metrics

  • For Interface Resources, a String attribute named __ifIndex is added to represent missing Interface ifIndex in String form.

Important Internal Changes

  • Sink internals have been updated to use Protobuf 3.

Changelog

Release 27.1.0

Release 27.1.0 contains a bunch of bug fixes, as well as a number of enhancements including a refactor of our BGP Monitoring Protocol integration.

The codename for 27.1.0 is Ravenous Bugblatter Beast of Traal.

Bug

  • Notification link in Admin menu goes to start page (Issue NMS-8184)

  • Change password does not go to Change password page (Issue NMS-8770)

  • Wrong label in event search menu (Issue NMS-8977)

  • Footer in Alarm view is broken (Issue NMS-9139)

  • Clarify clone detectors and policies (Issue NMS-9308)

  • REST API allows to import a requistion from XML with same services (Issue NMS-12739)

  • Broken provisiond policy does not appear in the logs (Issue NMS-12834)

  • Vacuumd throws NullPointer Exception on startup (Issue NMS-12884)

  • Change installation guide to run as non-root user (Issue NMS-12989)

  • Cortex plugin shows only data for counters in a window > 3 hours (Issue NMS-13014)

  • Timezone and Grafana Dashboard fields not preserved when editing a scheduled report (Issue NMS-13064)

  • Flow Rest API: Name lookup is happening per bucket and not per interval (Aggregation Only) (Issue NMS-13065)

  • Make OpenNMS compile on Apple Silicon (Issue NMS-13079)

  • Fix NPE in Vaccumd (Issue NMS-13098)

  • Make sequence number trackin thread-save and patient about out-of-order (Issue NMS-13106)

  • Vmware-importer requisition meta-data lost at import (Issue NMS-13109)

  • XSS in user management (Issue NMS-13125)

  • No option provided to change the number of records per page in Events ILP and Events/Alarms ILP under Topology (Issue NMS-13137)

  • The OpenNMS Web User Interface Has Experienced an Error observed when searching for a Event under Event notifications (Issue NMS-13148)

  • Node’s sub-option 'Availability' exceeds table alignment and overlaps next table of 'Notifications' under Topology section (Issue NMS-13153)

  • Newts Cache priming flag is inverted (Issue NMS-13156)

  • Dependabot: Upgrade Apache POI to 3.17 (CVE-2017-12626) (Issue NMS-13161)

Enhancement

  • Cloning of Foreign Source Definition (Issue NMS-8854)

  • Depend on haveged (and supply it in our repo) (Issue NMS-8959)

  • Rename button on Threshold Groups screen (Issue NMS-12606)

  • Upgrade Kafka components to 2.7.0 (Issue NMS-12704)

  • Handle all possible values of ifOperStatus in the SNMP Interface Poller (Issue NMS-12980)

  • Remote poller: review all documentation to update RP references (Issue NMS-12879)

  • Persist basic BMP messages in OpenNMS (Issue NMS-12949)

  • Handle stats for BMP (Issue NMS-12952)

  • Get dashboards from OpenBMP working (Issue NMS-12953)

  • Enhance Routes with ASN info (Issue NMS-12963)

  • Enhance Routes with WhoIs info (Issue NMS-12964)

  • Enhance Routes with RPKI info (Issue NMS-12965)

  • Create confd templates to handle Slack properties (Issue NMS-12982)

  • Create confd templates to handle Mattermost properties (Issue NMS-13004)

  • Updating UI to clarify rescan process (Issue NMS-13040)

  • Update PostgreSQL JDBC drivers (Issue NMS-13049)

  • Move Stats handling to TimeScaleDB (Issue NMS-13061)

  • Upgrade Container base images (Issue NMS-13071)

  • Add SNMP Data Collection example to Horizon Docs (Issue NMS-13107)

  • Nephron: Replace JacksonJsonCoder for FlowSummaries (Issue NMS-13115)

  • Nephron: fix rounding errors in flow sampling (Issue NMS-13116)

  • Create smoke test that verifies all Reload daemon are successful (Issue NMS-13117)

  • Searching node link informations (Issue NMS-13129)

  • add service status to rest /info API (Issue NMS-13135)

  • Update Kafka settings for multiple instances documentation (Issue NMS-13136)

  • Move some model objects from opennms-webapp to opennms-web-api (Issue NMS-13168)

  • create a table to show related events in the alarm detail view (Issue NMS-13170)

Release 27.0.5

Release 27.0.5 contains just a couple of bug fixes, most importantly a fix to provisioning that could break update to existing nodes when scanning.

The codename for 27.0.5 is Vogon.

Bug

  • Flow Rest API: Name lookup is happening per bucket and not per interval (Aggregation Only) (Issue NMS-13065)

  • Provisioning stopped working after upgrade to 27.0.4 (Issue NMS-13128)

Release 27.0.4

Release 27.0.4 contains a number of bug fixes relating to WMI, the Minion, flows, reports, JEXL processing, and more, as well as a few small enhancements.

The codename for 27.0.4 is Towel.

Bug

  • Searching for event context that contains single quotes is not possible (Issue NMS-12517)

  • WmiCollector does not work on Minion (Issue NMS-12944)

  • Discrepancy between Flows "top N" and SNMP for same interface (Issue NMS-12967)

  • Increase length of snmpinterfaces.snmpifdescr (Issue NMS-12976)

  • requisition meta data are deleted if node meta data defined with "db only" synchronize (Issue NMS-12990)

  • Timezone and date range inconsistencies when scheduling database reports associated with Grafana dashboards. (Issue NMS-13070)

  • Exception messages during node import (log noise) (Issue NMS-13082)

  • Fix vmwareconfigbuilder script to run with JDK9+ (Issue NMS-13084)

  • Keep and adjust flows with negative duration (Issue NMS-13088)

  • SFlow enhancment is not functional (Issue NMS-13093)

  • Karaf Poller commands won’t work if Telemetryd is disabled (Issue NMS-13094)

  • JEXL expression handling updates (Issue NMS-13103)

  • Telemetryd: Reload daemon always fails and stops Temetryd (Issue NMS-13112)

Enhancement

  • Optionally silence file not found warnings for JICMP, JRRD when properties are not set (Issue NMS-13081)

  • Update opennms.spec to take advantage of maven smart builder plugin (Issue NMS-13083)

Release 27.0.3

Release 27.0.3 is the fourth release in the Horizon 27 series.

It contains a number of mostly esoteric bug fixes, and a few small enhancements.

The codename for 27.0.3 is Dolphins.

Bug

  • Prometheus Collector attempting to persist non-integer values to counters (Issue NMS-13007)

  • OpenNMS fails to start, when more than one active listener is referencing the same parser (Issue NMS-13034)

  • ArrayIndexOutOfBoundsException thrown by the SNMP Interface Poller (Issue NMS-13042)

  • Karaf command 'snmp-fit' not functional (Issue NMS-13055)

  • Long datasource names are truncated and are not accessable on read (Issue NMS-13060)

  • The makerpm.sh script requires the mingw32-nsis package (Issue NMS-13069)

Enhancement

  • Depend on haveged (and supply it in our repo) (Issue NMS-8959)

  • Import OBMP Grafana dashboards to OpenNMS (Issue NMS-12954)

  • Inconsistent breadcrumbs on Locations/Minions (Issue NMS-13067)

Release 27.0.2

Release 27.0.2 is the third release in the Horizon 27 series.

It contains a few bug fixes including a critical fix to a regression in 27.0.1 that would keep OpenNMS from starting if you had existing reports in the database.

BREAKING: If you created any reports with 27.0.1, they will need to be recreated.

The codename for 27.0.2 is Deep Thought.

Bug

  • Bouncycastle JAR version 1.67 breaks CIFS Monitor (Issue NMS-13035)

  • report timezone changes break reading pre-existing reports from Quartz (Issue NMS-13037)

Enhancement

  • Add "geohash" support to the meta-data DSL (Issue NMS-13036)

Release 27.0.1

Release 27.0.1 is the second release in the Horizon 27 series.

It contains a number of bugfixes including a critical CVE fix for Jetty, as well as a number of other smaller changes and improvements.

The codename for 27.0.1 is Pan-Galactic Gargle Blaster.

Bug

  • No support for TLS v1.3 in MailTransportMonitor (Issue NMS-12733)

  • Timezone inconsistency when generating PDF reports from Grafana dashboards (Issue NMS-12930)

  • RRD files for SNMP data are not created until a Service Restart (Issue NMS-12974)

  • Nephron Stability Issues at Scale (Issue NMS-12975)

  • typo in BMP sample (Issue NMS-12984)

  • Upgrade script does not migrate VMware metadata (Issue NMS-12985)

  • VMware datacollection failed (Issue NMS-12986)

  • Monitoring VMware VMs and hosts stopped working in 27.0.0 (Issue NMS-12988)

  • Unable to enable Jaeger tracing in Sentinel (Issue NMS-12998)

  • Update typo in BMP docs (Issue NMS-13002)

  • CVE-2020-27216: Jetty webserver vulnerability (Issue NMS-13009)

  • JCifs leaks memory after upgrade (Issue NMS-13011)

  • When using a custom prefix, the Elasticsearch Forwarder for events and situation-feedback creates a wrong template. (Issue NMS-13017)

  • Check flow sequence numbers to detect missing packets (Issue NMS-13024)

Enhancement

  • Add gNMI support for OpenConfig (Issue NMS-12915)

  • TSS: Cortex Plugin: Use REST API for reading timeseries (Issue NMS-12938)

  • sFlow Ingress / Egress (Issue NMS-12955)

  • Provisioning introduction (Issue NMS-12994)

  • Document Configure Discovery process (Issue NMS-12995)

  • Document requisition process (Issue NMS-12996)

  • backport Minion certificate management to Horizon 27 (Issue NMS-13000)

  • flowStartMilliseconds/flowEndMilliseconds for NetFlow v9 (Issue NMS-13006)

  • Add clock skew correction mechanism (Issue NMS-13023)

Release 27.0.0

Release 27.0.0 is the first release in the Horizon 27 series.

It is a major enhancement release with many bug fixes and a ton of new features, most notably the removal of the Remote Poller in favor of a new Minion-based workflow called Application Perspective Monitoring.

The codename for 27.0.0 is Zaphod.

Bug

  • Investigate if the InstallerDB could be replaced with something simpler (Issue NMS-12079)

  • Limit RPC threads on Minion using bulkhead pattern (Issue NMS-12391)

  • RRD-to-Newts converter only handles AVERAGE RRAs (Issue NMS-12722)

  • Fix docs warnings for resource-types, time series config and thresholding (Issue NMS-12770)

  • Traps Not Associated With Node (Issue NMS-12797)

  • Netflow timestamps incorrectly calculated on interfaces with MPLS (Issue NMS-12800)

  • SNMPv3 Traps Cause Errors From Single Node to Minion (Issue NMS-12802)

  • Hardcoded path to bash (Issue NMS-12840)

  • Remote Poller: add remote outages to UI (Issue NMS-12842)

  • Remote Poller: add remote polling details to service page (Issue NMS-12843)

  • Remove polling package selection from application (Issue NMS-12844)

  • Remote Poller: remove not longer used role ROLE_REMOTING (Issue NMS-12849)

  • Remote Poller: remove references to old remote poller from documentation (Issue NMS-12850)

  • Remote Poller: remove scanreports UI and ReST endpoint (Issue NMS-12851)

  • Remote Poller: remove old Distributed Status Summary UI (Issue NMS-12852)

  • Incorrect TSS settings (Issue NMS-12862)

  • Remote Poller: Fix event definition for remote poller nodeLostService events (Issue NMS-12871)

  • Remote Poller: NPE while attempting to start RemotePollerNG (Issue NMS-12877)

  • Unable to install feature 'dominion-grpc-client' (Issue NMS-12880)

  • Remote Poller: Clear outages after removing from application (Issue NMS-12881)

  • Perspective Poller shows in tracing as RemotePollerNG (Issue NMS-12882)

  • Application Perspective Monitoring: NPE when starting OpenNMS (Issue NMS-12887)

  • Application Perspective Monitoring: OpenNMS refuses to start if service is referenced by two applications (Issue NMS-12889)

  • Telemetryd: a lot of InstanceAlreadyExistsExceptions when starting OpenNMS with default configuration (Issue NMS-12896)

  • Update link to In Memory TS DB (Issue NMS-12912)

  • Unable to poll Vcenter CIM - Calling something in OpenJDK11 that has been removed. (Issue NMS-12919)

  • Application link on start page redirects to start page (Issue NMS-12921)

  • Event Translator debug logging is incorrect (Issue NMS-12959)

  • service starts / restarts work but spit out an error if configured to wait for startup (Issue NMS-12966)

  • Display the alarm status correctly in topology map for applications (Issue NMS-12968)

  • Topology Application Map: Outage Table: Clicking on a service should show the outages of the service (Issue NMS-12970)

  • Alarm (v1 & v2) ReST Service PUT Can’t PUT Multiple Things (Issue NMS-12979)

Enhancement

  • Add custom string attributes based on indirect and complex SNMP Indices (Issue NMS-8484)

  • Trailing ", \" in report definitions throws not helpful error message (Issue NMS-10526)

  • nodeDeleted event should contain more information (Issue NMS-10554)

  • Netscaler vServer SNMP collection and graph definitions (Issue NMS-12269)

  • Upgrade Sink API to Proto3 (Issue NMS-12602)

  • Add selenium test for password change (Issue NMS-12645)

  • Re-enable Central Search functionality (Issue NMS-12646)

  • Remove remote-poller runtimes from the build (Issue NMS-12653)

  • Update remote-poller model to link service to locations instead of individual RPs (Issue NMS-12654)

  • Rewrite the remote poller backend to use Minion (Issue NMS-12655)

  • Dynamic service scheduling: Service lifetime (Issue NMS-12676)

  • Dynamic service scheduling: Polling package association (Issue NMS-12677)

  • Dynamic service scheduling: Poller-Configuration change (Issue NMS-12678)

  • Update database and send events only on status changes (Issue NMS-12679)

  • Get new remote poller running as daemon (Issue NMS-12683)

  • Add upgrade task to enable RemotePollerNG (Issue NMS-12684)

  • Confd templates for Minion configuration (Karaf) (Issue NMS-12687)

  • Support hostnames resolution when using aggregated flows (Issue NMS-12692)

  • Use CollectionSetBuilder to persist response times (Issue NMS-12698)

  • Create SQL index for optimizing poll result lookups (Issue NMS-12710)

  • Apply thresholding for remote poller response times (Issue NMS-12721)

  • Meta-data tag enhancements to Time Series Storage API (Issue NMS-12730)

  • remove the java ("windows") installer (Issue NMS-12742)

  • Add ReST interface for Remote Poller (Issue NMS-12760)

  • Anomaly Detection Experiment (Issue NMS-12773)

  • Anomaly Detection - Get the consumer working (Issue NMS-12774)

  • Migrated VMware asset data to metadata entries (Issue NMS-12781)

  • RemotePoller: only schedule services that are part of an application (Issue NMS-12785)

  • Use newer protocol versions for remote DCOM WMI (Issue NMS-12788)

  • Add support for OpenConfig Service in Detector/Poller (Issue NMS-12789)

  • Remote Poller: Add outages for remote poller status changes (Issue NMS-12792)

  • Add support for meta-data on single-DS threshold definitions (Issue NMS-12794)

  • Update Users chapter (Issue NMS-12809)

  • Remote Poller: Modify model to new structure (Issue NMS-12820)

  • Remote Poller: Change monitoring locations UI to reflect model changes (Issue NMS-12821)

  • Remote Poller: Change Admin UI to reflect Application model changes (Issue NMS-12822)

  • Application Topology Provider Status (Issue NMS-12823)

  • Allow meta-data in notifications (Issue NMS-12827)

  • Add events for new Application Model (Issue NMS-12838)

  • Remote Poller: camel case in new column definitions causing problems (Issue NMS-12841)

  • Remote Poller: Remove LocationSpecificStatus (Issue NMS-12859)

  • Remote Poller: Documentation (Issue NMS-12860)

  • Remote Poller: Fix the remote flag (Issue NMS-12861)

  • Remote Poller: Cleanup (Issue NMS-12864)

  • Remote Poller: Clearify perspecive labeling (Issue NMS-12865)

  • Remote Poller: Details pages show remote outages in recent outages list (Issue NMS-12866)

  • Remote Poller: Add graph definitions for response times (Issue NMS-12868)

  • Remote Poller: Add distributed tracing (Issue NMS-12869)

  • Remote Poller: Rename to Perspective Poller (Issue NMS-12874)

  • Remote Poller: Make backend more resilient (Issue NMS-12878)

  • Application Perspective Monitoring: create Application status page (Issue NMS-12886)

  • Document the MailTransportMonitor (Issue NMS-12892)

  • Additions to Application Perspective Monitoring docs (Issue NMS-12893)

  • Filter outages table in Application Topology View (Issue NMS-12897)

  • Add logging configuration for Perspective Poller (Issue NMS-12910)

  • Allow to navigate to monitored items in application status view (Issue NMS-12913)

  • Allow setting meta data in a requisition (Issue NMS-12918)

  • Create a report that matches Horizon 27.0.0 Jira issues with merged pull requests in GitHub (Issue NMS-12922)

  • Update Copyright notice for 2020 (Issue NMS-12933)

  • SSLCertMonitor should include more details about the expir(ing|ed) certificate in reason codes (Issue NMS-12948)

  • Update Maximum PostgreSQL to allow PostgreSQL 13 (Issue NMS-12958)

  • Create Horizon 27 Release Notes (Issue NMS-12961)

  • Identify message broker strategies in web "about" page (Issue NMS-12971)