Copyright © 2004-2020 The OpenNMS Group, Inc.

Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.1 or any later version published by the Free Software Foundation; with no Invariant Sections, with no Front-Cover Texts and with no Back-Cover Texts. A copy of the license is available at http://www.gnu.org/copyleft/fdl.html OpenNMS is the creation of numerous people and organizations, operating under the umbrella of the OpenNMS project. The source code is published under the GNU Affero GPL, version 3 or later and is Copyright © 2002-2020 The OpenNMS Group, Inc.

The current corporate sponsor of OpenNMS is The OpenNMS Group, which also owns the OpenNMS trademark.

Please report any omissions or corrections to this document by creating an issue at http://issues.opennms.org.

What’s New in OpenNMS Horizon 27

System Requirements

  • Java 8 through 11: OpenNMS Horizon 27 runs on JDK 8 through 11. We recommend the most recent version of OpenJDK 11.

  • PostgreSQL 10 or higher: Horizon 27 requires any supported version of PostgreSQL from 10 up to (and including) 13.

Breaking Changes

Remote Poller Removed

The remote poller has been removed from Horizon 27. It has been replaced with new infrastructure for perspective-based monitoring that runs on Minions.

VMware Assets

VMware-related data is not longer stored in node assets. The install command will move any existing VMware asset data to the node’s metadata when you upgrade.

New Features and Improvements

Karaf Shell

A number of useful commands have been added or updated in the Karaf shell.

  • alarm-list: list current alarms (optionally filtered by UEI)

  • event-list: list events (optionally filtered by UEI)

  • ip-addr-range: generate a list of IP addresses to use in other commands

  • reload-daemon: now accepts an optional configuration file to target for reload

  • send-event: additional options supported to match the CLI version of send-event more closely

  • snmp-fit: can now optionally send a newSuspect event on success

  • snmp-get: like snmp-walk, but fetches a single MIB object and prints the result

Minion and Sentinel

  • The Minion confd configuration support now handles a bunch more options, including setting Jetty, RMI, and ssh host and port, Secure Credentials Vault configuration, and environment variables and options for the JVM. For details, see the CONFD_README file.

  • It’s now possible to override entries in a Sentinel’s system.properties with a file called custom.system.properties

Application Perspective Monitoring

Application Perspective Monitoring replaces the old Remote Poller.

It uses the Minion infrastructure to monitor a service’s availability from these different perspectives. When a service is not responsive an outage will be generated with the corresponding perspective.

A number of provisioning, polling, and web UI changes have been made to accomodate this new infrastructure.

Provisioning

  • Node scan has been updated to limit the number of concurrent RPC requests so as not to overload communication with Minions.

  • New policies have been added for creating node and interface metadata during provisioning.

Polling and Monitoring

  • WMI/SMB support has been updated to use a wider set of encryption types.

  • Data collection configuration has been added for Netscaler vServer.

Thresholding

  • It is now possible to use metadata in threshold evaluation.

RPC, Telemetry, Flows, and BMP

  • Improvements have been made for supporting aggregate flow metrics.

  • Flow DNS enrichment has been enhanced to improve performance.

  • Aggregate flow queries now support resolving hostnames if available.

  • Sink RPC APIs have been enhanced to expose metrics to Karaf for tracing and performance tuning.

  • The OpenConfig integration now supports gNMI, in addition to JTI.

Notifications

  • It is now possible to use node-, interface-, and service-level metadata in notifications.

ReST API

  • Events for applicationCreated/applicationDeleted are now sent when applications are added or removed through the ReST API.

  • A new v2 API for retrieving application status has been added.

Web UI Improvements

  • A new universal search bar has been added which allows for searching alarms, nodes, and more from the top of any page in the UI.

Documentation Updates

Documentation improvements and additions have been made in a number of areas:

  • Application Perspective Monitoring

  • Data Collection Configuration

  • Metadata

  • Mail Transport Monitor

  • Minion/Kafka RPC Tuning

  • Thresholding

  • Users Configuration, Duty Schedules, and Security Roles

…​plus TONS of smaller readability improvements.

Important Internal Changes

  • Sink internals have been updated to use Protobuf 3.

Changelog

Release 27.0.3

Release 27.0.3 is the fourth release in the Horizon 27 series.

It contains a number of mostly esoteric bug fixes, and a few small enhancements.

The codename for 27.0.3 is Dolphins.

Bug

  • Prometheus Collector attempting to persist non-integer values to counters (Issue NMS-13007)

  • OpenNMS fails to start, when more than one active listener is referencing the same parser (Issue NMS-13034)

  • ArrayIndexOutOfBoundsException thrown by the SNMP Interface Poller (Issue NMS-13042)

  • Karaf command 'snmp-fit' not functional (Issue NMS-13055)

  • Long datasource names are truncated and are not accessable on read (Issue NMS-13060)

  • The makerpm.sh script requires the mingw32-nsis package (Issue NMS-13069)

Enhancement

  • Depend on haveged (and supply it in our repo) (Issue NMS-8959)

  • Import OBMP Grafana dashboards to OpenNMS (Issue NMS-12954)

  • Inconsistent breadcrumbs on Locations/Minions (Issue NMS-13067)

Release 27.0.2

Release 27.0.2 is the third release in the Horizon 27 series.

It contains a few bug fixes including a critical fix to a regression in 27.0.1 that would keep OpenNMS from starting if you had existing reports in the database.

BREAKING: If you created any reports with 27.0.1, they will need to be recreated.

The codename for 27.0.2 is Deep Thought.

Bug

  • Bouncycastle JAR version 1.67 breaks CIFS Monitor (Issue NMS-13035)

  • report timezone changes break reading pre-existing reports from Quartz (Issue NMS-13037)

Enhancement

  • Add "geohash" support to the meta-data DSL (Issue NMS-13036)

Release 27.0.1

Release 27.0.1 is the second release in the Horizon 27 series.

It contains a number of bugfixes including a critical CVE fix for Jetty, as well as a number of other smaller changes and improvements.

The codename for 27.0.1 is Pan-Galactic Gargle Blaster.

Bug

  • No support for TLS v1.3 in MailTransportMonitor (Issue NMS-12733)

  • Timezone inconsistency when generating PDF reports from Grafana dashboards (Issue NMS-12930)

  • RRD files for SNMP data are not created until a Service Restart (Issue NMS-12974)

  • Nephron Stability Issues at Scale (Issue NMS-12975)

  • typo in BMP sample (Issue NMS-12984)

  • Upgrade script does not migrate VMware metadata (Issue NMS-12985)

  • VMware datacollection failed (Issue NMS-12986)

  • Monitoring VMware VMs and hosts stopped working in 27.0.0 (Issue NMS-12988)

  • Unable to enable Jaeger tracing in Sentinel (Issue NMS-12998)

  • Update typo in BMP docs (Issue NMS-13002)

  • CVE-2020-27216: Jetty webserver vulnerability (Issue NMS-13009)

  • JCifs leaks memory after upgrade (Issue NMS-13011)

  • When using a custom prefix, the Elasticsearch Forwarder for events and situation-feedback creates a wrong template. (Issue NMS-13017)

  • Check flow sequence numbers to detect missing packets (Issue NMS-13024)

Enhancement

  • Add gNMI support for OpenConfig (Issue NMS-12915)

  • TSS: Cortex Plugin: Use REST API for reading timeseries (Issue NMS-12938)

  • sFlow Ingress / Egress (Issue NMS-12955)

  • Provisioning introduction (Issue NMS-12994)

  • Document Configure Discovery process (Issue NMS-12995)

  • Document requisition process (Issue NMS-12996)

  • backport Minion certificate management to Horizon 27 (Issue NMS-13000)

  • flowStartMilliseconds/flowEndMilliseconds for NetFlow v9 (Issue NMS-13006)

  • Add clock skew correction mechanism (Issue NMS-13023)

Release 27.0.0

Release 27.0.0 is the first release in the Horizon 27 series.

It is a major enhancement release with many bug fixes and a ton of new features, most notably the removal of the Remote Poller in favor of a new Minion-based workflow called Application Perspective Monitoring.

The codename for 27.0.0 is Zaphod.

Bug

  • Investigate if the InstallerDB could be replaced with something simpler (Issue NMS-12079)

  • Limit RPC threads on Minion using bulkhead pattern (Issue NMS-12391)

  • RRD-to-Newts converter only handles AVERAGE RRAs (Issue NMS-12722)

  • Fix docs warnings for resource-types, time series config and thresholding (Issue NMS-12770)

  • Traps Not Associated With Node (Issue NMS-12797)

  • Netflow timestamps incorrectly calculated on interfaces with MPLS (Issue NMS-12800)

  • SNMPv3 Traps Cause Errors From Single Node to Minion (Issue NMS-12802)

  • Hardcoded path to bash (Issue NMS-12840)

  • Remote Poller: add remote outages to UI (Issue NMS-12842)

  • Remote Poller: add remote polling details to service page (Issue NMS-12843)

  • Remove polling package selection from application (Issue NMS-12844)

  • Remote Poller: remove not longer used role ROLE_REMOTING (Issue NMS-12849)

  • Remote Poller: remove references to old remote poller from documentation (Issue NMS-12850)

  • Remote Poller: remove scanreports UI and ReST endpoint (Issue NMS-12851)

  • Remote Poller: remove old Distributed Status Summary UI (Issue NMS-12852)

  • Incorrect TSS settings (Issue NMS-12862)

  • Remote Poller: Fix event definition for remote poller nodeLostService events (Issue NMS-12871)

  • Remote Poller: NPE while attempting to start RemotePollerNG (Issue NMS-12877)

  • Unable to install feature 'dominion-grpc-client' (Issue NMS-12880)

  • Remote Poller: Clear outages after removing from application (Issue NMS-12881)

  • Perspective Poller shows in tracing as RemotePollerNG (Issue NMS-12882)

  • Application Perspective Monitoring: NPE when starting OpenNMS (Issue NMS-12887)

  • Application Perspective Monitoring: OpenNMS refuses to start if service is referenced by two applications (Issue NMS-12889)

  • Telemetryd: a lot of InstanceAlreadyExistsExceptions when starting OpenNMS with default configuration (Issue NMS-12896)

  • Update link to In Memory TS DB (Issue NMS-12912)

  • Unable to poll Vcenter CIM - Calling something in OpenJDK11 that has been removed. (Issue NMS-12919)

  • Application link on start page redirects to start page (Issue NMS-12921)

  • Event Translator debug logging is incorrect (Issue NMS-12959)

  • service starts / restarts work but spit out an error if configured to wait for startup (Issue NMS-12966)

  • Display the alarm status correctly in topology map for applications (Issue NMS-12968)

  • Topology Application Map: Outage Table: Clicking on a service should show the outages of the service (Issue NMS-12970)

  • Alarm (v1 & v2) ReST Service PUT Can’t PUT Multiple Things (Issue NMS-12979)

Enhancement

  • Add custom string attributes based on indirect and complex SNMP Indices (Issue NMS-8484)

  • Trailing ", \" in report definitions throws not helpful error message (Issue NMS-10526)

  • nodeDeleted event should contain more information (Issue NMS-10554)

  • Netscaler vServer SNMP collection and graph definitions (Issue NMS-12269)

  • Upgrade Sink API to Proto3 (Issue NMS-12602)

  • Add selenium test for password change (Issue NMS-12645)

  • Re-enable Central Search functionality (Issue NMS-12646)

  • Remove remote-poller runtimes from the build (Issue NMS-12653)

  • Update remote-poller model to link service to locations instead of individual RPs (Issue NMS-12654)

  • Rewrite the remote poller backend to use Minion (Issue NMS-12655)

  • Dynamic service scheduling: Service lifetime (Issue NMS-12676)

  • Dynamic service scheduling: Polling package association (Issue NMS-12677)

  • Dynamic service scheduling: Poller-Configuration change (Issue NMS-12678)

  • Update database and send events only on status changes (Issue NMS-12679)

  • Get new remote poller running as daemon (Issue NMS-12683)

  • Add upgrade task to enable RemotePollerNG (Issue NMS-12684)

  • Confd templates for Minion configuration (Karaf) (Issue NMS-12687)

  • Support hostnames resolution when using aggregated flows (Issue NMS-12692)

  • Use CollectionSetBuilder to persist response times (Issue NMS-12698)

  • Create SQL index for optimizing poll result lookups (Issue NMS-12710)

  • Apply thresholding for remote poller response times (Issue NMS-12721)

  • Meta-data tag enhancements to Time Series Storage API (Issue NMS-12730)

  • remove the java ("windows") installer (Issue NMS-12742)

  • Add ReST interface for Remote Poller (Issue NMS-12760)

  • Anomaly Detection Experiment (Issue NMS-12773)

  • Anomaly Detection - Get the consumer working (Issue NMS-12774)

  • Migrated VMware asset data to metadata entries (Issue NMS-12781)

  • RemotePoller: only schedule services that are part of an application (Issue NMS-12785)

  • Use newer protocol versions for remote DCOM WMI (Issue NMS-12788)

  • Add support for OpenConfig Service in Detector/Poller (Issue NMS-12789)

  • Remote Poller: Add outages for remote poller status changes (Issue NMS-12792)

  • Add support for meta-data on single-DS threshold definitions (Issue NMS-12794)

  • Update Users chapter (Issue NMS-12809)

  • Remote Poller: Modify model to new structure (Issue NMS-12820)

  • Remote Poller: Change monitoring locations UI to reflect model changes (Issue NMS-12821)

  • Remote Poller: Change Admin UI to reflect Application model changes (Issue NMS-12822)

  • Application Topology Provider Status (Issue NMS-12823)

  • Allow meta-data in notifications (Issue NMS-12827)

  • Add events for new Application Model (Issue NMS-12838)

  • Remote Poller: camel case in new column definitions causing problems (Issue NMS-12841)

  • Remote Poller: Remove LocationSpecificStatus (Issue NMS-12859)

  • Remote Poller: Documentation (Issue NMS-12860)

  • Remote Poller: Fix the remote flag (Issue NMS-12861)

  • Remote Poller: Cleanup (Issue NMS-12864)

  • Remote Poller: Clearify perspecive labeling (Issue NMS-12865)

  • Remote Poller: Details pages show remote outages in recent outages list (Issue NMS-12866)

  • Remote Poller: Add graph definitions for response times (Issue NMS-12868)

  • Remote Poller: Add distributed tracing (Issue NMS-12869)

  • Remote Poller: Rename to Perspective Poller (Issue NMS-12874)

  • Remote Poller: Make backend more resilient (Issue NMS-12878)

  • Application Perspective Monitoring: create Application status page (Issue NMS-12886)

  • Document the MailTransportMonitor (Issue NMS-12892)

  • Additions to Application Perspective Monitoring docs (Issue NMS-12893)

  • Filter outages table in Application Topology View (Issue NMS-12897)

  • Add logging configuration for Perspective Poller (Issue NMS-12910)

  • Allow to navigate to monitored items in application status view (Issue NMS-12913)

  • Allow setting meta data in a requisition (Issue NMS-12918)

  • Create a report that matches Horizon 27.0.0 Jira issues with merged pull requests in GitHub (Issue NMS-12922)

  • Update Copyright notice for 2020 (Issue NMS-12933)

  • SSLCertMonitor should include more details about the expir(ing|ed) certificate in reason codes (Issue NMS-12948)

  • Update Maximum PostgreSQL to allow PostgreSQL 13 (Issue NMS-12958)

  • Create Horizon 27 Release Notes (Issue NMS-12961)

  • Identify message broker strategies in web "about" page (Issue NMS-12971)