Copyright © 2004-2020 The OpenNMS Group, Inc.

Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.1 or any later version published by the Free Software Foundation; with no Invariant Sections, with no Front-Cover Texts and with no Back-Cover Texts. A copy of the license is available at http://www.gnu.org/copyleft/fdl.html OpenNMS is the creation of numerous people and organizations, operating under the umbrella of the OpenNMS project. The source code is published under the GNU Affero GPL, version 3 or later and is Copyright © 2002-2020 The OpenNMS Group, Inc.

The current corporate sponsor of OpenNMS is The OpenNMS Group, which also owns the OpenNMS trademark.

Please report any omissions or corrections to this document by creating an issue at http://issues.opennms.org.

What’s New in OpenNMS Horizon 25

System Requirements

  • Java 8 through 11: OpenNMS Horizon 25 runs on JDK 8 through 11. We recommend the most recent version of OpenJDK 11.

  • Default Heap Size: The default heap size is now 2GB.

  • PostgreSQL 10 or higher: Horizon 25 requires any supported version of PostgreSQL 10 or higher.

Breaking Changes

The Flow Classification UI accepted invalid ip address values, e.g. 10,192.1,168.1,5.1-160. When upgrading OpenNMS Horizon existing Flow Classification Rules may be considered invalid. In this case they are silently ignored by the Flow Classification Engine. In order to assure no invalid Flow Classification Rules exist, please run the OSGi shell command opennms-classification:list-invalid-rules to list all invalid rules. If there are any invalid rules, they must be manually fixed using the Flow Classification UI. For more details, refer to issue NMS-12422.

PostgreSQL 10

OpenNMS Horizon 25 is supported on PostgreSQL 10 or later.

If you are upgrading from an OpenNMS using an older PostgreSQL version, you can use the pg_upgrade tool to migrate. For details on the migration process, see the PostgreSQL pg_upgrade documentation.

If you are running on a distribution that does not provide PostgreSQL 10 packages by default, you can download packages provided by PostgreSQL instead.

Elasticsearch 7.x Support

All of the features that leverage integrations with Elasticsearch i.e. event & alarm history, flows & situation feedback have been updated to support Elasticsearch 7.x. Elasticsearch versions before 7.x are no longer supported.

Given the pace of changes and the number of breaking changes between major versions of Elastisearch, we will focus on supporting a single major version of Elasticsearch per release moving forward.

Flows

While working on improvements to flow support we have modified the index templates and it is no longer possible to query flows stored in indices created with prior version of OpenNMS. We will revisit ways to migrate these in a future release.

It is now possible to use CIDR ip expressions when defining Classification Rules for Flows.

Minion Packages Combined

The Minion RPM and Debian/Ubuntu packages have been combined into a single package. Previously it was possible to just install the core container and optionally a set of features on top, but in the real world no one used this, and it’s still possible to choose what features are used with configuration.

If you were previously only installing the sub-packages and not installing the wrapper opennms-minion RPM package, you may need to run this command to upgrade cleanly: 
$ yum swap 'opennms-minion*' -- install opennms-minion

Sentinel Now Requires an ID and Location

The documentation for setting up a Sentinel already suggested setting ID and Location, but now they are required for tracing.

Helm 4

Technically not a part of OpenNMS, but it’s important to note that Helm 4 introduces a datasource that replaces the "Faults" datasource. Migration of existing dashboards requires a few manual steps; documentation on how to do so will be available in the latest Helm documentation once Helm 4 goes live.

Removed from Horizon 25

  • The Alarm Change Notifier plugin has been removed due to performance problems. Use the Alarm History function instead.

  • The resourcecli tool has been removed in favor of Karaf shell commands to perform the same operations.

Event (and Alarm) Date/Time Formats

Events and alarms previously used the current locale for formatting timestamps. As of Horizon 25.1.0, we now format dates as ISO-8601 date-time with offsets (eg, 2011-12-03T10:15:30+01:00) and we expect dates in incoming events to be the same. This reduces the amount of code we run through trying to figure out date formats while parsing events, and simplifies things to work no matter what locale you are using.

New Features and Improvements

Docker Containers

  • Docker containers are now built automatically as part of our CI process.

  • The horizon Docker container now runs as non-root by default.

Poller

  • Pollerd now reloads properly with a daemonReload event.

  • Filters that use <include-url> now reload when Pollerd configuration reloads.

  • It is now possible to configure the poller with wildcard configurations that match multiple services. For details, see the Admin Guide section on Service Patterns.

Thresholding

  • Thresholding state is now persisted to the database, and can also be shared with Sentinel.

  • Thresholding can now be applied to streaming telemetry data, including scaling to multiple Sentinel instances.

  • Thresholding expressions now support meta-data

  • Threshold state can be managed through the Karaf shell.

Flows

A large number of improvements have been made to flow support.

  • Flow classification rules are now configurable with custom priorities and custom groups can be created.

  • The node list, node details, and resource graph pages indicate whether flows are available.

  • Basic clock skew detection for flow timestamps has been introduced.

  • Flows can be grouped by monitoring location.

ReST queries and Helm have been enhanced to support new flow features:

  • Support has been added for querying traffic for specific applications over time.

  • Support has been added for querying a list of "Top N" hosts, similar to applications and conversations.

  • Hostnames can now be shown in Helm rather than just IP addresses, when available.

Elasticsearch

  • It is now possible to set a custom prefix for the indices, which allows for multiple OpenNMS instances to share the same cluster.

Detectors, Monitors, and Collectors

  • TTLs can be customized per-detector, monitor, or collector.

  • A JMS/ActiveMQ detector and monitor have been added.

  • The TLS Certificate Monitor now supports using STARTTLS to negotiate.

Reporting

  • The scheduled reporting UI has been updated and improved.

  • It is now possible to configure OpenNMS to integrate with Grafana and schedule PDF reports generated from Grafana dashboards.

  • Time zone is now considered when scheduling reports.

  • Scheduled reports now support multiple recipients.

New or Updated Karaf Shell Commands

The following new commands have been added to support new or existing functionality:

  • events:send - send an OpenNMS event

  • opennms-dns:stress - stress-test the new internal DNS resolver

  • opennms-kv-blob:get - print a record from the blob store

  • opennms-kv-blob:put - inserts a string into the blob store

  • opennms-kv-blob:benchmark - benchmark the internal OpenNMS blob store

  • opennms-kv-json:get - print a record from the JSON store

  • opennms-kv-blob:put - inserts a record into the JSON store

  • opennms-measurements:show-measurements - show measurements data

  • opennms-measurements:show-newts-samples - show raw sample data from Newts

  • opennms-measurements:show-resources - show/filter the resource tree

  • opennms-measurements:delete-resource - delete the measurements and meta-data for a resource

  • opennms-snmp:fit - given an IP address, "fit" that address to an SNMP config profile

  • opennms-snmp:remove-from-definition - remove an IP address from an SNMP config definition

  • opennms-threshold-states:enumerate - list threshold states

  • opennms-threshold-states:details - show the details of a specific threshold state

  • opennms-threshold-states:clear - clear a specific threshold state

  • opennms-threshold-states:clear-all - reset all threshold states

  • provision:import-requisition - import a requisition from a URL

Additionally, a few commands have been updated to have opennms- as their prefix:

  • snmp:show-config → opennms-snmp:show-config

  • snmp:walk → opennms-snmp:walk

UI Improvements

  • The UI has gone through a refresh, moving to Bootstrap 4 and vastly cleaning up our HTML to make it easier to make further improvements going forward.

  • "Graph All" now works properly even when there are a large number of resources to graph.

  • Node, interface, and service meta-data is now configurable in the requisition UI.

  • The Backshift graph zooming regression introduced with the web asset refactor has been fixed.

  • The rendered graph page can now be searched/filtered to pare down results, and graphs are now lazily loaded.

  • The alarm list can now be filtered by surveillance category.

Development Considerations

Internals

A ton of other internal improvements have been made:

  • The complete node object is now exposed to Scriptd and script policies.

  • A bunch more things have been added to the Karaf health:check command-line.

  • Elasticsearch client performance has been improved by enabling compression where possible.

  • A new DNS resolution service has been introduced internally that should speed up hostname lookups.

  • A new key/value store was added, for internal use in things like thresholding persistence and user session data.

  • The snmp-config.xml file now supports "profiles" for influencing the behavior of IPs that don’t match existing definitions. For details, see the Admin Guide section on SNMP Profiles.

  • Our embedded Drools has been updated to the latest 7.x release.

  • More instrumentation has been added to the Jaeger OpenTracing support introduced in Horizon 24.

  • A ton of performance improvements have been made in various parts of the codebase.

  • 2 new roles have been added:

  • ROLE_REPORT_DESIGNER - use the ReST APIs or UI for manipulating reports and report schedules

  • ROLE_FLOW_MANAGER - use the Rest APIs or UI for managing flow classification rules

Kafka Producer Metrics

  • For Interface Resources, a String attribute named __ifIndex is added to represent missing Interface ifIndex in String form.

Changelog

Release 25.0.0

Release 25.0.0 is the first release in the Horizon 25 series.

It contains a huge number of bug fixes and enhancements, most notably massive thresholding improvements, big changes to netflow and other streaming telemetry data, reporting and UI updates, plus much more.

The codename for 25.0.0 is Baozi.

Bug

  • "Graph all" fails with 'Request Header Fields Too Large' error (Issue NMS-8712)

  • reload Pollerd doesn’t work correctly (Issue NMS-9181)

  • No class found exception in OSGi for org.osgi.service.jdbc.DataSourceFactory (Issue NMS-9341)

  • GenericResourceType is altering index names on class object initialization (Issue NMS-9702)

  • Updating external lists referenced by include-url requires restart OpenNMS (Issue NMS-10071)

  • Telephone PIN Field in User Details is Misplaced (Issue NMS-10073)

  • HealthCheck should not fail for ElasticSearch if flows are not enabled (Issue NMS-10498)

  • Syslog incorrect dates being parsed into database (Issue NMS-10605)

  • Zooming with Backshift is broken (Issue NMS-10635)

  • MonitoredServiceDaoIT test fail due to database (Issue NMS-10643)

  • EventdIT test failure (Issue NMS-10644)

  • DuplicatePrimaryAddressIT logs a failure due to logging assertion (Issue NMS-10645)

  • Reflected file download vulnerability in /api/v2/scanreports (Issue NMS-10703)

  • Insecure cache-control for session cookies (Issue NMS-10704)

  • Reflected XSS vulnerability in notification/detail.jsp and outage/detail.htm (Issue NMS-10707)

  • Kafka alarm producer synchronization fails due to missing event (Issue NMS-10724)

  • Documentation broken (Issue NMS-10759)

  • [circleci] Building container images fails intermittently (Issue NMS-10834)

  • Fix version range for maven-jaxb-schemagen-plugin version in root pom (Issue NMS-11760)

  • Sentinel container does not contain integration API in maven repository (Issue NMS-11766)

  • Windows is failing tests due to spurious cleanup errors (Issue NMS-12102)

  • AS is stored as signed int in flow doc (Issue NMS-12105)

  • Add local cache to reverse hostname resolution in flows (Issue NMS-12117)

  • karaf.log appears on the root file system when running Minion/Sentinel on Ubuntu/Debian. (Issue NMS-12125)

  • Sentinel cannot persist Telemetry when running under OracleJDK 11 (Issue NMS-12126)

  • Cannot debug Telemetry persistence on Sentinel (Issue NMS-12127)

  • Outage timeline is empty (Issue NMS-12160)

  • Minion not showing label correctly and label cannot be edited (Issue NMS-12164)

  • WsManWQLDetector, ActiveMQDetector missing setBeanProperties (Issue NMS-12175)

  • sFlow adapter drops flows due to NPE (Issue NMS-12185)

  • BulkRequests should log more details of why it failed (Issue NMS-12188)

  • The Cron Expression is only shown for reports which are created with the new UI (Issue NMS-12189)

  • Fix typo in sonar.sh (Issue NMS-12196)

  • Minion memory leak related to new DNS lookup functionality (Issue NMS-12207)

  • JMX Metrics for Newts disappeared (Issue NMS-12216)

  • Can’t change Minion location in the Provisiond UI (Issue NMS-12217)

  • Changing a nodes location breaks the service polling schedule (Issue NMS-12219)

  • Wrong PID in opennms.pid (Issue NMS-12226)

  • interfaceresolv event token is incorrect (Issue NMS-12227)

  • Kafka RPC: Slow callbacks cause lag on response topics (Issue NMS-12232)

  • WS-MAN doesn’t work with JDK 11 (Issue NMS-12235)

  • Kafka RPC: StackOverflowError while unmarshaling causes processing to halt (Issue NMS-12236)

  • Ops wallboard hanging due to Vaadin session deadlock (Issue NMS-12237)

  • Fix flapping DatabaseReportPageIT (Issue NMS-12241)

  • Invalid timestamps used for CollectionSets generated by telemetryd (Issue NMS-12242)

  • DNS resolved hostname should always be reflected in 'iphostname' field in ipinterface table (Issue NMS-12249)

  • Cannot change the type of a scheduled outage (Issue NMS-12255)

  • DefaultReportWrapperService: failed to run or render report (Issue NMS-12258)

  • Rewrite test in ScheduleOutageIT for new UI (Issue NMS-12264)

  • vmwarereqtool freezes and never return to the prompt after is executed (Issue NMS-12265)

  • EnhancedLinkd-Thread-x-of-x - Null Pointer Exception (Issue NMS-12267)

  • ScriptPolicy scripts encounter LazyInitializationException on calling node.getSnmpInterface() (Issue NMS-12268)

  • Remove old comment from data source configuration (Issue NMS-12270)

  • Fix usage description of vmwarereqtool (Issue NMS-12275)

  • Poller take a long time to startup/initialize on large systems (Issue NMS-12285)

  • Ordering of classification groups does not work (Issue NMS-12286)

  • Default settings in container image don’t write RRD files (Issue NMS-12290)

  • Grafana report generation is timing out (Issue NMS-12297)

  • Default "90% Interface Throughput" Threshold Doesn’t Work (Issue NMS-12301)

  • ConcurrentModificationException when cloning events (Issue NMS-12306)

Enhancement

  • Remove alarm-change-notifier plugin (Issue NMS-10658)

  • Enforce VertexRef to be immutable (Issue NMS-11138)

  • View traffic for specific hosts over time (flows) (backend work) (Issue NMS-11153)

  • Initial CircleCI pipeline (Issue NMS-11157)

  • Update flow indicators for SNMP interfaces to link to deep dive tool (Issue NMS-11169)

  • Refactor GraphProvider to return Graph instead of being the Graph itself (Issue NMS-11264)

  • Design new header for PDF reports (Issue NMS-11305)

  • Edges should allow to contain vertices of different namespaces (Issue NMS-11343)

  • View traffic for specific conversations over time (flows) (backend work) (Issue NMS-11345)

  • Core structure/API of new Graph Service (Issue NMS-11366)

  • Implement domain specific graph objects in New Graph service (Issue NMS-11408)

  • View traffic for specific applications over time (flows) - back end work (Issue NMS-11458)

  • Use composite aggregation instead of convo_key field from flow documents (Issue NMS-11473)

  • Provide new GraphProvider implementation for the ApplicationTopologyProvider (Issue NMS-11506)

  • Indicators for nodes with flow data (Issue NMS-11523)

  • The GraphService interface should return GenericGraph instead of Graph<?, ?> (Issue NMS-11554)

  • Provide capability to expose GraphContainerProviders with only one graph (SingleGraphContainerProvider) (Issue NMS-11574)

  • Make GraphRepository accessible via OSGI (Issue NMS-11575)

  • Make GraphRepository persist collections (Issue NMS-11624)

  • Implement ReST Service for the new Graph Service (Issue NMS-11636)

  • Search for nodes that have flow data (Issue NMS-11654)

  • Refactoring Docker Container Images towards Horizon 25 (Issue NMS-11719)

  • Run container image as non-root (Issue NMS-11720)

  • Replace custom bash templating with Confd and streamline entrypoint script (Issue NMS-11721)

  • Replace current (Spring Webflow) Reporting UI with an angular implementation (Issue NMS-11723)

  • CircleCI (Issue NMS-11830)

  • Thresholding for streaming metrics & more (Issue NMS-11834)

  • Project Drift Enhancements (Issue NMS-11836)

  • Grafana Integration (Issue NMS-11844)

  • Enterprise Reporting Enhancements (Issue NMS-11861)

  • Grouped flow classification rules (Issue NMS-11975)

  • Make Graph immutable (Issue NMS-11985)

  • Skew detection for flow timestamps (Issue NMS-11993)

  • Run system tests using OpenJDK 11 (Issue NMS-11995)

  • Top N hosts (flows) (Issue NMS-11998)

  • Dynamically change a node’s location (Issue NMS-12053)

  • Customize priority of flow classification rules (Issue NMS-12058)

  • Flow classification role (Issue NMS-12065)

  • Add support for sending events from a Minion (Issue NMS-12078)

  • Hostname resolution for flows (Issue NMS-12087)

  • Provide search capabilities (Issue NMS-12092)

  • KPIs for resource graphs (Issue NMS-12095)

  • Migrate system tests to use TestContainers framework (Issue NMS-12107)

  • Remove hack from opennms-spring-extender to expose ServiceRegistry (Issue NMS-12122)

  • Upgrade Bootstrap to latest 4.3.x version (Issue NMS-12124)

  • Allow tarball installation in Docker images for Horizon (Issue NMS-12133)

  • Add the ability to trigger a requisition import via the Karaf shell (Issue NMS-12135)

  • Store and read thresholding configuration from PostgreSQL (Issue NMS-12140)

  • Add support for sending events from Sentinel (Issue NMS-12141)

  • Stickiness for distributed processing in telemetryd (Issue NMS-12142)

  • Share thresholding state across Sentinels (Issue NMS-12143)

  • Create Jasper report templates for Grafana dashboards (Issue NMS-12144)

  • Improve Grafana report rendering speed (Issue NMS-12145)

  • Implement easier datetime selection (Issue NMS-12147)

  • Consider timezone when scheduling reports (Issue NMS-12148)

  • Allow multiple recipients when delivering/scheduling a report (Issue NMS-12149)

  • Add ability to edit existing scheduled reports (Issue NMS-12150)

  • Allow for searching a dashboard (Issue NMS-12152)

  • Easier Schedule Definition (Issue NMS-12154)

  • Improve performance of hostname resolution for flows (Issue NMS-12157)

  • Remove Legacy Database Reports (Web Flow) (Issue NMS-12158)

  • UsersRestService: /users/whoami returns 404 on ldap authenticated user (not in users.xml) (Issue NMS-12159)

  • User conceptualizes profile-fitting for SNMP across enterprise (Issue NMS-12167)

  • User provisions nodes with SNMP profiles in place (Issue NMS-12169)

  • Grafana reporting: Render exceptions in images instead of aborting (Issue NMS-12170)

  • Collect and graph packet reception rates for telemetryd listeners (Issue NMS-12182)

  • Allow developer to push OCI’s to docker hub from arbitrary branches (Issue NMS-12187)

  • Enable flow UDP listener by default on Minion and OpenNMS (Issue NMS-12190)

  • Upgrade Drools to latest 7.x release (Issue NMS-12191)

  • Grafana Reporting: Add time to date range in report header of template (Issue NMS-12200)

  • Show configured rule groups (Issue NMS-12212)

  • Edit (CRUD) rule groups via rest & gui (Issue NMS-12213)

  • Add reply-to field to notification emails (Issue NMS-12224)

  • Ensure the reporting jar for jasperstudio is still working and containing all the relevant jars (Issue NMS-12229)

  • Verify Database Reporting Documentation is still up to date (Issue NMS-12230)

  • Dependency based testing (Issue NMS-12233)

  • Update minimum required version of PostgresSQL for H25 (Issue NMS-12238)

  • Report scheduler should be disableable (Issue NMS-12240)

  • Add support for meta-data fields in thresholding expressions (Issue NMS-12247)

  • Manage persisted threshold state via shell (Issue NMS-12248)

  • Increase default heap size to 2GB (Issue NMS-12276)

  • Expose Kafka Lag on Sink/RPC through Karaf shell (Issue NMS-12295)

  • Expose RPC/Sink Metrics on Karaf shell (Issue NMS-12296)

  • Expose health:check via ReST (Issue NMS-12298)

  • Add a JMS broker monitor (Issue NMS-7188)

  • Expose the complete OnmsNode object in Scriptd to avoid LazyInitializationExceptions (Issue NMS-8294)

  • Document RPC TTL tuning (Issue NMS-10376)

  • Redesign default threshold events (Issue NMS-10672)

  • Filter alarm list by surveillance categories (Issue NMS-10695)

  • use sevntu-checkstyle-maven-plugin from maven-central (Issue NMS-10717)

  • use org.asteriskjava:asterisk-java from maven-central (Issue NMS-10718)

  • Angular requests do not handle 401 responses from the OpenNMS backend (Issue NMS-10719)

  • Increase logging clarity for flow processing (Issue NMS-10723)

  • Support compression in JestClient for Elasticsearch (Issue NMS-10859)

  • Publish Docker images from CircleCI to DockerHub (Issue NMS-11722)

  • Improve the sample content for the NXOS GPB Parser Groovy Script. (Issue NMS-11759)

  • Design thresholding support for telemetryd on Sentinel (Issue NMS-11776)

  • Thresholding support for push metrics (Issue NMS-11797)

  • View traffic for specific conversations over time (flows) (frontend work) (Issue NMS-12074)

  • Provide possibility to add Grafana Instances (Issue NMS-12075)

  • View traffic for specific applications over time (flows) - front end work (Issue NMS-12076)

  • View traffic for specific hosts over time (flows) (frontend work) (Issue NMS-12077)

  • Support wildcards in service names (Issue NMS-12080)

  • Refactor threshd out of opennms-services (Issue NMS-12081)

  • Configurable index name for Event & Alarm Forwarder (Issue NMS-12082)

  • Add a UI to edit meta-data in requisition (Issue NMS-12083)

  • Custom TTLs for detectors, monitors and collectors (Issue NMS-12096)

  • Update Elasticsearch integrations to support ES 7.x (Issue NMS-12099)

  • Update Webpack Build to Latest Babel and TypeScript (Issue NMS-12130)

  • Enhance CircleCI workflow to reduce build times and tests (Issue NMS-12136)

  • Add SonarQube Code Coverage to CircleCI Workflow (Issue NMS-12163)

  • Create feature to install thresholding service on sentinel (Issue NMS-12180)

  • Postgres KeyValueStore implementation via JDBC (Issue NMS-12181)

  • Update warmerge plugin dependency to 0.5 (Issue NMS-12194)

  • Enhance Jaeger OpenTracing Logging with tag for thread name (Issue NMS-12195)

  • Upgrade Kafka components to 2.3.0 (Issue NMS-12203)

  • Add enhancements to DNS Resolver and Circuit Breaker (Issue NMS-12215)

  • Leverage telemetry sequencing for better thresholding performance (Issue NMS-12243)

  • Remove session lock from Drools in alarmd (Issue NMS-12244)

  • Karaf shell command for benchmarking performance of BlobStore (Issue NMS-12245)

  • Add support for STARTTLS to TLS certificate monitor (Issue NMS-12252)

  • Resource Graphs : Graph Selection fails to render when many resources are selected (Issue NMS-12256)

  • split shell utilities into a separate module (Issue NMS-12266)

  • allow using fetched RPMs in build_container_image.sh (Issue NMS-12271)

  • Add documentation about classification rule & group sorting (Issue NMS-12278)

  • Migrate resourcecli to the Karaf shell (Issue NMS-12280)

  • Add Karaf shell command to view raw sample stored in Newts (Issue NMS-12281)

  • Be able to execute ICMP requests through Java when running as non-root (Issue NMS-12283)

  • Update Minion packaging to use single RPM/Deb (Issue NMS-12300)