Install Minion

Objectives

  • Install a Horizon Minion on one of the supported operating systems

  • Secure access to the Karaf shell with encrypted passwords

  • Configure connection to message broker for communication with the Horizon core instance

  • Verify the setup

Requirements

  • Horizon core instance running and configured with a message broker

  • Linux physical server or a virtual machine running a supported Linux operating system

  • Internet access to download the installation packages

  • Java installed OpenJDK 11

  • DNS configured so that localhost and your server’s host name resolve properly

  • Horizon core instance runs on latest stable release

  • Minion server can access the Horizon core instance via REST (8980/tcp) and desired message broker nodes (for example, ActiveMQ 61616/tcp, Apache Kafka 9092/tcp)

  • System user with administrative permissions (sudo) to perform the installation tasks

  • If you are running the services in Docker you also need Docker Compose for the service stacks from our examples

If you run Debian, you have to install and configure sudo yourself (see the Debian Wiki).

Install the Minion package

  • CentOS/RHEL 8

  • CentOS/RHEL 7

  • Ubuntu

  • Debian

  • Docker

Install OpenJDK 11 JRE runtime
sudo dnf -y install java-11-openjdk-headless
Add repository and import GPG key
sudo dnf -y install https://yum.opennms.org/repofiles/opennms-repo-stable-rhel8.noarch.rpm
sudo rpm --import https://yum.opennms.org/OPENNMS-GPG-KEY
Install the Horizon Minion package
sudo dnf -y install opennms-minion
Disable the OpenNMS Horizon repository after installation to prevent unwanted upgrades when upgrading other packages on the server. After upgrade, Horizon requires manual steps to upgrade configuration files or migrate database schemas to a new version. For this reason, we recommended that you exclude the Horizon packages from update except when you plan to perform an upgrade.
Disable auto updates for Horizon Minion
sudo dnf config-manager --disable opennms-repo-stable-*
Verify directory structure with the tree command
sudo dnf -y install tree
tree /opt/minion -L 1
Directory structure after successful installation
/opt/minion
├── bin
├── COPYING
├── deploy
├── etc
├── lib
├── repositories
└── system
Enable Horizon Minion on system boot and start immediately
sudo systemctl enable --now minion
Install OpenJDK 11 JRE runtime
sudo yum -y install java-11-openjdk-headless
Add repository and import GPG key
sudo yum -y install https://yum.opennms.org/repofiles/opennms-repo-stable-rhel7.noarch.rpm
sudo rpm --import https://yum.opennms.org/OPENNMS-GPG-KEY
Install the Horizon Minion package
sudo yum -y install opennms-minion
Disable the OpenNMS Horizon repository after installation to prevent unwanted upgrades when upgrading other packages on the server. After upgrade, Horizon requires manual steps to upgrade configuration files or migrate database schemas to a new version. For this reason, we recommend that you exclude the Horizon packages from update except when you plan to perform an upgrade.
Disable auto updates for Horizon Minion
sudo yum -y install yum-utils
sudo yum-config-manager --disable opennms-repo-stable-*
Verify directory structure with the tree command
sudo yum -y install tree
tree /opt/minion -L 1
Directory structure after successful installation
/opt/minion
├── bin
├── COPYING
├── deploy
├── etc
├── lib
├── repositories
└── system
Enable Horizon Minion on system boot and start immediately
sudo systemctl enable --now minion
Add OpenNMS repository GPG key
sudo apt-key adv --fetch-keys https://debian.opennms.org/OPENNMS-GPG-KEY
Add apt repository
sudo add-apt-repository -s 'deb https://debian.opennms.org stable main'
You can safely ignore the message about conflicting distributions (expected stable but got opennms-xx).
Installing the of Horizon Minion
sudo apt -y install opennms-minion
Disable the OpenNMS Horizon repository after installation to prevent unwanted upgrades when upgrading other packages on the server. After upgrade, Horizon requires manual steps to upgrade configuration files or migrate database schemas to a new version. For this reason, we recommend that you exclude the Horizon packages from update except when you plan to perform an upgrade.
sudo apt-mark hold opennms-minion
Verify directory structure with the tree command
sudo apt -y install tree
tree /usr/share/minion -L 1
Directory structure after successful installation
/usr/share/minion
├── bin
├── data -> /var/lib/minion/data
├── deploy -> /var/lib/minion/deploy
├── etc -> /etc/minion
├── lib
├── repositories
└── system
Enable Horizon Minion on system boot and start immediately
sudo systemctl enable --now minion
Install gnupg and add OpenNMS repository GPG key
sudo apt -y install gnupg ca-certificates
sudo apt-key adv --fetch-keys https://debian.opennms.org/OPENNMS-GPG-KEY
Add apt repository
sudo apt -y install software-properties-common
sudo add-apt-repository -s 'deb https://debian.opennms.org stable main'
sudo apt update
You can safely ignore the message about conflicting distributions (expected stable but got opennms-xx).
Install the Horizon Minion package
sudo apt -y install opennms-minion
Disable the OpenNMS Horizon repository after installation to prevent unwanted upgrades when upgrading other packages on the server. After upgrade, Horizon requires manual steps to upgrade configuration files or migrate database schemas to a new version. For this reason, we recommend that you exclude the Horizon packages from update except when you plan to performing an upgrade.
sudo apt-mark hold opennms-minion
Verify directory structure with the tree command
sudo apt -y install tree
tree /usr/share/minion -L 1
Directory structure after successful installation
/usr/share/minion
├── bin
├── data -> /var/lib/minion/data
├── deploy -> /var/lib/minion/deploy
├── etc -> /etc/minion
├── lib
├── repositories
└── system
Enable Horizon Minion on system boot and start immediately
sudo systemctl enable --now minion
Create a project directory for Horizon core and create a docker-compose.yml file.
mkdir minion
cd minion
vi docker-compose.yml
---
version: '3'

services:
  minion:
    image: opennms/minion:bleeding
    container_name: minion(1)
    environment:
      TZ: 'America/New_York'(2)
      JAVA_MIN_MEM: 512M(3)
      JAVA_MAX_MEM: 2048M(4)
    command: ["-c"]
    volumes:
      - ./minion-config.yaml:/opt/minion/minion-config.yaml(5)
      - ./scv.jce:/opt/minion/scv.jce(6)
    healthcheck:
      test: "/health.sh"(7)
      interval: 30s
      timeout: 6s
      retries: 3
    ports:(8)
      - '514:1514/udp'
      - '162:1162/udp'
      - '8201:8201/tcp'
1 Friendly container name
2 Set the time zone and the Minion configuration via environment variables. To list all available time zones, use timedatectl list-timezones.
3 Optional. Use to control the minimal Java heap size.
4 Optional. Use to control the maximum Java heap size.
5 Configuration file for connectivity and features
6 Keystore file with enrypted credentials for authenticating REST and Broker endpoints
7 Run our health check to indicate the Minion is ready. It uses the opennms:health-check internally running in Karaf.
8 Publish ports for Syslog, SNMP trap listener, and the SSH access to the Karaf shell.

The Minion process in the container runs as a non-root user and can’t open privileged ports < 1024. We open the listener for privileged (514/udp, 162/udp) and don’t have to give additional permissions to the Minion process in the container.

Some environments do not allow the use of ICMP (ping) for non-root users. Horizon Minion runs as a non-privileged user in the container with userid=10001, groupid=10001. You can verify with sysctl -n net.ipv4.ping_group_range if the group id 10001 is in the allowed group range. If not, add the following lines in your service definition in the docker-compose.yml file.
sysctls:
  net.ipv4.ping_group_range: "10001 10001"
Create a file with the configuration for connectivity and features
---
http-url: 'http://core-instance-ip:8980/opennms'(1)
broker-url: 'failover:tcp://core-instance-ip:61616'(2)
id: 'my-minion'(3)
location: 'my-location'(4)

netmgt:
  syslog:(5)
    syslog.listen.interface: "0.0.0.0"
    syslog.listen.port: 1514

  traps:(6)
    trapd.listen.interface: "0.0.0.0"
    trapd.listen.port: 1162
1 Replace the REST endpoint URL that goes to your Horizon Core instance.
2 Replace the broker endpoint URL that goes to your Horizon Core instance. If you have ActiveMQ with SSL running, replace tcp with ssl.
3 Replace my-location with a location name that represents the remote location where the Minion is running.
4 Optional. You can set a unique human-readable Minion identifier; if not set, a unique identifier is generated.
5 Enable the Syslog listener on the Minion on port 1514/udp.
6 Enable the SNMP trap listener on the Minion on port 1162/udp.
To run with Apache Kafka or configure flow listeners, see the configuration reference in the Confd readme.
Initialize the keystore with credentials
docker-compose run -v $(pwd):/keystore minion -s
Validate your Docker Compose file
docker-compose config -q
Start the service in background
docker-compose up -d
Verify the connectivity and the configuration with the health check
docker-compose exec minion bin/client opennms:health-check
The endpoints should be reachable and the bundles should be installed successful
Connecting to OpenNMS ReST API   [ Success  ]
Verifying installed bundles      [ Success  ]
Connecting to JMS Broker         [ Success  ]
=> Everything is awesome

Secure access to Karaf shell

Change the default user/password admin/admin for the Karaf shell and encrypt it.
  • Linux

  • Docker

Login to the Karaf shell
ssh -p 8201 admin@localhost
Enable password encryption using SHA-512
config:edit org.apache.karaf.jaas
config:property-set encryption.enabled true
config:property-set encryption.algorithm SHA-512
config:update
Set a new secure admin password
jaas:realm-manage --index 1 --realm karaf
jaas:user-add admin my-secure-password(1)
jaas:update
1 Replace my-secure-password with a strong password

Logout and try login to verify your new password is set.

The only way to change the admin password and enable encryption is to inject configuration files into the container. Commands are executed in the Docker project directory for the Minion.

Create the configuration file to enable encryption
vi org.apache.karaf.jaas.cfg
Enable encryption for the admin password
encryption.enabled = true
encryption.prefix = {CRYPT}
encryption.suffix = {CRYPT}
encryption.algorithm = SHA-512
encryption.encoding = hexadecimal
Create a property file for the SSH user for the Karaf shell
vi users.properties
Set a secure admin password for Karaf
admin = new-karaf-password,_g_:admingroup(1)
_g_\:admingroup = group,admin,manager,viewer,systembundles,ssh(2)
1 Replace the string new-karaf-password with your new password in plaintext.
2 Assign permissions to allow administrative tasks and access via SSH.
Set Minion process as owner and restrictive file permissions
chown 10001:10001 users.properties
chmod 600 users.properties
Inject the configuration files
volumes:
  - ./minion-config.yaml:/opt/minion/minion-config.yaml
  - ./scv.jce:/opt/minion/scv.jce
  - ./users.properties:/opt/minion/etc/users.properties
  - ./org.apache.karaf.jaas.cfg:/opt/minion/etc/org.apache.karaf.jaas.cfg
Restart the Minion container
docker-compose restart minion

Changes to the password or encryption algorithm are applied immediately. You do not need to restart the Minion.

By default, the Karaf Shell is restricted to 127.0.0.1. To enable remote access, set sshHost=0.0.0.0 in org.apache.karaf.shell.cfg. The change is applied immediately and a Minion restart is not required. If you have a firewall running on your host, allow 8201/tcp to grant access to the Karaf shell.

Configure connectivity to the Core instance

Set configuration in the Horizon etc directory. We reference etc relative to the OpenNMS Horizon Core home directory. Depending on your operating system, the home directory is /usr/share/opennms for Debian/Ubuntu or /opt/opennms for CentOS/RHEL.

  • Kafka

  • ActiveMQ

  • gRPC

  • AWS SQS

Create a file to prevent installing ActiveMQ features on Minion start-up
sudo vi etc/featuresBoot.d/disable-activemq.boot
Add the following lines to disable ActiveMQ features and save the file
!minion-jms
!opennms-core-ipc-rpc-jms
!opennms-core-ipc-sink-camel
Create a file to install Kafka features on Minion start-up
sudo vi etc/featuresBoot.d/kafka.boot
Add the following lines to install the remote producer call (RPC) and sink feature for Kafka on Minion start-up and save the file
opennms-core-ipc-rpc-kafka
opennms-core-ipc-sink-kafka
Configure the Kafka features and the Minion location via the Karaf shell
ssh -p 8201 admin@localhost
Configure the Minion location and REST endpoint
config:edit org.opennms.minion.controller(1)
config:property-set location my-location(2)
config:property-set http-url http://core-instance-ip:8980/opennms(3)
config:update(4)
1 Edit the Minion configuration.
2 Replace my-location with a location name that represents the remote location where the Minion is running.
3 Replace the example http-url with the URL of your Horizon Core instance.
4 Save the configuration.

By default, the Minion generates a unique ID. Provide a human-readable Minion identifier yourself with config:property-set id my-minion-name.

Configure the credentials for the REST endpoint and exit Karaf shell
opennms:scv-set opennms.http my-minion-user my-minion-password(1)
1 Set the credentials for the REST endpoint created in your Horizon Core instance.

The credentials are encrypted on disk in etc/scv.jce.

Configure the Kafka endpoints for RPC feature
config:edit org.opennms.core.ipc.rpc.kafka
config:property-set bootstrap.servers my-kafka-ip-1:9092,my-kafka-ip-2:9092(1)
config:update
1 Connect to the specified Kafka nodes and adjust the IPs or FQDNs with the Kafka port (9092) accordingly.
Configure the Kafka endpoints for sink feature
config:edit org.opennms.core.ipc.sink.kafka
config:property-set bootstrap.servers my-kafka-ip-1:9092,my-kafka-ip-2:9092(1)
config:update
1 Connect to the specified Kafka nodes and adjust the IPs or FQDNs with the Kafka port (9092) accordingly.

If you set more than one Kafka node as bootstrap.servers. the driver attempts to connect to the first entry. If that is successful, the whole broker topology will be discovered and will be known by the client. The other entries are used only if the connection to the first entry fails.

Ensure you use the FQDN or IP for your Kafka nodes as configured as advertised listener.

Exit the Karaf shell with Ctrl+d

Restart the Minion to apply the configuration
sudo systemctl restart minion
Verify the configuration using the health check in the Karaf shell
ssh -p 8201 admin@localhost
Run the health check command
opennms:health-check
Verify all components are configured properly
Verifying the health of the container

Connecting to OpenNMS ReST API           [ Success  ]
Verifying installed bundles              [ Success  ]
Connecting to Kafka from RPC             [ Success  ]
Connecting to Kafka from Sink Producer   [ Success  ]

=> Everything is awesome
Connect to the Karaf shell with user admin and password admin
ssh -p 8201 admin@localhost
Configure REST endpoints, ActiveMQ and remote location name
config:edit org.opennms.minion.controller(1)
config:property-set location my-location(2)
config:property-set http-url http://core-instance-ip:8980/opennms(3)
config:property-set broker-url failover:tcp://core-instance-ip:61616(4)
config:update(5)
1 Edit the Minion configuration.
2 Replace my-location with a location name that represents the remote location where the Minion is running.
3 Replace the REST endpoint URL that goes to your Horizon Core instance.
4 Replace the broker URL that goes to your Horizon Core instance. If you have ActiveMQ with SSL running, replace tcp with ssl.
5 Save the configuration.

By default, the Minion generates a unique ID. Provide a human-readable Minion identifier yourself with config:property-set id my-minion-name

Configure the credentials and exit Karaf shell
opennms:scv-set opennms.http my-minion-user my-minion-password(1)
opennms:scv-set opennms.broker my-minion-user my-minion-password(2)
1 Set the credentials for the REST endpoint created in your Horizon Core instance.
2 Set the credentials for the ActiveMQ message broker.

The credentials are encrypted on disk in etc/scv.jce.

Exit the Karaf shell with Ctrl+d

Restart the Minion to apply the configuration
sudo systemctl restart minion
Verify the configuration using the health check in the Karaf shell
ssh -p 8201 admin@localhost
Run the health check to verify connectivity
opennms:health-check
The result should show success for each component
Connecting to OpenNMS ReST API   [ Success  ]
Verifying installed bundles      [ Success  ]
Connecting to JMS Broker         [ Success  ]
=> Everything is awesome
Create a file to prevent installing ActiveMQ features on Minion start-up
sudo vi etc/featuresBoot.d/disable-activemq.boot
Add the following lines to disable ActiveMQ features and save the file
!minion-jms
!opennms-core-ipc-rpc-jms
!opennms-core-ipc-sink-camel
Create a file to install gRPC features on start-up
sudo vi etc/featuresBoot.d/grpc.boot
Add the gRPC client features
opennms-core-ipc-grpc-client
Configure the gRPC client and the Minion location via the Karaf shell
ssh -p 8201 admin@localhost
Configure the Minion location and REST endpoint
config:edit org.opennms.minion.controller(1)
config:property-set location my-location(2)
config:update(3)
1 Edit the Minion configuration.
2 Replace my-location with a location name that represents the remote location where the Minion is running.
3 Save the configuration.

By default, the Minion generates a unique ID. Provide a human-readable Minion identifier yourself with config:property-set id my-minion-name

Configure the gRPC endpoint
config:edit org.opennms.core.ipc.grpc.client
config:property-set host core-instance-ip(1)
config:property-set port 8990(2)
config:update(3)
1 Set the host to connect to the gRPC server running on the Horizon Core instance. Replace the core-instance-ip accordingly.
2 Set the port of the gRPC server, which is 8990 by default.
3 Save the configuration.
Restart the Minion to apply the changes
systemctl restart minion
Verify the configuration using the health check in the Karaf shell
ssh -p 8201 admin@localhost
Run the health check command
opennms:health-check
Verify all components are configured properly
admin@minion> opennms:health-check
Verifying the health of the container

Connecting to OpenNMS ReST API   [ Success  ]
Verifying installed bundles      [ Success  ]
Connecting to gRPC IPC Server    [ Success  ]

=> Everything is awesome

Optional. To enable TLS for gRPC you must provide certificate files and enable it. The commands for TLS appear below.

Connect to the Karaf shell
ssh -p 8201 admin@localhost
Configure TLS and certificate parameters
config:edit org.opennms.core.ipc.grpc.client
config:property-set tls.enabled true(1)
config:property-set trust.cert.filepath /custom-path/ca.crt(2)
config:property-set client.cert.filepath /custom-path/client.crt(3)
config:property-set client.private.key.filepath /custom-path/client.pem(4)
config:update(5)
1 Enable TLS for the gRPC server.
2 Set the path to your CA certificate file.
3 Set the path to your client certificate file.
4 Set the path client certificate key file.
5 Save and update the configuration.

This is optional, and you can set a maximum message size for gRPC. The maximum size must be the same on the Horizon Core instance. The default message size is 10 MiB.

Configure maximum message size for gRPC in the Karaf shell
config:edit org.opennms.core.ipc.grpc.client
config:property-set max.message.size 10485760
config:update
Restart the Horizon Core instance to apply changes
sudo systemctl restart opennms
Create a file to install AWS SQS features on Minion start-up
sudo vi etc/featuresBoot.d/aws-sqs.boot
Add the following lines to disable ActiveMQ and load AWS SQS
!minion-jms
!opennms-core-ipc-rpc-jms
!opennms-core-ipc-sink-camel
opennms-core-ipc-rpc-aws-sqs
opennms-core-ipc-sink-aws-sqs
Configure the AWS SQS via the Karaf shell
ssh -p 8201 admin@localhost
config:edit org.opennms.core.ipc.aws.sqs(1)
config:property-set aws_region us-east-1(2)
config:property-set aws_access_key_id my-access-key(3)
config:property-set aws_secret_access_key my-secret-access-key(4)
config:property-set sink.FifoQueue false(5)
config:update(6)
1 Configure the AWS SQS feature.
2 Set AWS SQS region.
3 The AWS SQS access key
4 The AWS SQS secret for the access key
5 If you require consistent ordering of incoming messages, use FIFO queues. The default is false and must match the Core instance setting.
6 Save and update the configuration.
Restart the Minion to apply the configuration
sudo systemctl restart minion
Verify the configuration using the health check in the Karaf Shell
ssh -p 8201 admin@localhost
Run the health check command
opennms:health-check
Verify all components are configured properly
Verifying the health of the container

Connecting to OpenNMS ReST API           [ Success  ]
Verifying installed bundles              [ Success  ]
Connecting to AWS-SQS from RPC           [ Success  ]
Connecting to AWS-SQS from Sink Producer [ Success  ]

=> Everything is awesome
You can find available configuration parameters in the Amazon Simple Queue Service reference section.

Check Minion in the Core web UI

  1. Log in to the web UI as an administrative user.

  2. Click the gears icon and choose Manage Minions in the Distributed Monitoring area.

After a few minutes your Minion should be provisioned automatically and the status should be "up".