Installing Minion Objectives Install a Horizon Minion on one of the following operating systems Secure access with encrypted passwords to the Karaf shell Configure connection to message broker for communication with the Horizon core instance Verify the setup Requirements The Horizon core instance is running and configured with a message broker Linux physical server or a virtual machine running a supported Linux operating system Internet access to download the installation packages Java installed OpenJDK 8, OpenJDK 11 DNS is configured, localhost and your server’s host name is resolved properly Horizon core instance runs on latest stable release Minion server can access the Horizon core instance via REST (8980/tcp) and desired message broker nodes (e.g., ActiveMQ 61616/tcp, Apache Kafke 9092/tcp) System user with administrative permissions (sudo) to perform the installation tasks If you are running the services in Docker you also need Docker Compose for the service stacks from our examples. If you run Debian, you have to install and configure sudo yourself (see the Debian Wiki). Installing the Minion package CentOS/RHEL 8 CentOS/RHEL 7 Ubuntu Debian Docker Install OpenJDK 11 JRE runtime sudo dnf -y install java-11-openjdk-headless Add repository and import GPG key sudo dnf -y install https://yum.opennms.org/repofiles/opennms-repo-stable-rhel7.noarch.rpm sudo rpm --import https://yum.opennms.org/OPENNMS-GPG-KEY Install the Horizon Minion package sudo dnf -y install opennms-minion Disable the OpenNMS Horizon repository after installation to prevent unwanted upgrades when upgrading other packages on the server. After upgrade, Horizon requires manual steps to upgrade configuration files or migrate database schemas to a new version. For this reason, it is recommended to exclude the Horizon packages from update except when you are planning on performing an upgrade. Disable auto updates for Horizon Minion sudo dnf config-manager --disable opennms-repo-stable-* Verify directory structure with the tree command sudo dnf -y install tree tree /opt/minion -L 1 Directory structure after successful installation /opt/minion ├── bin ├── COPYING ├── deploy ├── etc ├── lib ├── repositories └── system Enable Horizon Minion on system boot and start immediately sudo systemctl enable --now minion Install OpenJDK 11 JRE runtime sudo yum -y install java-11-openjdk-headless Add repository and import GPG key sudo yum -y install https://yum.opennms.org/repofiles/opennms-repo-stable-rhel7.noarch.rpm sudo rpm --import https://yum.opennms.org/OPENNMS-GPG-KEY Install the Horizon Minion package sudo yum -y install opennms-minion Disable the OpenNMS Horizon repository after installation to prevent unwanted upgrades when upgrading other packages on the server. After upgrade, Horizon requires manual steps to upgrade configuration files or migrate database schemas to a new version. For this reason, it is recommended to exclude the Horizon packages from update except when you are planning on performing an upgrade. Disable auto updates for Horizon Minion sudo yum -y install yum-utils sudo yum-config-manager --disable opennms-repo-stable-* Verify directory structure with the tree command sudo yum -y install tree tree /opt/minion -L 1 Directory structure after successful installation /opt/minion ├── bin ├── COPYING ├── deploy ├── etc ├── lib ├── repositories └── system Enable Horizon Minion on system boot and start immediately sudo systemctl enable --now minion Add OpenNMS repository GPG key sudo apt-key adv --fetch-keys https://debian.opennms.org/OPENNMS-GPG-KEY Add apt repository sudo add-apt-repository -s 'deb https://debian.opennms.org stable main' The message with conflicting distributions stable but got opennms-xx can be safely ignored. Installing the of Horizon Minion sudo apt -y install opennms-minion Disable the OpenNMS Horizon repository after installation to prevent unwanted upgrades when upgrading other packages on the server. After upgrade, Horizon requires manual steps to upgrade configuration files or migrate database schemas to a new version. For this reason, it is recommended to exclude the Horizon packages from update except when you are planning on performing an upgrade. sudo apt-mark hold opennms-minion Verify directory structure with the tree command sudo apt -y install tree tree /usr/share/minion -L 1 Directory structure after successful installation /usr/share/minion ├── bin ├── data -> /var/lib/minion/data ├── deploy -> /var/lib/minion/deploy ├── etc -> /etc/minion ├── lib ├── repositories └── system Enable Horizon Minion on system boot and start immediately sudo systemctl enable --now minion Install gnupg and add OpenNMS repository GPG key sudo apt -y install gnupg sudo apt-key adv --fetch-keys https://debian.opennms.org/OPENNMS-GPG-KEY Add apt repository sudo apt -y install software-properties-common sudo add-apt-repository -s 'deb https://debian.opennms.org stable main' sudo apt update The message with conflicting distributions stable but got opennms-xx can be safely ignored. Installing the of Horizon Minion sudo apt -y install opennms-minion Disable the OpenNMS Horizon repository after installation to prevent unwanted upgrades when upgrading other packages on the server. After upgrade, Horizon requires manual steps to upgrade configuration files or migrate database schemas to a new version. For this reason, it is recommended to exclude the Horizon packages from update except when you are planning on performing an upgrade. sudo apt-mark hold opennms-minion Verify directory structure with the tree command sudo apt -y install tree tree /usr/share/minion -L 1 Directory structure after successful installation /usr/share/minion ├── bin ├── data -> /var/lib/minion/data ├── deploy -> /var/lib/minion/deploy ├── etc -> /etc/minion ├── lib ├── repositories └── system Enable Horizon Minion on system boot and start immediately sudo systemctl enable --now minion Create a project directory for Horizon Core and create a docker-compose.yml file. mkdir minion cd minion vi docker-compose.yml --- version: '3' services: minion: image: opennms/minion:bleeding container_name: minion(1) environment: TZ: 'America/New_York'(2) JAVA_MIN_MEM: 512M(3) JAVA_MAX_MEM: 2048M(4) command: ["-c"] volumes: - ./minion-config.yaml:/opt/minion/minion-config.yaml(5) - ./scv.jce:/opt/minion/scv.jce(6) healthcheck: test: "/health.sh"(7) interval: 30s timeout: 6s retries: 3 ports:(8) - '514:1514/udp' - '162:1162/udp' - '8201:8201/tcp' 1 Friendly container name 2 Set the time zone and the Minion configuration via environment variables. To list all available time zones you can use timedatectl list-timezones 3 This is optional and can be used to control the minimal Java heap size 4 This is optional and can be used to control the maximum Java heap size 5 Configuration file for connectivity and features 6 Keystore file with enrypted credentials for authenticating REST and Broker endpoints 7 Run our health check to indicate the Minion is ready. It uses the opennms:health-check internally running in Karaf 8 Publish ports for Syslog, SNMP trap listener and the SSH access to the Karaf Shell The Minion process in the container runs as a non-root user and can’t open privileged ports < 1024. We open the listener for privileged (514/udp, 162/udp) and don’t have to give the Minion process in the container additional permissions. In some environments it is not allowed to use ICMP (ping) for non-root users. Horizon Minion runs as a non-privileged user in the container with userid=10001, groupid=10001. You can verify with sysctl -n net.ipv4.ping_group_range if the group id 10001 is in the allowed group range. If not you can add the following lines in your service definition in the docker-compose.yml file. sysctls: net.ipv4.ping_group_range: "10001 10001" Create a file with the configuration for connectivity and features --- http-url: 'http://core-instance-ip:8980/opennms'(1) broker-url: 'failover:tcp://core-instance-ip:61616'(2) id: 'my-minion'(3) location: 'my-location'(4) netmgt: syslog:(5) syslog.listen.interface: "0.0.0.0" syslog.listen.port: 1514 traps:(6) trapd.listen.interface: "0.0.0.0" trapd.listen.port: 1162 1 Replace the REST endpoint URL which goes to your Horizon Core instance 2 Replace the Broker endpoint URL which goes to your Horizon Core instance. If you have ActiveMQ with SSL running replace tcp with ssl. 3 Replace my-location with a location name representing your remote location the Minion is running 4 This optional, you can set a unique human readable Minion identifier, if not set a unique identifier is generated 5 Enable the Syslog listener on the Minion on port 1514/udp 6 Enable the SNMP trap listener on the Minion on port 1162/udp If want to run with Apache Kafka or configure flow listeners you can find the configuration reference in the Confd readme. Initialize the keystore with credentials docker-compose run -v $(pwd):/keystore minion -s Validate your Docker Compose file docker-compose config -q Start the service in background docker-compose up -d Verify the connectivity and the configuration with the health check docker-compose exec minion bin/client opennms:health-check The endpoints should be reachable and the bundles should be installed successful Connecting to OpenNMS ReST API [ Success ] Verifying installed bundles [ Success ] Connecting to JMS Broker [ Success ] => Everything is awesome Secure Access to Karaf Shell Change the default user/password admin/admin for the Karaf shell and encrypt it. CentOS/RHEL 7/8 Debian/Ubuntu Docker Enable password encryption sudo vi /opt/minion/etc/org.apache.karaf.jaas.cfg # # Boolean enabling / disabling encrypted passwords # encryption.enabled = true(1) #... encryption.algorithm = SHA-512(2) 1 Enable password encryption from false to true 2 Set a secure encryption algorithm like SHA-512 As soon the file is saved, Karaf will immediately encrypt the password in users.properties. Set a secure admin password for Karaf sudo vi /opt/minion/etc/users.properties # All users, groups, and roles entered in this file are available after Karaf startup # and modifiable via the JAAS command group. These users reside in a JAAS domain # with the name "karaf". # # OPENNMS: Change the admin user from 'karaf' to 'admin' admin = {CRYPT}C7AD...{CRYPT},_g_:admingroup(1) 1 Replace the whole string {CRYPT}C7AD…{CRYPT} with your new password in plainttext. As soon you save the file the password will be SHA-512 encrypted. Set restrictive file permissions sudo chmod 600 /opt/minion/etc/users.properties Enable password encryption sudo vi /usr/share/minion/etc/org.apache.karaf.jaas.cfg # # Boolean enabling / disabling encrypted passwords # encryption.enabled = true(1) #... encryption.algorithm = SHA-512(2) 1 Enable password encryption from false to true 2 Set a secure encryption algorithm like SHA-512 As soon the file is saved, Karaf will immediately encrypt the password in users.properties. Set a secure admin password for Karaf sudo vi /usr/share/minion/etc/users.properties # All users, groups, and roles entered in this file are available after Karaf startup # and modifiable via the JAAS command group. These users reside in a JAAS domain # with the name "karaf". # # OPENNMS: Change the admin user from 'karaf' to 'admin' admin = {CRYPT}C7AD...{CRYPT},_g_:admingroup(1) 1 Replace the whole string {CRYPT}C7AD…{CRYPT} with your new password in plainttext. As soon you save the file the password will be SHA-512 encrypted. Set restrictive file permissions sudo chmod 600 /usr/share/minion/etc/users.properties The only way changing the admin password and enable encryption is by injecting configuration files into the container. Commands are executed in the Docker project directory for the Minion. Create the configuration file to enable encryption vi org.apache.karaf.jaas.cfg Enable encryption for the admin password encryption.enabled = true encryption.prefix = {CRYPT} encryption.suffix = {CRYPT} encryption.algorithm = SHA-512 encryption.encoding = hexadecimal Create a property file for the SSH user for the Karaf Shell vi users.properties Set a secure admin password for Karaf admin = my-new-password,_g_:admingroup(1) _g_\:admingroup = group,admin,manager,viewer,systembundles,ssh(2) 1 Replace the string my-karaf-password with your new password in plainttext 2 Assign permissions to allow administrative tasks and access via SSH Set Minion process as owner and restrictive file permissions chown 10001:10001 users.properties chmod 600 users.properties Inject the configuration files with the volumes: - ./minion-config.yaml:/opt/minion/minion-config.yaml - ./scv.jce:/opt/minion/scv.jce - ./users.properties:/opt/minion/etc/users.properties - ./org.apache.karaf.jaas.cfg:/opt/minion/etc/org.apache.karaf.jaas.cfg Restart the Minion container docker-compose restart minion Changing the password or encryption algorithm get applied immediately. You do not need to restart the Minion. By default, the Karaf Shell is restricted to 127.0.0.1. To enable remote access, set sshHost=0.0.0.0 in org.apache.karaf.shell.cfg. The change is applied immediately and a Minion restart is not required. If you have firewall running on your host, allow 8201/tcp to grant access to the Karaf Shell. Configure Connectivity to the Core Instance Configurations has to be made in the Horizon etc directory. We reference etc relative to the OpenNMS Horizon core home directory. Depending on your operating system the home directory is /usr/share/opennms for Debian/Ubuntu or /opt/opennms for CentOS/RHEL. Kafka ActiveMQ gRPC AWS SQS Create a file to prevent installing ActiveMQ features on Minion startup sudo vi etc/featuresBoot.d/disable-activemq.boot Add the following lines to disable ActiveMQ features and save the file !minion-jms !opennms-core-ipc-rpc-jms !opennms-core-ipc-sink-camel Create a file to install Kafka features on Minion startup sudo vi etc/featuresBoot.d/kafka.boot Add the following lines to install the remote producer call (RPC) and sink feature for Kafka on Minion startup and save the file opennms-core-ipc-rpc-kafka opennms-core-ipc-sink-kafka Configure the Kafka features and the Minion location via the Karaf Shell ssh -p 8201 admin@localhost Configure the Minion location and REST endpoint config:edit org.opennms.minion.controller(1) config:property-set location my-location(2) config:property-set http-url http://core-instance-ip:8980/opennms(3) config:update(4) 1 Edit the minion configuration 2 Replace my-location with a location name representing your remote location the Minion is running 3 Replace the example http-url with the URL of your Horizon Core instance 4 Save the configuration By default the Minion generates a unique id. If you want to provide a human readable Minion identifier yourself with config:property-set id my-minion-name Configure the credentials for the REST endpoint and exit Karaf shell opennms:scv-set opennms.http my-minion-user my-minion-password(1) 1 Set the credentials for the REST endpoint created in your Horizon Core instance The credentials are encrypted on disk in etc/scv.jce. Configure the Kafka endpoints for RPC feature config:edit org.opennms.core.ipc.rpc.kafka config:property-set bootstrap.servers my-kafka-ip-1:9092,my-kafka-ip-2:9092(1) config:update 1 Connect to the following Kafka nodes and adjust the IPs or FQDNs with the Kafka port (9092) accordingly. Configure the Kafka endpoints for Sink feature config:edit org.opennms.core.ipc.sink.kafka config:property-set bootstrap.servers my-kafka-ip-1:9092,my-kafka-ip-2:9092(1) config:update 1 Connect to the following Kafka nodes and adjust the IPs or FQDNs with the Kafka port (9092) accordingly. If you set more than one Kafka node as bootstrap.servers the driver will attempt to connect to the first entry. If that is successful the whole broker topology will be discovered and will be known by the client. The other entries are only used if the connection to the first entry fails. Ensure you use the FQDN or IP for your Kafka nodes as configured as advertised listener. Exit the Karaf Shell with Ctrl+d Restart the Minion to apply the configuration sudo systemctl restart minion Verify the configuration using the health check in the Karaf Shell ssh -p 8201 admin@localhost Run the health check command opennms:health-check Verify if all components are configured properly Verifying the health of the container Connecting to OpenNMS ReST API [ Success ] Verifying installed bundles [ Success ] Connecting to Kafka from RPC [ Success ] Connecting to Kafka from Sink Producer [ Success ] => Everything is awesome Connect to the Karaf shell with user admin and password admin ssh -p 8201 admin@localhost Configure REST endpoints, ActiveMQ and remote location name config:edit org.opennms.minion.controller(1) config:property-set location my-location(2) config:property-set http-url http://core-instance-ip:8980/opennms(3) config:property-set broker-url failover:tcp://core-instance-ip:61616(4) config:update(5) 1 Edit the minion configuration 2 Replace my-location with a location name representing your remote location the Minion is running 3 Replace the REST endpoint URL which goes to your Horizon Core instance 4 Replace the Broker URL which goes to your Horizon Core instance. If you have ActiveMQ with SSL running replace tcp with ssl. 5 Save the configuration By default the Minion generates a unique id. If you want to provide a human readable Minion identifier yourself with config:property-set id my-minion-name Configure the credentials and exit Karaf shell opennms:scv-set opennms.http my-minion-user my-minion-password(1) opennms:scv-set opennms.broker my-minion-user my-minion-password(2) 1 Set the credentials for the REST endpoint created in your Horizon Core instance 2 Set the credentials for the ActiveMQ message broker The credentials are encrypted on disk in etc/scv.jce. Exit the Karaf Shell with Ctrl+d Restart the Minion to apply the configuration sudo systemctl restart minion Verify the configuration using the health check in the Karaf Shell ssh -p 8201 admin@localhost Run the health check to verify connectivity opennms:health-check The result should show Success for each component Connecting to OpenNMS ReST API [ Success ] Verifying installed bundles [ Success ] Connecting to JMS Broker [ Success ] => Everything is awesome Create a file to prevent installing ActiveMQ features on Minion startup sudo vi etc/featuresBoot.d/disable-activemq.boot Add the following lines to disable ActiveMQ features and save the file !minion-jms !opennms-core-ipc-rpc-jms !opennms-core-ipc-sink-camel Create a file to install gRPC features on startup sudo vi etc/featuresBoot.d/grpc.boot Add the gRPC client features opennms-core-ipc-grpc-client Configure the gRPC client and the Minion location via the Karaf Shell ssh -p 8201 admin@localhost Configure the Minion location and REST endpoint config:edit org.opennms.minion.controller(1) config:property-set location my-location(2) config:update(3) 1 Edit the minion configuration 2 Replace my-location with a location name representing your remote location the Minion is running 3 Save the configuration By default the Minion generates a unique id. If you want to provide a human readable Minion identifier yourself with config:property-set id my-minion-name Configure the gRPC endpoint config:edit org.opennms.core.ipc.grpc.client config:property-set host core-instance-ip(1) config:property-set port 8990(2) config:update(3) 1 Set the host to connect to the gRPC server running on the Horizon Core instance, replace the core-instance-ip accordingly 2 Set the port of the gRPC server which is by default 8990 3 Save the configuration Restart the Minion to apply the changes systemctl restart minion Verify the configuration using the health check in the Karaf Shell ssh -p 8201 admin@localhost Run the health check command opennms:health-check Verify if all components are configured properly admin@minion> opennms:health-check Verifying the health of the container Connecting to OpenNMS ReST API [ Success ] Verifying installed bundles [ Success ] Connecting to gRPC IPC Server [ Success ] => Everything is awesome This optional, if you want to enable TLS for gRPC you have to provide certificate files and enable it. The commands for TLS are described below. Connect to the Karaf Shell ssh -p 8201 admin@localhost Configure TLS and certificate parameters config:edit org.opennms.core.ipc.grpc.client config:property-set tls.enabled true(1) config:property-set trust.cert.filepath /custom-path/ca.crt(2) config:property-set client.cert.filepath /custom-path/client.crt(3) config:property-set client.private.key.filepath /custom-path/client.pem(4) config:update(5) 1 Enable TLS for the gRPC server 2 Set the path to your CA certificate file 3 Set the path to your client certificate file 4 Set the path client certificate key file 5 Save and update the configuration This is optional and you can set a maximum message size for gRPC. The maxium size has to be the same on the Horizon Core instance as well. If you don’t set a maximum message size the default is 10 MiB. Configure maximum message size for gRPC in the Karaf Shell config:edit org.opennms.core.ipc.grpc.client config:property-set max.message.size 10485760 config:update Apply the changes with restarting Horizon Core instance sudo systemctl restart opennms Create a file to install AWS SQS features on Minion startup sudo vi etc/featuresBoot.d/aws-sqs.boot Add the following lines to disable ActiveMQ and load AWS SQS !minion-jms !opennms-core-ipc-rpc-jms !opennms-core-ipc-sink-camel opennms-core-ipc-rpc-aws-sqs opennms-core-ipc-sink-aws-sqs Configure the AWS SQS via the Karaf Shell ssh -p 8201 admin@localhost config:edit org.opennms.core.ipc.aws.sqs(1) config:property-set aws_region us-east-1(2) config:property-set aws_access_key_id my-access-key(3) config:property-set aws_secret_access_key my-secret-access-key(4) config:property-set sink.FifoQueue false(5) config:update(6) 1 Configure the AWS SQS feature 2 Set AWS SQS region 3 The AWS SQS access key 4 The AWS SQS secret for the access key 5 If consistent ordering of incoming messages is required, FIFO queues can be used, the default is false and must match with the Core instance setting 6 Save and update the configuration Restart the Minion to apply the configuration sudo systemctl restart minion Verify the configuration using the health check in the Karaf Shell ssh -p 8201 admin@localhost Run the health check command opennms:health-check Verify if all components are configured properly Verifying the health of the container Connecting to OpenNMS ReST API [ Success ] Verifying installed bundles [ Success ] Connecting to AWS-SQS from RPC [ Success ] Connecting to AWS-SQS from Sink Producer [ Success ] => Everything is awesome You can find available configuration parameters in the Amazon Simple Queueing Services reference section. Check Minion in the Core web UI Log in to the web UI as an administrative user. Click the gears icon and choose Manage Minions in the Distributed Monitoring area. After a few minutes your Minion should be provisioned automatically and the status should be up. Minimum System Requirements Installing JDBC Driver