What is it for?

Zenoss Cloud is an optional infrastructure monitoring service offering managed by the ITS Campus Solutions Monitoring Team. The service provides basic monitoring for CPU, Disk, Memory and Network utilization for networked devices (including hosts.) Many advanced and non-standard services (such as web monitoring and metrics generation) are not officially supported by the monitoring team.

Getting Started

Enrolling a GROUP into Zenoss Cloud

If your academic unit or group is not set up in Zenoss Cloud and you would like it to be, a few hurdles need to be cleared before adding users and hosts to Zenoss Cloud. If you just need to add a new user to an existing group, skip ahead.

It is the responsibility of customers to inform the Monitoring Team if the name of the group changes, or moves under a different sponsor, or if new team members need to be added or removed. While the Groups do use AD for authentication and access, these users and groups have the layers across the platform that need to be maintained by admins in order for multi-tenancy to work with adfs. For this reason, users cannot supply their own AD group.

Group Name:
EXAMPLE: FAS →ITS→ Campus Solutions → ESS → Application & Delivery Development

Members:
Standard UT emails for all users of all services within the group

Setting up monitoring and alerting for Services in a Group

Second, we need to set up alerting for your devices. As a courtesy, the Monitoring Team will add devices for newly enrolled Groups, but users are expected to add, remove, and maintain their own monitors.

Under any given Group, there could be any number of SERVICES. These Services are defined separately from each other. All members of the Group a service belongs to will have console access to the related monitors, but the recipients of alerts will be specified by a single group email maintained by the customer.

Users are expected to maintain their own monitors, including adding and deleting servers, but the Monitoring Team will add them in one go as a courtesy for new services, as batch adding can only be done by users via the API. Hosts added in this way will be set in Pre-Production or to Out-Of-Service by default if they are TEST or DEV hosts (*d0* or *t0* hostnames).

Zenoss Cloud has limited resources, so TEST and DEV monitors are only to be used for testing changes to services within the application lifecycle and set back to Out-of-Service to free up resources for others. If users abuse this or if resources no longer allow this to be sustainable, TEST and DEV hosts will be removed by admins.

For those needing full-time monitoring of TEST and DEV hosts, consider the Monitoring Team's sister service, Thousand Eyes.

Accessing Zenoss Cloud

Once you have created an account, and the test monitor has been verified working, log into the Cloud instance at https://ut.zenoss.io/cz0/zport/dmd/itinfrastructure with your UT email.

You'll then see the familiar interface of the Collection Zone, analogous to the Resource Manager in the on-prem Zenoss instance.

Adding Monitors

Click on the "+" sign in the Infrastructure view and in the pulldown, select, Add a Single Device.

Hostname or IP Address:Always use full FQDN. Never use IP addresses as it creates huge problems later down the road.
Device Class: Choose between:

• /Server/Linux/UT/$GROUPNAME
• /Server/Microsoft/Windows/UT/$GROUPNAME
• /Ping/UT/$GROUPNAME

Collector: AUSTIN 
Never choose localhost.

Production Status: I keep this as Pre-Production until ready to begin monitoring, but one can leave it as Production if they want monitoring to begin immediately. TEST and DEV hosts should be kept in Out-of-Production except when testing code changes as part of the development cycle. Users are on the honor system to keep them in this state to ensure there are ample resources for everyone.

Device Priority: Typically Normal. High and Highest will provoke USC Operator calls when alerts trigger IF the hosts are added to a service within Sysmod.

Systems: This is essential as alerting is mapped to this location. Choose the directory you want your hosts to be found within Zenoss 

Leave everything else as default and click Add.

Removing Monitors

To remove a monitor, click and highlight in the Infrastructure view and click the "-" button.

Click "Remove"

Managing Monitors

Changing Production State

Production states are for specific situations:

Pre-Production -- Monitor is being set up.  Not actively alerting, but will collect metrics and monitor. After two weeks, it should be set to either Production or Out-of-Service. Hosts left in Pre-Production for more than two weeks will be set to Production by policy.

Production – Actively monitoring, collecting metrics and alerting.

Maintenance – Monitor is in a designated Maintenance Window. It will continue to monitor and collect metrics but will not alert. Like Pre-Production, it will revert to Production after two weeks.

Out-of-Service – Monitor is neither in Production, Pre-Production, or Maintenance, but is not intended to be Decommissioned. It will not collect metrics, monitor, or alert. When in doubt, use this state. This is the default for DEV and TEST hosts except for testing code changes as part of the development cycle.

Decommissioned – Monitor is at end-of-life. It will not collect metrics, monitor, or alert.

Monitors left in Maintenance or Pre-Production for more than 2 weeks will be put into Production. TEST and DEV hosts left in Production will be set to Out-of-Service as resources demand.

Changing Priority

Priority can be changed from the default Normal to High or Highest, which can get UDC Operators to call the registrants for the service in Sysmod.

This can be done individually, with the drop down menu under the production state.

From the individual view of a device, click on Device Administration in the left-hand column.

At the top of the next page, under Maintenance Windows, click the "+" to create a maintenance window.

From here you can create a maintenance window of your choosing, and you can create as many of them as you like, though it should be noted that users should verify and add the weekly Maintenance Window set by ITS by first looking that information up in SCCM.

Changing IP, Renaming Hostnames

If a host changes names or is repurposed with a new OS or has a new OS installed, REMOVE the monitor and create a new one. 

If the only change is IP, however, select the "*" in the bottom left, and select Reset/Change IP Address. Leave it blank and click save.

Always create monitors with FQDN. Never, ever insert IP numbers. It will work, but it is very difficult to support.

Changing Device Type

Sometimes you might have a Windows or Linux monitor that is pingable but otherwise isn't working due to some configuration on the server having recently changed.

You'll often want to set it to ping-only in order to maintain monitoring until the issue is addressed.

Conversely, you might have a monitor that is Ping-only that you may want to change to Windows or Linux.

Go to the lower left of the detail page of the specific monitor, and select the star "*" menu and choose Change Device Class.

That should open a box to select the specific device class you want to change the device to. It should only allow you to select the device classes your group owns.

Once you have changed the device class, you can test that everything works as it should by clearing all alerts associated with the device, and remodel by choosing Modeling--> Model Device.

Managing Alerts

Advanced Topics

To create an API key, log into ut.zenoss.io (aka the Cloud Layer).

• Look in the upper right for the pulldown with your name.
• Click on it.
• Choose "Settings."
• On the left side, click "Collection Zone API Keys"
• On the right side, click the "Add Key" button.

This will bring up the API ADD CLIENT page. Enter the name associated with the account and click SAVE. This will create an API user with the same permissions of the account creating it.

To see what others have done with the API:

• Michael Kaczmarczik from the UT Web team has written up a deep dive into the API, including using it with Ansible.
• Rabindra Kar wrote up how to do REST API calls.
• For other questions regarding the API, we recommend reading official documentation published by the vendor and otherwise contacting the Monitoring Team.

Creating and Using Service Accounts With the API

• Users will need to request an AD account from the AD team to create a service account to use with Zenoss Cloud, for the singular purpose of generating an API key not tied to a specific user.
• That account will be added as a member of their Group and will have the same access as any other user, but will be unable to access the console as it does not have the means to authenticate via MFA.
• Users will need to make a request to the monitoring team to grant temporary access for a brief window in order to disable MFA for a service account so a user can access the console with the account and generate an API key.

Other Functions