Troubleshooting Zenoss Cloud

Since migrating to the new Cloud service, many are reporting various new alerts that were not previously seen on-prem. Understand there has been a jump in multiple releases of Zenoss, as well as the fact that Cloud is set up to model a much wider set of metrics for both Windows and Linux devices which means there are many things now being monitored by default that were not previously. 

The following is a list of current outstanding issues as yet not fully resolved by Zenoss, with links to both the open Zenoss ticket (requires access to Zenoss client portal) and related SNOW events and/or the reporting users. Issues that have a Zenoss defect assigned are verified Zenoss bugs to be addressed in future updates. 

All outstanding issues were verified by Zenoss, and as of March 23, 2023 remain in their bugtracker.

USERS EXPERIENCING ERRORS WHEN CLICKING EVENT DETAIL LINK IN ALERT EMAILS

Users report links sent by alert emails not working correctly when clicked, showing errors in Zenoss Cloud. It's a known bug that affects web-based emails clients, O365 in particular. No known workaround is yet known by the vendor -- users using web mail clients are advised to log in directly to Zenoss Cloud to view the errors.

Mail team has recently whitelisted Zenoss in O365, but it did not provide a fix.

WORKAROUND FOR DISK SPACE BUG

Discovered by and workaround submitted to Zenoss by Mic. This is the infamous 1000000% disk space full alert found even before on-prem and is not considered related to the other issues. Mic's findings have been included into the Zenoss internal JIRA.

Mic Kaczmarczik

SSH Collector Key Fails with RHEL 9

In the process of testing SSH-based monitoring, it was discovered that the public key worked fine with RHEL 7/8 hosts, but not RHEL 9. THis is because RHEL 9 uses a stricter security stance than older versions.

According to Armando Rivera, a similar issue was seen with the Nessus client, which prompted Tenable to upgrade to a 4096-bit key in order to play nice with newer Linux releases.

Zenoss has aknowledged the issue and has said an upgrade to the key is coming in future upgrades.

Zach Helke

WinRS Not Encrypted

RHEL 5, which originally hosted the original on-Prem instance of Zenoss, had issues communicating using the default Kerberos encryption in WinRM. At the time the workaround was to use encrypted https, which would have been difficult to maange for individual hosts, prompting an ISO-sanctioned decision to run unencrypted, and out standard GPO for setting up Zenoss for Windows turns off encryption.

According to notes from Zenoss on the Windows ZenPack, this issue was resolved with RHEL6, and it should no longer be neccessary. We are seeking verification and oversignt from Zenoss but want to to test re-introducing encryption back to our WinRM setups.

Zach Helke

Solved Issues

Reported by

Missed Runs /SNMP issues on Linux devices

INC1021235 Grady Bailey

Emily Magee Feb 14, 2022

Monitoring Team Feb 2, 2022

Issues with device transforms / recreating on-prem behavior in Cloud

This issue was first seen as device transforms that previously worked on-prem not working correctly in Cloud. This includes the format of alert messages, thresholds, and downgrading some things to warnings. Scope has been widened to look at all aspects of the on-prem configuration to determine why it is not working correctly in Cloud.

On Feb 17, the vendor asked to collect customized thresholds and transforms from our TEST on-prem instance. After allowed them to do so and they attempted to apply some of these changes without informing us, resulting in the instance going down for several hours. That circumstance has now been added to the scope of this ticket.

On Feb 23, and March 1, during the services weekly MW, Zenoss has applied incremental changes to the backend in an effort to get it to the desired state without bringing it down.

INC1013090 Drew Coyle

INC1020917 Michael Kae Wang

INC1019947 Brick Jones

INC0990957 Tori Brown

Monitoring Team September1, 2021

ISSUE WITH API IN SCRIPTING

DBAs utilizing the API discovered errors when trying to run curl commands and use with Ansible. Issue is verified by Zenoss.

Kevin Chang
James Alexander