Anatomy of a must-gather - Analyzing real customer issues

Welcome to Red Hat One 2025 and this exciting, hand-on, lab experience. Throughout this lab, we will explore a wide variety of tools available to analyze, search and debug OpenShift cluster data to find patterns, issues and other useful information that will help you engage with your customers.

This lab does not use any running OpenShift cluster, it instead focuses on offline debugging using the most common and easiest to collect files our customers provide when they run into issues.

openshift-logo

Lab structure

This lab is broken down into a set of 10 unique modules that explore a variety of tools. There is no specific ordering or pre-requisite to move between each module, but we do recommend you review Module 1 and Module 2 first.

Once you have reviewed Module 1 and Module 2, feel free to continue to Module 3 or pick one that interests you the most.

  1. Module 1 introduces all of the tools we will be using

  2. Module 2 does a deeper dive on the functionality in the omc tool

  3. Module 3 uses omc to explore an issue with a vSphere IPI cluster

  4. Module 4 uses kubectl-dev_tool to explore a cluster with an overloaded OpenShift API

  5. Module 5 uses ocp_insights.sh to proactively look for performance issues with OpenShift Insights data

  6. Module 6 uses etcd-ocp-diag.sh to review etcd logs for performance related issues

  7. Module 7 uses omc to review a stuck Cluster Upgrade

  8. Module 8 uses omc to review installed OLM Operators

  9. Module 9 uses omc and manual checks on the cluster must-gather in order to explore an OCP networking issue

  10. Module 10 uses koff to get and inspect data from an etcd database snapshot