Intro to omc

In this module we will at a subset of the features in omc that allow you to quickly review an OpenShift Must-Gather. omc has similar commands as the oc client so those will not be covered and instead we will cover the omc specific commands.

omc `use` command

To get started you want to verify that omc is installed in your path and working
Once you have confirmed this, you want to call the omc use command to tell omc the must-gather to utilize for review
Then run the omc get clusterversion to verify that you are using the must-gather

Example 1. omc use

$ omc version
omc version: v3.7.4
hash: 45e91af
https://github.com/gmeghnag/omc

cd ~/Module2/

$ omc use sno_demo
Must-Gather  : /home/lab-user/Module2/sno_demo/quay-io-openshift-release-dev-ocp-v4-0-art-dev-sha256-2de07af89683678ae6bb7a939615fc0d4ced7fe185add38b050f2c6f60023b6f
Project      : default
ApiServerURL : https://api.cluster-6fmht.dynamic.redhatworkshops.io:6443
Platform     : None
ClusterID    : 1304b08d-2a73-410f-98ed-342233008cb0

$ omc get clusterversion
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.16.20   True        False         20h     Cluster version is 4.16.20

omc `certs inspect` command

The omc certs inspect command allows you to inspect all certificates in ConfigMaps and Secrets that are in the must-gather
Additionally, the command also highlights CertificateSigngingRequest which can help resolve issues related to nodes joining a cluster.

Example 2. certs

$ omc certs inspect
NAME                     KIND                        AGE    CERTTYPE      SUBJECT                                                                                        NOTBEFORE                       NOTAFTER
csr-zwmnc                CertificateSigningRequest   53m    ca-bundle     CN=system:multus:ocp4-2nvq7-worker-xdwch,O=system:multus                                     2024-08-14 14:27:20 +0000 UTC   2024-08-15 14:27:20 +0000 UTC

omc `ectd` command

The etcd command allows you to view the health and the status of your etcd cluster.
It includes two options, etcd health and etcd status

Example 3. etcd

$ omc etcd health
+---------------------------+--------+-------------+-------+
|         ENDPOINT          | HEALTH |    TOOK     | ERROR |
+---------------------------+--------+-------------+-------+
| https://10.36.18.138:2379 | true   | 26.26127ms  |       |
| https://10.36.18.23:2379  | true   | 24.586699ms |       |
| https://10.36.18.22:2379  | true   | 28.410291ms |       |
+---------------------------+--------+-------------+-------+

$ omc etcd status
+---------------------------+------------------+---------+----------------+----------+-----------+------------+-----------+------------+--------------------+--------+
|         ENDPOINT          |        ID        | VERSION | DB SIZE/IN USE | NOT USED | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+---------------------------+------------------+---------+----------------+----------+-----------+------------+-----------+------------+--------------------+--------+
| https://10.36.18.138:2379 | 39ddb61a8f3bee1d | 3.5.13  | 325 MB/207 MB  | 37%      | true      | false      |       176 |  873809134 |          873809134 |        |
| https://10.36.18.22:2379  | a2b2eb5b34d6a292 | 3.5.13  | 323 MB/207 MB  | 36%      | false     | false      |       176 |  873809136 |          873809136 |        |
| https://10.36.18.23:2379  | dbb6cf331005ae32 | 3.5.13  | 291 MB/207 MB  | 30%      | false     | false      |       176 |  873809138 |          873809138 |        |
+---------------------------+------------------+---------+----------------+----------+-----------+------------+-----------+------------+--------------------+--------+

omc `haproxy` command

The haproxy backends command displays all of the haproxy configured backends

Example 4. haproxy

$ omc haproxy backends
NAMESPACE		NAME					INGRESSCONTROLLER	SERVICES				    PORT		TERMINATION
aap			    frost-prod				default			    frost-prod-service			http(8052)	edge/Redirect
ecomm			app-api-blue-p4lb5		default			    ecomm-api-blue				https(8443)	reencrypt/Redirect
ecomm			app-api-prod-kg8l6		default			    plaid-api-prod				https(8443)	passthrough/Redirect

omc `machine-config` command

The omc machine-config diff command allows you to compare two MachineConfig objects. This is different from the built-in omc get machineconfigs which will only display the MachineConfigs.
The diff option will open the selected machine-configs in vim-diff to allow quick review of the two MachineConfigs to help quickly identify all changes.

Example 5. machine-config

$ omc get machineconfigs -n openshift-machineconfig-operator
NAME                                               GENERATEDBYCONTROLLER                      IGNITIONVERSION   AGE
00-master                                          20e767851f25a12f9be5701012abb881ac33ab25   3.4.0             8d
00-worker                                          20e767851f25a12f9be5701012abb881ac33ab25   3.4.0             8d
...
99-worker-generated-registries                     20e767851f25a12f9be5701012abb881ac33ab25   3.4.0             8d
rendered-master-c5f327c05f9e70b9ef90ad15b0610341   20e767851f25a12f9be5701012abb881ac33ab25   3.4.0             8d
rendered-worker-cf392af65e20a73197f046cacaf156e5   20e767851f25a12f9be5701012abb881ac33ab25   3.4.0             8d

$ omc machine-config diff rendered-master-c5f327c05f9e70b9ef90ad15b0610341 rendered-worker-cf392af65e20a73197f046cacaf156e5

omc `node-logs` command

The omc node-logs command lists the node-logs collected from the OpenShift control-plane nodes and are in the must-gather.
By running omc node-logs crio, for example, will then output crio-logs for all of the control-plane nodes.

Example 6. node-logs

$ omc node-logs
The following node services logs are available to be read:

- NetworkManager
- crio
- kubelet
- machine-config-daemon-firstboot
- machine-config-daemon-host
- openvswitch
- ostree-finalize-staged
- ovs-configuration
- ovs-vswitchd
- ovsdb-server
- rpm-ostreed

is it possible to read the content by executing 'omc node-logs <SERVICE>'.

$ omc node-logs kubelet | head -n 10
Aug 07 15:25:30.970016 ocp4-2nvq7-master-0 kubenswrapper[1916]: I0807 15:25:30.969893    1916 kubelet_getters.go:187] "Pod status updated" pod="openshift-kube-apiserver/kube-apiserver-ocp4-2nvq7-master-0" status=Running
Aug 07 15:25:30.970016 ocp4-2nvq7-master-0 kubenswrapper[1916]: I0807 15:25:30.970031    1916 kubelet_getters.go:187] "Pod status updated" pod="openshift-vsphere-infra/coredns-ocp4-2nvq7-master-0" status=Running
Aug 07 15:25:30.971039 ocp4-2nvq7-master-0 kubenswrapper[1916]: I0807 15:25:30.970074    1916 kubelet_getters.go:187] "Pod status updated" pod="openshift-vsphere-infra/haproxy-ocp4-2nvq7-master-0" status=Running
Aug 07 15:25:30.971039 ocp4-2nvq7-master-0 kubenswrapper[1916]: I0807 15:25:30.970121    1916 kubelet_getters.go:187] "Pod status updated" pod="openshift-kube-controller-manager/kube-controller-manager-ocp4-2nvq7-master-0" status=Running
Aug 07 15:25:30.971039 ocp4-2nvq7-master-0 kubenswrapper[1916]: I0807 15:25:30.970159    1916 kubelet_getters.go:187] "Pod status updated" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ocp4-2nvq7-master-0" status=Running
Aug 07 15:25:30.971039 ocp4-2nvq7-master-0 kubenswrapper[1916]: I0807 15:25:30.970180    1916 kubelet_getters.go:187] "Pod status updated" pod="openshift-vsphere-infra/keepalived-ocp4-2nvq7-master-0" status=Running
Aug 07 15:25:30.971039 ocp4-2nvq7-master-0 kubenswrapper[1916]: I0807 15:25:30.970216    1916 kubelet_getters.go:187] "Pod status updated" pod="openshift-kube-scheduler/openshift-kube-scheduler-ocp4-2nvq7-master-0" status=Running
Aug 07 15:25:30.971039 ocp4-2nvq7-master-0 kubenswrapper[1916]: I0807 15:25:30.970240    1916 kubelet_getters.go:187] "Pod status updated" pod="openshift-etcd/etcd-ocp4-2nvq7-master-0" status=Running
Aug 07 15:25:56.027736 ocp4-2nvq7-master-1 kubenswrapper[1927]: I0807 15:25:56.027649    1927 kubelet_getters.go:187] "Pod status updated" pod="openshift-kube-controller-manager/kube-controller-manager-ocp4-2nvq7-master-1" status=Running
Aug 07 15:25:56.028592 ocp4-2nvq7-master-1 kubenswrapper[1927]: I0807 15:25:56.027867    1927 kubelet_getters.go:187] "Pod status updated" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ocp4-2nvq7-master-1" status=Running

omc `ovn` command

The omc ovn subnets command will output all of the ovn subnets on the cluster.

Example 7. ovn

$ omc ovn subnets
HOST/NODE                       ROLE                          HOST IP-ADDRESSES               PRIMARY IF-ADDRESS   HOST GATEWAY-IP   NODE SUBNET
control-plane-cluster-6fmht-1   control-plane,master,worker   10.10.10.10/24,192.168.1.2/24   10.10.10.10/24       10.10.10.1        10.132.0.0/23

omc `prometheus` command

The omc prometheus command provides several options to output Prometheus alertgroup, alertrule, and target.

Example 8. prometheus

$ omc prometheus alertgroup | head -n 10
GROUP                                               FILENAME                                                                                                                AGE
CloudCredentialOperator                             openshift-cloud-credential-operator-cloud-credential-operator-alerts-2b1b6efc-359d-41f1-910c-f759091ea8db.yaml          27s
cluster-machine-approver.rules                      openshift-cluster-machine-approver-machineapprover-rules-559e1f58-cf67-435f-8e25-8fe67acc824f.yaml                      14s
node-tuning-operator.rules                          openshift-cluster-node-tuning-operator-node-tuning-operator-2ed91e6f-a85e-48fe-bc8d-1df61349ecb2.yaml                   1s
SamplesOperator                                     openshift-cluster-samples-operator-samples-operator-alerts-07e868fe-c246-493c-b948-963979fb222e.yaml                    28s
default-storage-classes.rules                       openshift-cluster-storage-operator-prometheus-39ea760b-44d6-4c6d-b9c8-698cfed53b24.yaml                                 7s
storage-operations.rules                            openshift-cluster-storage-operator-prometheus-39ea760b-44d6-4c6d-b9c8-698cfed53b24.yaml                                 7s
storage-selinux.rules                               openshift-cluster-storage-operator-prometheus-39ea760b-44d6-4c6d-b9c8-698cfed53b24.yaml                                 11s
cluster-operators                                   openshift-cluster-version-cluster-version-operator-af01a96b-d635-43af-935d-8c09f1b4ef0e.yaml                            24s
cluster-version                                     openshift-cluster-version-cluster-version-operator-af01a96b-d635-43af-935d-8c09f1b4ef0e.yaml                            26s

$ omc prometheus alertrule | head -n 10
RULE                                                    SEVERITY   STATE      AGE   ALERTS   ACTIVE SINCE
CloudCredentialOperatorTargetNamespaceMissing           warning    inactive   27s   0        ----
CloudCredentialOperatorProvisioningFailed               warning    inactive   27s   0        ----
CloudCredentialOperatorDeprovisioningFailed             warning    inactive   27s   0        ----
CloudCredentialOperatorInsufficientCloudCreds           warning    inactive   27s   0        ----
CloudCredentialOperatorStaleCredentials                 warning    inactive   27s   0        ----
MachineApproverMaxPendingCSRsReached                    warning    inactive   14s   0        ----
NTOPodsNotReady                                         warning    inactive   1s    0        ----
NTODegraded                                             warning    inactive   1s    0        ----
SamplesRetriesMissingOnImagestreamImportFailing         warning    inactive   28s   0        ----

$ omc prometheus target | head -n 10
TARGET                                                   SCRAPE URL                                        HEALTH   LAST ERROR
openshift-apiserver-operator-5b89bd7bb8-z69dz            https://10.132.0.12:8443/metrics                  up
apiserver-66dcdc546c-vxms2                               https://10.132.0.144:17698/metrics                up
apiserver-66dcdc546c-vxms2                               https://10.132.0.144:8443/metrics                 up
authentication-operator-595d65667-92gcg                  https://10.132.0.26:8443/metrics                  up
oauth-openshift-545bf7bdf7-6n8xd                         https://10.132.0.253:6443/metrics                 up
cloud-credential-operator-65d6f5df6d-wknks               https://10.132.0.47:8443/metrics                  up
machine-approver-7d57ddd485-f6cv6                        https://10.10.10.10:9192/metrics                  up
cluster-node-tuning-operator-56f7cbd8bc-k8qgq            https://10.132.0.24:60000/metrics                 up
cluster-samples-operator-dbfb4c7b-jhqz6                  https://10.132.0.48:60000/metrics                 up

Intro to omc

omc use command

omc certs inspect command

omc ectd command

omc haproxy command

omc machine-config command

omc node-logs command

omc ovn command

omc prometheus command