Verify the Changes and Conclusion

Step 4: Verify the Changes

Once the nodes have been updated, you can verify that rsyslog is running and generating logs.

Check Node Status: Monitor the nodes to ensure they reboot successfully and return to a Ready state. Because this is a single-node OpenShift demo environment, the web console will be unavailable for a while.
```
oc get nodes
```
Inspect a Node: SSH into the node. In a single-node OpenShift cluster, you can get the node name and debug it with the following command.
```
oc debug node/$(oc get nodes -o jsonpath='{.items[0].metadata.name}')
```

Check for Log Files: List the contents of /var/log. With rsyslog running, you should see classic log files like messages, secure, and boot.log.

chroot /host
ls -l /var/log

Example Output:

total 53604
lrwxrwxrwx.  1 root        root                  39 Sep  1 04:58 README -> ../../usr/share/doc/systemd/README.logs
drwx------.  2 root        root                  23 Sep  1 04:58 audit
-rw-------.  1 root        root                8055 Sep  3 13:20 boot.log
-rw-rw----.  1 root        utmp                   0 Sep  1 04:58 btmp
...
-rw-------.  1 root        root            24620123 Sep  3 13:28 messages
...
-rw-------.  1 root        root                5169 Sep  3 13:20 secure
...

4. Verify System Services and OS Layer

After the node has rebooted, it’s crucial to verify that the new OS layer is active and the rsyslog service is running as expected.

a. Check the `rpm-ostree` Status

The rpm-ostree status command confirms which OS deployment is currently active. The output should show that the system is booted into your custom-layered image.

Verify the active OS deployment.

rpm-ostree status

Example Output Analysis:

State: idle
Deployments:
● ostree-unverified-registry:quay.io/wangzheng422/qimgs:rhcos-4.18-rsyslog-2025.09.03-v02
                   Digest: sha256:51221e7c01b3bb77cb53d5cc77eb7a089e900b2d0186454a9964b04b11d6c491
                  Version: 418.94.202508060022-0 (2025-09-03T13:12:38Z)

The ● symbol indicates that the currently running deployment is the one from your custom image (quay.io/wangzheng422/qimgs:rhcos-4.18-rsyslog-2025.09.03-v02).
The State: idle confirms that there are no ongoing rpm-ostree operations.

b. Check the `rsyslog` Service Status

Next, verify that the rsyslog service was started automatically and is running without errors.

Check the status of the rsyslog service.

systemctl status rsyslog

Example Output Analysis:

● rsyslog.service - System Logging Service
     Loaded: loaded (/usr/lib/systemd/system/rsyslog.service; enabled; preset: enabled)
     Active: active (running) since Wed 2025-09-03 14:20:16 UTC; 34min ago
...
   Main PID: 2034 (rsyslogd)
...
Sep 03 14:20:16 master-03-demo systemd[1]: Starting System Logging Service...
Sep 03 14:20:16 master-03-demo rsyslogd[2034]: [origin software="rsyslogd" swVersion="8.2412.0-1.el9" x-pid="2034" x-info="https://www.rsyslog.com"] start
Sep 03 14:20:16 master-03-demo systemd[1]: Started System Logging Service.
Sep 03 14:20:17 master-03-demo rsyslogd[2034]: message too long (8461) with configured size 8096, begin of message is: time="2025-09-03 13:49:49.187181490Z" level=info msg="Current CRI-O c>
Sep 03 14:20:18 master-03-demo rsyslogd[2034]: imjournal from <master-03-demo:kubenswrapper>: begin to drop messages due to rate-limiting
Sep 03 14:20:18 master-03-demo rsyslogd[2034]: imjournal: journal files changed, reloading...  [v8.2412.0-1.el9 try https://www.rsyslog.com/e/0 ]
Sep 03 14:30:21 master-03-demo rsyslogd[2034]: imjournal: 20204 messages lost due to rate-limiting (20000 allowed within 600 seconds)

Active: active (running) confirms the service is operational.
The log entries show the service starting successfully.
Important: The output also reveals potential issues that may need attention in a production environment:
message too long (8461) with configured size 8096: This indicates that some log messages from CRI-O are larger than the default rsyslog message size limit. You may need to adjust the $MaxMessageSize setting in /etc/rsyslog.conf if you need to capture these large messages in their entirety.
imjournal: … messages lost due to rate-limiting: This warning means that the journal is producing logs faster than rsyslog is configured to process them, causing some messages to be dropped. To mitigate this, you can adjust the rate-limiting settings in /etc/rsyslog.conf (e.g., by increasing $imjournalRatelimit.Burst and $imjournalRatelimit.Interval).

This level of verification ensures not only that the package is installed but also that the service is functioning correctly and is configured appropriately for the cluster’s workload.

Conclusion

You have successfully customized an RHCOS image by layering the rsyslog package, pushed it to a container registry, and applied it to your OpenShift cluster using a MachineConfig. This process demonstrates the power of CoreOS layering for extending the functionality of OpenShift nodes while maintaining the core principles of an immutable infrastructure.