Module 3 - Trouble Shooting Performance and Implementing a Standard Operating Environment
Introduction
Administrators will often need to maintain a Satellite Server for a long period of time, as the Satellite Server becomes an integral part of their organization’s infrastructure.
The primary goal of this scenario is to investigate performance and maintenance configurations for the Satellite Server, correct misconfigurations, and automate maintenance cycles as part of the Standard Operating Environment Method of deploying Red Hat Satellite.
Executing the Lab Script
In this exercise setup, you first need to run the command to break the lab from the Bastion server:
ansible-navigator run --inventory /tmp/hosts --eei quay.io/agnosticd/ee-ubi9-aap2.5:1.0.1 rh1-lab19-satellite/Module_3/break_module3.yml -m stdout
Satellite Performance Tuning
Scenario
Other teams in your organization have notified you that the Satellite Server does not seem to be performing well. The Server has had additional memory and CPU added to the virtual machine as your organization provisions thousands of hosts and performance managing the hosts is still slow. You have been asked to implement tuning based on the additional hardware of the Satellite Server.
Login to the satellite
server from the bastion
:
ssh satellite
Inspect the new number of CPU cores
nproc
Inspect the new amount of memory
free -h
Inspect the current tuning
sudo grep tuning /etc/foreman-installer/scenarios.d/last_scenario.yaml
Based upon the results of the above commands, the Satellite has 8 cores and 62 Gi of RAM. The recommended tuning profile is Medium, which does not match the default tuning profile that is currently applied.
Apply the Medium tuning profile
sudo satellite-installer --tuning medium
Verify that the Profile has been applied
sudo grep tuning /etc/foreman-installer/scenarios.d/last_scenario.yaml
Automate the Cleanup of old content views
The number of Content View versions will continue to grow as Satellite continues to be utilized in an organization. In the past, manual or scripted intervention was required to clean these obsoleted Content Views up. Today, Satellite has automated tooling to perform these actions.
Scenario
The infrastructure team in your organization has been experimenting with different combinations of packages and content filters in the Satellite Server. This has created a large number of content view versions on the Satellite that are obsolete.
View the content view versions on the Satellite and the Lifecycle Environments in which they are published:
sudo hammer content-view version list --content-view "RHEL9-CV" --organization "Default Organization"
Note that the RHEL9-CV Content View has 15 published versions.
Let’s clean up these up:
sudo hammer content-view purge --name "RHEL9-CV" --organization "Default Organization"
View that the content views versions have been purged:
sudo hammer content-view version list --content-view "RHEL9-CV" --organization "Default Organization"
By default, this will keep the content-view versions that are promoted to Lifecycle Environments, and 3 unused versions. After consulting with the infrastructure team, they have agreed that they only need 1 unused version. Modify the previous command, by adding "--count 1" to the previous command to keep only 1 unused version
sudo hammer content-view purge --name "RHEL9-CV" --organization "Default Organization" --count 1
Verify that the content-view versions have been purged: View that the content views versions have been purged:
sudo hammer content-view version list --content-view "RHEL9-CV" --organization "Default Organization"
Cleanup Orphaned Pulp Content
Content accumulates on a Satellite Server and the on-disk size of content appears to constantly grow. When an administrator deletes a Content View or disables a Repository that disk is not immediately reclaimed. This content is called Orphaned Pulp Content.
Satellite implements a Cronjob in /etc/cron.d/katello to remove the orphaned content, but in many cases, such as a full or near full /var/lib/pulp volume it is useful too to execute the command ad-hoc.
Scenario
In the last scenario, you pruned many content views, there is likely orphaned content on the Satellite Server that needs to be removed.
View the Cronjob on the Satellite:
sudo cat /etc/cron.d/katello
The Script for initiating the removal of orphaned content automatically executes at 10PM local time every Sunday.
Check the amount of unutilized disk in /var/lib/pulp
sudo du -s /var/lib/pulp
Execute the command to clean up orphaned content
sudo foreman-rake katello:delete_orphaned_content RAILS_ENV=production
We can then Query the remove orphans task to check its status:
sudo -i
certs="--cacert /etc/pki/katello/certs/katello-server-ca.crt --cert /etc/foreman/client_cert.pem --key /etc/foreman/client_key.pem"
hname=$(hostname -f)
curl -s $certs https://${hname}/pulp/api/v3/tasks/?name=pulpcore.app.tasks.orphan.orphan_cleanup | json_reformat
Successful output will look like this:
{ "pulp_href": "/pulp/api/v3/tasks/0192d0da-4100-74f4-8e3c-72ce95ddf6e4/", "pulp_created": "2024-10-28T02:00:17.409026Z", "state": "completed", "name": "pulpcore.app.tasks.orphan.orphan_cleanup", "logging_cid": "ac115363680b4b04ba86d5a510a81f39", "created_by": "/pulp/api/v3/users/2/", "started_at": "2024-10-28T02:00:21.480234Z", "finished_at": "2024-10-28T02:00:28.576079Z", "error": null, "worker": null, "parent_task": null, "child_tasks": [ ], "task_group": null, "progress_reports": [ { "message": "Clean up orphan Content", "code": "clean-up.content", "state": "completed", "total": 2599, "done": 2599, "suffix": null }, { "message": "Clean up orphan Artifacts", "code": "clean-up.artifacts", "state": "completed", "total": 17, "done": 17, "suffix": null } ], "created_resources": [ ], "reserved_resources_record": [ "/api/v3/orphans/cleanup/", "shared:/pulp/api/v3/domains/0191ddfd-4882-782d-b5c5-910afce923db/" ] }
Then check the size of /var/lib/pulp once the task is completed:
sudo du /var/lib/pulp
Automate Cleanup of Old Tasks
Cleaning the tasks on the Satellite server is an important task that should be enabled. If not done over a long period of time, tasks will accumulate, which will negatively impact the performance of the Satellite server. This also affects the upgrade of the Satellite server when there is a large number of tasks that are present on the satellite server which is not cleaned. This generally creates issues during upgrades, when commands to clean the system are run, but due to a large number of tasks, there isn’t enough space present so the cleaning also fails resulting in failing of Satellite upgrades
Scenario
Every upgrade of the Satellite Server you have noticed that the satellite-maintain command has reported that there are many old tasks to clean up. This has extended the upgrade window of the Satellite Server and has also impacted the performance of searching for the status of Remote Execution jobs on the Satellite.
Task
Check to see if automatic task cleanup is enabled
sudo satellite-installer --help | grep foreman-plugin-tasks-automatic-cleanup
--foreman-plugin-tasks-automatic-cleanup Enable automatic task cleanup using a cron job (current: false)
Note that the automatic cleanup is disabled, enable it.
sudo satellite-installer --foreman-plugin-tasks-automatic-cleanup true
View the schedule for the execution of the task cleanup
sudo satellite-installer --help | grep foreman-plugin-tasks-cron-line
--foreman-plugin-tasks-cron-line Cron line defining when the cleanup cron job should run (current: "45 19 * * *")
This means that the automated cronjob will run at 7:45PM every day.
Scheduling Capsule Syncs
By default, the Capsule Server will automatically synchronize new and updated content views that are in the Capsule’s assigned lifecycle environments. This is usually beneficial, but when making large numbers of changes to content views, it may be beneficial to schedule the Capsule synchronizations at the end of the publication and promotion of the content views.
Open the Tasks UI in the Satellite Web UI by clicking on Monitor > Tasks
Publish and Promote a content view:
sudo hammer content-view publish --name "RHEL9-CV" --organization "Default Organization" --lifecycle-environments Development,Test,Production
In the Satellite Web UI, you may observe that the Content View Publish and Promote tasks are created, as well as a sync task for the capsule server.
To disable this functionality:
sudo hammer settings set --id foreman_proxy_content_auto_sync --value false
Then Publish a Content View:
sudo hammer content-view publish --name "RHEL9-CV" --organization "Default Organization" --lifecycle-environments Development,Test,Production
Observe that in the Satellite Web UI that a Capsule Synchronization task is not created.
Execute a capsule sync task now that the content views have completed publishing and promoting.
sudo hammer capsule content synchronize --id 2
Observe that in the Satellite Web UI that a Capsule Synchronization task has been created and completes quickly.
Once this completes, enable the function for automatic content synchronization for Capsules:
sudo hammer settings set --id foreman_proxy_content_auto_sync --value true
This lab is now complete.