Module 2 - Troubleshooting Client Connectivity

In this module, we will explore scenarios affecting connectivity between Satellite server, Satellite capsule, client and remote execution.

Environment

  • bastion - RHEL 9 jump host

  • satellite - Red Hat Satellite Server 6.15

  • capsule - Red Hat Satellite Capsule 6.15

  • node2 - RHEL 9 client


Scenario 1: Troubleshoot client registration errors

Connectivity issues prevent clients from accessing Red Hat content including security fixes which is a compliance risk.
In addition, the client does not benefit from the intelligence provided by Red Hat Insights.

The objective of this scenario is to investigate and resolve issues which prevents a client from registering with or accessing content from Red Hat Satellite.

Access the bastion node through the terminal on the right, or using SSH and switch to root user.

sudo -i

Run the following command to introduce the registration failure.

lab client_registration

Task: Client registration

  • Login to node2

ssh root@node2

  • Display registration status. The command is expected to return a failure output.

subscription-manager status

  • Stop the stuck command by pressing Ctrl + C.

  • Examine the subscription management log

tail /var/log/rhsm/rhsm.log

  • Inspect the rhsm.conf file

cat /etc/rhsm/rhsm.conf

  • We can see that custom proxy settings are in use.

grep ^proxy /etc/rhsm/rhsm.conf

Output

   proxy_hostname = proxy.example.com
   proxy_scheme = http
   proxy_port = 8080

  • Remove the custom proxy settings

subscription-manager config --server.proxy_hostname "" --server.proxy_port "" --server.proxy_user "" --server.proxy_password ""

  • Edit the rhsm.conf file to replace the false hostname with the correct FQDN of the Satellite server.

sed -i 's/hostname =.*/hostname = {satellite_public_hostname}/' /etc/rhsm/rhsm.conf
sed -i 's/baseurl =.*/baseurl = https://{satellite_public_hostname}/pulp/content' /etc/rhsm/rhsm.conf

  • On satellite, generate a Global Registration command (insecure option used to disable SSL validation)

hammer host-registration generate-command \
--activation-keys "My_Activation_Key" \
--insecure true

  • Paste the registration command on node2. Note: Do not copy this command, use the command generated by Satellite The registration is expected to fail with error.

set -o pipefail && curl -sS --insecure 'https://{satellite_public_hostname}/register?activation_keys=GENERAL&force=1&location_id=2&organization_id=1&setup_insights=0&setup_remote_execution=1&setup_remote_execution_pull=0' -H 'Authorization: Bearer TOKEN' | bash

Output:

  # Running registration
  #
  This system is currently not registered.
  All local data removed
  subscription-manager is already installed!
  Proxy error: unable to connect to proxy.example.com:8080: Name or service not known (error code -2)

  • Run the registration command. This time the registration should be successful.

set -o pipefail && curl -sS --insecure 'https://{satellite_public_hostname}/register?activation_keys=GENERAL&force=1&location_id=2&organization_id=1&setup_insights=0&setup_remote_execution=1&setup_remote_execution_pull=0' -H 'Authorization: Bearer TOKEN' | bash

Output:

# Running registration
#
subscription-manager is already installed!
The system has been registered with ID: 213874ca-6ccf-4e5b-a3f1-147d8f1c521b
The registered system name is: {node2_public_hostname}

  • Display registration status. The command is expected to run successfully.

subscription-manager status

Output:

+-------------------------------------------+
   System Status Details
+-------------------------------------------+
Overall Status: Disabled
Content Access Mode is set to Simple Content Access. This host has access to content, regardless of subscription status.

System Purpose Status: Disabled


Scenario 2: Troubleshoot Remote Execution

Remote execution enables administrators to run tasks simultaneously on multiple hosts using Ansible or shell scripts.
Failure to execute causes significant disruption, especially in large Red Hat Satellite deployments.

The objective of this scenario is to investigate and remediate issues preventing remote execution of jobs.

Access the bastion node through the terminal on the right, or using SSH and switch to root user.

sudo -i

Task: Remote execution

  • On the Satellite server UI, create a Remote execution job to execute on node2.

  • Navigate to Monitor → Jobs → Run job.

    Job Category: Commands
Job template: Run Command - Script Default

  • Click Next

    Target hosts and input: node2
command: date

  • Click Run on selected hosts

  • Examine the job output. The job fails because the Remote Execution SSH public key used by the Satellite server does not exist on node2.

  • On satellite, copy the SSH public key to node2.

scp /var/lib/foreman-proxy/ssh/id_rsa_foreman_proxy.pub node2:~/.ssh/authorized_keys

  • Re-run the Remote Execution job on node2. The job should run successfully.


Scenario 3: Troubleshoot Capsule Connectivity

Capsules servers mirror content from Satellite server, bringing content and Satellite services closer to clients in distinct geographical or logical locations.
Connectivity issues between Satellite and Capsules can result in corrupt or inconsistent data being served to clients.

The objective of this scenario is to investigate and remediate issues affecting connectivity between clients and Red Hat Satellite Capsule.

Access the bastion node through the terminal on the right, or using SSH and switch to root user.

sudo -i

Run the following command to introduce Capsule connectivity failure.

lab break client_capsule

Task: Capsule connectivity

  • On the Satellite server UI, check the status of the Capsule server. Navigate to Infrastructure → Capsules, then click on capsule

  • Notice that the Communication status is marked with a red X.

capsule failure


  • On satellite, check certificate exchange with capsule. Notice the Capsule features are not listed.

curl --cert /etc/foreman/client_cert.pem --key /etc/foreman/client_key.pem --cacert /etc/foreman/proxy_ca.pem https://{capsule_public_hostname}:9090/features | python3 -m json.tool

  • On capsule, examine the foreman-proxy log.

# grep -i '\[E]' /var/log/foreman-proxy/proxy.log

Output:

2024-12-19T21:32:22  [E] <Errno::ENOENT> No such file or directory @ rb_sysopen - /etc/foreman-proxy/foreman_ssl_cert.pem
2024-12-19T21:32:22  [E] <Errno::ENOENT> No such file or directory @ rb_sysopen - /etc/foreman-proxy/foreman_ssl_cert.pem
2024-12-19T21:32:22  [E] <Errno::ENOENT> No such file or directory @ rb_sysopen - /etc/foreman-proxy/foreman_ssl_cert.pem

  • On capsule, inspect the certificates directory. Notice that the foreman_ssl_cert.pem certificate file is missing

# ls -l /etc/foreman-proxy/

Output:

total 20
-rw-r--r--. 1 root root             0 Dec 20  2023 migration_state
drwxr-xr-x. 2 root root          4096 Dec 19 21:07 settings.d
-rw-r-----. 1 root foreman-proxy 3477 Dec 19 21:05 settings.yml
-r--r-----. 1 root foreman-proxy 2496 Dec 19 21:05 ssl_ca.pem
-r--r--r--. 1 root foreman-proxy 2175 Dec 19 21:05 ssl_cert.pem
-r--r-----. 1 root foreman-proxy 3272 Dec 19 21:05 ssl_key.pem

  • On satellite, generate new certificates for the Capsule.

capsule-certs-generate \
--foreman-proxy-fqdn capsule.example.com \
--certs-tar /root/capsule_cert/capsule.example.com-certs.tar

Output:

satellite-installer --scenario capsule \
--certs-tar-file "/root/capsule_cert/capsule.example.com-certs.tar" \
--foreman-proxy-register-in-foreman "true" \
--foreman-proxy-foreman-base-url "https://satellite.example.com" \
--foreman-proxy-trusted-hosts "satellite.example.com" \
--foreman-proxy-trusted-hosts "capsule.example.com" \
--foreman-proxy-oauth-consumer-key "s97QxvUAgFNAQZNGg4F9zLq2biDsxM7f" \
--foreman-proxy-oauth-consumer-secret "6bpzAdMpRAfYaVZtaepYetomgBVQ6ehY"

  • Copy the certificate bundle to capsule.

scp /root/capsule_cert/capsule.example.com-certs.tar \
root@capsule.example.com:/root/capsule.example.com-certs.tar

  • On capsule, run satellite-installer command generated in the previous step.

satellite-installer --scenario capsule \
--certs-tar-file "/root/capsule_cert/capsule.example.com-certs.tar" \
--foreman-proxy-register-in-foreman "true" \
--foreman-proxy-foreman-base-url "https://satellite.example.com" \
--foreman-proxy-trusted-hosts "satellite.example.com" \
--foreman-proxy-trusted-hosts "capsule.example.com" \
--foreman-proxy-oauth-consumer-key "s97QxvUAgFNAQZNGg4F9zLq2biDsxM7f" \
--foreman-proxy-oauth-consumer-secret "6bpzAdMpRAfYaVZtaepYetomgBVQ6ehY"

  • On capsule, restart the foreman-proxy service.

systemctl restart foreman.proxy

  • On satellite, check certificate exchange with the Capsule. This time, the Capsule features are listed.

curl --cert /etc/foreman/client_cert.pem --key /etc/foreman/client_key.pem --cacert /etc/foreman/proxy_ca.pem https://{capsule_public_hostname}:9090/features | python3 -m json.tool

Output:

[
"container_gateway",
"dynflow",
"logs",
"pulpcore",
"registration",
"script",
"templates"
]

  • On the Satellite server UI, check the status of the Capsule server. Navigate to Infrastructure → Capsules, then click on capsule

  • Notice that the Communication status is marked with a green check.

capsule fixed

This lab is complete.