Repository Validation Walkthrough

Environment

Satellite v6.15, Capsule v6.15, RHEL 9 registered client
Satellite can access cdn.redhat.com
Satellite has repositories synced:
- BaseOS
- AppStream
- Red Hat Satellite Client 6 for RHEL 9 x86_64 RPMs
Satellite has additional Lifecycle Environments (LCE):
- Development
- Test
- Production
Capsule has the above LCEs associated with it
A Content view with the above repositories created published and promoted to LCEs above
Clients have the repos above enabled on it

Objectives:

To provide a method for a user to validate the information they are seeing on Satellite and Clients as accurate using tools outside of Satellite. The goal of this lesson is to provide a user with a deeper understanding of how to query the package counts from the Red Hat CDN, Satellite, Capsule, and Client.

How do we validate the package count of the above repositories on CDN?
How do we validate the package count of the Satellite repositories with the CDN count?
How do we validate the package count of the repositories on a Capsule with the count from the Satellite?
How do we validate the package count of the repositories on a client with the package count from the Capsule and Satellite?

Special Considerations: We’ll only focus on the Red Hat Enterprise Linux 9 for x86_64 - AppStream RPMs 9 repository package count for simplification. However, these steps could be used for any RPM-based repository.

Obtaining Client Package Count

Let’s start first by clarifying environmental parameters that could affect the package count on a RHEL system.

Where is the system obtaining content from (Red Hat Satellite, Red Hat CDN, Public/Private Repo, Local filesystem, etc)?
```
For our use case, this will be a Red Hat Satellite Capsule.
```
Does the system have any local restriction that would skew the package count for the system? (DNF excludes, DNF plugins, Subscription Manager release set, etc)?
```
For our use case, the client will have no local restrictions
```

If the system is registered to a Red Hat Satellite, is the system assigned to a content view with any filtering of content?

For our use case, the client will be assigned to the "RHEL9-CV" content view in the "Development" lifecycle environment(LCE) available on the Capsule with no content view filters.

To obtain the package count of the enabled repositories on a RHEL system, first log into the client Node1 from the bastion, become the root user, and then execute the dnf repolist -v as seen below:

ssh node1

sudo -i

dnf repolist -v

The output seen should resemble a package count similar to the output below:

# dnf repolist -v
Loaded plugins: builddep, changelog, config-manager, copr, debug, debuginfo-install, download, generate_completion_cache, groups-manager, needs-restarting, playground, product-id, repoclosure, repodiff, repograph, repomanage, reposync, subscription-manager, system-upgrade, uploadprofile
Updating Subscription Management repositories.
DNF version: 4.14.0
cachedir: /var/cache/dnf
Red Hat Satellite Client 6 for RHEL 9 x86_64 (RPMs)                                                                                        115 kB/s | 3.8 kB     00:00
Red Hat Enterprise Linux 9 for x86_64 - BaseOS (RPMs)                                                                                      120 kB/s | 4.1 kB     00:00
Red Hat Enterprise Linux 9 for x86_64 - AppStream (RPMs)                                                                                   137 kB/s | 4.5 kB     00:00
Repo-id            : rhel-9-for-x86_64-appstream-rpms
Repo-name          : Red Hat Enterprise Linux 9 for x86_64 - AppStream (RPMs)
Repo-revision      : 1736435364
Repo-updated       : Thu 09 Jan 2025 03:09:23 PM UTC
Repo-pkgs          : 21,828
Repo-available-pkgs: 20,920
Repo-size          : 78 G
Repo-baseurl       : https://capsule.example.com/pulp/content/Default_Organization/Development/RHEL9-CV/content/dist/rhel9/9/x86_64/appstream/os
Repo-expire        : 1 second(s) (last: Thu 09 Jan 2025 08:06:33 PM UTC)
Repo-filename      : /etc/yum.repos.d/redhat.repo

Repo-id            : rhel-9-for-x86_64-baseos-rpms
Repo-name          : Red Hat Enterprise Linux 9 for x86_64 - BaseOS (RPMs)
Repo-revision      : 1736296840
Repo-updated       : Wed 08 Jan 2025 12:40:39 AM UTC
Repo-pkgs          : 8,265
Repo-available-pkgs: 8,265
Repo-size          : 22 G
Repo-baseurl       : https://capsule.example.com/pulp/content/Default_Organization/Development/RHEL9-CV/content/dist/rhel9/9/x86_64/baseos/os
Repo-expire        : 1 second(s) (last: Thu 09 Jan 2025 08:06:33 PM UTC)
Repo-filename      : /etc/yum.repos.d/redhat.repo

Repo-id            : satellite-client-6-for-rhel-9-x86_64-rpms
Repo-name          : Red Hat Satellite Client 6 for RHEL 9 x86_64 (RPMs)
Repo-revision      : 1714430944
Repo-updated       : Mon 29 Apr 2024 10:49:04 PM UTC
Repo-pkgs          : 32
Repo-available-pkgs: 32
Repo-size          : 105 M
Repo-baseurl       : https://capsule.example.com/pulp/content/Default_Organization/Development/RHEL9-CV/content/dist/layered/rhel9/x86_64/sat-client/6/os
Repo-expire        : 1 second(s) (last: Thu 09 Jan 2025 08:06:33 PM UTC)
Repo-filename      : /etc/yum.repos.d/redhat.repo
Total packages: 30,125

The difference between Repo-pkgs and Repo-available-pkgs is due to AppStream module stream enablement. At the date of this lab creation, the repo has 21,828 packages, but the client can only access 20,920 packages.

With the client using a content view, we know that the client is restricted to the content available based on the following 2 factors:

When was the content view published?
When was the last successful sync on the Satellite for the "Red Hat Enterprise Linux 9 for x86_64 - AppStream RPMs 9" repository before the content view publish?

Accessing the Red Hat CDN

Next, let’s verify what the Red Hat CDN provides for this repository. Lucky for us, the repository’s BaseURL path naming convention mimics the CDN’s naming convention. We can see on the client that the "Repo-baseurl" for this repository is: https://capsule.example.com/pulp/content/Default_Organization/Library/content/dist/rhel9/9/x86_64/appstream/os

By replacing the first portion of the Repo-baseurl: https://capsule.example.com/pulp/content/Default_Organization/Library

with the URL for the CDN: https://cdn.redhat.com/

you will successfully have the RHEL 9 AppStream CDN repository URL for use: https://cdn.redhat.com/content/dist/rhel9/9/x86_64/appstream/os

To access this repository you will need an entitlement-certificate/subscription that provides access to the product Red Hat Enterprise Linux for x86_64 (productID: 479). This can be extracted from the manifest that is uploaded to the Satellite as part of the post-installation steps (https://access.redhat.com/solutions/7075209), or you can use the entitlement certificate that is provided to a Satellite that is registered to the Red Hat Customer Portal.

Since our Satellite is registered to the Customer Portal, we will use the local entitlement certificate assigned to the Satellite server from subscription manager. Use the below command to log into the Satellite server, from the Bastion server, and find the entitlement certificate and key provided to it:

ssh satellite

sudo -i

ls /etc/pki/entitlement/

Your output should show an 18-digit filename followed by .pem and -key.pem like the example below:

# ls /etc/pki/entitlement/
450425603410326691-key.pem  450425603410326691.pem

This is the entitlement certificate and key that will be used to communicate with the Red Hat CDN. The CA certificate used for communication with the CDN is located at /etc/rhsm/ca/redhat-uep.pem.

By using the information we have found we are now able to access the RHEL 9 AppStream repository on the CDN. Use the command syntax below to build your curl command to query the CDN:

curl --cacert <CA CERT> --cert <ENTITLEMENT CERT> --key <ENTITLEMENT KEY> <CDN URL>

Based on the information provided from my example output my command would look like this:

curl --cacert /etc/rhsm/ca/redhat-uep.pem \
--cert /etc/pki/entitlement/450425603410326691.pem \
--key /etc/pki/entitlement/450425603410326691-key.pem \
https://cdn.redhat.com/content/dist/rhel9/9/x86_64/appstream/os/

Using this command should provide you with HTML output like the following:

<!DOCTYPE html>
<html lang="en">

    <head>
        <meta charset="utf-8">
        <title>repository index</title>
    </head>
    <body>
        <h1>repository index</h1>
        <div class="header">

        </div>
        <pre>

   <a href="Packages/">Packages/</a>
   <a href="repodata/">repodata/</a>
        </pre>
        <div class="footer">

        </div>
    </body>

To obtain the package count for a repository you would need to inspect the primary.xml file in the repodata. To ensure you get the correct primary.xml file as referenced by the repository, you can pull the primary.xml file’s name from the repodata/repomd.xml file first then make a 2nd request for the primary.xml file. You can use the following script to accomplish this:

CACERT='/etc/rhsm/ca/redhat-uep.pem'
ENTCERT=$(ls -1 /etc/pki/entitlement/* | grep -v key)
ENTKEY=$(ls -1 /etc/pki/entitlement/* | grep key)
REPOURL='https://cdn.redhat.com/content/dist/rhel9/9/x86_64/appstream/os/'
PRIMARYXML=$(curl -s --cacert $CACERT --cert $ENTCERT --key $ENTKEY $REPOURL"repodata/repomd.xml" | grep primary.xml | cut -d'"' -f2)

curl -s --cacert $CACERT --cert $ENTCERT --key $ENTKEY $REPOURL$PRIMARYXML | zgrep "metadata packages" | cut -d'"' -f2

Package Comparison

If the Repo-pkgs package count on the client matches the package count from the return of the curl commands in the script you ran, then you know that your Satellite and Capsule server has the latest available packages and they are being served to the client from the RHEL9-CV content view in the Development lifecycle environment on the Capsule server.

[root@satellite ~]# CACERT='/etc/rhsm/ca/redhat-uep.pem'
[root@satellite ~]# ENTCERT=$(ls -1 /etc/pki/entitlement/* | grep -v key)
[root@satellite ~]# ENTKEY=$(ls -1 /etc/pki/entitlement/* | grep key)
[root@satellite ~]# REPOURL='https://cdn.redhat.com/content/dist/rhel9/9/x86_64/appstream/os/'
[root@satellite ~]# PRIMARYXML=$(curl -s --cacert $CACERT --cert $ENTCERT --key $ENTKEY $REPOURL"repodata/repomd.xml" | grep primary.xml | cut -d'"' -f2)
[root@satellite ~]#
[root@satellite ~]# curl -s --cacert $CACERT --cert $ENTCERT --key $ENTKEY $REPOURL$PRIMARYXML | zgrep "metadata packages" | cut -d'"' -f2
21815

[root@node1 ~]# dnf repolist -v rhel-9-for-x86_64-appstream-rpms | grep "Repo-pkgs"
Red Hat Enterprise Linux 9 for x86_64 - AppStre  82 kB/s | 4.5 kB     00:00
Repo-pkgs          : 21,815

However, this is rarely the case. With a newer product such as RHEL 9, updates are frequently released, typically showing the RHEL 9 client missing 1 or more available updates. This is where it is important to understand your system’s update policy/schedule.

Validating Satellite Package Count

Knowing that the RHEL 9 AppStream repository should have the same number of packages as the CDN, the first action should be to check the package count on the Satellite for the "Red Hat Enterprise Linux 9 for x86_64 - AppStream RPMs 9" and initiate a sync for the repository if it varies. This should update the repository locally with the same package information as the Red Hat CDN.

Use the below command to initiate the repository sync on the Satellite server:

hammer repository synchronize --name "Red Hat Enterprise Linux 9 for x86_64 - AppStream RPMs 9" --product "Red Hat Enterprise Linux for x86_64" --organization "Default Organization"

Once the repository has synced successfully, you can query for the repository count from the Satellite using the hammer command below:

hammer repository info --name "Red Hat Enterprise Linux 9 for x86_64 - AppStream RPMs 9" --product "Red Hat Enterprise Linux for x86_64" --organization "Default Organization" --fields "Content counts/packages"

Once you have confirmed the package count for the "Red Hat Enterprise Linux 9 for x86_64 - AppStream RPMs 9" repository matches that of the package count from the curl command performed on the CDN, it’s time to update the content view associated with the client.

Before publishing the content view, it is good practice to check the content view for any filtering that may have been applied to the content view previously and adjust the filters as needed to ensure packages are included/excluded as expected. For this example, there are no content view filters implemented so the package count on the client using the content view should be identical to that of the Satellite. Use the command below to check the content view filters for the RHEL9-CV content view:

hammer content-view filter list --content-view "RHEL9-CV" --organization "Default Organization"

The output should show the headers of the columns used to identify the content view filters, but no additional rows should be listed like the example below:

# hammer content-view filter list --content-view "RHEL9-CV" --organization "Default Organization"
----------|------|-------------|------|----------
FILTER ID | NAME | DESCRIPTION | TYPE | INCLUSION
----------|------|-------------|------|----------

Next, publish the content view and promote it to the lifecycle assigned to the client. To know which lifecycle environment the client is assigned to, run the below command on the Satellite to view the client’s lifecycle:

hammer host list --search name~node --fields "Name,Content view,Lifecycle environment"

The output should look like the following:

---------------------|--------------|----------------------
NAME                 | CONTENT VIEW | LIFECYCLE ENVIRONMENT
---------------------|--------------|----------------------
node1.jtxlz.internal | RHEL9-CV     | Development
node2.jtxlz.internal | RHEL9-CV     | Development
node3.jtxlz.internal | RHEL9-CV     | Development
---------------------|--------------|----------------------

Now that we know the client is assigned to the Development lifecycle environment, we know we can publish a new version of the RHEL9-CV content view and promote it to the Development lifecycle. Run the following command to perform this action:

hammer content-view publish --name "RHEL9-CV" --organization "Default Organization" --lifecycle-environments "Development"

The Satellite’s setting foreman_proxy_content_auto_sync is True (True by default) so the Satellite will initiate a Capsule sync to all Capsule servers that are assigned the Development lifecycle environment. This helps eliminate additional steps the user would take to sync the content to the Capsule server.

Validating the Capsule Content

After the Capsule sync has completed you could view the Satellite WebUI or use the hammer command to query the package count for the repository on the Capsule. However, this doesn’t actually query the Capsule for its package count. This provides a package count based on what the Satellite believes it to have.

Additionally, you could use a client to query the repository to see the package count, but then you are assuming the client is accessing the newly updated repository that was just synced (which it should be). So how can we query the Capsule server for the package count of the newly synced repo for its package count?

The easiest method (without having to install any additional packages) would be to use the Pulp service’s API on the Capsule. To query this information from the API you will need to know the HREF for the repository that the Satellite synced to on the Capsule server. This information can be found in the Capsule sync task that was initiated by the content view publish (if the content was synced and not skipped). Or you can locate the Backend Identifier value of the repository from the Satellite WebUI > Content > Products > Red Hat Enterprise Linux for x86_64 > Repositories > Red Hat Enterprise Linux 9 for x86_64 - AppStream RPMs 9.

Using the Backend Identifier value, use the simple curl command below to view the Capsule’s Pulp API response for the repository’s package count:

BACKEND_ID=<YOUR ID>; for i in $(curl -s --cert /etc/foreman/client_cert.pem --key /etc/foreman/client_key.pem https://capsule.$(hostname -d)/pulp/api/v3/repositories/ | python3 -m json.tool | grep -C3 1-RHEL9-CV-Development-$BACKEND_ID | grep latest_version_href | cut -d'"' -f4); do curl -s --cert /etc/foreman/client_cert.pem --key /etc/foreman/client_key.pem https://capsule.$(hostname -d)$i | python3 -m json.tool | grep -A1 -e 'rpm.package"' -e 'added"' -e 'present"'|grep -v -E 'advisory|metadata'; done

An example of the output, is seen below:

        "added": {
--
            "rpm.package": {
                "count": 21815,
--
        "present": {
--
            "rpm.package": {
                "count": 21815,

Finally, we can double-check the client is capable of seeing the same package count as seen from the API call to the Capsule using the same command as we did before. Run the following command on the RHEL 9 client:

dnf repolist -v

Conclusion

At this point we have come full circle.

We first started with a single repository seen by the client of a Capsule on a Satellite that downloads the RPMs from the Red Hat CDN. We then verified the packages on the same repository from the CDN with what the client was seeing. To ensure we see the same packages on the client’s repository as we do on the CDN, we went through the steps of checking and updating the repositories on both the Satellite and Capsule and finally checked again the packages on the client. With all numbers matching we can safely assume that this client has all available and latest packages from the AppStream repository available to it as the Red Hat CDN provides.