Lab: Create RHEL for Edge Images with MicroShift

Estimated reading time: 15 minutes.

Objective: Build RHEL for Edge Images with MicroShift, using Image Builder, and publish them on remote OSTree repositories.

Before you Begin

You need a few machines to perform the hands-on activities in this course.

A development machine with RHEL and unrestricted sudo, where you will install Image Builder and RPM-OSTree tools, create KVM VMs using Libvirt, and also run the OpenShift client to access your MicroShift instances in other machines.
A package server machine already configured to serve DNF repositories for RHEL, Fast Datapath for RHEL, and Red Hat OpenShift Container Platform.
A web server machine with RHEL and unrestricted sudo, to host OSTree repositories
A mirror registry machine with RHEL and unrestricted sudo, already populated with the container images required by MicroShift and sample applications.

Make sure that your package server machine and mirror registry machine are properly configured and verified by following the instructions from the previous lab of this course.

These instructions were tested on RHEL 9.5 but should work with minimal or no change on newer and older RHEL 9.x releases.

If you are using the course environment, you will log in to the workstation VM as the user student with password student. The workstation VM is your development machine. You will start SSH sessions from the workstation VM to the servera VM, which is your web server machine and also your mirror registry machine, using the same user.

You will also create a local KVM virtual machine on your development machine, and we refer to that VM as your edge machine.

Be sure you execute each step on the correct machine. If a step is not explicit about the machine it should be performed, it is using the same machine as its previous step.

Instructions

On your development machine, verify that you have all prerequisites from previous labs.

Check that the Image Builder service is running and your unprivileged user has access to it.

$ systemctl is-active osbuild-composer.socket
active
$ composer-cli compose types
ami
edge-ami
edge-commit
edge-container
edge-installer
...

Check that the Image Builder service is configured with repository overrides for RHEL packages.

$ composer-cli sources list
appstream
baseos
$ composer-cli sources info baseos
check_gpg = true
...
url = "http://content.example.com/rhel9.5/x86_64/dvd/BaseOS"

Check that you can access your web server machine.

$ curl -s http://servera.lab.example.com | head -n 5
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN" "http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">

<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en">
        <head>
                <title>Test Page for the HTTP Server on Red Hat Enterprise Linux</title>

Check that your web server machine is already serving an OSTree repository.

$ curl http://servera.lab.example.com/repo/config
[core]
repo_version=1
mode=archive-z2

If the output of the previous command shows an HTTP 404 error, as follows, it means your web server machine is NOT serving any OSTree repository and this lab will show how to configure it.

$ curl http://servera.lab.example.com/repo/config
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>404 Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL was not found on this server.</p>
</body></html>

Check that your unprivileged user has access to Libvirt session VMs.

$ virsh uri
qemu:///session
$ virsh nodeinfo
CPU model:           x86_64
...

Check that you have a valid pull secret for your your mirror registry machine and that you can access it with TLS validation enabled.

$ podman search --authfile mirror-pull-secret servera.lab.example.com:8443/
NAME                                                                 DESCRIPTION
servera.lab.example.com:8443/lvms4/lvms-rhel9-operator
servera.lab.example.com:8443/openshift-release-dev/ocp-v4.0-art-dev
servera.lab.example.com:8443/rhel9/mysql-80
servera.lab.example.com:8443/flozanorht/php-ubi
servera.lab.example.com:8443/ubi9/ubi

Check that output from the previous command shows that the mirror registry is already populated with MicroShift release images (openshift-release-dev/ocp-v4.0-art-dev) and LVM Storage operator images (lvms4/lvms-rhel9-operator).

Configure the Image Builder service to use the mirror registry.
1. Create an Image Builder worker configuration file that refers to credentials for accessing the mirror registry.
  
  You already configured your development machine with CA certificates and credentials to access the mirror registry in a previous lab.
  $ sudo mkdir /etc/osbuild-worker $ cat > osbuild-worker.toml << EOF [containers] auth_file_path = "/etc/osbuild-worker/containers-auth.json" EOF $ sudo cp osbuild-worker.toml /etc/osbuild-worker/osbuild-worker.toml $ sudo cp mirror-pull-secret /etc/osbuild-worker/containers-auth.json
2. Configure the container runtime to get MicroShift release images from the mirror registry.
  
  The registry configuration file is the same one you already used to configure MicroShift in a previous lab.
  $ wget -q https://raw.githubusercontent.com/RedHatQuickCourses/rhde-build-samples/refs/heads/main/microshift/999-microshift-mirror.conf $ sudo cp 999-microshift-mirror.conf /etc/containers/registries.conf.d
3. Disable signature verification on container images required by MicroShift by downloading a https://github.com/RedHatQuickCourses/rhde-build-samples/blob/main/microshift/containers-policy.json.nosigs[container image policy file/ from the course samples git repository.
  $ wget -q https://raw.githubusercontent.com/RedHatQuickCourses/rhde-build-samples/refs/heads/main/microshift/containers-policy.json.nosigs $ sudo cp containers-policy.json.nosigs /etc/containers/policy.json
4. Reboot your development machine so its Image Builder service reads its new configuration files.

Configure the Image Builder service with package sources for the RPM repositories required by MicroShift

Download the package sources TOML files from the course samples git repository.

$ wget -q https://raw.githubusercontent.com/RedHatQuickCourses/rhde-build-samples/refs/heads/main/sources/openshift.toml
$ wget -q https://raw.githubusercontent.com/RedHatQuickCourses/rhde-build-samples/refs/heads/main/sources/fastdata.toml

Configure your Image Builder service with the additional package sources.

These package sources refer to the same RPM repositories that you already used in the previous chapter to deploy MicroShift on package-based RHEL.

$ composer-cli sources add openshift.toml
$ composer-cli sources add fastdata.toml
$ composer-cli sources list
appstream
baseos
fastdata
openshift
$ composer-cli sources info openshift
...
url = "http://content.example.com/rhde/rpms/rhocp-4.17-for-rhel-9-x86_64-rpms"
...

Download and inspect an Image Builder blueprint for a preconfigured MicroShift instance.

Download the sample blueprint from the course samples git repository. It is a long blueprint but, assuming that you performed all activities from the first Red Hat Device Edge course and also from the previous chapter of this course, there should be no surprises.
```
$ wget -q https://raw.githubusercontent.com/RedHatQuickCourses/rhde-build-samples/refs/heads/main/blueprints/rhel9-microshift.toml
```

Review the rhel9-microshift.toml blueprint and make sure you understand its customizations:

The code snippets here help you locate the relevant sections in the blueprint, but they do not list the entirety of most sections. Follow along with a text editor and navigate through the blueprint file.

Set the hostname to ushift where the letter "u" stands for the Greek letter "micro".
```
[customizations]
hostname = "ushift"
```
Install the MicroShift packages and dependencies.
```
[[packages]]
name = "microshift"
```

Enable two first boot services, which are defined later in the blueprint.

[customizations.services]
enabled = ["sshd", "microshift", "firstboot-microshift", "firstboot-microshift-user" ]

Expose the Kubernetes API endpoint.

[customizations.firewall]
ports = ["6443:tcp"]

Add a pull secret for the mirror registry.

[[customizations.files]]
path = "/etc/crio/openshift-pull-secret"
...

Add a CA certificate for the mirror registry.

[[customizations.files]]
path = "/etc/pki/ca-trust/source/anchors/quay-rootCA.pem"
...

Add a registry configuration which redirects MicroShift container images to the mirror registry.

[[customizations.files]]
path = "/etc/containers/registries.conf.d/999-microshift-mirror.conf"
...

Add an image policy which disables signature validation.

[[customizations.files]]
path = "/etc/containers/policy.json"
...

Embed the MicroShift release container images in the system image.

[[containers]]
source = "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:98bcfe497a8550cff543321ba5c6535b9823aa4b091daf89ebba0bbcafa19208"

[[containers]]
source = "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:1b0a5de3da8895f72deed91a79f4ecf83464370ce253494d28b8c98a1586bfc0"
...

Embed a test application container in the system image.

[[containers]]
source = "servera.lab.example.com:8443/ubi9/ubi"

Embed a first boot service which configures the system’s firewall for the Kubernetes pod and service virtual networks.
```
[[customizations.files]]
path = "/etc/systemd/system/firstboot-microshift.service"
...
```
Embed a script which copies the autogenerated kubeconfig files to the core user home directory.
```
[[customizations.files]]
path = "/etc/microshift/firstboot-microshift-user"
...
```
The script starts before MicroShift is fully initialized so it was coded to wait until MicroShift autogenerates the kubeconfig files.

Invoke the previous script from another first boot service.

[[customizations.files]]
path = "/etc/systemd/system/firstboot-microshift-user.service"
...

Build an edge system image.

Add the CA certificate for the mirror registry to the blueprint. Because it was generated when you installed the mirror registry in a previous lab, we cannot have it in the course samples repository.

Open the rhel9-microshift.toml file in a text editor and replace the REPLACE_QUAY_CA text with the entire contents of the quay-rootCA.pem file.

Push the blueprint and check that Image Builder can resolve all RPM packages from it.

$ composer-cli blueprints push rhel9-microshift.toml
$ composer-cli blueprints depsolve rhel9-microshift | grep microshift
blueprint: rhel9-microshift v0.1.0
    microshift-4.17.3-202410241116.p0.gec0b5ea.assembly.4.17.3.el9.x86_64
    microshift-greenboot-4.17.3-202410241116.p0.gec0b5ea.assembly.4.17.3.el9.noarch
    microshift-networking-4.17.3-202410241116.p0.gec0b5ea.assembly.4.17.3.el9.x86_64
    microshift-selinux-4.17.3-202410241116.p0.gec0b5ea.assembly.4.17.3.el9.noarch

If you get errors related to downloading container images, it is probably because you forgot to reboot your development machine after configuring it for accessing the mirror registry.

Start a compose for an edge commit image.

Save the UUID of the compose in a shell variable so you can use it in other commands.

$ composer-cli compose start-ostree rhel9-microshift edge-commit --ref rhel/9/x86_64/ushift
Warning: Please note that user customizations on "edge-commit" image type are deprecated and will be removed in the near future

Compose be27c45f-e1ea-42eb-94b3-a0789243bea4 added to the queue
$ UUID=be27c45f-e1ea-42eb-94b3-a0789243bea4

You can ignore the warning about "user customizations" because the blueprint does not include any.

Wait until the compose finishes. Be patient, it will take a few minutes.

$ composer-cli compose list
ID                                     Status     Blueprint          Version   Type
be27c45f-e1ea-42eb-94b3-a0789243bea4   RUNNING    rhel9-microshift   0.1.0     edge-commit
...
$ composer-cli compose list
ID                                     Status     Blueprint          Version   Type
be27c45f-e1ea-42eb-94b3-a0789243bea4   FINISHED   rhel9-microshift   0.1.0     edge-commit
...

Copy the edge commit image to your web server machine.

$ composer-cli compose image $UUID
$ scp $UUID-commit.tar servera:~

On your web server machine, publish your new edge system image.

Copy and paste the UUID from your development machine.
```
$ UUID=be27c45f-e1ea-42eb-94b3-a0789243bea4
```

Extract your edge commit image and check it contains an OSTree repository.

$ mkdir delete-me
$ tar xf $UUID-commit.tar -C delete-me/
$ ostree refs --repo delete-me/repo
rhel/9/x86_64/ushift

If your web server already contains an OSTree repository from activities from previous Red Hat Device Edge courses, pull your edge container image into the same repository.

Do NOT run the pull-local command if you do not have an OSTree repository in the web server!
```
$ ostree --repo=/var/www/html/repo refs
rhel/9/x86_64/edge
$ sudo ostree pull-local --repo=/var/www/html/repo delete-me/repo
265 metadata, 649 content objects imported; 0 bytes content written
```

If your web server does NOT contain an OSTree repository, just copy your edge commit image to the web server.

If you copy OSTree repository files over another OSTree repository, you may lose access to other OSTree commits previously stored in the OSTree repository. Only perform a file copy to an empty OSTree repository directory.

$ ostree --repo=/var/www/html/repo refs
error: opening repo: opendir(/var/www/html/repo): No such file or directory
$ sudo cp -r delete-me/repo /var/www/html

On your development machine, download and inspect an Image Builder blueprint and a kickstart file for an edge installer image.

Download the sample installer blueprint from the course samples git repository. It is a minimal blueprint, without any customizations.

$ wget -q https://raw.githubusercontent.com/RedHatQuickCourses/rhde-build-samples/refs/heads/main/blueprints/rhel9-microshift-installer.toml

Download the sample installer kickstart from the course samples git repository.

$ wget -q https://raw.githubusercontent.com/RedHatQuickCourses/rhde-build-samples/refs/heads/main/ks/rhel9-microshift-installer.ks

Review the rhel9-microshift-installer.ks kickstart and make sure you understand it’s instructions.

The code snippets here help you locate the relevant sections in the kickstart file, but they do not list the entirety of the file. Follow along with a text editor and navigate through the kickstart file.

Partition the root disk with LVM, but taking only 10G for the root partition, in order to leave empty space in the rhel volume group for use by MicroShift’s LVM Storage operator.

zerombr
clearpart --all --initlabel
part /boot/efi --fstype=efi --size=200
part /boot --fstype=xfs --asprimary --size=800
part pv.01 --grow
volgroup rhel pv.01
logvol / --vgname=rhel --fstype=xfs --size=10240 --name=root

Deploy the OSTree commit embedded in the installation media.

ostreesetup --nogpg --osname=rhel --remote=edge --url=file:///run/install/repo/ostree/repo --ref=rhel/9/x86_64/ushift

Create an initial user with a known password, unlimited sudo, and SSH key.

%post --log=/var/log/anaconda/post-install.log --erroronfail

# local user for console login and SSH
useradd -g wheel core
echo "core:redhat123" | chpasswd
mkdir /home/core/.ssh
cat > /home/core/.ssh/authorized_keys << EOFSSH
REPLACE_WTH_SSH_PUB_KEY
EOFSSH

%end

Production systems would NOT enable password login. We left it here as a convenience for the learner, so you can log in at the VM console for troubleshooting.

Create an SSH key for use with your edge devices.
```
$ ssh-keygen -N '' -f edge-key -C 'initial key for edge devices'
Generating public/private rsa key pair.
...
```
If you get an error, because you already have an edge-key generated from previous Red Hat Device Edge courses, just ignore the error and continue with the next step.

Embed your new SSH key in the kickstart file.

$ SSH_PUB_KEY=$( cat edge-key.pub )
$ sed -i "s|REPLACE_WTH_SSH_PUB_KEY|$SSH_PUB_KEY|" rhel9-microshift-installer.ks

Build an edge installer image and download it as an ISO file.

Push the blueprint for your edge installer image.

$ composer-cli blueprints push rhel9-microshift-installer.toml

Start a compose for your edge installer image.

$ composer-cli compose start-ostree microshift-installer edge-installer --ref rhel/9/x86_64/ushift --url http://servera.lab.example.com/repo/
Compose 170edc4e-9384-4e86-b616-7c09eaa718e4 added to the queue
$ UUID=170edc4e-9384-4e86-b616-7c09eaa718e4

Wait until your compose finishes. It will take a few minutes.

$ composer-cli compose list
ID                                     Status     Blueprint              Version   Type
170edc4e-9384-4e86-b616-7c09eaa718e4   RUNNING    microshift-installer   0.1.0     edge-installer
...
$ composer-cli compose list
ID                                     Status     Blueprint              Version   Type
170edc4e-9384-4e86-b616-7c09eaa718e4   FINISHED   microshift-installer   0.1.0     edge-installer

Download your edge installer image.

$ composer-cli compose image $UUID
170edc4e-9384-4e86-b616-7c09eaa718e4-installer.iso

Add your custom kickstart file to your installer ISO file.

Install the Lorax tools for manipulating Anaconda installation media.
```
$ sudo dnf install lorax
...
Complete!
```

Create a new ISO file including your custom kickstart file.

$ mkksiso --ks rhel9-microshift-installer.ks $UUID-installer.iso rhel9-microshift.iso
...
Writing to '/home/student/rhel9-microshift.iso' completed successfully.

On your web server machine, stop the mirror registry for Red Hat OpenShift, so you can prove that your edge installer image can actually work in disconnected edge sites.
```
$ sudo systemctl stop quay-app quay-pod quay-redis
$ sudo systemctl is-active quay-app quay-pod quay-redis
failed
inactive
inactive
```
The failed state is expected for the quay-app service.
On your development machine, create a KVM virtual machine to test your edge installer image.
1. Set a shell variable with the disk label of your installer image, by copy-and-paste from the output of the iso-info command.
  $ iso-info rhel9-microshift.iso ... Volume : RHEL-9-5-0-BaseOS-x86_64 No Joliet extensions $ LABEL=RHEL-9-5-0-BaseOS-x86_64
2. Create a KVM virtual machine from your installer image. The installation should proceed unattended until you get a login prompt.
  
  You could use different virt-install commands or the Cockpit web UI. The use of --location and --extra-arg in the following command enables the VM to run with a serial console, so you don’t need to leave your shell and don’t need to open a graphical console for your edge machine.
  $ virt-install --name edge-microshift-1 --os-variant rhel9.5 \ --memory 4096 --vcpus 2 --disk size=20 --graphics=none --network bridge=virbr0 \ --location /home/student/rhel9-microshift.iso \ --extra-arg inst.ks=hd:LABEL=$LABEL:/rhel9-microshift-installer.ks \ --extra-arg console=ttyS0 -v ... ushift login:
  You may need to press Enter to see the login prompt, after the VM stops displaying console messages.
3. Detach from the VM console, by pressing Ctrl+], and start an SSH session to your edge machine.
  $ ssh -i edge-key core@ushift

On your edge machine, verify that MicroShift is fully initialized.

Check that MicroShift is healthy and all its pods are ready and running. Beware it may take a while for MicroShift to finish starting all its pods.

$ export KUBECONFIG=~/local-admin
$ oc get node
NAME     STATUS   ROLES                         AGE   VERSION
ushift   Ready    control-plane,master,worker   10m   v1.30.5
$ oc get pod -A
NAMESPACE                  NAME                                       READY   STATUS    RESTARTS   AGE
kube-system                csi-snapshot-controller-69ddff88c8-6g4wr   1/1     Running   0          4m20s
kube-system                csi-snapshot-webhook-74dc497864-xgjzz      1/1     Running   0          4m24s
openshift-dns              dns-default-tj2wf                          2/2     Running   0          4m5s
openshift-dns              node-resolver-lfg6k                        1/1     Running   0          4m21s
openshift-ingress          router-default-575b4fc7-cnnsc              1/1     Running   0          4m19s
openshift-ovn-kubernetes   ovnkube-master-xc84v                       4/4     Running   0          4m21s
openshift-ovn-kubernetes   ovnkube-node-mgsz2                         1/1     Running   0          4m21s
openshift-service-ca       service-ca-9db855698-pwbfg                 1/1     Running   0          4m19s
openshift-storage          lvms-operator-7f544467bc-94227             1/1     Running   0          4m22s
openshift-storage          vg-manager-5n494                           1/1     Running   0          3m55s

If you are too quick, you may get a NotReady state from oc get node, or a "no resources found" error from oc get pod. If any of this happens, just wait a few seconds and try again.

As an additional check, create a test pod from the sample application image we included in the blueprint.
```
$ oc run shell -it --restart Never --image-pull-policy IfNotPresent --image servera.lab.example.com:8443/ubi9/ubi -- rpm -q redhat-release
redhat-release-9.5-0.6.el9.x86_64
$ oc delete pod shell
pod "shell" deleted
```
If you create a pod with the default image pull policy of Always it will fail because we stopped the mirror registry.

You succeed in provisioning an edge device, from an edge installer image, in an air-gapped environment: you only need the installation media to fully provision a ready-to-use RHEL for Edge system running a MicroShift instance.

What’s Next

This was the final activity of this course. If you wish, you can create service accounts and kubeconfig files for developer access using the same steps from previous labs already demonstrated on package-based RHEL.