Live Migration failed

Scenario

Using live migration, the user is attempting to migrate a virtual machine from one node to another.

04 step5 1
The primary goal is to investigate the cause of the live migration failure and take corrective action to successfully migrate the exercise6 virtual machine to another node. Your task is to identify the root cause and resolve the issue.

The steps to fix exercise6 are:

Console

  1. Login to OpenShift console using the assigned user account.

{openshift_console_url}

Username
userx
Password
{password}

  1. Go to Virtualization - VirtualMachines - Select the project for the assigned user account - look for VM called exercise6.

01 step2

  1. Open the Virtual Machine exercise6 to get it’s details.

02 step3

  1. Try to migrate the virtual machine using Live Migration. Migration is going to fail.

03 step4

  1. Examine the logs of the virt-launcher-exercise6 pod to identify the root cause.

    • Go to the project for the assigned user account and then look for the failed pod.

04 step5 1

  • Check the pod logs. The issue occurs because the virtual machine disks are configured as read-only, preventing migration.

{"component":"virt-launcher","level":"error","msg":"Operation not supported: Cannot migrate empty or read-only disk vdb","pos":"qemuMigrationDstStartNBDServer:628","subcomponent":"libvirt","thread":"33","timestamp":"2024-11-20T11:44:26.459000Z"}

  1. Stop the virtual machine to edit it’s configuration.

  1. Edit the virtual machine and remove the fields readonly: true from the VM disks. Go to the YAML section and change the field spec.template.spec.domain.devices.disks.readonly to false.

        devices:
          disks:
            - disk:
                bus: virtio
              name: rootdisk
            - disk:
                bus: virtio
                readonly: false
              name: cloudinitdisk
          interfaces:

  1. Start again the virtual machine. Check the node where the virtual machine is placed.

05 step8

  1. Migrate virtual machine.

  1. The virtual machine should be migrated to another node successfully.

06 step10
Run grade to validate the exercise
lab grade exercise6

Command line (CLI)

  1. Login to Openshift server API using the assigned user account with oc command if not logged in.

OpenShift login command
{login_command}

  1. Go to the assigned namespace-userx

oc project namespace-userx

  1. Check there is a virtual machine running in this namespace. The output should be similar to the following one:

oc get vmi
NAME        AGE   PHASE     IP            NODENAME                 READY
exercise6   11m   Running   10.132.2.46   worker-cluster-1   True

  1. Migrate the virtual machine using the virtctl command.

virtctl migrate exercise6

  1. Check the migration status by retreiving the VirtualMachineInstanceMigration object. The output should indicate that the migration is in the Failed phase.

oc get VirtualMachineInstanceMigration
NAME                        PHASE    VMI
kubevirt-migrate-vm   Failed   exercise6

  1. Stop the virtual machine.

virtctl stop exercise6

  1. Edit the virtual machine and remove the fields readonly: true from the VM disks. Change the field spec.template.spec.domain.devices.disks.readonly to false.

oc edit vm exercise6
        devices:
          disks:
            - disk:
                bus: virtio
              name: rootdisk
            - disk:
                bus: virtio
                readonly: false
              name: cloudinitdisk
          interfaces:

  1. Start again the virtual machine.

virtctl start exercise6

  1. Check the node where the virtual machine is placed.

oc get vmi
NAME        AGE   PHASE     IP            NODENAME                 READY
exercise6   98s   Running   10.132.2.48   worker-cluster-vmnjk-1   True

  1. Migrate the virtual machine.

virtctl migrate exercise6

  1. Check the migration status by retreiving the VirtualMachineInstanceMigration object. Now, the output should indicate that the migration is in the Succeeded phase.

oc get VirtualMachineInstanceMigration
NAME                        PHASE         VMI
kubevirt-migrate-vm     Succeeded   exercise6

  1. Check the node where the vortual machine is now placed

oc get vmi
NAME        AGE     PHASE     IP             NODENAME                 READY
exercise6   3m50s   Running   10.135.1.196   worker-cluster-vmnjk-3   True

What you learned

In this exercise, you learned that virtual machines readonly disks attached cannot be live migrated between nodes in OpenShift Virtualization. This limitation occurs because live migration involves creating a new instance of the virtual machine on the target node, which fails due to the RWO disk’s single-node access restriction.