Remove KVM VBDP Disk

A note about removing virtual disks from KVM guests when using virtio-blk-data-plane.

When a disk is removed from a KVM guest using the virt-manager GUI tool, and if virtio-blk-data-plane is being used, an error will occur when restarting the KVM guest as shown below.

gstanden@vmem1:~$ virsh start oracle651

error: Failed to start domain oracle651

error: internal error: process exited while connecting to monitor: qemu-system-x86_64: -set device.virtio-disk2.scsi=off: there is no device "virtio-disk2" defined

gstanden@vmem1:~$ virsh edit oracle651

This occurs because virt-manager has no knowledge of virtio-blk-data-plane code in the XML file defining the VM. The code for the virtio-blk-data-plane feature must be removed manually as shown below. Remove the bolded entries as shown below using the error message shown above to indicate which "Disk#" is referenced by the deleted disk, in this case "Disk2".

gstanden@vmem1:~$ cat oracle651.xml

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>

<name>oracle651</name>

<uuid>9a017de6-d362-4269-90fe-09331460aca0</uuid>

<memory unit='KiB'>1048576</memory>

<currentMemory unit='KiB'>1048576</currentMemory>

<vcpu placement='static'>8</vcpu>

<resource>

<partition>/machine</partition>

</resource>

<os>

<type arch='x86_64' machine='pc-i440fx-trusty'>hvm</type>

<boot dev='hd'/>

<boot dev='cdrom'/>

<bootmenu enable='yes'/>

</os>

<features>

<acpi/>

<apic/>

<pae/>

</features>

<clock offset='utc'/>

<on_poweroff>destroy</on_poweroff>

<on_reboot>restart</on_reboot>

<on_crash>restart</on_crash>

<devices>

<emulator>/usr/bin/kvm-spice</emulator>

<disk type='file' device='disk'>

<driver name='qemu' type='raw' cache='none' io='native'/>

<source file='/var/lib/libvirt/images/oracle651-0.img'/>

<target dev='vda' bus='virtio'/>

<address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>

</disk>

<disk type='file' device='disk'>

<driver name='qemu' type='raw' cache='none' io='native'/>

<source file='/var/lib/libvirt/images/oracle651-3.img'/>

<target dev='vdd' bus='virtio'/>

<address type='pci' domain='0x0000' bus='0x00' slot='0x0b' function='0x0'/>

</disk>

<disk type='file' device='disk'>

<driver name='qemu' type='raw' cache='none' io='native'/>

<source file='/var/lib/libvirt/images/oracle651-4.img'/>

<target dev='vde' bus='virtio'/>

<address type='pci' domain='0x0000' bus='0x00' slot='0x0c' function='0x0'/>

</disk>

<disk type='block' device='cdrom'>

<driver name='qemu' type='raw'/>

<target dev='hdc' bus='ide' tray='open'/>

<readonly/>

<address type='drive' controller='0' bus='1' target='0' unit='0'/>

</disk>

<controller type='usb' index='0'>

<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>

</controller>

<controller type='pci' index='0' model='pci-root'/>

<controller type='ide' index='0'>

<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>

</controller>

<interface type='bridge'>

<mac address='52:54:00:66:26:a0'/>

<source bridge='sw1'/>

<virtualport type='openvswitch'>

<parameters interfaceid='0efa5f2a-cbf2-46df-9ff2-66bddb09218d'/>

</virtualport>

<target dev='s4'/>

<model type='virtio'/>

<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>

</interface>

<interface type='bridge'>

<mac address='52:54:00:65:c2:e5'/>

<source bridge='sw2'/>

<virtualport type='openvswitch'>

<parameters interfaceid='bd4a4643-dae2-488c-8d84-24c388941207'/>

</virtualport>

<target dev='t4'/>

<model type='virtio'/>

<address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>

</interface>

<interface type='bridge'>

<mac address='52:54:00:87:44:e1'/>

<source bridge='sw3'/>

<virtualport type='openvswitch'>

<parameters interfaceid='8a67e2de-d3b5-4fa8-af93-3e115e7f70e3'/>

</virtualport>

<target dev='w4'/>

<model type='virtio'/>

<address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>

</interface>

<serial type='pty'>

<target port='0'/>

</serial>

<console type='pty'>

<target type='serial' port='0'/>

</console>

<input type='mouse' bus='ps2'/>

<input type='keyboard' bus='ps2'/>

<graphics type='vnc' port='-1' autoport='yes' listen='127.0.0.1'>

<listen type='address' address='127.0.0.1'/>

</graphics>

<sound model='ich6'>

<address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>

</sound>

<video>

<model type='cirrus' vram='9216' heads='1'/>

<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>

</video>

<memballoon model='virtio'>

<address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>

</memballoon>

</devices>

<seclabel type='dynamic' model='apparmor' relabel='yes'/>

<qemu:commandline>

<qemu:arg value='-set'/>

<qemu:arg value='device.virtio-disk0.scsi=off'/>

<qemu:arg value='-set'/>

<qemu:arg value='device.virtio-disk0.config-wce=off'/>

<qemu:arg value='-set'/>

<qemu:arg value='device.virtio-disk0.x-data-plane=on'/>

<qemu:arg value='-set'/>

<qemu:arg value='device.virtio-disk2.scsi=off'/>

<qemu:arg value='-set'/>

<qemu:arg value='device.virtio-disk2.config-wce=off'/>

<qemu:arg value='-set'/>

<qemu:arg value='device.virtio-disk2.x-data-plane=on'/>

<qemu:arg value='-set'/>

<qemu:arg value='device.virtio-disk2.logical_block_size=4096'/>

<qemu:arg value='-set'/>

<qemu:arg value='device.virtio-disk2.physical_block_size=4096'/>

<qemu:arg value='-set'/>

<qemu:arg value='device.virtio-disk3.scsi=off'/>

<qemu:arg value='-set'/>

<qemu:arg value='device.virtio-disk3.config-wce=off'/>

<qemu:arg value='-set'/>

<qemu:arg value='device.virtio-disk3.x-data-plane=on'/>

<qemu:arg value='-set'/>

<qemu:arg value='device.virtio-disk3.physical_block_size=4096'/>

<qemu:arg value='-set'/>

<qemu:arg value='device.virtio-disk4.scsi=off'/>

<qemu:arg value='-set'/>

<qemu:arg value='device.virtio-disk4.config-wce=off'/>

<qemu:arg value='-set'/>

<qemu:arg value='device.virtio-disk4.x-data-plane=on'/>

<qemu:arg value='-set'/>

<qemu:arg value='device.virtio-disk4.physical_block_size=4096'/>

</qemu:commandline>

</domain>

gstanden@vmem1:~$

Once these references to the removed disk have been removed, the KVM guest will start normally with no errors. Startup the KVM guest as shown below, and locate the deleted disk image files and delete them. By starting up the KVM guest, ownership will change for active disks. Also, size of disk can be used to help ensure the correct disks are being removed. Finally, naming of the disk can also be used by comparing the XML before and after the drop of the disk to ensure the correct disks are being deleted. For safety, it is best to "mv" the disks to another location first, rather than dropping them, to first verify that the correct disks are being "rm" from the OS permanently.

root@vmem1:/var/lib/libvirt/images# mv oracle651-2.img oracle651-1.img.dropped

root@vmem1:/var/lib/libvirt/images# mv oracle651-2.img oracle651-2.img.dropped

root@vmem1:/var/lib/libvirt/images# exit

gstanden@vmem1:~$ virsh start oracle651

Domain oracle651 started

gstanden@vmem1:~$ sudo su -

root@vmem1:~# cd /var/lib/libvirt/images

root@vmem1:/var/lib/libvirt/images# ls -lrt oracle651*

-rw------- 1 root root 10737418240 Aug 11 18:45 oracle651-2.img.dropped

-rw-r--r-- 1 root root 10737418240 Aug 11 18:56 oracle651-1.img.dropped

-rw------- 1 libvirt-qemu kvm 21474836480 Aug 23 12:46 oracle651-3.img

-rw------- 1 libvirt-qemu kvm 21474836480 Aug 23 12:46 oracle651-4.img

-rw-r--r-- 1 libvirt-qemu kvm 21474836480 Aug 23 12:46 oracle651-0.img

root@vmem1:/var/lib/libvirt/images# rm oracle651-2.img.dropped oracle651-1.img.dropped

Done.