Understanding KubeVirt's Host Kernel and Userspace Dependencies | KubeVirt.io

KubeVirt runs each VM inside a virt-launcher pod. QEMU and libvirt are userspace processes scheduled like any other container, with host kernel devices (/dev/kvm, /dev/vhost-net, /dev/net/tun) mapped in. Like any workload that probes kernel device capabilities, a mismatch between the userspace stack and the host kernel can cause failures. What makes the virtualization case distinct is that those probe results feed directly into guest device feature negotiation and live-migration Application Binary Interface (ABI), so a kernel/userspace mismatch can surface not just as startup instability, but as broken live migrations across virt-launcher versions.
Upstream KubeVirt builds default virt-launcher images with Enterprise Linux (EL) userspace and runs CI against EL host kernels. When this EL userspace runs on non-EL host kernels, subtle compatibility assumptions can break in ways upstream CI never sees. A recent issue, kubevirt/kubevirt#16386, brought this reality into sharp focus.
During upgrades from KubeVirt v1.6 to v1.7+ on Ubuntu worker nodes, live migrations failed across thousands of running VMs:
The bit difference 0x1c0000000000000 corresponds to three virtio-net feature bits for UDP Segmentation Offload (USO):
Upstream KubeVirt release images are built on CentOS Stream 9, and upstream CI runs on EL9 host kernels (5.14).
Because CI nodes run kernel 5.14, QEMU’s TAP probe always failed. USO was never enabled in CI, so migration tests passed cleanly when the downstream QEMU patch dropped the feature. Only clusters running modern non-EL host kernels encountered the breakage.
virt-launcher is a translation layer between the host node and the guest OS:
Compatibility requires alignment across all three boundaries:
Distributions address this by aligning the entire stack, ensuring the host kernel and the containerized virtualization userspace are built from the same distribution base.
Harvester (built on SUSE / Rancher) does not run upstream CentOS Stream images on its nodes. Instead:
OpenShift Virtualization (Red Hat) takes a similar approach within the OpenShift platform:
In both cases, because the host kernel and the containerized hypervisor share the same distribution base, they avoid the impedance mismatch that arises from running userspace built for one kernel generation on a host running a different one.
We are working to make these boundaries explicit:
Documenting assumptions is essential, but documentation alone does not prevent regressions.
Upstream maintainers validate release artifacts built on Enterprise Linux userspace and tested on EL host kernels. Upstream cannot realistically absorb the testing matrix and debugging burden of arbitrary host distributions without dedicated capacity.
If you rely on KubeVirt on non-EL hosts (Ubuntu, Debian, Flatcar, Talos, etc.):
We are a Cloud Native Computing Foundation incubating project.