KVM from the Ground Up: VMX, EPT and a Guest in a Hundred Lines of C

A virtual machine is often described as "a computer inside a computer", which is true and completely unhelpful when you want to know what happens on an out instruction. This post goes one level down. It covers what Intel's VT-x (and its AMD counterpart, SVM) actually adds to the processor, how KVM exposes that through /dev/kvm, and finishes with a working program that runs a few bytes of guest code and prints their output. Along the way it becomes clear why "VM escape" is mostly a story about device emulation rather than about the CPU.

What the hardware adds

Before VT-x, running an unmodified OS as a guest meant trapping or rewriting every privileged instruction in software. VT-x makes the processor itself aware of two operating modes:

• VMX root operation is where the hypervisor runs. It looks like a normal CPU with a few extra instructions (VMXON, VMLAUNCH, VMRESUME, VMPTRLD, VMREAD, VMWRITE).
• VMX non-root operation is where the guest runs. It has all four privilege rings, so a guest kernel runs in ring 0 and believes it owns the machine, but a configurable set of events cause a VM exit: control transfers to the hypervisor, with the full guest state saved into a per-vCPU structure called the VMCS (virtual machine control structure).

The VMCS is the heart of it. It holds the guest's register state, the host's state to restore on exit, and the controls that decide what exits: which exceptions, which MSR accesses, whether CPUID or HLT or I/O instructions trap, whether external interrupts are delivered to the guest or to the host. KVM fills it in, and the CPU consults it on every entry and exit.

Not every exit leaves the kernel. KVM handles the frequent, simple ones itself: CPUID, most MSRs, the local APIC timer, EPT violations that just need a page mapped. Only exits that need a device model, such as port or memory-mapped I/O to an emulated device, return to the userspace VMM through the ioctl(KVM_RUN) call. The cost of an exit, a few thousand cycles for the round trip into the kernel and far more for a trip to userspace, is what drives everything in virtualization performance work: paravirtualized drivers, posted interrupts, and the big-page EPT mappings discussed next.

Memory: two page tables at once

A guest kernel manages its own page tables and expects to write to CR3. Letting it do so while also isolating it from the host requires a second level of translation, which Intel calls Extended Page Tables (EPT) and AMD calls nested page tables.

The guest translates virtual to guest-physical addresses with its own tables. The hardware then translates guest-physical to host-physical through EPT, whose root (the EPTP) sits in the VMCS. On a TLB miss the page walker has to do both: each guest page-table entry it reads is itself a guest-physical address that must go through EPT first. For $n$-level tables on both sides this costs up to

against 4 for a native walk. This is why 2 MiB and 1 GiB EPT mappings matter so much more than huge pages do on bare metal, and why KVM goes to some trouble to keep them.

Where do EPT entries come from? The VMM allocates ordinary memory with mmap and registers it with KVM_SET_USER_MEMORY_REGION, saying "guest-physical range X maps to this buffer in my address space". KVM fills EPT entries lazily: the first guest touch of a page causes an EPT violation exit, KVM resolves the host page (faulting it in if needed) and installs the mapping, and the guest resumes without noticing. EPT entries carry their own read/write/execute bits, which KVM uses for dirty-page tracking during migration and for keeping MMIO regions unmapped so that every access becomes an exit the device model can see.

A guest in a hundred lines

The KVM API is small enough to use directly. The classic demonstration, described in the LWN article on the KVM API, runs a dozen bytes of 16-bit real-mode code that adds two numbers and writes the digit to the serial port at I/O address 0x3f8. The program below is a from-scratch version of that idea.

The guest code, hand-assembled:

static const uint8_t guest_code[] = {
	0xba, 0xf8, 0x03,	/* mov  $0x3f8, %dx   ; serial port */
	0x00, 0xd8,		/* add  %bl, %al      ; al = al + bl */
	0x04, '0',		/* add  $'0', %al     ; make it ASCII */
	0xee,			/* out  %al, (%dx)    ; VM exit: KVM_EXIT_IO */
	0xb0, '\n',		/* mov  $'\n', %al */
	0xee,			/* out  %al, (%dx)    ; second exit */
	0xf4,			/* hlt                ; KVM_EXIT_HLT */
};

The host side does exactly five things: open /dev/kvm, create a VM, give it memory, create a vCPU with initial registers, and loop on KVM_RUN handling exits.

int kvm = open("/dev/kvm", O_RDWR | O_CLOEXEC);
int vm  = ioctl(kvm, KVM_CREATE_VM, 0);

/* 4 KiB of guest RAM at guest-physical 0x1000 */
void *mem = mmap(NULL, 0x1000, PROT_READ | PROT_WRITE, MAP_SHARED | MAP_ANONYMOUS, -1, 0);
memcpy(mem, guest_code, sizeof guest_code);
struct kvm_userspace_memory_region region = {
	.slot = 0, .guest_phys_addr = 0x1000, .memory_size = 0x1000,
	.userspace_addr = (uint64_t)mem,
};
ioctl(vm, KVM_SET_USER_MEMORY_REGION, &region);

int vcpu = ioctl(vm, KVM_CREATE_VCPU, 0);
size_t run_sz = ioctl(kvm, KVM_GET_VCPU_MMAP_SIZE, NULL);
struct kvm_run *run = mmap(NULL, run_sz, PROT_READ | PROT_WRITE, MAP_SHARED, vcpu, 0);

struct kvm_sregs sregs;
ioctl(vcpu, KVM_GET_SREGS, &sregs);
sregs.cs.base = 0; sregs.cs.selector = 0;	/* flat real mode */
ioctl(vcpu, KVM_SET_SREGS, &sregs);
struct kvm_regs regs = { .rip = 0x1000, .rax = 2, .rbx = 2, .rflags = 0x2 };
ioctl(vcpu, KVM_SET_REGS, &regs);

for (;;) {
	ioctl(vcpu, KVM_RUN, NULL);
	switch (run->exit_reason) {
	case KVM_EXIT_HLT:
		return 0;
	case KVM_EXIT_IO:
		if (run->io.direction == KVM_EXIT_IO_OUT && run->io.port == 0x3f8)
			fwrite((char *)run + run->io.data_offset, run->io.size, run->io.count, stdout);
		break;
	default:
		fprintf(stderr, "unhandled exit %u\n", run->exit_reason);
		return 1;
	}
}

// SPDX-License-Identifier: MIT
// Build: gcc -O2 -Wall -o tinyvm tinyvm.c   Run: ./tinyvm   (expects "4")
#include <fcntl.h>
#include <linux/kvm.h>
#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/ioctl.h>
#include <sys/mman.h>
#include <unistd.h>

static const uint8_t guest_code[] = {
	0xba, 0xf8, 0x03, 0x00, 0xd8, 0x04, '0', 0xee, 0xb0, '\n', 0xee, 0xf4,
};

static void die(const char *what)
{
	perror(what);
	exit(1);
}

int main(void)
{
	int kvm = open("/dev/kvm", O_RDWR | O_CLOEXEC);
	if (kvm < 0)
		die("open /dev/kvm");
	if (ioctl(kvm, KVM_GET_API_VERSION, NULL) != KVM_API_VERSION)
		die("KVM_GET_API_VERSION mismatch");

	int vm = ioctl(kvm, KVM_CREATE_VM, 0);
	if (vm < 0)
		die("KVM_CREATE_VM");

	void *mem = mmap(NULL, 0x1000, PROT_READ | PROT_WRITE,
			 MAP_SHARED | MAP_ANONYMOUS, -1, 0);
	if (mem == MAP_FAILED)
		die("mmap guest memory");
	memcpy(mem, guest_code, sizeof guest_code);

	struct kvm_userspace_memory_region region = {
		.slot = 0,
		.guest_phys_addr = 0x1000,
		.memory_size = 0x1000,
		.userspace_addr = (uint64_t)mem,
	};
	if (ioctl(vm, KVM_SET_USER_MEMORY_REGION, &region) < 0)
		die("KVM_SET_USER_MEMORY_REGION");

	int vcpu = ioctl(vm, KVM_CREATE_VCPU, 0);
	if (vcpu < 0)
		die("KVM_CREATE_VCPU");

	int run_sz = ioctl(kvm, KVM_GET_VCPU_MMAP_SIZE, NULL);
	struct kvm_run *run = mmap(NULL, run_sz, PROT_READ | PROT_WRITE,
				   MAP_SHARED, vcpu, 0);
	if (run == MAP_FAILED)
		die("mmap kvm_run");

	struct kvm_sregs sregs;
	if (ioctl(vcpu, KVM_GET_SREGS, &sregs) < 0)
		die("KVM_GET_SREGS");
	sregs.cs.base = 0;
	sregs.cs.selector = 0;
	if (ioctl(vcpu, KVM_SET_SREGS, &sregs) < 0)
		die("KVM_SET_SREGS");

	struct kvm_regs regs = {
		.rip = 0x1000, .rax = 2, .rbx = 2, .rflags = 0x2,
	};
	if (ioctl(vcpu, KVM_SET_REGS, &regs) < 0)
		die("KVM_SET_REGS");

	for (;;) {
		if (ioctl(vcpu, KVM_RUN, NULL) < 0)
			die("KVM_RUN");
		switch (run->exit_reason) {
		case KVM_EXIT_HLT:
			return 0;
		case KVM_EXIT_IO:
			if (run->io.direction == KVM_EXIT_IO_OUT &&
			    run->io.port == 0x3f8)
				fwrite((char *)run + run->io.data_offset,
				       run->io.size, run->io.count, stdout);
			break;
		case KVM_EXIT_FAIL_ENTRY:
			fprintf(stderr, "entry failed, reason 0x%llx\n",
				(unsigned long long)run->fail_entry.hardware_entry_failure_reason);
			return 1;
		default:
			fprintf(stderr, "unhandled exit reason %u\n", run->exit_reason);
			return 1;
		}
	}
}

# Host checks: is virtualization on, and can this user open the device?
grep -cE 'vmx|svm' /proc/cpuinfo        # > 0 means the CPU supports it
ls -l /dev/kvm                          # usually root:kvm, mode 0660
lsmod | grep -E '^kvm'                  # kvm_intel or kvm_amd loaded
gcc -O2 -Wall -o tinyvm tinyvm.c && ./tinyvm
# 4

Run it and the guest computes 2 + 2, writes '4' and a newline to port 0x3f8, and halts. Each out is a VM exit that lands in our switch; the halt is a third exit that ends the loop. Our program is, in the smallest possible way, a hypervisor: it owns the guest's memory and its entire view of the outside world.

TIP

Everything a real VMM does is an elaboration of this loop. QEMU's main loop, Firecracker's vCPU threads and crosvm all sit on ioctl(KVM_RUN) and dispatch on exit_reason. Reading the kvm_run structure in linux/kvm.h tells you the full set of exits a device model has to be prepared for.

Where "VM escape" actually happens

An escape means guest code gaining execution in the VMM process or the host kernel. The hardware boundary itself (VMX, EPT) has had very few bugs. The overwhelming majority of real escapes have come from the row in bold: device emulation. A guest driver writes a descriptor that claims a 64 KiB buffer but points at 16 bytes; the emulated NIC copies 64 KiB; the VMM's heap is corrupted. The emulated device is just a C program parsing attacker-controlled bytes, with all the usual consequences, and virtio makes this explicit by placing the shared descriptor rings in guest memory where the guest can change them while the host is reading.

The defensive conclusions follow directly:

• Minimise the device model. Firecracker exposes a handful of virtio devices and nothing legacy; there is no floppy controller to have a bug in.
• Sandbox the VMM. QEMU runs under seccomp filters, with its own user, often with SELinux or AppArmor profiles; a compromised VMM process should not be able to reach other guests or the host.
• Read guest memory once. Copy descriptors out of shared memory before validating them, so the guest cannot change a length after the check.
• Prefer memory-safe device code. The rust-vmm crates used by Firecracker and crosvm exist because the bug class is so predictable.
• Keep the kernel surface small. vhost moves virtio data paths into the host kernel for speed, which is a trade: fewer exits, but a bug now lands in ring 0.

Further reading

• The Definitive KVM API Documentation, every ioctl used above and the layout of struct kvm_run.
• Josh Triplett, Using the KVM API, LWN, the original walk-through of a minimal KVM program.
• The x86 KVM shadow MMU, the kernel's own description of how EPT and shadow paging are managed.
• QEMU Security, the project's threat model and the recommended sandboxing of the VMM process.
• Virtual I/O Device (VIRTIO) Version 1.2, the specification for the shared-memory rings every modern VMM parses.
• Firecracker, a minimal-device-model VMM designed around the attack surface argument.