Most of what makes kernel code different from user code fits in one sentence: there is no process around you to die on your behalf. A NULL dereference in a user program kills that program. The same dereference in a module takes a kernel thread down with it, taints the kernel, and may leave locks held forever. Learning to write a small module and, more importantly, to read what the kernel prints when it fails is the fastest route to understanding why the user/kernel boundary is guarded the way it is.

DANGER

Do this in a virtual machine or on a board you can power-cycle. A buggy module can corrupt the filesystem of the machine it runs on, and the deliberate crash later in this post is exactly that kind of bug. Snapshot the VM before loading anything.

Lab setup

  1. Boot a VM with a distribution kernel and install the matching headers: sudo apt install build-essential linux-headers-$(uname -r) on Debian or Ubuntu, sudo dnf install kernel-devel-$(uname -r) on Fedora.
  2. Confirm the build tree exists: ls /lib/modules/$(uname -r)/build should show a Makefile and .config.
  3. Check whether the kernel enforces module signatures: cat /sys/module/module/parameters/sig_enforce prints N on most desktop kernels. If it prints Y (Secure Boot with lockdown), you will need to sign the module or test in a VM with Secure Boot off.
  4. Open a second terminal running sudo dmesg -w so that kernel messages appear as they happen.
  5. Build, load, inspect, unload, as shown below.

hello.c and its Makefile

// SPDX-License-Identifier: GPL-2.0
#include <linux/module.h>
#include <linux/init.h>

static int __init hello_init(void)
{
	pr_info("hello: loaded into kernel %s\n", init_uts_ns.name.release);
	return 0;
}

static void __exit hello_exit(void)
{
	pr_info("hello: unloaded\n");
}

module_init(hello_init);
module_exit(hello_exit);
MODULE_LICENSE("GPL");
MODULE_DESCRIPTION("Smallest possible out-of-tree module");

Out-of-tree modules are built by the kernel's own build system (kbuild). The Makefile simply hands control to it:

obj-m += hello.o

KDIR ?= /lib/modules/$(shell uname -r)/build

all:
	$(MAKE) -C $(KDIR) M=$(CURDIR) modules

clean:
	$(MAKE) -C $(KDIR) M=$(CURDIR) clean

make
sudo insmod hello.ko
sudo dmesg | tail -n 2
#  [  142.118205] hello: loaded into kernel 6.8.0-xx-generic
lsmod | grep hello
modinfo hello.ko          # license, description, vermagic
sudo rmmod hello

Two things are worth noticing. MODULE_LICENSE("GPL") is not decoration: without a GPL-compatible licence the kernel marks itself tainted (P) and refuses to export GPL-only symbols to you. And vermagic in modinfo must match the running kernel exactly, which is why the module has to be rebuilt for every kernel update.

printk levels

pr_info is a wrapper around printk with a log level prefix. The level decides whether the message reaches the console immediately and how dmesg -l filters it.

Level Macro Helper Typical use
0 KERN_EMERG pr_emerg system is unusable
1 KERN_ALERT pr_alert action must be taken immediately
2 KERN_CRIT pr_crit critical conditions
3 KERN_ERR pr_err error conditions
4 KERN_WARNING pr_warn warning conditions
5 KERN_NOTICE pr_notice normal but significant
6 KERN_INFO pr_info informational
7 KERN_DEBUG pr_debug compiled out unless DEBUG or dyndbg

Crossing the boundary: a misc device

A module that only prints is not very interesting. Let us give it a device node so that user space can talk to it, which immediately raises the question of how kernel code may touch a pointer handed to it by a process.

A write() crossing the user/kernel boundary User space (ring 3) low half of the address space 0x0000_0000_0000_0000 … 0x0000_7fff_ffff_ffff process char buf[] = "hello"; write(fd, buf, 6); buf is a user pointer: valid only in this process libc wrapper mov eax, 1 (SYS_write); syscall return value 6 on success, or -1 with errno = EFAULT a bad pointer costs the caller an error, not the machine privilege boundary Kernel (ring 0) high half, shared by every process: one bug is fatal to all entry_SYSCALL_64 → ksys_write → vfs_write → hello_write(file, ubuf, 6, off) copy_from_user(kbuf, ubuf, 6) 1. access_ok: range must lie in the user half 2. copy with a fixup entry in the exception table: a page fault lands in the fixup, not in an oops 3. returns the number of bytes NOT copied SMAP / PAN: plain *ubuf from kernel code faults kernel buffer char kbuf[64]; kbuf[n] = '\0'; bounded by min(len, sizeof kbuf - 1) before the copy sysret
A write() on /dev/hello. The user pointer is only meaningful inside the calling process, and the kernel must assume it is hostile: unmapped, pointing into kernel memory, or changing underneath it.
// SPDX-License-Identifier: GPL-2.0
#include <linux/module.h>
#include <linux/miscdevice.h>
#include <linux/fs.h>
#include <linux/uaccess.h>
#include <linux/minmax.h>

static char kbuf[64];
static size_t kbuf_len;

static ssize_t hello_write(struct file *f, const char __user *ubuf,
			   size_t len, loff_t *off)
{
	size_t n = min(len, sizeof(kbuf) - 1);

	if (copy_from_user(kbuf, ubuf, n))
		return -EFAULT;
	kbuf[n] = '\0';
	kbuf_len = n;
	pr_info("hello: got %zu bytes: %s\n", n, kbuf);
	return len;	/* consume everything, like /dev/null does */
}

static ssize_t hello_read(struct file *f, char __user *ubuf,
			  size_t len, loff_t *off)
{
	return simple_read_from_buffer(ubuf, len, off, kbuf, kbuf_len);
}

static const struct file_operations hello_fops = {
	.owner = THIS_MODULE,
	.read  = hello_read,
	.write = hello_write,
};

static struct miscdevice hello_dev = {
	.minor = MISC_DYNAMIC_MINOR,
	.name  = "hello",
	.fops  = &hello_fops,
	.mode  = 0666,
};

static int __init hello_init(void)
{
	int ret = misc_register(&hello_dev);

	if (ret)
		return ret;
	pr_info("hello: /dev/hello ready\n");
	return 0;
}

static void __exit hello_exit(void)
{
	misc_deregister(&hello_dev);
}

module_init(hello_init);
module_exit(hello_exit);
MODULE_LICENSE("GPL");
MODULE_DESCRIPTION("Misc device demonstrating copy_from_user");

make && sudo insmod hello.ko
echo -n "kernel space" > /dev/hello
cat /dev/hello; echo
sudo dmesg | tail -n 1
#  [  310.442871] hello: got 12 bytes: kernel space

Why copy_from_user and not memcpy

The __user annotation on ubuf is checked by sparse, not by the compiler, and it marks the pointer as belonging to another address space. Three things go wrong if you dereference it directly:

  • The page may not be mapped. A user program can pass any value, including 0x41414141. A plain load from kernel mode would fault, and a fault in kernel mode is an oops. copy_from_user registers its copying instructions in the kernel's exception table, so a fault inside it is redirected to a fixup that returns the number of bytes left uncopied, and the driver returns -EFAULT instead of crashing.
  • The pointer may point into the kernel. Before the copy, access_ok() rejects addresses in the kernel half. Without that check, a write() with a crafted pointer would let a process read kernel memory out through your device, or overwrite it through read().
  • Hardware now refuses. On x86 with SMAP, or arm64 with PAN, a kernel-mode access to a user page faults unless user access was explicitly enabled around the copy. copy_from_user does that enabling (stac/clac on x86) for exactly the instructions that need it.

The length check matters too. min(len, sizeof(kbuf) - 1) is what keeps a long write() from running off the end of kbuf; copy_from_user trusts the length you give it.

Breaking it on purpose

Replace the start of hello_write with a deliberate bug and reload:

	char *p = NULL;

	*p = 'x';		/* deliberate NULL write */

make && sudo insmod hello.ko
echo hi > /dev/hello
# bash: echo: write error: ... (the shell may hang here; look at dmesg)

The kernel kills the writing process and prints an oops. It looks intimidating, but it follows a fixed layout:

Anatomy of a kernel oops BUG: kernel NULL pointer dereference, address: 0000000000000000 #PF: supervisor write access in kernel mode #PF: error_code(0x0002) - not-present page Oops: 0002 [#1] PREEMPT SMP NOPTI CPU: 1 PID: 1337 Comm: bash Tainted: G O 6.8.0 #1 RIP: 0010:hello_write+0x2a/0x60 [hello] Code: 48 89 e5 41 54 53 ... <c6> 04 25 00 00 00 00 ... RSP: 0018:ffffc9000061fe90 EFLAGS: 00010246 RAX: 0000000000000000 RBX: ffff888103a4c000 ... FS: 00007f2d1c1f0740 GS: ffff88813bc80000 ... CR2: 0000000000000000 CR3: 0000000104f2a000 Call Trace: <TASK> vfs_write+0xc6/0x390 ksys_write+0x6d/0xf0 do_syscall_64+0x5c/0x90 entry_SYSCALL_64_after_hwframe+0x6e/0xd8 </TASK> Modules linked in: hello(O) ... ---[ end trace 0000000000000000 ]--- What and where address 0 (or a small number) means a NULL dereference, maybe through a struct field Page-fault flags write, from kernel mode, page not present Taint and oops count O = out-of-tree module; [#1] = first oops, trust later ones less Faulting instruction symbol+offset/size [module]; map it with objdump -dS hello.ko or decode_stacktrace.sh Registers CPU state at the fault, to read next to the disassembly; CR2 repeats the faulting address Call trace innermost frame first; read upward from the syscall entry to see how we got here Modules which .ko files were loaded; (O) marks ours
Reading an oops: the first line names the fault, RIP names the instruction, and the call trace, innermost first, explains how execution got there.
A full oops for the deliberate NULL write (trimmed) 
[  412.901512] BUG: kernel NULL pointer dereference, address: 0000000000000000
[  412.901519] #PF: supervisor write access in kernel mode
[  412.901521] #PF: error_code(0x0002) - not-present page
[  412.901523] PGD 0 P4D 0
[  412.901526] Oops: 0002 [#1] PREEMPT SMP NOPTI
[  412.901530] CPU: 1 PID: 1337 Comm: bash Tainted: G           O       6.8.0-xx-generic #1
[  412.901534] RIP: 0010:hello_write+0x2a/0x60 [hello]
[  412.901539] Code: 48 89 e5 41 54 53 48 83 ec 10 ... <c6> 04 25 00 00 00 00 78 ...
[  412.901542] RSP: 0018:ffffc9000061fe90 EFLAGS: 00010246
[  412.901544] RAX: 0000000000000000 RBX: ffff888103a4c000 RCX: 0000000000000003
[  412.901560] CR2: 0000000000000000 CR3: 0000000104f2a000 CR4: 0000000000750ef0
[  412.901562] Call Trace:
[  412.901563]  <TASK>
[  412.901566]  vfs_write+0xc6/0x390
[  412.901570]  ksys_write+0x6d/0xf0
[  412.901573]  do_syscall_64+0x5c/0x90
[  412.901577]  entry_SYSCALL_64_after_hwframe+0x6e/0xd8
[  412.901581]  </TASK>
[  412.901582] Modules linked in: hello(O) ...
[  412.901600] ---[ end trace 0000000000000000 ]---

The exact addresses and offsets will differ on your kernel; the shape will not.

Reading it top to bottom:

  • BUG: line. The class of fault and the address. Address 0 or a small number is a NULL dereference, possibly through a struct field at that offset. A large kernel-looking address suggests use-after-free or a wild pointer.
  • #PF flags. Whether it was a read or write, in user or kernel mode, and whether the page was absent or had the wrong permissions. "Supervisor write" to an absent page is our bug exactly.
  • Tainted: O means an out-of-tree module is loaded, P a proprietary one, D a previous oops. Upstream developers will ask you to reproduce without taint.
  • RIP: symbol+offset/size [module]. Everything you need to find the instruction.
  • Call Trace. Innermost frame first. Entries marked ? are guesses from scanning the stack and may be stale.

To map hello_write+0x2a back to source, build with debug info (add ccflags-y += -g to the Makefile) and ask the debugger, or disassemble:

gdb -batch -ex 'list *(hello_write+0x2a)' hello.ko
objdump -dS hello.ko | less        # search for <hello_write>, count to +0x2a
# For traces from the main kernel image, the tree ships a helper:
./scripts/decode_stacktrace.sh vmlinux < oops.txt

After an oops the module is still loaded and rmmod will likely refuse because a reference count was never dropped. That is the point of the [#1]: the kernel is now in an unknown state, and the only clean exit is a reboot, which in the lab means restoring the snapshot.

What this teaches about kernel attack surface

Everything that made the lab fragile is what an attacker looks for in real drivers: a length that is trusted, a user pointer that is dereferenced directly, a copy whose return value is ignored. The kernel's defences are aimed at the same places. Distributions sign modules and enable lockdown so arbitrary code cannot be loaded. CONFIG_STRICT_DEVMEM, SMAP and PAN make the hardware enforce the boundary that copy_from_user respects by convention. And the oops itself is a defensive feature: dying loudly with a trace is better than continuing with corrupted state, which is why panic_on_oops=1 is often set on production servers. Reading one oops end to end is worth more than any amount of theory about why these mechanisms exist.

Further reading