Panic (gpf) in early boot after upgrading FreeBSD 10.4 -> 11.2 on Ganeti

[Konstantin Belousov]

On Sun, Nov 11, 2018 at 11:24:48PM -0500, Rob Austein wrote:
> Belated upgrade (don't ask) of a pair of FreeBSD 10.4 VMs to 11.2.
> Each VM got as far as:
> 
>   freebsd-update -r 11.2-RELEASE update
>   freebsd-update install
>   reboot
> 
> Each VM got an immediate kernel panic after the reboot (log below).
> 
> The two VMs are basically identical at the system level, but run in
> separate Ganeti clusters on opposite coasts, so no hardware in common.
> A dozen or so other VMs run in each cluster without issues (including
> at least one other FreeBSD 11.2 VM), and the VMs I'm trying to upgrade
> have also been just fine until now, so the problem seems unlikely to
> be hardware per se.
> 
> GENERIC, amd64, UFS2, no non-/boot/kernel modules, one CPU per VM.
> The only things even slightly unusual about these VMs are:
> 
> * They're running in Ganeti clusters:
>   * Ganeti version 2.15.2
>   * KVM hypervisor
> 
> * They use virtio, so:
>   * Disk is vtbd0
>   * Net is vtnet0
> 
> Log of attempted boot with new kernel:
> 
>     /boot/kernel/kernel text=0x1547d48 data=0x144138+0x4e9818 syms=[0x8+0x16aef8+0x8+0x183f99]
>     Booting...
>     Copyright (c) 1992-2018 The FreeBSD Project.
>     Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
> 	    The Regents of the University of California. All rights reserved.
>     FreeBSD is a registered trademark of The FreeBSD Foundation.
>     FreeBSD 11.2-RELEASE-p4 #0: Thu Sep 27 08:16:24 UTC 2018
> 	root at amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64
>     FreeBSD clang version 6.0.0 (tags/RELEASE_600/final 326565) (based on LLVM 6.0.0)
>     VT(vga): text 80x25
>     CPU: QEMU Virtual CPU version 2.1.2 (2666.81-MHz K8-class CPU)
>       Origin="GenuineIntel"  Id=0x663  Family=0x6  Model=0x6  Stepping=3
>       Features=0x783fbfd<FPU,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE,SSE2>
>       Features2=0x80a02001<SSE3,CX16,x2APIC,POPCNT,HV>
>       AMD Features=0x20100800<SYSCALL,NX,LM>
>       AMD Features2=0x1<LAHF>
>     Hypervisor: Origin = "KVMKVMKVM"
>     real memory  = 4294967296 (4096 MB)
>     avail memory = 4088406016 (3899 MB)
>     Event timer "LAPIC" quality 100
>     ACPI APIC Table: <BOCHS  BXPCAPIC>
>     ioapic0 <Version 1.1> irqs 0-23 on motherboard
>     kernel trap 9 with interrupts disabled
> 
> 
>     Fatal trap 9: general protection fault while in kernel mode
>     cpuid = 0; apic id = 00
>     instruction pointer	= 0x20:0xffffffff810e9ae6
>     stack pointer	        = 0x28:0xffffffff82272c20
>     frame pointer	        = 0x28:0xffffffff82272c80
>     code segment		= base 0x0, limit 0xfffff, type 0x1b
> 			    = DPL 0, pres 1, long 1, def32 0, gran 1
>     processor eflags	= resume, IOPL = 0
>     current process		= 0 (swapper)
>     trap number		= 9
>     panic: general protection fault
>     cpuid = 0
>     KDB: stack backtrace:
>     #0 0xffffffff80b3d587 at kdb_backtrace+0x67
>     #1 0xffffffff80af6b27 at vpanic+0x177
>     #2 0xffffffff80af69a3 at panic+0x43
>     #3 0xffffffff80f77fdf at trap_fatal+0x35f
>     #4 0xffffffff80f7759e at trap+0x5e
>     #5 0xffffffff80f57fbc at calltrap+0x8
>     #6 0xffffffff810ec5f3 at apic_setup_io+0x53
>     #7 0xffffffff80a92898 at mi_startup+0x118
>     #8 0xffffffff8031002c at btext+0x2c
>     Uptime: 1s
>     Automatic reboot in 15 seconds - press a key on the console to abort
>     --> Press a key on the console to reboot,
>     --> or switch off the system now.
> 
> Goggling turned up a few theories about bad memory and incompatible
> changes to video drivers, none of which seem likely to apply here.
> 
> Cluebat, please, somebody?
Try to issue the following commands at the loader prompt:
	set hw.x2apic_enable=0
	boot