Xen-4.16.0 + FreeBSD-13.1 dom0 fails on large ADM64 system
Date: Sat, 13 Apr 2024 05:57:32 UTC
Hello. I'm trying to load xen on a large AMD64 server, with 600G of RAM and 56 CPUs. I'm using a xen-4.16 image with FreeBSD-13.1, a duplicate image, in fact, with a couple of other machines I have running in production and which have been running without issues for a couple of years. I think I'm running into some defined limits in the sense that I think the machine has more of something than xen is configured for. Unfortunately, pouring through the source doesn't clue me into exactly what's rong, except I think it has something to do with mmio allocations. Can someone take a look at the boot message below and give a clue as to what might be wrong? Do I need to recompile xen with some new limits, turn something on or off in BIOS or change some parameter on the xen command line? Or, is this something that requires a newer version of Xen? Any thoughts would be greatly appreciated. The error is: Failed to identity map [ffffffffc7ffb, ffffffffc7ffb] for d0: -22 that's EINVAL, from sys/errno.h, suggesting something is out of range. -thanks -Brian [25;0HLoading Xen kernel... /boot/xen data=0x2659c8+0x13e638 |/ Loading kernel... -\|/boot/kernel/kernel /-\|/-\|/-\|/-size=0x1bfbfc0 \|/-\Loading configured modules... /boot/kernel/geom_mirror.ko size 0x20c80 at 0x21a1000 |/etc/hostid size=0x25 /boot/entropy size=0x1000 //boot/kernel/ipmi.ko size 0x11968 at 0x21c3000 loading required module 'smbus' -/boot/kernel/smbus.ko size 0x3cb0 at 0x21d5000 \/boot/kernel/tpm.ko size 0xad70 at 0x21d9000 |/boot/firmware/intel-ucode.bin /size=0x303800 [03;00H [01;01H Xen 4.16.0 (XEN) Xen version 4.16.0 (buhrow@) (FreeBSD clang version 13.0.0 (git@github.com:llvm/llvm-project.git llvmorg-13.0.0-0-gd7b669b3a303)) debug=n Thu Jun 2 11:30:17 PDT 2022 (XEN) Latest ChangeSet: (XEN) Bootloader: FreeBSD Loader (XEN) Command line: dom0_mem=65536m dom0_max_vcpus=8 dom0=pvh mmio-relax=true pv-l1tf=off,domu=off console=com2,vga com2=115200,8n1 (XEN) Xen image load base address: 0 (XEN) Video information: (XEN) VGA is text mode 80x25, font 8x16 (XEN) VBE/DDC methods: none; EDID transfer time: 1 seconds (XEN) EDID info not retrieved because no DDC retrieval method detected (XEN) Disc information: (XEN) Found 1 MBR signatures (XEN) Found 1 EDD information structures (XEN) Xen-e820 RAM map: (XEN) [0000000000000000, 00000000000997ff] (usable) (XEN) [0000000000099800, 000000000009ffff] (reserved) (XEN) [00000000000e0000, 00000000000fffff] (reserved) (XEN) [0000000000100000, 0000000078e70fff] (usable) (XEN) [0000000078e71000, 0000000079775fff] (reserved) (XEN) [0000000079776000, 0000000079d37fff] (ACPI NVS) (XEN) [0000000079d38000, 000000008fffffff] (reserved) (XEN) [00000000fed1c000, 00000000fed44fff] (reserved) (XEN) [00000000ff000000, 00000000ffffffff] (reserved) (XEN) [0000000100000000, 000000807fffffff] (usable) (XEN) New Xen image base address: 0x78800000 (XEN) ACPI: RSDP 000F05B0, 0024 (r2 SUPERM) (XEN) ACPI: XSDT 797EE0B0, 00DC (r1 1072009 AMI 10013) (XEN) ACPI: FACP 79824B18, 010C (r5 SUPERM SMCI--MB 1072009 AMI 10013) (XEN) ACPI: DSDT 797EE220, 368F3 (r2 SUPERM SMCI--MB 1072009 INTL 20091013) (XEN) ACPI: FACS 79D36F80, 0040 (XEN) ACPI: APIC 79824C28, 0374 (r3 SUPERM SMCI--MB 1072009 AMI 10013) (XEN) ACPI: FPDT 79824FA0, 0044 (r1 SUPERM SMCI--MB 1072009 AMI 10013) (XEN) ACPI: FIDT 79824FE8, 009C (r1 SUPERM SMCI--MB 1072009 AMI 10013) (XEN) ACPI: SPMI 79825088, 0041 (r5 SUPERM SMCI--MB 0 AMI. 0) (XEN) ACPI: MCFG 798250D0, 003C (r1 SUPERM SMCI--MB 1072009 MSFT 97) (XEN) ACPI: UEFI 79825110, 0042 (r1 SUPERM SMCI--MB 1072009 0) (XEN) ACPI: HPET 79825158, 0038 (r1 SUPERM SMCI--MB 1 INTL 20091013) (XEN) ACPI: MSCT 79825190, 0090 (r1 SUPERM SMCI--MB 1 INTL 20091013) (XEN) ACPI: NFIT 79825220, 18028 (r1 0 0) (XEN) ACPI: SLIT 7983D248, 0030 (r1 SUPERM SMCI--MB 1 INTL 20091013) (XEN) ACPI: SRAT 7983D278, 1158 (r3 SUPERM SMCI--MB 1 INTL 20091013) (XEN) ACPI: WDDT 7983E3D0, 0040 (r1 SUPERM SMCI--MB 0 INTL 20091013) (XEN) ACPI: SSDT 7983E410, 1717F (r2 SUPERM PmMgt 1 INTL 20120913) (XEN) ACPI: NITR 79855590, 0071 (r2 SUPERM SMCI--MB 1 INTL 20091013) (XEN) ACPI: SSDT 79855608, 264C (r2 SUPERM SpsNm 2 INTL 20120913) (XEN) ACPI: SSDT 79857C58, 0064 (r2 SUPERM SpsNvs 2 INTL 20120913) (XEN) ACPI: PRAD 79857CC0, 0102 (r2 SUPERM SMCI--MB 2 INTL 20120913) (XEN) ACPI: DMAR 79857DC8, 0138 (r1 SUPERM SMCI--MB 1 INTL 20091013) (XEN) ACPI: HEST 79857F00, 027C (r1 SUPERM SMCI--MB 1 INTL 1) (XEN) ACPI: BERT 79858180, 0030 (r1 SUPERM SMCI--MB 1 INTL 1) (XEN) ACPI: ERST 798581B0, 0230 (r1 SUPERM SMCI--MB 1 INTL 1) (XEN) ACPI: EINJ 798583E0, 0130 (r1 SUPERM SMCI--MB 1 INTL 1) (XEN) System RAM: 524174MB (536754216kB) (XEN) Domain heap initialised DMA width 32 bits (XEN) x2APIC mode is already enabled by BIOS. (XEN) ACPI: 32/64X FACS address mismatch in FADT - 79d36f80/0000000000000000, using 32 (XEN) IOAPIC[0]: apic_id 1, version 32, address 0xfec00000, GSI 0-23 (XEN) IOAPIC[1]: apic_id 2, version 32, address 0xfec01000, GSI 24-47 (XEN) IOAPIC[2]: apic_id 3, version 32, address 0xfec40000, GSI 48-71 (XEN) CPU0: 1200 ... 2600 MHz (XEN) xstate: size: 0x340 and states: 0x7 (XEN) Speculative mitigation facilities: (XEN) Hardware hints: (XEN) Hardware features: IBPB IBRS STIBP SSBD L1D_FLUSH MD_CLEAR (XEN) Compiled-in support: INDIRECT_THUNK SHADOW_PAGING (XEN) Xen settings: BTI-Thunk RETPOLINE, SPEC_CTRL: IBRS- STIBP- SSBD-, Other: IBPB L1D_FLUSH VERW BRANCH_HARDEN (XEN) L1TF: believed vulnerable, maxphysaddr L1D 46, CPUID 46, Safe address 300000000000 (XEN) Support for HVM VMs: MSR_SPEC_CTRL RSB EAGER_FPU MD_CLEAR (XEN) Support for PV VMs: MSR_SPEC_CTRL EAGER_FPU MD_CLEAR (XEN) XPTI (64-bit PV only): Dom0 enabled, DomU enabled (with PCID) (XEN) PV L1TF shadowing: Dom0 disabled, DomU disabled (XEN) Using scheduler: SMP Credit Scheduler rev2 (credit2) (XEN) Initializing Credit2 scheduler (XEN) Platform timer is 14.318MHz HPET (XEN) Detected 2600.012 MHz processor. (XEN) Intel VT-d iommu 0 supported page sizes: 4kB, 2MB, 1GB (XEN) Intel VT-d iommu 1 supported page sizes: 4kB, 2MB, 1GB (XEN) Intel VT-d Snoop Control enabled. (XEN) Intel VT-d Dom0 DMA Passthrough not enabled. (XEN) Intel VT-d Queued Invalidation enabled. (XEN) Intel VT-d Interrupt Remapping enabled. (XEN) Intel VT-d Posted Interrupt not enabled. (XEN) Intel VT-d Shared EPT tables enabled. (XEN) I/O virtualisation enabled (XEN) - Dom0 mode: Relaxed (XEN) Interrupt remapping enabled (XEN) Enabled directed EOI with ioapic_ack_old on! (XEN) ENABLING IO-APIC IRQs (XEN) -> Using old ACK method (XEN) Allocated console ring of 128 KiB. (XEN) VMX: Supported advanced features: (XEN) - APIC MMIO access virtualisation (XEN) - APIC TPR shadow (XEN) - Extended Page Tables (EPT) (XEN) - Virtual-Processor Identifiers (VPID) (XEN) - Virtual NMI (XEN) - MSR direct-access bitmap (XEN) - Unrestricted Guest (XEN) - APIC Register Virtualization (XEN) - Virtual Interrupt Delivery (XEN) - Posted Interrupt Processing (XEN) - VMCS shadowing (XEN) - VM Functions (XEN) - Virtualisation Exceptions (XEN) - Page Modification Logging (XEN) HVM: ASIDs enabled. (XEN) VMX: Disabling executable EPT superpages due to CVE-2018-12207 (XEN) HVM: VMX enabled (XEN) HVM: Hardware Assisted Paging (HAP) detected (XEN) HVM: HAP page sizes: 4kB, 2MB, 1GB (XEN) Brought up 56 CPUs (XEN) Scheduling granularity: cpu, 1 CPU per sched-resource (XEN) Dom0 has maximum 1320 PIRQs (XEN) Failed to identity map [ffffffffc7ffb, ffffffffc7ffb] for d0: -22 (XEN) setup of vPCI for d0 failed: -22 (XEN) ----[ Xen-4.16.0 x86_64 debug=n Not tainted ]---- (XEN) CPU: 0 (XEN) RIP: e008:[<ffff82d040256a35>] drivers/vpci/header.c#modify_bars+0x285/0x350 (XEN) RFLAGS: 0000000000010202 CONTEXT: hypervisor (XEN) rax: ffff83203ff9d501 rbx: 0000000000000000 rcx: ffff83203ff9d4c0 (XEN) rdx: ffff83203ff9d5b0 rsi: 0000000000000000 rdi: ffff8320277bd000 (XEN) rbp: 000000000000002c rsp: ffff82d04046f718 r8: ffff82d040578068 (XEN) r9: ffff83201739a8d0 r10: 0180000000000000 r11: ffff82d0402561f0 (XEN) r12: ffff83203ff9d4d0 r13: 00000000000000d0 r14: 0000000000000000 (XEN) r15: 000000000000002c cr0: 000000008005003b cr4: 00000000001526e0 (XEN) cr3: 0000000078c62000 cr2: 000000000000002c (XEN) fsb: 0000000000000000 gsb: 0000000000000000 gss: 0000000000000000 (XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: 0000 cs: e008 (XEN) Xen code around <ffff82d040256a35> (drivers/vpci/header.c#modify_bars+0x285/0x350): (XEN) 49 8b 9c 24 c0 00 00 00 <f6> 04 2b 02 74 e5 4c 8b 6c 2b ec 48 8b 44 2b f4 (XEN) Xen stack trace from rsp=ffff82d04046f718: (XEN) 0100000000000206 ffff83201739a7e0 0000000000000100 ffff83201739a001 (XEN) 0000000700000024 ffff83203ff9d5b0 ffff83203ff9d5b0 ffff83201739cf10 (XEN) 0000000000000030 00000000c7e04001 0000000100000005 0000000000000000 (XEN) ffff83203ff9d5b0 ffff82d040256512 000000071739cf10 ffff83201739cf28 (XEN) 0000000000000000 0000000000000000 0000003000000040 ffff83201739cf10 (XEN) 0000000000000006 0000000000000078 0000000000000090 0000000000000002 (XEN) 000000000000001f 0000000000000018 ffff82d0403b5578 ffff83203ff9d5b0 (XEN) 0000000000000000 ffff82d0402552ec 0000000000000000 ffff82d040464e00 (XEN) ffff82d0403de6e0 0000000000000000 ffff8320277bd000 ffff83203ff9d5f0 (XEN) ffff83203ff9d5b0 ffff82d0403e5579 ffff82e000000000 00000001277bd000 (XEN) ffff83203fff5670 ffff8320277bd160 ffff83203fff5670 ffff82d04046fa48 (XEN) 0000000000000000 ffff8320277bd000 ffff8320277bd000 ffff830000097f20 (XEN) 0000006ff8e0a000 ffff82d0403ddfc7 ffff82d0404507a8 ffff82d04046fa48 (XEN) 0000000000000000 0000000001000000 ffff830000097f20 ffff82d0403ed8f7 (XEN) ffff82d0405a3ad0 0000000000000042 0000000000000000 0000000000000000 (XEN) ffff82d040442650 0000000000000000 ffff830000097f20 ffff82d040402916 (XEN) 0000000000000000 0000008080000000 0000000000000000 0000000000000000 (XEN) 0000000000000000 0000000008080000 0000000000000002 ffff82d0403fb480 (XEN) 0000000000000000 0100000000000000 0000000000000003 ffffffed00000100 (XEN) 0000000000000000 ffff830000097f10 00000000ffffff01 0100000000000000 (XEN) Xen call trace: (XEN) [<ffff82d040256a35>] R drivers/vpci/header.c#modify_bars+0x285/0x350 (XEN) [<ffff82d040256512>] S drivers/vpci/header.c#init_bars+0x322/0x350 (XEN) [<ffff82d0402552ec>] S lib/xxhash64.c#vpci_add_handlers+0x9c/0xe0 (XEN) [<ffff82d0403de6e0>] S drivers/passthrough/vtd/iommu.c#setup_hwdom_device+0/0x10 (XEN) [<ffff82d0403e5579>] S lib/xxhash64.c#setup_hwdom_pci_devices+0x1b9/0x2e0 (XEN) [<ffff82d0403ddfc7>] S drivers/passthrough/vtd/iommu.c#intel_iommu_hwdom_init+0x17/0x100 (XEN) [<ffff82d0403ed8f7>] S lib/xxhash64.c#dom0_construct_pvh+0x217/0x1a60 (XEN) [<ffff82d040402916>] S lib/xxhash64.c#acpi_scan_nodes+0x266/0x370 (XEN) [<ffff82d0403fb480>] S lib/xxhash64.c#numa_initmem_init+0x2e0/0x440 (XEN) [<ffff82d040226b81>] S common/page_alloc.c#alloc_heap_pages+0x4a1/0x6b0 (XEN) [<ffff82d0403275ac>] S lib/xxhash64.c#send_IPI_mask+0x16c/0x1f0 (XEN) [<ffff82d040282ef0>] S arch/x86/genapic/x2apic.c#send_IPI_mask_x2apic_cluster+0/0x1f0 (XEN) [<ffff82d040248a2f>] S lib/xxhash64.c#vcpu_wake+0x15f/0x1f0 (XEN) [<ffff82d0402fcde1>] S lib/xxhash64.c#reprogram_timer+0x31/0xc0 (XEN) [<ffff82d04022ddf4>] S common/timer.c#timer_softirq_action+0x1f4/0x2d0 (XEN) [<ffff82d04022afc1>] S lib/xxhash64.c#process_pending_softirqs+0x61/0xc0 (XEN) [<ffff82d04040762c>] S lib/xxhash64.c#construct_dom0+0x8c/0xc0 (XEN) [<ffff82d0403ff6c0>] S arch/x86/setup.c#create_dom0+0x300/0x370 (XEN) [<ffff82d0403ff238>] S lib/xxhash64.c#__start_xen+0x30b8/0x3160 (XEN) [<ffff82d0402000ef>] S __high_start+0x4f/0x60 (XEN) (XEN) Pagetable walk from 000000000000002c: (XEN) L4[0x000] = 000000203ffe7063 ffffffffffffffff (XEN) L3[0x000] = 000000203ffe6063 ffffffffffffffff (XEN) L2[0x000] = 000000203ffe5063 ffffffffffffffff (XEN) L1[0x000] = 0000000000000000 ffffffffffffffff (XEN) (XEN) **************************************** (XEN) Panic on CPU 0: (XEN) FATAL PAGE FAULT (XEN) [error_code=0000] (XEN) Faulting linear address: 000000000000002c (XEN) **************************************** (XEN) (XEN) Reboot in five seconds... (XEN) Resetting with ACPI MEMORY or I/O RESET_REG.