From nobody Fri May 06 12:13:48 2022 X-Original-To: bugs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 52E911ABB6C4 for ; Fri, 6 May 2022 12:13:48 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4KvqH00PnPz4yWy for ; Fri, 6 May 2022 12:13:48 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id E25AE1BC84 for ; Fri, 6 May 2022 12:13:47 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 246CDlHh099462 for ; Fri, 6 May 2022 12:13:47 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 246CDljF099461 for bugs@FreeBSD.org; Fri, 6 May 2022 12:13:47 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: bugs@FreeBSD.org Subject: [Bug 263814] panic: GPF in cpu_search_highest if all cores in a domain are disabled Date: Fri, 06 May 2022 12:13:48 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: new X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: dgmorris@earthlink.net X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: bugs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_id short_desc product version rep_platform op_sys bug_status bug_severity priority component assigned_to reporter attachments.mimetype attachments.created Message-ID: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Bug reports List-Archive: https://lists.freebsd.org/archives/freebsd-bugs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-bugs@freebsd.org MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1651839228; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=6ODHBJ6dmgGIlC73OA/CWhnkhcibF7tBGAZyYIKHTok=; b=D7rPZjLGtBoZLBPTq3Zb6lsvzAo2QsYEZC+m7Ivpg4N60K5l7OyPz0Y9WZrZxxpcOcARXn C9sUKALpNoL4FdYoE9yO9PncrHLQjyeUUkqs8aM/zflaRwPsSIhW7zUPnm79j0M+nBEJvD HyEGBllvQ9hRdTsQ3W4q/0et/m/Kn/ZalUstm1GZqq3GBYF3KI+C4iM3osZ0Nj7rINXGC9 /5sAqHj04YInPPZtTfrcUFu/0l8O4uh/Q9T9O8YzO4Wr0Ukluk9sUYUreJIUudEvRoOQt2 znVACsWoFX5oDhyVpRNevEPf9SheB1FqPYQ9aY+QoPhPS+8PvFSzTzaRoP13MA== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1651839228; a=rsa-sha256; cv=none; b=B+h3ggIMeSITCfaXDAAKtntsgzpY/vsqPsnnLXcuOJXpXuzHOo1siiWBM9R03SW7lhQUVT W15TeklSevXhLEHOJAGQbyujw1unh3UZFEvCssnB3a+4u1tKzxahhf7AMwOxYcEHIpJ+ct 6dDFX9IeNCPGOZ2fhTyew7+HZiaArt/AQbK+w5zqpxcFqwIYbOOv/G8IdoMqTz28AyvL2n 3ZluwuhtJfdHuHZwemdkPv6nquJz+kgZ2C5+7xWkgrK1qQLM/nSgrkxQflbc0cxmZEvbx2 e/GRvtOyRWhNR/slQY8NODCI985lpufRodrliaLBYIN3vT4Hp8XXsB+L0PAnlg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D263814 Bug ID: 263814 Summary: panic: GPF in cpu_search_highest if all cores in a domain are disabled Product: Base System Version: CURRENT Hardware: amd64 OS: Any Status: New Severity: Affects Some People Priority: --- Component: kern Assignee: bugs@FreeBSD.org Reporter: dgmorris@earthlink.net Attachment #233763 text/plain mime type: Created attachment 233763 --> https://bugs.freebsd.org/bugzilla/attachment.cgi?id=3D233763&action= =3Dedit 2 domain KVM node definition Looking into a hang at boot in intr_next_cpu() for a product based off of FreeBSD (and a bit behind Top Of Branch, I'm afraid) when performance folks wanted to check NUMA effects on an AWS i4i.32xlarge (https://aws.amazon.com/ec2/instance-types/i4i/) instance by disabling one socket via setting hint.lapic.X.disabled for all cores of the socket. (Note= : If there are better ways, I'm certainly open to hear them - I'll freely admit = this may have a "don't do that" aspect to it.. but raising the issue so that can= be considered). This will reproduce with a local KVM configured to present as a two domain = (at least) NUMA system. Attaching XML for virtual machine in case it is needed/helps. Please forgive any oddities within it, it is a local testing environment I hack around on a lot. When checking how Top-of-Branch upstream handles things (to see if we could just cherry-pick fixes back), we get a panic setting up scheduling instead (this is with a downloaded qcow2.xz for FreeBSD14-CURRENT, plugged in as the boot drive for the same KVM used to replicate): FreeBSD/SMP: Multiprocessor System Detected: 16 CPUs FreeBSD/SMP: 2 package(s) x 8 core(s) root@freebsd:~ # uname -a FreeBSD freebsd 14.0-CURRENT FreeBSD 14.0-CURRENT #0 main-n255198-1907e1c07= c3: Thu May 5 07:52:56 UTC 2022=20=20=20=20 root@releng1.nyi.freebsd.org:/usr/obj/usr/src/amd64.amd64/sys/GENERIC amd64 root@freebsd:~ # sysctl kern.vm_guest kern.vm_guest: kvm root@freebsd:~ # bash [root@freebsd ~]# for i in {8..15};do echo hint.lapic.$i.disabled=3D1; done= >> /boot/loader.conf [root@freebsd ~]# cat /boot/loader.conf hint.lapic.8.disabled=3D1 hint.lapic.9.disabled=3D1 hint.lapic.10.disabled=3D1 hint.lapic.11.disabled=3D1 hint.lapic.12.disabled=3D1 hint.lapic.13.disabled=3D1 hint.lapic.14.disabled=3D1 hint.lapic.15.disabled=3D1 At the reboot: Loading kernel... /boot/kernel/kernel text=3D0x189e30 text=3D0xe405f8 text=3D0x6b8114 data=3D= 0x140 data=3D0x1c8240+0x436dc0 0x8+0x198600+0x8+0x1b88b0 Loading configured modules... /etc/hostid size=3D0x25 /boot/entropy size=3D0x1000 staging 0xb4000000 (not copying) tramp 0xbdd00000 PT4 0xbdcf7000 Start @ 0xffffffff8038a000 ... EFI framebuffer information: addr, size 0xc4000000, 0x1d5000 dimensions 800 x 600 stride 800 masks 0x00ff0000, 0x0000ff00, 0x000000ff, 0xff000000 GDB: no debug ports present KDB: debugger backends: ddb KDB: current backend: ddb ---<>--- Copyright (c) 1992-2022 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 14.0-CURRENT #0 main-n255198-1907e1c07c3: Thu May 5 07:52:56 UTC 2= 022 root@releng1.nyi.freebsd.org:/usr/obj/usr/src/amd64.amd64/sys/GENERIC a= md64 FreeBSD clang version 13.0.0 (git@github.com:llvm/llvm-project.git llvmorg-13.0.0-0-gd7b669b3a303) WARNING: WITNESS option enabled, expect reduced performance. VT(efifb): resolution 800x600 CPU: Intel(R) Xeon(R) W-2295 CPU @ 3.00GHz (3000.05-MHz K8-class CPU) Origin=3D"GenuineIntel" Id=3D0x50657 Family=3D0x6 Model=3D0x55 Steppi= ng=3D7 =20 Features=3D0x1f83fbff =20 Features2=3D0xfffab223 AMD Features=3D0x2c100800 AMD Features2=3D0x121 Structured Extended Features=3D0xd19f47ab Structured Extended Features2=3D0x804 Structured Extended Features3=3D0xac000400 XSAVE Features=3D0xf IA32_ARCH_CAPS=3D0xeb AMD Extended Feature Extensions ID EBX=3D0x100d000 VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr Hypervisor: Origin =3D "KVMKVMKVM" real memory =3D 16777216000 (16000 MB) avail memory =3D 16193560576 (15443 MB) Event timer "LAPIC" quality 600 ACPI APIC Table: FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs FreeBSD/SMP: 2 package(s) x 8 core(s) FreeBSD/SMP Online: 1 package(s) x 8 core(s) random: registering fast source Intel Secure Key RNG random: fast provider: "Intel Secure Key RNG" random: unblocking device. ioapic0 irqs 0-23 Launching APs: 2 7 1 5 4 6 3 Fatal trap 9: general protection fault while in kernel mode cpuid =3D 2; apic id =3D 02 instruction pointer =3D 0x20:0xffffffff80c4426a stack pointer =3D 0x28:0xfffffe001b5aedc0 frame pointer =3D 0x28:0xfffffe001b5aee00 code segment =3D base 0x0, limit 0xfffff, type 0x1b =3D DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags =3D interrupt enabled, resume, IOPL =3D 0 current process =3D 11 (idle: cpu2) trap number =3D 9 panic: general protection fault cpuid =3D 2 time =3D 1 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe001b5ae= be0 vpanic() at vpanic+0x17f/frame 0xfffffe001b5aec30 panic() at panic+0x43/frame 0xfffffe001b5aec90 trap_fatal() at trap_fatal+0x385/frame 0xfffffe001b5aecf0 calltrap() at calltrap+0x8/frame 0xfffffe001b5aecf0 --- trap 0x9, rip =3D 0xffffffff80c4426a, rsp =3D 0xfffffe001b5aedc0, rbp = =3D 0xfffffe001b5aee00 --- cpu_search_highest() at cpu_search_highest+0xfa/frame 0xfffffe001b5aee00 cpu_search_highest() at cpu_search_highest+0x80/frame 0xfffffe001b5aee50 sched_idletd() at sched_idletd+0x377/frame 0xfffffe001b5aeef0 fork_exit() at fork_exit+0x80/frame 0xfffffe001b5aef30 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe001b5aef30 --- trap 0, rip =3D 0, rsp =3D 0, rbp =3D 0 --- KDB: enter: panic [ thread pid 11 tid 100005 ] Stopped at kdb_enter+0x32: movq $0,0x127c753(%rip) db> x/s version version: FreeBSD 14.0-CURRENT #0 main-n255198-1907e1c07c3: Thu May 5 07:52:56 UTC 2022\012=20=20=20 root@releng1.nyi.freebsd.org:/usr/obj/usr/src/amd64.amd64/sys/GENERIC\012 db> --=20 You are receiving this mail because: You are the assignee for the bug.=