From nobody Tue Jan 11 10:07:28 2022 X-Original-To: xen@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id CCAA3193D301 for ; Tue, 11 Jan 2022 10:07:30 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4JY5wL5Qqqz4k2X; Tue, 11 Jan 2022 10:07:30 +0000 (UTC) (envelope-from avg@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1641895650; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=oKlvkHgVLxDqQouW77Ct9xqcuNrG693kGpZ1R+9cHns=; b=p34C0u9ZlfIua+MCY/4jpbdFdttgNk8evIKga6HEYW3m3UbuiRX+IAzd48inMUx87DHD79 /mK9dLldHK0/JqijDnI8FLEnj5NpgbHCwAHdbjS0d5QR37ZPx1F88qOREgZTrZd7bjC4JU aV2taMs6XqIDztkQCXPGGmWLBQs6N64Bl2kpfHhKA8l7GD5ZycAe+MABqHAKR2KmHWVBhX lHYmIdWnKWzJuiRrF9NgSNlIdYhjTzbTtEbGeFVHS70RVthU/T7DjxWcy+/4lz32BHShD6 ojwA9Z/uC+SmsChoG0q+JsTotT/ryBd33J3DWN96wrB0kfqQW9ld2BBH3RE3vg== Received: from [192.168.0.88] (unknown [195.64.148.76]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) (Authenticated sender: avg/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id 3F1D23279; Tue, 11 Jan 2022 10:07:30 +0000 (UTC) (envelope-from avg@FreeBSD.org) Message-ID: Date: Tue, 11 Jan 2022 12:07:28 +0200 List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-xen List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-xen@freebsd.org X-BeenThere: freebsd-xen@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:91.0) Gecko/20100101 Firefox/91.0 Thunderbird/91.4.1 Subject: Re: VCPUOP_send_nmi returns -38 Content-Language: en-US From: Andriy Gapon To: xen@freebsd.org Cc: =?UTF-8?Q?Roger_Pau_Monn=c3=a9?= References: <3f9f173b-40b9-0180-404d-52fa56dde45f@FreeBSD.org> In-Reply-To: <3f9f173b-40b9-0180-404d-52fa56dde45f@FreeBSD.org> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1641895650; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=oKlvkHgVLxDqQouW77Ct9xqcuNrG693kGpZ1R+9cHns=; b=RAiVKwJvY/IjwIpt3wKBc1QnDUdm0vDDqytJah48N82emIppPRHg/ET03r3hE9zczQAsMv Wvp81fye/hI077Zwm3Uz15IY04cLaG0VHjvGXGA8xlZquoTgK5HUo8xuZtwf5OcwLe2g92 FZt2Y5tvoaaCA/nLEWgmPXL0cg/2cGHzwzQI29cIu73+JSZGwfDYJi2a8txhemp9za4ho9 oR+FZlyVfpBzoJ2vKK2URl8fGt7AVV0hmeFRDYlk/qmFlDm1J2cNnYWNuwHIv7Mie2gmp4 kzjLeesQspanvZWR2/78nauntq3kxEhkNuJ2lttwsWUT91FeedVQZJZGiEf3cA== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1641895650; a=rsa-sha256; cv=none; b=Z4qHQhsbrd8YWqh/HIc2ieIa3rv52voKyn9zPVrW8r6jk3t60Bo5v/gz/e9rpwxisuEaeV FhwMUklGubfnbhNGWIMrRrV60N9IkHuWOnPG9bGBRnkEBS73OOWcjryTnJwk/TXkaWHcc7 csWhyJC8YZzIVzrudH8Hl3dDyE7qU0LIzMgmtt1sK/LfjZayRTxbeGVsstM0kG5PT9+6GP KScYLyJWhBljwYXcY5FwQ1qP/hPWpc1XFO1eNaDkXWlfc2LFCxAxjqRaT2lcpvzjNcibeR vi6XJxVHsX0b+BwkpUicK43i4he1rIBqFrQhXXkQ79gV4vAIZUKpZap6tP7b9g== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N On 11/01/2022 11:50, Andriy Gapon wrote: > > Recently I got a report of crashes related to using procstat -k on one of our > systems.  The system runs FreeBSD 12.2 on an AWS Xen-based instance (can get > more specifics about it later). The instance type is t2.large. Here are all lines from verbose boot that mention Xen: XEN: Hypervisor version 4.2 detected. Disabling MSI-X interrupt migration due to Xen hypervisor bug. XEN: disabling emulated disks XEN: disabling emulated nics Hypervisor: Origin = "XenVMMXenVMM" x2APIC available but disabled due to running under XEN ACPI APIC Table: Xen interrupts: unable to register PIRQ EOI map Xen interrupt system initialized ACPI: RSDP 0x00000000000EA020 000024 (v02 Xen ) ACPI: XSDT 0x00000000FC00E2A0 000054 (v01 Xen HVM 00000000 HVML 00000000) ACPI: FACP 0x00000000FC00DF60 0000F4 (v04 Xen HVM 00000000 HVML 00000000) ACPI: DSDT 0x00000000FC0021C0 00BD19 (v02 Xen HVM 00000000 INTL 20090123) ACPI: APIC 0x00000000FC00E060 0000D8 (v02 Xen HVM 00000000 HVML 00000000) ACPI: HPET 0x00000000FC00E1B0 000038 (v01 Xen HVM 00000000 HVML 00000000) ACPI: WAET 0x00000000FC00E1F0 000028 (v01 Xen HVM 00000000 HVML 00000000) ACPI: SSDT 0x00000000FC00E220 000031 (v02 Xen HVM 00000000 INTL 20090123) ACPI: SSDT 0x00000000FC00E260 000033 (v02 Xen HVM 00000000 INTL 20090123) acpi0: on motherboard xenpci0: port 0xc000-0xc0ff mem 0xf2000000-0xf2ffffff irq 28 at device 3.0 on pci0 xenpv0: on motherboard granttable0: on xenpv0 xen_et0: on xenpv0 Event timer "XENTIMER" frequency 1000000000 Hz quality 950 Timecounter "XENTIMER" frequency 1000000000 Hz quality 950 xen_et0: registered as a time-of-day clock, resolution 0.000001s xenstore0: on xenpv0 xsd_dev0: on xenpv0 evtchn0: on xenpv0 privcmd0: on xenpv0 gntdev0: on xenpv0 debug0: on xenpv0 xenballoon0: on xenstore0 on xenstore0 xs_dev0: on xenstore0 xenbusb_front0: on xenstore0 xn0: at device/vif/0 on xenbusb_front0 xenbusb_back0: on xenstore0 xbd0: 40960MB at device/vbd/768 on xenbusb_front0 xbd6: 141312MB at device/vbd/51808 on xenbusb_front0 xbd1: 204800MB at device/vbd/51728 on xenbusb_front0 xbd2: 204800MB at device/vbd/51744 on xenbusb_front0 xen_et0: providing initial system time > It immediately reminded me of an older issue (Subject: Xen (HVM) and NMI) where > the root cause was that NMIs were delivered as regular interrupts. > But 12.2 has the newer code that delivers NMIs as NMIs. > > After some investigation it became evident that NMIs are not delivered at all. > I modified send_nmi() in sys/x86/xen/xen_apic.c to capture and report errors > from HYPERVISOR_vcpu_op(VCPUOP_send_nmi) calls. > That revealed that the call returns -38 which appears to mean ENOSYS. > > I am not sure what that could mean. > Perhaps NMI is somehow disabled in the Xen configuration (for that specific > instance type)? > I am out of better ideas. > > P.S. > It appears that FreeBSD does not expect that an IPI, including NMI, can fail. > So, there is no way to propagate the error to callers. > I think that we could either printf it or, perhaps, even panic on such a failure. > -- Andriy Gapon