From nobody Thu Jul 21 18:47:42 2022 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4LphQR102cz4XHHy for ; Thu, 21 Jul 2022 18:47:43 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4LphQQ74Ssz3TNX for ; Thu, 21 Jul 2022 18:47:42 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4LphQQ65tXzXSJ for ; Thu, 21 Jul 2022 18:47:42 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 26LIlgHZ047430 for ; Thu, 21 Jul 2022 18:47:42 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 26LIlgHw047429 for virtualization@FreeBSD.org; Thu, 21 Jul 2022 18:47:42 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 265196] talos linux vms hang on reboot at the com ports, need to reboot the host to clear it up Date: Thu, 21 Jul 2022 18:47:42 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bhyve X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: jhb@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1658429263; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=iE+tcDg/xmos55WLzWfW00o8fzBOjOvY6JgHwBo1Ckw=; b=QCVNKHANiISRU8HKmTg9x3EpxdM2uN6zzZji0GruzZwU5FnGbLNXiwMT6EX8pRCp5nIE2u V70R+jdH69XtAghCmIwi/Fgh1CntKtRvVweXm9A9t4YIRgzte1/weLjDL9AViNBXEAIFMs pTjge26lDE3NSDlQ5hxxKGg1ULpNSmsROHOMuLCmTjoECGoKI03ICB3Gq8i3MKmrgnNUKk sWNUllwwE/0RHSaqqqUoufoqYKQa6I3ZgYbrh87h362fSNTMlmy0mtY1Uieo+o9e507OIE d2COJLf07i1bi51h9w2PsEeCR0sdHx63BeCM4vc37ANDf8y5dIz4iTjywB2uvA== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1658429263; a=rsa-sha256; cv=none; b=Mccotb+OnjGM1WbItPDxEkJ6lSghUs6uZQXamQVe5c2yxO3hxfwfcbNMh5N3rP1SMelXNI rf4eP5PtPFD6FwUAg5+cJYddPQbp8m2i+/qC99f7ninA+LYQRAm6Pyj3Oywb2nHIwwFkNz Eve8J66Io5jaEroXvylGXPHLJ5hLsWf1/aPHLiGT4TWfojleqV6K7g0mJzbR89p+fst5US EUqEJCTFYNVJ8DYv4VtE/V4gDjQE0Hw273SSANRV2qOrWYBeRVy4AUWqpnpEhZAoFJaJF/ UBUwgNjtlxg1BDb+jqSRssycBVC86mSqPseFROYdTPEybGHIN8CTi5pEeAlOiQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D265196 --- Comment #23 from John Baldwin --- So in the case that bhyvectl hangs, from the procstat -kk output, bhyvectl = is waiting because some other process has the /dev/vmm/ file still ope= n.=20 For that case, you can try using 'sudo procstat -af | grep ' to see which processes still have it open preventing bhyvectl from exiting. For the case where you had a bhyve exit of 4 and an error of 'vm_open: No s= uch file or directory', that may be a race between the async destroy used on 13= .1 for bhyve (but since fixed in 14 and stable/13 so that bhyvectl will now sl= eep waiting for the --destroy request to end before returning). They return value of 134 is due to abort() and is the triple fault case you have logs of in bhyve.log. A triple fault isn't a crash of bhyve, that is a bit of an old-school way to reboot an x86 computer. It's perhaps a bit odd that a Linux guest would use that to reboot vs more conventional means.=20 However, you shouldn't have to reboot the host machine just because the gue= st exits due to a triple fault. You should be able to restart the VM again without rebooting the host. Here I use "host" to mean the FreeBSD machine running bhyve VMs. Looking again, it seems like the talos upgrade is perhaps trying to use kex= ec to upgrade instead of a real reboot, and that the second Linux kernel is perhaps crashing (and not trying to use a triple fault to reboot). Given t= he turn around times for VM booting, you don't really need kexec for VMs. If = you want to debug this you will have to debug the crash that happens in the sec= ond Linux kernel. It may be that there is something bhyve isn't emulating quite right that results in the triple fault, but it will be hard to know what th= at is from the bhyve side. I would see if there's a way to configure talos to= not use kexec and just use "plain" reboots for upgrades instead. --=20 You are receiving this mail because: You are the assignee for the bug.=