From nobody Sat Nov 27 07:40:57 2021 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id A3F0F18B7E56 for ; Sat, 27 Nov 2021 07:41:08 +0000 (UTC) (envelope-from david@crossfamilyweb.com) Received: from mail.dcrosstech.com (rrcs-24-97-5-250.nys.biz.rr.com [24.97.5.250]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "mail.dcrosstech.com", Issuer "DCrossTech.com LLC CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4J1NpC74q6z3nYG for ; Sat, 27 Nov 2021 07:41:07 +0000 (UTC) (envelope-from david@crossfamilyweb.com) X-Virus-Scanned: amavisd-new at dcrosstech.com Received: from winry.priv.dcrosstech.com (d130.office.dcrosstech.com [10.1.12.130]) (authenticated bits=0) by mail.dcrosstech.com (8.15.2/8.15.2) with ESMTPSA id 1AR7evCE018586 (version=TLSv1.3 cipher=TLS_AES_128_GCM_SHA256 bits=128 verify=NO) for ; Sat, 27 Nov 2021 07:40:58 GMT (envelope-from david@crossfamilyweb.com) X-Authentication-Warning: mail.priv.dcrosstech.com: Host d130.office.dcrosstech.com [10.1.12.130] claimed to be winry.priv.dcrosstech.com To: freebsd-hackers@freebsd.org From: "David E. Cross" Subject: bhyve -D not cleaning up after itself Message-ID: Date: Sat, 27 Nov 2021 02:40:57 -0500 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:78.0) Gecko/20100101 Thunderbird/78.11.0 List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="------------A8B7A2064EAB048E5235D381" Content-Language: en-US X-Rspamd-Queue-Id: 4J1NpC74q6z3nYG X-Spamd-Bar: + Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of david@crossfamilyweb.com designates 24.97.5.250 as permitted sender) smtp.mailfrom=david@crossfamilyweb.com X-Spamd-Result: default: False [1.92 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; ARC_NA(0.00)[]; FREEFALL_USER(0.00)[david]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; HAS_XAW(0.00)[]; TO_DN_NONE(0.00)[]; NEURAL_SPAM_MEDIUM(1.00)[1.000]; RCPT_COUNT_ONE(0.00)[1]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; R_SPF_ALLOW(-0.20)[+mx]; NEURAL_SPAM_LONG(1.00)[1.000]; DMARC_NA(0.00)[crossfamilyweb.com]; NEURAL_SPAM_SHORT(0.22)[0.220]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; ASN(0.00)[asn:11351, ipnet:24.97.0.0/16, country:US]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[] X-ThisMailContainsUnwantedMimeParts: Y This is a multi-part message in MIME format. --------------A8B7A2064EAB048E5235D381 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit I have noticed for awhile that bhyve -D doesn't seem to actually do what is claimed  (to destroy a VM on guest initiated power-off).  This evening I decided to ktrace it to see if I was just not getting something about how this was supposed to work, and found:  68613 vcpu 0   CALL __sysctlbyname(0x1ebcdb20a133,0xe,0,0,0x1ebce4ba60f0,0x9)  68613 vcpu 0   SCTL "hw.vmm.destroy"  68613 vcpu 0   RET   __sysctlbyname -1 errno 1 Operation not permitted  68613 vcpu 0   CALL  exit(0x1) Reading quickly the kernel code for vm_destroy(), I find 2 candidates: static int vmm_priv_check(struct ucred *ucred) {         if (jailed(ucred) &&             !(ucred->cr_prison->pr_allow & pr_allow_flag))                 return (EPERM);         return (0); } This doesn't seem to be it, my process is not jailed. That leads to the only other (I think) call in sysctl_vmm_destroy that could return EPERM: error = sysctl_handle_string(oidp, buf, buflen, req); But I am just not seeing it.  Also this EXACT same call works from the context of bhyvectl --vm=FOO --destroy, run from the same shell as the bhyve process that just terminated.  Is the 'ctx' somehow incorrect in bhyve?  I is used earlier in that function, so I am assuming it is right? Any help appreciated! --------------A8B7A2064EAB048E5235D381--