Re: Can't shutdown FreeBSD domU

On Mon, Jan 10, 2022 at 04:58:40PM +0100, Roger Pau Monné wrote:
> On Sat, Dec 04, 2021 at 10:33:37AM +0100, Freddy DISSAUX wrote:
> > Hello,
> > 
> > My dom0 is a alpine linux 3.15 with xen 4.15:
> > 
> > dsx@xen>cat /etc/alpine-release 
> > 3.15.0
> > dsx@xen>apk info xen
> > xen-4.15.1-r2 description:
> > Xen hypervisor
> > 
> > xen-4.15.1-r2 webpage:
> > https://www.xenproject.org/
> > 
> > xen-4.15.1-r2 installed size:
> > 262 MiB
> > 
> > All domU are in pvh mode. I can shutdown domU alpine linux, domU netbsd
> > but not domU freebsd:
> > 
> > dsx@xen>doas xl create al30.cfg ; sleep 60 ; doas xl list ; doas xl shutdown --wait al30 ; sleep 2; doas xl list
> > Parsing config from al30.cfg
> > Name                                        ID   Mem VCPUsStateTime(s)
> > Domain-0                                     0  1022     4     r-----    2581.2
> > al30                                        66  1023     1     -b----       2.1
> > Shutting down domain 66
> > Waiting for 1 domains
> > Domain 66 has been shut down, reason code 0
> > Name                                        ID   Mem VCPUsStateTime(s)
> > Domain-0                                     0  1022     4     r-----    2581.7
> > dsx@xen>doas xl create nbsd20.cfg ; sleep 60 ; doas xl list ; doas xl shutdown --wait nbsd20 ; sleep 2 ; doas xl list
> > Parsing config from nbsd20.cfg
> > WARNING: msr_relaxed will be removed in future versions.
> > If it fixes an issue you are having please report to xen-devel@lists.xenproject.org.
> > Name                                        ID   Mem VCPUsStateTime(s)
> > Domain-0                                     0  1022     4     r-----    2583.2
> > nbsd20                                      67  2047     2     -b----       2.2
> > Shutting down domain 67
> > Waiting for 1 domains
> > Domain 67 has been shut down, reason code 0
> > Name                                        ID   Mem VCPUsStateTime(s)
> > Domain-0                                     0  1022     4     r-----    2583.7
> > dsx@xen>doas xl create fbsd40.cfg ; sleep 60 ; doas xl list ; doas xl shutdown --wait fbsd40 ; sleep 2 ; doas xl list
> > Parsing config from fbsd40.cfg
> > Name                                        ID   Mem VCPUsStateTime(s)
> > Domain-0                                     0  1019     4     r-----    2584.7
> > fbsd40                                      68  2048     2     -b----       4.5
> > Shutting down domain 68
> > Waiting for 1 domains
> > ^C
> > dsx@xen>doas xl list
> > Name                                        ID   Mem VCPUsStateTime(s)
> > Domain-0                                     0  1019     4     r-----    2585.0
> > fbsd40                                      68  2048     2     r-----     170.6
> > 
> > 
> > FreebSD domU config:
> > 
> > name     = "fbsd40"
> > maxvcpus = 2
> > memory   = 2048
> > vif      = [ 'bridge=br40' ]
> > type     = "pvh"
> > kernel   = "/home/dsx/kernel"
> > extra    = 'vfs.root.mountfrom=ufs:ufs/rootfs,boot_verbose=1'
> > disk     = [ 'format=raw, vdev=xvda, access=w, target=/home/dsx/FreeBSD-13.0-RELEASE-amd64.raw' ]
> > 
> > 
> > dmesg: http://download.bsdsx.fr/xen/dmesg.txt
> > 
> > Same problem with FreeBSD dom0:
> > 
> > dsx@ni3>uname -smr
> > FreeBSD 13.0-RELEASE-p4 amd64
> > dsx@ni3>pkg info 'xen-*'
> > xen-kernel-4.15.0_1
> > xen-tools-4.15.0_2
> > 
> > Any hint ?
> 
> This is a known problem. IIRC it's all caused by PVH having no ACPI
> SCI interrupt. Will try to find some time to look into it and
> hopefully find a solution.

The following patch should fix the issue, will test it doesn't cause
issues for HVM guests. Note that migration of PVH guests won't work
also.

Thanks, Roger.
---
diff --git a/sys/dev/xen/control/control.c b/sys/dev/xen/control/control.c
index 426682a8f2d9..b42b52a62411 100644
--- a/sys/dev/xen/control/control.c
+++ b/sys/dev/xen/control/control.c
@@ -347,17 +347,12 @@ xctrl_crash()
 }
 
 static void
-xen_pv_shutdown_final(void *arg, int howto)
+shutdown_final(void *arg, int howto)
 {
-	/*
-	 * Inform the hypervisor that shutdown is complete.
-	 * This is not necessary in HVM domains since Xen
-	 * emulates ACPI in that mode and FreeBSD's ACPI
-	 * support will request this transition.
-	 */
-	if (howto & (RB_HALT | RB_POWEROFF))
+	/* Inform the hypervisor that shutdown is complete. */
+	if (howto & RB_POWEROFF)
 		HYPERVISOR_shutdown(SHUTDOWN_poweroff);
-	else
+	else if (howto & RB_POWERCYCLE)
 		HYPERVISOR_shutdown(SHUTDOWN_reboot);
 }
 
@@ -453,9 +448,8 @@ xctrl_attach(device_t dev)
 	xctrl->xctrl_watch.max_pending = 1;
 	xs_register_watch(&xctrl->xctrl_watch);
 
-	if (xen_pv_shutdown_handler())
-		EVENTHANDLER_REGISTER(shutdown_final, xen_pv_shutdown_final, NULL,
-		                      SHUTDOWN_PRI_LAST);
+	EVENTHANDLER_REGISTER(shutdown_final, shutdown_final, NULL,
+	    SHUTDOWN_PRI_LAST);
 
 	return (0);
 }
diff --git a/sys/x86/include/xen/xen-os.h b/sys/x86/include/xen/xen-os.h
index 9d0964c6631d..655961da22fd 100644
--- a/sys/x86/include/xen/xen-os.h
+++ b/sys/x86/include/xen/xen-os.h
@@ -56,13 +56,6 @@ xen_has_percpu_evtchn(void)
 	return (!xen_hvm_domain() || xen_vector_callback_enabled);
 }
 
-static inline bool
-xen_pv_shutdown_handler(void)
-{
-
-	return (xen_pv_domain());
-}
-
 static inline bool
 xen_pv_disks_disabled(void)
 {