Re: Can we "pause" in loader and ddb>?

On Mon, Nov 21, 2022 at 09:37:36PM +0200, Konstantin Belousov wrote:
> On Mon, Nov 21, 2022 at 02:06:47PM -0500, Mark Johnston wrote:
> > On Mon, Nov 21, 2022 at 06:42:44PM +0000, Bjoern A. Zeeb wrote:
> > > Hi,
> > > 
> > > having a VM sitting (a) in loader prompt or (b) in ddb> makes the fans
> > > go loud on a test-laptop which makes one wonder if we can avoid that.
> > > Note -- I assume also on real HW though that's less likely observered
> > > here.
> > > 
> > > For example I am sitting in a 4 vCPU bhyve current in ddb> and two
> > > threads are at 100% on the base system.  Do we need to heat up the
> > > planet doing that or are there alternatives?

I forgot to mention: in this situation my "solution" is to attach gdb to
the running VM, which I usually want to do anyway.  This causes all vCPU
threads to pause.

> > > I haven't looke dta the
> > > code in ages and I assume we need to poll in these situations on the
> > > console and for interactivity often enough?  Can we "pause"?   And why
> > > would two [v]CPUs be at 100% and not just one as I would expect all but
> > > one to be stopped?
> > 
> > When DDB is active, one CPU is going to be spinning in cncheckc()
> > waiting for input, and the others will be in cpustop_handler().  For the
> > latter, there is an option to use MWAIT/MONITOR to pause the CPU instead
> > of calling pause in a loop, but bhyve hides the MWAIT/MONITOR CPU
> > capability from guests (I'm not sure exactly why), so they just execute
> > "pause" in a loop, which won't help the apparent CPU usage.
> We do not want the guest thread eating CPU time in MWAIT, it is the host
> scheduler that should decide that there is nothing to do and schedule
> idle thread.
> 
> In other words, MWAIT should be emulated, perhaps by revoking the monitoring
> page on MONITOR, and waking up the MWAIT-ing thread on write to the armed
> range.
> 
> Additional complication is that the range to arm is usually equal to the cache
> line, doing dumb emulation by interpreting any write to the monitor page
> might be too harsh.
> 
> There is also non-trivial cooperation with atomic interrupt delivery.
> 
> These complexities are perhaps the reason to not enable M* for guests.

I see.  Maybe instead we could allow stopped CPUs to execute "hlt" with
interrupts disabled, and when exiting DDB, the last CPU resumes all of
the others by raising NMIs.  Then only one vCPU thread will be consuming
host CPU time when in DDB.

> > Note that when the kernel panics, cpustop_handler() will execute "hlt",
> > which should result in a vmexit if you started bhyve with -H, and you'll
> > see only one vCPU thread consuming 100% when sitting at the DDB prompt.
> > But when you enter DDB without a panic, we need some mechanism to resume
> > the stopped CPUs when exiting DDB.
>