Bhyve: Investigating poor guest performance when host is busy

[Stefan Andritoiu]

Problem intro:
- Started a make -j 20 buildworld on the host to peg the cpus
- Started a 2 and 4 vcpu FreeBSD guest and time how long it takes to boot up
- Started a 1 vcpu FreeBSD guest and time how long it takes to boot
up. It is always much faster than the 2 or 4 vcpus.

My suspicion:
I suspect that the ULE scheduler makes some bad decisions with
scheduling because it is not aware that bhyve threads are actually
running vcpus on top of them, and Lock-holder-preemption may occur.
The classic way to solve this is with gang scheduling to ensure that
all vcpus are running simultaneously.

1. Can anyone tell me what the cause might be? What may be happening?
2. Do you know if there is currently any work in investigation this
problem? Or anything related?
3. Is Gang Scheduling or Coscheduling implemented in FreeBSD?
4. Do you know of any other solution to this kind of problem?
5. Can you recommend me any papers/videos/links in anyway related to this?

I'm sorry if the question seem vague. But I don't understand the
problem so well either and I'm relatively new to FreeBSD