Extreme console latency during disk IO (8.0-RC1, previous releases also affected according to others)

[Robert Watson]

On Tue, 13 Oct 2009, Ivan Voras wrote:

> Thomas Backman wrote:
>> I'm copying this over from the freebsd-performance list, as I'm looking for 
>> a few more opinions - not on the problems *I* am having, but rather to 
>> check whether the problem is universal or not, and if not, find a possible 
>> common factor. In other words: I want to hear about your experiences, *good 
>> or bad*!
>> 
>> Here's the original thread (not from the beginning, though): 
>> http://lists.freebsd.org/pipermail/freebsd-performance/2009-October/003843.html
>> 
>> Long story short, my version: when the disk is stressed hard enough, 
>> console IO becomes COMPLETELY unbearable. 10+ seconds to switch between 
>> windows in screen(1), running (or even typing) simple commands, etc. This 
>> happens both via SSH and the serial console.
>
> Hmm, this looks familiar - I've noticed it before on the physical (VGA) 
> console and I notice it all the time under VMWare. It sort of looks like 
> disk IO really blocks network IO in this case - I use the VMs over ssh.

Real hardware and virtual hardware have vastly different performance 
properties, so I'd be careful not to assume that the issue described by the 
original reporter and the issue you're experiencing are the same.  In our 
kernel, low level network protocols will essentially always take precedence 
over disk I/O activity.  So on face value "disk IO really blocks network IO" 
is highly unlikely.

There are two much more likely possibilities: (1) poor VM implementation 
causes the virtual CPU to be suspended behind synchronous host OS I/O or (2) 
the network stack is running fine but the interactive user application is 
getting I/O or locks scheduled behind a bulk process.

A useful diagnostic here is to compare the behavior of three kinds of network 
latency tests:

(1) ping from the host OS to the guest OS
(2) netperf TCP_RR from the host OS to the guest OS
(3) ssh interactive latency

If (1) is highly variable during I/O, it's almost certainly a property of the 
VM technology you're using, and there's nought to be done about it in the 
guest OS.

If (2) but not (1) is highly variable, it may well be a scheduling issue, 
although under high memory pressure you couldn't rule out paging out of 
netserver pages/etc causing latency.

If (3) but not (1) or (2) is highly variable, it's most likely an I/O 
scheduling issue, perhaps caused by priority inversion on lockmgr locks on a 
vnode, disk I/O scheduling leading to starvation, etc.

Robert