g_vfs_done

[Carole Macheret]

Hi Scott,

Thanks a lot for your advice, we have finally run some tests with the following setting changed: kern.cam.da.retry_count=100 (in /etc/sysctl.conf)
Now the FreeBSD virtual machines doesn't freeze anymore after loosing the disks during the IPstor failover.

Best regards

Carole Macheret


>>> Scott Long <scottl at samsco.org> 14.08.2008 19:07 >>>
Carole Macheret wrote:
> Hello,
> 
> We are using FreeBSD 7.0-RELEASE #1 running Squid and Zabbix on vmware ESX 3.0.2 and our vmware ESX servers access our SAN through IpStor cluster (Storage virtualization and mirroring). 
> 
> We have 2 storages (EVA 6100) and the IpStor solution allows us to mirror disks on both EVAs.
> 
> We have a problem with both the Zabbix and Squid FreeBSD virtual machines, when the virtual machine is loosing its disks (EVA controller reboot or ipstor cluster failover), we have several "g_vfs_done() : da1s1d[WRITE(offset=2312431234, length=12453)] error= 5" errors then the host is definitively frozen. The disk loss lasts 1-5 seconds. Windows virtual machines do freeze during the loss then continue working. On Windows we had to specify a longer timeout for local disk in registry.
> 
> Does anybody has an idea what could be tuned to avoid this problem ?
> 
> Attached you can find the dmesg and a screenshot of the g_vfs_done error...
> 
> Thanks in advance for your help
> 

So the virtual disks that the FreeBSD images are using in VMWare are on
an IpStor, and those periodically go away, yes?  What's probably
happening is that the VMWare host is triggering an event in the FreeBSD
client VM that essentially is making the virtual disks go away.  Inside
the FreeBSD VM, the SCSI layer tries to talk to the disk and gets a
selection timeout since the disk is no longer there.  It doesn't know
that this is a temporary state, and it declares the I/O as failed.  At
that point, the BSD VM gets upset and everything gets bad.

There is a property called kern.cam.da.default_timeout.  It's set to
60 seconds, but I don't think that it will help you in this case, since
it's likely that the i/o is failing because of a selection timeout, not
because the virtual disk is slow in completing the i/o.  The
kern.cam.da.retry_count property is set to 5, and changing it might help
since it might be able to force enough retries to give time for the
virtual disk to come back.  Try the following command on a running
system:

sysctl kern.cam.da.retry_count=100

This will allow for about 25 seconds worth of retries (a selection
attempt takes 250ms, so you'll get about 4 retries per second).  If
this doesn't work, try configuring VMWare to give you a serial console
that you can capture on the host, then set bootverbose during boot and
send me the log once the problem happens.

Scott