[ATA] and re(4) stability issues

[Pyun YongHyeon]

On Wed, Dec 10, 2008 at 09:59:35AM +0100, Victor Balada Diaz wrote:
 > On Wed, Dec 10, 2008 at 03:12:26PM +0900, Pyun YongHyeon wrote:
 > > On Tue, Dec 09, 2008 at 07:52:37PM +0100, Victor Balada Diaz wrote:
 > >  > Hello,
 > >  > 
 > >  > I got various machines[1] at hetzner.de and I've been having problems
 > >  > with interrupts on FreeBSD 7.0 and now FreeBSD 7.1 -BETA2 in amd64. I've
 > >  > been trying to narrow the problem so someone more knowledgeable than me
 > >  > is able to fix it. This mail is an other attempt to ask a question
 > >  > with regards ATA code to see if this time i got something.
 > >  > 
 > >  > For the ones that don't actually know what happened:
 > >  > 
 > >  > With FreeBSD 7.0 -RELEASE for amd64 and default kernel
 > >  > the system shared re0 interrupt with OHCI and this caused
 > >  > re(4) to corrupt packets and create interrupt storms. Tried
 > > 
 > > re(4) in 7.0-RELEASE had bus_dma(9) bug which could be easily
 > > triggered on systems with > 4GB memory. But I dont' know whether
 > > this is related with interrupt storms.
 > > 
 > >  > updating to 7.1 -BETA2 and still had some problems with it.
 > >  > 
 > >  > I've opened the PR kern/128287[2] and Remko quickly answered
 > >  > with a workaround: that workaround was removing USB support from
 > >  > my kernel. I did it and re(4) wasn't sharing interrupts anylonger,
 > >  > and the interrupt storms were gone. Now sometime later the interface
 > >  > goes up and down from time to time, but less often. Also sometimes
 > >  > the machine losts the network interface but continues to work.
 > >  > 
 > > 
 > > It seems that your controller supports MSI so you can set a tunable
 > > hw.re.msi_disable to 0 to enable MSI. With MSI you can remove
 > > interrupt sharing(e.g. add hw.re.msi_disable="0" to
 > > /boot/loader.conf file.) However there were several issues on re(4)
 > > w.r.t MSI so it was off by default.
 > 
 > This is undocumented and with sysctl -a i can't find the tunable. Is this
 > a HEAD feature or it's also in 7.1 -BETA2? Should i add

Yeah it's an undocmented feature. But most drivers written by me
have similar kobs. Both HEAD and stable/7 including 7.1 BETA2 have
the tunable.

 > hw.re_msi_disable="0" to /boot/loader.conf?
   ^^^^^^^^^^^^^^^^^^^^^
   Shoule be hw.re.msi_disable="0"
 > 

Yes, just add it to /boot/loader.conf. Note, you should not disable
system-wide MSI control(e.g. hw.pci.enable_msi == 1).

 > This was sharing interrupt with USB, does USB need any special MSI handling
 > or with re using MSI is enough to not share the interrupt?

If re(4) can use MSI, you don't need to worry about interrupt
sharing with USB. Check the output of "vmstat -i". You normally get
an irq256 or higher for MSI enabled driver.

 > 
 > 
 > > 
 > >  > I know it continues to work because some days later i can see that
 > >  > it tried to deliver the status reports but was unable to resolve the
 > >  > aliases hostnames. I can't ping the machine and i know the network
 > >  > is OK. If i reboot the machine everything is working again.
 > >  > 
 > > 
 > > Recently I've made small changes to re(4) which may help to detect
 > > link state change event. Would you try re(4) in HEAD?
 > 
 > Can i just drop HEAD's /stable/7/sys/dev/re/ in -STABLE and test that

Yes, you can. It should build without problems. Just replace re(4) on
stable/7 with HEAD version.

 > or do i need to test the whole HEAD kernel?
 > 

No you don't have to that.

-- 
Regards,
Pyun YongHyeon