NFS stalling on 8.1-STABLE

[Mark Morley]

On Sun, 15 Aug 2010 23:35:50 -0700 Jeremy Chadwick <freebsd at jdc.parodius.com> wrote:
>On Thu, Aug 12, 2010 at 10:35:49AM -0700, Mark Morley wrote:
>> I have five front end web servers that all mount their content from
>> the same server via NFS.  If I stress the link on any one of the
>> machines (eg: copy a large directory with a lot of files to/from the
>> mounted file system) the client will pause.  That is, all processes
>> trying to access that mount will freeze.  The log files with hundreds
>> or thousands of nfs server not responding / is alive again messages.
>> After 60 seconds it returns to normal, unless the load is still there
>> in which case it continues to pause.
>>
>> This has only started happening since I upgraded the client machines
>> to 8.1-STABLE (previously four of them were 8.0 and one was 7.3).  The
>> server is 7.1-RELEASE-p11.  No other changes have taken place in terms
>> of hardware or software or mount options, etc.
>>
>> All nics involved are gigabit em cards, and they are on a private
>> network (web access to the boxes is via an external interface).
>
>Are there any indications in dmesg that the NIC is responsible, e.g.
>interface down/up, etc.?

No, nothing like that.

>Does switching to UDP-based NFS solve the problem for you?

Trying that now for the past 24 hours or so.  Four of the machine seem ok so far, but the fifth one has started dropping the mount entirely.  Access to it gives an "Input / output error" message.  Forcing a dismount and remounting brings it back.

>What OS version (uname -a) and NIC are used on the NFS server?

FreeBSD xxx 7.1-RELEASE-p11 FreeBSD 7.1-RELEASE-p11 #0: Wed May 26 03:20:59 PDT 2010
root at xxx:/usr/obj/usr/src/sys/CUSTOM  i386

NICs are em

>Can you please provide the following output from one of the client
>machines running 8.1-STABLE with gigE em(4)?  You can X-out machine
>names, MAC addresses, and IP addresses/netblocks if need be.
>
>* uname -a

FreeBSD xxx 8.1-STABLE FreeBSD 8.1-STABLE #0: Tue Jul 27 16:27:44 PDT 2010
root at xxx:/usr/obj/usr/src/sys/CUSTOM  amd64

>* ifconfig emX  (where X is the interface number which would be
>  used for NFS)

em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC>
ether 00:0e:0c:85:d5:0d
inet 192.168.1.30 netmask 0xffffff00 broadcast 192.168.1.255
media: Ethernet 1000baseT <full-duplex>
status: active

>* netstat -idn -I emX

Name    Mtu Network       Address              Ipkts Ierrs Idrop    Opkts Oerrs  Coll Drop
em0    1500 <Link#1>      00:0e:0c:85:d5:0d 39913814     2     0 39949943     0     0    0
em0    1500 192.168.1.0/2 192.168.1.30      39944016     -     - 39949664     -     -    -


>* pciconf -lvc  (provide only the data for emX please)

em0 at pci0:1:6:0: class=0x020000 card=0x13768086 chip=0x107c8086 rev=0x05 hdr=0x00
vendor     = 'Intel Corporation'
device     = 'Gigabit Ethernet Controller (Copper) rev 5 (82541PI)'
class      = network
subclass   = ethernet
cap 01[dc] = powerspec 2  supports D0 D3  current D0
cap 07[e4] = PCI-X supports 2048 burst read, 1 split transaction


>* vmstat -i

interrupt                          total       rate
irq1: atkbd0                         239          0
irq16: em0                      36746591        883
irq18: em1                      12658607        304
irq21: ohci0                           2          0
irq22: ehci0                      528002         12
irq23: atapci1                   2334936         56
cpu0: timer                     83207296       2000
cpu1: timer                     83207289       2000
Total                          218682962       5256

>* sysctl hw.pci

hw.pci.usb_early_takeover: 1
hw.pci.honor_msi_blacklist: 1
hw.pci.enable_msix: 1
hw.pci.enable_msi: 1
hw.pci.do_power_resume: 1
hw.pci.do_power_nodriver: 0
hw.pci.enable_io_modes: 1
hw.pci.default_vgapci_unit: -1
hw.pci.host_mem_start: 2147483648
hw.pci.mcfg: 1

>* As root, run "sysctl dev.em.X.stats=1" then do "dmesg" and
>  provide the output for NIC statistics (will start with "emX:")

em0: Excessive collisions = 0
em0: Sequence errors = 0
em0: Defer count = 52
em0: Missed Packets = 0
em0: Receive No Buffers = 0
em0: Receive Length Errors = 0
em0: Receive errors = 1
em0: Crc errors = 1
em0: Alignment errors = 0
em0: Collision/Carrier extension errors = 0
em0: RX overruns = 0
em0: watchdog timeouts = 0
em0: RX MSIX IRQ = 0 TX MSIX IRQ = 0 LINK MSIX IRQ = 0
em0: XON Rcvd = 54
em0: XON Xmtd = 0
em0: XOFF Rcvd = 54
em0: XOFF Xmtd = 0
em0: Good Packets Rcvd = 39915088
em0: Good Packets Xmtd = 39951839

Mark