Intel em0: watchdog timeout

[Kirk Davis]

Hi,
	I have a FreeBSD server running Quagga as a BGP router.  It has
a number of interfaces in it both bce and em.  The most heavily used
interfaces are starting to give me watchdog timeout errors just in the
last week.  We normally sustain about 300Mb/s on both of these
interfaces but in the last week this now up to 380Mb/s.

	This is a Intel Pro/1000 PT dual interface PCI-E card. There is
two of them in the server.  The server is a Dell 2950

	Searching the mailing list and checking on google has not turned
up much. Since this is our main router it is difficult to test with.  I
have seen one message that suggests trying to set hw.em.rxd=1024 and
hw.em.txd=1024 in loader.conf and another that suggested turning off
but none this has not made any difference.

	The odd thing is that this just started.  This box has been up
and running fine for a while. The only thing different on our network
had been an increase in the bandwidth.

	Any idea where I go from here to trouble shoot this?

# uname -a
FreeBSD inet-gw.epsb.ca 7.1-STABLE FreeBSD 7.1-STABLE #3: Mon Mar 23
16:08:53 MDT 2009
root at inet-gw-test.epsb.ca:/usr/obj/usr/src/sys/DELL2950  amd64

# tail /var/log/messages
Feb 19 12:26:04 inet-gw kernel: em0: watchdog timeout -- resetting
Feb 19 12:26:04 inet-gw kernel: em0: link state changed to DOWN
Feb 19 12:26:07 inet-gw kernel: em0: link state changed to UP
Feb 19 12:26:08 inet-gw kernel: em0: link state changed to DOWN
Feb 19 12:26:10 inet-gw kernel: em0: link state changed to UP
Feb 19 14:44:20 inet-gw kernel: em0: watchdog timeout -- resetting
Feb 19 14:44:20 inet-gw kernel: em0: link state changed to DOWN
Feb 19 14:44:23 inet-gw kernel: em0: link state changed to UP
Feb 19 15:05:03 inet-gw kernel: em2: watchdog timeout -- resetting
Feb 19 15:05:03 inet-gw kernel: em2: link state changed to DOWN
Feb 19 15:05:05 inet-gw kernel: em2: link state changed to UP
Feb 19 15:07:39 inet-gw kernel: em2: watchdog timeout -- resetting
Feb 19 15:07:39 inet-gw kernel: em2: link state changed to DOWN
Feb 19 15:07:42 inet-gw kernel: em2: link state changed to UP

# from /var/run/dmesg.boot
em0: <Intel(R) PRO/1000 Network Connection 6.9.6> port 0xdce0-0xdcff mem
0xd5ee0000-0xd5efffff,0xd5ec0000-0xd5edffff irq 17 at device 0.0 on pci8
em0: Using MSI interrupt
em0: [FILTER]
em0: Ethernet address: 00:15:17:a6:ae:94
em2: <Intel(R) PRO/1000 Network Connection 6.9.6> port 0xcce0-0xccff mem
0xde3e0000-0xde3fffff,0xde3c0000-0xde3dffff irq 16 at device 0.0 on
pci10
em2: Using MSI interrupt
em2: [FILTER]
em2: Ethernet address: 00:15:17:a6:af:d6

# pciconf -lv
em0 at pci0:8:0:0: class=0x020000 card=0x135e8086 chip=0x105e8086 rev=0x06
hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'PRO/1000 PT'
    class      = network
    subclass   = ethernet
em2 at pci0:10:0:0:        class=0x020000 card=0x135e8086 chip=0x105e8086
rev=0x06 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = 'PRO/1000 PT'
    class      = network
    subclass   = ethernet

# netstat -bdhI em2 2
            input          (em2)           output
   packets  errs      bytes    packets  errs      bytes colls drops
       65K     0        72M        51K     0       9.4M     0     0 
       69K     0        78M        52K     0       8.5M     0     0 
       76K     0        88M        55K     0        11M     0     0 
       74K     0        85M        54K     0        10M     0     0 
       78K     0        91M        56K     0       9.0M     0     0 
       75K     0        86M        54K     0       8.7M     0     0 
       74K     0        85M        54K     0       9.2M     0     0 
       75K     0        86M        56K     0        10M     0     0 
       78K     0        88M        55K     0        12M     0     0 
       78K     0        90M        58K     0        12M     0     0 
       76K     0        87M        54K     0        10M     0     0 
       79K     0        91M        56K     0        10M     0     0 


---- Kirk
------------------------------------------------------------------------
--------
Kirk Davis
Senior Network Analyst, ITS
Edmonton Public Schools
One Kingsway Ave. 
Edmonton, Alberta, Canada
T5H 4G9