Intel 82574L interface wedging on em 7.1.9/7.2.3 when MSIX enabled

[Mike Tancsa]

On 10/26/2011 7:33 AM, Jason Wolfe wrote:
> Hooman,
> 
> I have run with dev.em.X.flow_control=0, which should have the same result
> as hw.em.fc_setting=0, and net.inet.tcp.tso is also 0.  I'm not sure the
> remaining options would be able to produce the scenario I'm seeing, but I'm
> open to giving it a try with no options on the interfaces.  I've also added
> ifconfig output to the collection.
> 
> options=219b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_MAGIC>
> ifconfig emX -rxcsum -txcsum -vlanhwtag -tso -wol
> options=88<VLAN_MTU,VLAN_HWCSUM>
> 
> It's always TX, but these servers push ~12x what they receive, so I'm
> guessing it could happen to either buffer given the right traffic patterns.
>  While looking through commits I also found a patch to add a couple sysctls
> for em, which I'm adding - http://freshbsd.org/commit/freebsd/r223676

Can you provide some more details as to how traffic flows on these servers ? Are they going through them, or are they generating the traffic ?  I wonder given a smaller CPU, it would be easier to trigger the condition somehow.  We recently got a Soekris 6501 which has onboard 4 such em nics, but has just a 1Ghz Atom.


em0 at pci0:5:0:0: class=0x020000 card=0x00008086 chip=0x10d38086 rev=0x00 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82574L Gigabit Network Connection'
    class      = network
    subclass   = ethernet
    bar   [10] = type Memory, range 32, base 0xa1000000, size 131072, enabled
    bar   [18] = type I/O Port, range 32, base 0x2000, size 32, enabled
    bar   [1c] = type Memory, range 32, base 0xa1020000, size 16384, enabled
    cap 01[c8] = powerspec 2  supports D0 D3  current D0
    cap 05[d0] = MSI supports 1 message, 64 bit 
    cap 10[e0] = PCI-Express 1 endpoint max data 128(256) link x1(x1)
    cap 11[a0] = MSI-X supports 5 messages in map 0x1c enabled
em1 at pci0:6:0:0: class=0x020000 card=0x00008086 chip=0x10d38086 rev=0x00 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82574L Gigabit Network Connection'
    class      = network
    subclass   = ethernet
    bar   [10] = type Memory, range 32, base 0xa2000000, size 131072, enabled
    bar   [18] = type I/O Port, range 32, base 0x3000, size 32, enabled
    bar   [1c] = type Memory, range 32, base 0xa2020000, size 16384, enabled
    cap 01[c8] = powerspec 2  supports D0 D3  current D0
    cap 05[d0] = MSI supports 1 message, 64 bit 
    cap 10[e0] = PCI-Express 1 endpoint max data 128(256) link x1(x1)
    cap 11[a0] = MSI-X supports 5 messages in map 0x1c enabled
em2 at pci0:10:0:0:        class=0x020000 card=0x00008086 chip=0x10d38086 rev=0x00 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82574L Gigabit Network Connection'
    class      = network
    subclass   = ethernet
    bar   [10] = type Memory, range 32, base 0xa3000000, size 131072, enabled
    bar   [18] = type I/O Port, range 32, base 0x4000, size 32, enabled
    bar   [1c] = type Memory, range 32, base 0xa3020000, size 16384, enabled
    cap 01[c8] = powerspec 2  supports D0 D3  current D0
    cap 05[d0] = MSI supports 1 message, 64 bit 
    cap 10[e0] = PCI-Express 1 endpoint max data 128(256) link x1(x1)
    cap 11[a0] = MSI-X supports 5 messages in map 0x1c enabled
em3 at pci0:11:0:0:        class=0x020000 card=0x00008086 chip=0x10d38086 rev=0x00 hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82574L Gigabit Network Connection'
    class      = network
    subclass   = ethernet
    bar   [10] = type Memory, range 32, base 0xa4000000, size 131072, enabled
    bar   [18] = type I/O Port, range 32, base 0x5000, size 32, enabled
    bar   [1c] = type Memory, range 32, base 0xa4020000, size 16384, enabled
    cap 01[c8] = powerspec 2  supports D0 D3  current D0
    cap 05[d0] = MSI supports 1 message, 64 bit 
    cap 10[e0] = PCI-Express 1 endpoint max data 128(256) link x1(x1)
    cap 11[a0] = MSI-X supports 5 messages in map 0x1c enabled

soekris6501# vmstat -i
interrupt                          total       rate
irq4: uart0                         4993          0
cpu0:timer                      84947808         71
irq256: ahci0                      59195          0
irq257: em0:rx 0                 1946546          1
irq258: em0:tx 0                  629707          0
irq259: em0:link                       2          0
Total                           87588251         73
soekris6501# 
-- 
-------------------
Mike Tancsa, tel +1 519 651 3400
Sentex Communications, mike at sentex.net
Providing Internet services since 1994 www.sentex.net
Cambridge, Ontario Canada   http://www.tancsa.com/