Frequent hickups on the networking layer

Tue Apr 28 12:19:47 UTC 2015

I've disabled TSO a while ago, after my networking stopped working completely, with `ifconfig em0 -tso`.

em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
 options=4209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWTSO>
 ether 00:25:90:dc:0f:a2
 nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
 media: Ethernet autoselect (1000baseT <full-duplex>)
 status: active

nfsv212: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
 options=3<RXCSUM,TXCSUM>
 ether 00:25:90:dc:0f:a2
 inet6 fe80::225:90ff:fedc:fa2%nfsv212 prefixlen 64 scopeid 0x6 
 inet6 fd::212:31:3:111:fffe prefixlen 64 
 nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
 media: Ethernet autoselect (1000baseT <full-duplex>)
 status: active
 vlan: 212 parent interface: em0

nfsv308: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
 options=3<RXCSUM,TXCSUM>
 ether 00:25:90:dc:0f:a2
 inet 10.38.0.253 netmask 0xffffff00 broadcast 10.38.0.255 
 inet6 fe80::225:90ff:fedc:fa2%nfsv308 prefixlen 64 scopeid 0x8 
 inet6 fd:308::30 prefixlen 64 
 nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
 media: Ethernet autoselect (1000baseT <full-duplex>)
 status: active
 vlan: 308 parent interface: em0

Or should I disable VLAN_HWTSO as well. If so, how?

Met vriendelijke groeten,

-- 
Kerio Operator in de Cloud? https://www.kerioindecloud.nl/
Mark Schouten  | Tuxis Internet Engineering
KvK: 61527076 | http://www.tuxis.nl/
T: 0318 200208 | info at tuxis.nl

 Van:   Chris Forgeron <cforgeron at acsi.ca> 
 Aan:   Mark Schouten <mark at tuxis.nl>, "freebsd-net at FreeBSD.org" <freebsd-net at FreeBSD.org> 
 Verzonden:   28-4-2015 13:58 
 Onderwerp:   RE: Frequent hickups on the networking layer 

What network care are you using? 

There have been a few reports of issues with TSO, if you check the list. I had some myself a while ago, but are now resolved thanks to a few helpful folk here. 

You could increase your mbufs, but if the problem is a leak/error in the stack, then you're just delaying the behaviour.  

-----Original Message----- 
From: owner-freebsd-net at freebsd.org [mailto:owner-freebsd-net at freebsd.org] On Behalf Of Mark Schouten 
Sent: Tuesday, April 28, 2015 5:48 AM 
To: freebsd-net at FreeBSD.org 
Subject: Frequent hickups on the networking layer 

Hi, 

I've got a FreeBSD 10.1-RELEASE box running with iscsi on top of ZFS. I've had some major issues with it where it would stop processing traffic for a minute or two, but that's 'fixed' by disabling TSO. I do have frequent iscsi errors, which are luckily fixed on the iscsi layer, but they do cause an occasional errormessage on both the iscsi client and server. Also, I see input errors on the FreeBSD server, but I'm unable to find out what those are. I do see a relation between iscsi-errormessages and the number of ethernet inputerrors on the server. 

I saw this message [1] which made me have a look at `vmstat -z`, and that shows me the following: 

vmstat -z | head -n 1; vmstat -z | sort -k 6 -t , | tail -10 ITEM                   SIZE  LIMIT     USED     FREE      REQ FAIL SLEEP zio_data_buf_94208:   94208,      0,     162,       5,  135632,   0,   0 zio_data_buf_98304:   98304,      0,     118,       9,  101606,   0,   0 zio_link_cache:          48,      0,       6,   30870,24853549414,   0,   0 8 Bucket:                64,      0,     145,    2831,148672720,  11,   0 32 Bucket:              256,      0,     859,     731,231513474,  52,   0 mbuf_jumbo_9k:         9216, 604528,    7230,    2002,11764806459,108298123,   0 64 Bucket:              512,      0,     808,     352,147120342,16375582,   0 256 Bucket:            2048,      0,     500,      50,307051808,189685088,   0 vmem btag:               56,      0, 1671605, 1291509,198933250,36431,   0 128 Bucket:            1024,      0,     410,     106,65267164,772374,   0  

I am using jumboframes. Could it be that the inputerrors AND my frequent hickups come from all those failures to allocate 9k jumbo mbufs? And can I increase the in [1] mentioned sysctls at will? 

Thanks 

[1]: https://lists.freebsd.org/pipermail/freebsd-questions/2013-August/252827.html 

Met vriendelijke groeten, 

--  
Kerio Operator in de Cloud? https://www.kerioindecloud.nl/ 
Mark Schouten  | Tuxis Internet Engineering 
KvK: 61527076 | http://www.tuxis.nl/ 
T: 0318 200208 | info at tuxis.nl 

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4549 bytes
Desc: Electronic Signature S/MIME
URL: <http://lists.freebsd.org/pipermail/freebsd-net/attachments/20150428/9c86dc16/attachment.bin>