FreeBSD 10.1-REL - network unaccessible after high traffic

Fri Jun 12 12:29:23 UTC 2015

Well, even at low speed it could drop due to memory from what I've seen.

What was the last line from vmstat 5 before it locked up?

 I find that the em driver isn't crap, but there is a deeper problem inside
of FreeBSD that is being exposed now - For me it's due to faster network
connections.

 Are you using rsync to move the files?

On Fri, Jun 12, 2015 at 7:17 AM, Cs <bimmer at field.hu> wrote:

> it seems it's not memory related. Server just died a few minutes ago
> during transporting the backup (400GB) around 800Mbps speed..
> will disable remote backup, it's a shame that em driver is such a crap.
>
>
> 2015.06.08. 5:01 keltezéssel, Christopher Forgeron írta:
>
>> You know what helped me:
>>
>> 'vmstat 5'
>>
>> Leave that running. If the last thing on the console after a crash/hang is
>> vmstat showing 8k of memory left, then you're in the same problem-park as
>> me.
>>
>> My 10.1 96GiB RAM box is chewing ~8 GiB of RAM in less than 5 seconds, and
>> then crashing/panicking/hanging.
>>
>> There's others with this issues if you search for it; a sysctl
>> to vm.v_free_min to double or triple that value may help, but first let us
>> know if that's what is bonking your sever.
>>
>>
>>
>> On Sun, Jun 7, 2015 at 11:03 AM, Cs <bimmer at field.hu> wrote:
>>
>>  ok, just lowered it to 1500 but please also note that it was on 1500 for
>>> 2
>>> years
>>>
>>> 2015.06.07. 14:57 keltezéssel, Rick Macklem írta:
>>>
>>>  Since disabling TSO didn't help, you could try dropping to 1500mtu
>>>> on both interfaces. Some people run into problems when 9K jumbo clusters
>>>> fragment the kernel address space used to allocate mbufs.
>>>>
>>>> Good luck with it, rick
>>>>
>>>> ----- Original Message -----
>>>>
>>>>  Hi All,
>>>>>
>>>>> It worked fine for two weeks but I had a network outage 2 days ago
>>>>> then
>>>>> today. Tried to disable rxcsum and txcsum after the first one, didn't
>>>>> help. Don't know what else to do it's a shame that I can't use this
>>>>> card
>>>>> with fbsd i REALLY don't want to install linux instead but my
>>>>> production
>>>>> servers outages are not welcomed by the customers..
>>>>>
>>>>> 2015.05.26. 10:36 keltezéssel, Cs írta:
>>>>>
>>>>>  Thanks Mark, good idea. I found this thread which is exactly the
>>>>>> same
>>>>>> problem as mine:
>>>>>>
>>>>>>
>>>>>> https://forums.freebsd.org/threads/workaround-freebsd-10-1-sudden-network-down.49264/
>>>>>>
>>>>>> Will see if it helps in a couple weeks.
>>>>>>
>>>>>> Regards,
>>>>>> Csaba
>>>>>>
>>>>>> 2015.05.26. 10:30 keltezéssel, Mark Schouten írta:
>>>>>>
>>>>>>  Oh, didn't see your lowest remark. Then, the next thing that comes
>>>>>>> past here a few times per week is 'Try disabling TSO'.
>>>>>>>
>>>>>>>
>>>>>>> Met vriendelijke groeten,
>>>>>>>
>>>>>>> --
>>>>>>> Kerio Operator in de Cloud? https://www.kerioindecloud.nl/
>>>>>>> Mark Schouten  | Tuxis Internet Engineering
>>>>>>> KvK: 61527076 | http://www.tuxis.nl/
>>>>>>> T: 0318 200208 | info at tuxis.nl
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>     Van:   Cs <bimmer at field.hu>
>>>>>>>     Aan:   Mark Schouten <mark at tuxis.nl>
>>>>>>>     Cc:    <freebsd-net at freebsd.org>
>>>>>>>     Verzonden:   25-5-2015 11:12
>>>>>>>     Onderwerp:   Re: FreeBSD 10.1-REL - network unaccessible after
>>>>>>>     high
>>>>>>> traffic
>>>>>>>
>>>>>>> It was on 1500 for ~3 years :)
>>>>>>>     Regards,
>>>>>>> Csaba
>>>>>>>         On May 25, 2015, 10:30, at 10:30, Mark Schouten
>>>>>>>         <mark at tuxis.nl>
>>>>>>> wrote:
>>>>>>>
>>>>>>>  Try lowering your mtu to 1500, that worked miracles for me..
>>>>>>>>
>>>>>>>> --
>>>>>>>> Mark Schouten
>>>>>>>> Tuxis Internet Engineering
>>>>>>>> mark at tuxis.nl / 0318 200208
>>>>>>>>
>>>>>>>>   On 25 May 2015, at 09:36, "Cs" <bimmer at field.hu> wrote:
>>>>>>>>
>>>>>>>>>     Hi all,
>>>>>>>>>     I have two FreeBSd 10.1-RELEASE servers connected to each
>>>>>>>>>     other.
>>>>>>>>> They
>>>>>>>>>
>>>>>>>>>  were connected via cross link, but they are connected to a cisco
>>>>>>>> switch
>>>>>>>> now (the problem was the same with cross link too). When
>>>>>>>> transferring
>>>>>>>> huge files (50-500GB backup files) via Gigabit (it is important!)
>>>>>>>> the
>>>>>>>> network randomly dies. The backup runs every day/week and
>>>>>>>> sometimes the
>>>>>>>> connection is ok for months sometimes it happens twice a week.
>>>>>>>> When the
>>>>>>>> network dies I can log in to the server via IPMI and use the
>>>>>>>> console
>>>>>>>> everything is OK, but can't send anything out on the network.
>>>>>>>> ifconfig
>>>>>>>> em0 down/up doesn't help nor netif restart. The problem never
>>>>>>>> occured
>>>>>>>> when I used 100Mbit connection between them, but it was 3com NIC
>>>>>>>> (xl),
>>>>>>>> gigabit adapter is Intel (em0). When I limit the transfer rate
>>>>>>>> (rsync
>>>>>>>> bandwith limit or ipfw pipe) the problem is much more rare.
>>>>>>>>
>>>>>>>>      I tried to set these tuning parameters on both servers with
>>>>>>>>> different
>>>>>>>>>
>>>>>>>>>  buffer size but nothing helped:
>>>>>>>>
>>>>>>>>      # cat /etc/sysctl.conf
>>>>>>>>> security.bsd.see_other_uids=0
>>>>>>>>> net.inet.tcp.recvspace=512000
>>>>>>>>> net.route.netisr_maxqlen=2048
>>>>>>>>> kern.ipc.nmbclusters=1310720
>>>>>>>>> net.inet.tcp.sendbuf_max=16777216
>>>>>>>>> net.inet.tcp.recvbuf_max=16777216
>>>>>>>>> kern.ipc.soacceptqueue=32768
>>>>>>>>>     # cat /boot/loader.conf
>>>>>>>>> geom_mirror_load="YES" # RAID1 disk driver (see gmirror(8))
>>>>>>>>> ipfw_load="YES"
>>>>>>>>> net.inet.ip.fw.default_to_accept=1
>>>>>>>>> kern.maxusers=4096
>>>>>>>>> accf_data_load="YES"
>>>>>>>>>     The duplex settings are identical on both servers.
>>>>>>>>>     Server A:
>>>>>>>>> em1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0
>>>>>>>>> mtu
>>>>>>>>>
>>>>>>>>>  9000
>>>>>>>>
>>>>>>>>
>>>>>>>> options=4219b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_MAGIC,VLAN_HWTSO>
>>>>>>>>
>>>>>>>>
>>>>>>>>            ether 00:25:90:24:52:66
>>>>>>>>
>>>>>>>>>           inet x.x.x.x netmask 0xfffffe00 broadcast x.x.x.x
>>>>>>>>>           nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
>>>>>>>>>           media: Ethernet autoselect (1000baseT <full-duplex>)
>>>>>>>>>           status: active
>>>>>>>>>     Server B:
>>>>>>>>> em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0
>>>>>>>>> mtu
>>>>>>>>>
>>>>>>>>>  9000
>>>>>>>>
>>>>>>>>
>>>>>>>> options=4219b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_MAGIC,VLAN_HWTSO>
>>>>>>>>
>>>>>>>>
>>>>>>>>            ether 00:30:48:dd:fe:3e
>>>>>>>>
>>>>>>>>>           inet x.x.x.x netmask 0xfffffe00 broadcast x.x.x.x
>>>>>>>>>           nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
>>>>>>>>>           media: Ethernet autoselect (1000baseT <full-duplex>)
>>>>>>>>>           status: active
>>>>>>>>>     Today I tried to set mtu to 9000 but in tcpdump I see that
>>>>>>>>>     during
>>>>>>>>> scp
>>>>>>>>>
>>>>>>>>>  it is still 1500:
>>>>>>>>
>>>>>>>>        x.x.x.x.222 > x.x.x.x.37612: Flags [.], cksum 0xb6ee
>>>>>>>>>       (incorrect ->
>>>>>>>>>
>>>>>>>>>  0xda6f), seq 35749, ack 113701596, win 7986, options [nop,nop,TS
>>>>>>>> val
>>>>>>>> 3103966325 ecr 853712893], length 0
>>>>>>>>
>>>>>>>>  09:27:33.912354 IP (tos 0x8, ttl 64, id 1028, offset 0, flags
>>>>>>>>> [DF],
>>>>>>>>>
>>>>>>>>>  proto TCP (6), length 1500)
>>>>>>>>
>>>>>>>>  09:27:33.912358 IP (tos 0x8, ttl 64, id 1029, offset 0, flags
>>>>>>>>> [DF],
>>>>>>>>>
>>>>>>>>>  proto TCP (6), length 1500)
>>>>>>>>
>>>>>>>>        Any ideas? Thanks guys!
>>>>>>>>> _______________________________________________
>>>>>>>>> freebsd-net at freebsd.org mailing list
>>>>>>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-net
>>>>>>>>> To unsubscribe, send any mail to
>>>>>>>>>
>>>>>>>>>  "freebsd-net-unsubscribe at freebsd.org"
>>>>>>>>
>>>>>>>>  _______________________________________________
>>>>>>> freebsd-net at freebsd.org mailing list
>>>>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-net
>>>>>>> To unsubscribe, send any mail to
>>>>>>> "freebsd-net-unsubscribe at freebsd.org"
>>>>>>>
>>>>>>>
>>>>>>>   _______________________________________________
>>>>>>>
>>>>>> freebsd-net at freebsd.org mailing list
>>>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-net
>>>>>> To unsubscribe, send any mail to
>>>>>> "freebsd-net-unsubscribe at freebsd.org"
>>>>>>
>>>>>>  _______________________________________________
>>>>> freebsd-net at freebsd.org mailing list
>>>>> http://lists.freebsd.org/mailman/listinfo/freebsd-net
>>>>> To unsubscribe, send any mail to
>>>>> "freebsd-net-unsubscribe at freebsd.org"
>>>>>
>>>>>  _______________________________________________
>>> freebsd-net at freebsd.org mailing list
>>> http://lists.freebsd.org/mailman/listinfo/freebsd-net
>>> To unsubscribe, send any mail to "freebsd-net-unsubscribe at freebsd.org"
>>>
>>>  _______________________________________________
>> freebsd-net at freebsd.org mailing list
>> http://lists.freebsd.org/mailman/listinfo/freebsd-net
>> To unsubscribe, send any mail to "freebsd-net-unsubscribe at freebsd.org"
>>
>
> _______________________________________________
> freebsd-net at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "freebsd-net-unsubscribe at freebsd.org"
>