FreeBSD 10.1-REL - network unaccessible after high traffic
Cs
bimmer at field.hu
Fri Jun 12 16:07:50 UTC 2015
I'll take your advice and give it a shot, thanks :)
2015.06.12. 14:43 keltezéssel, Christopher Forgeron írta:
> Ah, but the 'why' will come later, after we know for sure what the
> 'what' is in your problem.
>
> I'm just pointing out the problems that I'm having, as yours sound
> similar. Once the box runs out of memory, all sorts of interesting
> things can happen. Perhaps that's not your case, but it's quite possible.
>
> Setup a remote terminal, do the copy again, and send in the last few
> lines of 'vmstat 5' after it's locked up, perhaps I can help.
>
> On Fri, Jun 12, 2015 at 9:39 AM, Cs <bimmer at field.hu
> <mailto:bimmer at field.hu>> wrote:
>
> but why is that machine runs fine except the network if it's
> memory related? swap didn't increased before the network outage.
>
>
> 2015.06.12. 14:37 keltezéssel, Christopher Forgeron írta:
>> rsycn burns memory - I'd say you have a good chance you're
>> running out of mem before it's replenished.
>>
>> For vmstat 5 - Don't run it on console. Connect via a second box
>> with ssh, and run it there - That way it's the last thing on the
>> ssh terminal screen when the box dies, and you'll have your proof.
>>
>> On Fri, Jun 12, 2015 at 9:31 AM, Cs <bimmer at field.hu
>> <mailto:bimmer at field.hu>> wrote:
>>
>> machine has been restarted before I could check the "vmstat
>> 5" output. Yep, it's rsync. Anyway I disabled the backup
>> transfer it'll solve, but I can't really accept this for
>> solution.
>>
>>
>> 2015.06.12. 14 <tel:2015.06.12.%2014>:29 keltezéssel,
>> Christopher Forgeron írta:
>>
>> Well, even at low speed it could drop due to memory from
>> what I've seen.
>>
>> What was the last line from vmstat 5 before it locked up?
>>
>> I find that the em driver isn't crap, but there is a
>> deeper problem inside
>> of FreeBSD that is being exposed now - For me it's due to
>> faster network
>> connections.
>>
>> Are you using rsync to move the files?
>>
>> On Fri, Jun 12, 2015 at 7:17 AM, Cs <bimmer at field.hu
>> <mailto:bimmer at field.hu>> wrote:
>>
>> it seems it's not memory related. Server just died a
>> few minutes ago
>> during transporting the backup (400GB) around 800Mbps
>> speed..
>> will disable remote backup, it's a shame that em
>> driver is such a crap.
>>
>>
>> 2015.06.08. 5:01 keltezéssel, Christopher Forgeron írta:
>>
>> You know what helped me:
>>
>> 'vmstat 5'
>>
>> Leave that running. If the last thing on the
>> console after a crash/hang is
>> vmstat showing 8k of memory left, then you're in
>> the same problem-park as
>> me.
>>
>> My 10.1 96GiB RAM box is chewing ~8 GiB of RAM in
>> less than 5 seconds, and
>> then crashing/panicking/hanging.
>>
>> There's others with this issues if you search for
>> it; a sysctl
>> to vm.v_free_min to double or triple that value
>> may help, but first let us
>> know if that's what is bonking your sever.
>>
>>
>>
>> On Sun, Jun 7, 2015 at 11:03 AM, Cs
>> <bimmer at field.hu <mailto:bimmer at field.hu>> wrote:
>>
>> ok, just lowered it to 1500 but please also
>> note that it was on 1500 for
>>
>> 2
>> years
>>
>> 2015.06.07. 14 <tel:2015.06.07.%2014>:57
>> keltezéssel, Rick Macklem írta:
>>
>> Since disabling TSO didn't help, you could
>> try dropping to 1500mtu
>>
>> on both interfaces. Some people run into
>> problems when 9K jumbo clusters
>> fragment the kernel address space used to
>> allocate mbufs.
>>
>> Good luck with it, rick
>>
>> ----- Original Message -----
>>
>> Hi All,
>>
>> It worked fine for two weeks but I
>> had a network outage 2 days ago
>> then
>> today. Tried to disable rxcsum and
>> txcsum after the first one, didn't
>> help. Don't know what else to do it's
>> a shame that I can't use this
>> card
>> with fbsd i REALLY don't want to
>> install linux instead but my
>> production
>> servers outages are not welcomed by
>> the customers..
>>
>> 2015.05.26. 10
>> <tel:2015.05.26.%2010>:36
>> keltezéssel, Cs írta:
>>
>> Thanks Mark, good idea. I found
>> this thread which is exactly the
>>
>> same
>> problem as mine:
>>
>>
>> https://forums.freebsd.org/threads/workaround-freebsd-10-1-sudden-network-down.49264/
>>
>> Will see if it helps in a couple
>> weeks.
>>
>> Regards,
>> Csaba
>>
>> 2015.05.26. 10
>> <tel:2015.05.26.%2010>:30
>> keltezéssel, Mark Schouten írta:
>>
>> Oh, didn't see your lowest
>> remark. Then, the next thing that
>> comes
>>
>> past here a few times per
>> week is 'Try disabling TSO'.
>>
>>
>> Met vriendelijke groeten,
>>
>> --
>> Kerio Operator in de Cloud?
>> https://www.kerioindecloud.nl/
>> Mark Schouten | Tuxis
>> Internet Engineering
>> KvK: 61527076 |
>> http://www.tuxis.nl/
>> T: 0318 200208 |
>> info at tuxis.nl
>> <mailto:info at tuxis.nl>
>>
>>
>>
>> Van: Cs
>> <bimmer at field.hu
>> <mailto:bimmer at field.hu>>
>> Aan: Mark Schouten
>> <mark at tuxis.nl
>> <mailto:mark at tuxis.nl>>
>> Cc:
>> <freebsd-net at freebsd.org
>> <mailto:freebsd-net at freebsd.org>>
>> Verzonden: 25-5-2015 11:12
>> Onderwerp: Re: FreeBSD
>> 10.1-REL - network
>> unaccessible after
>> high
>> traffic
>>
>> It was on 1500 for ~3 years :)
>> Regards,
>> Csaba
>> On May 25, 2015,
>> 10:30, at 10:30, Mark Schouten
>> <mark at tuxis.nl
>> <mailto:mark at tuxis.nl>>
>> wrote:
>>
>> Try lowering your mtu to
>> 1500, that worked miracles
>> for me..
>>
>> --
>> Mark Schouten
>> Tuxis Internet Engineering
>> mark at tuxis.nl
>> <mailto:mark at tuxis.nl> /
>> 0318 200208
>>
>> On 25 May 2015, at
>> 09:36, "Cs"
>> <bimmer at field.hu
>> <mailto:bimmer at field.hu>>
>> wrote:
>>
>> Hi all,
>> I have two
>> FreeBSd 10.1-RELEASE
>> servers connected to each
>> other.
>> They
>>
>> were connected via
>> cross link, but they
>> are connected to a cisco
>>
>> switch
>> now (the problem was the
>> same with cross link
>> too). When
>> transferring
>> huge files (50-500GB
>> backup files) via Gigabit
>> (it is important!)
>> the
>> network randomly dies.
>> The backup runs every
>> day/week and
>> sometimes the
>> connection is ok for
>> months sometimes it
>> happens twice a week.
>> When the
>> network dies I can log in
>> to the server via IPMI
>> and use the
>> console
>> everything is OK, but
>> can't send anything out
>> on the network.
>> ifconfig
>> em0 down/up doesn't help
>> nor netif restart. The
>> problem never
>> occured
>> when I used 100Mbit
>> connection between them,
>> but it was 3com NIC
>> (xl),
>> gigabit adapter is Intel
>> (em0). When I limit the
>> transfer rate
>> (rsync
>> bandwith limit or ipfw
>> pipe) the problem is much
>> more rare.
>>
>> I tried to set
>> these tuning parameters
>> on both servers with
>>
>> different
>>
>> buffer size but
>> nothing helped:
>>
>> # cat /etc/sysctl.conf
>>
>> security.bsd.see_other_uids=0
>> net.inet.tcp.recvspace=512000
>> net.route.netisr_maxqlen=2048
>> kern.ipc.nmbclusters=1310720
>> net.inet.tcp.sendbuf_max=16777216
>> net.inet.tcp.recvbuf_max=16777216
>> kern.ipc.soacceptqueue=32768
>> # cat
>> /boot/loader.conf
>> geom_mirror_load="YES" #
>> RAID1 disk driver
>> (see gmirror(8))
>> ipfw_load="YES"
>> net.inet.ip.fw.default_to_accept=1
>> kern.maxusers=4096
>> accf_data_load="YES"
>> The duplex
>> settings are
>> identical on both
>> servers.
>> Server A:
>> em1:
>> flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST>
>> metric 0
>> mtu
>>
>> 9000
>>
>>
>> options=4219b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_MAGIC,VLAN_HWTSO>
>>
>>
>> ether
>> 00:25:90:24:52:66
>>
>> inet
>> x.x.x.x netmask
>> 0xfffffe00 broadcast
>> x.x.x.x
>> nd6
>> options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
>> media:
>> Ethernet autoselect
>> (1000baseT <full-duplex>)
>> status: active
>> Server B:
>> em0:
>> flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST>
>> metric 0
>> mtu
>>
>> 9000
>>
>>
>> options=4219b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_MAGIC,VLAN_HWTSO>
>>
>>
>> ether
>> 00:30:48:dd:fe:3e
>>
>> inet
>> x.x.x.x netmask
>> 0xfffffe00 broadcast
>> x.x.x.x
>> nd6
>> options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
>> media:
>> Ethernet autoselect
>> (1000baseT <full-duplex>)
>> status: active
>> Today I tried to
>> set mtu to 9000 but
>> in tcpdump I see that
>> during
>> scp
>>
>> it is still 1500:
>>
>> x.x.x.x.222 >
>> x.x.x.x.37612: Flags [.],
>> cksum 0xb6ee
>>
>> (incorrect ->
>>
>> 0xda6f), seq 35749,
>> ack 113701596, win
>> 7986, options [nop,nop,TS
>>
>> val
>> 3103966325
>> <tel:3103966325> ecr
>> 853712893], length 0
>>
>> 09:27:33.912354 IP (tos
>> 0x8, ttl 64, id 1028,
>> offset 0, flags
>>
>> [DF],
>>
>> proto TCP (6),
>> length 1500)
>>
>> 09:27:33.912358 IP (tos
>> 0x8, ttl 64, id 1029,
>> offset 0, flags
>>
>> [DF],
>>
>> proto TCP (6),
>> length 1500)
>>
>> Any ideas? Thanks
>> guys!
>>
>> _______________________________________________
>> freebsd-net at freebsd.org
>> <mailto:freebsd-net at freebsd.org>
>> mailing list
>> http://lists.freebsd.org/mailman/listinfo/freebsd-net
>> To unsubscribe, send
>> any mail to
>>
>>
>> "freebsd-net-unsubscribe at freebsd.org
>> <mailto:freebsd-net-unsubscribe at freebsd.org>"
>>
>> _______________________________________________
>>
>> freebsd-net at freebsd.org
>> <mailto:freebsd-net at freebsd.org>
>> mailing list
>> http://lists.freebsd.org/mailman/listinfo/freebsd-net
>> To unsubscribe, send any mail to
>> "freebsd-net-unsubscribe at freebsd.org
>> <mailto:freebsd-net-unsubscribe at freebsd.org>"
>>
>>
>> _______________________________________________
>>
>> freebsd-net at freebsd.org
>> <mailto:freebsd-net at freebsd.org>
>> mailing list
>> http://lists.freebsd.org/mailman/listinfo/freebsd-net
>> To unsubscribe, send any mail to
>> "freebsd-net-unsubscribe at freebsd.org
>> <mailto:freebsd-net-unsubscribe at freebsd.org>"
>>
>> _______________________________________________
>>
>> freebsd-net at freebsd.org
>> <mailto:freebsd-net at freebsd.org>
>> mailing list
>> http://lists.freebsd.org/mailman/listinfo/freebsd-net
>> To unsubscribe, send any mail to
>> "freebsd-net-unsubscribe at freebsd.org
>> <mailto:freebsd-net-unsubscribe at freebsd.org>"
>>
>> _______________________________________________
>>
>> freebsd-net at freebsd.org
>> <mailto:freebsd-net at freebsd.org> mailing list
>> http://lists.freebsd.org/mailman/listinfo/freebsd-net
>> To unsubscribe, send any mail to
>> "freebsd-net-unsubscribe at freebsd.org
>> <mailto:freebsd-net-unsubscribe at freebsd.org>"
>>
>> _______________________________________________
>>
>> freebsd-net at freebsd.org
>> <mailto:freebsd-net at freebsd.org> mailing list
>> http://lists.freebsd.org/mailman/listinfo/freebsd-net
>> To unsubscribe, send any mail to
>> "freebsd-net-unsubscribe at freebsd.org
>> <mailto:freebsd-net-unsubscribe at freebsd.org>"
>>
>> _______________________________________________
>> freebsd-net at freebsd.org
>> <mailto:freebsd-net at freebsd.org> mailing list
>> http://lists.freebsd.org/mailman/listinfo/freebsd-net
>> To unsubscribe, send any mail to
>> "freebsd-net-unsubscribe at freebsd.org
>> <mailto:freebsd-net-unsubscribe at freebsd.org>"
>>
>> _______________________________________________
>> freebsd-net at freebsd.org <mailto:freebsd-net at freebsd.org>
>> mailing list
>> http://lists.freebsd.org/mailman/listinfo/freebsd-net
>> To unsubscribe, send any mail to
>> "freebsd-net-unsubscribe at freebsd.org
>> <mailto:freebsd-net-unsubscribe at freebsd.org>"
>>
>>
>> _______________________________________________
>> freebsd-net at freebsd.org <mailto:freebsd-net at freebsd.org>
>> mailing list
>> http://lists.freebsd.org/mailman/listinfo/freebsd-net
>> To unsubscribe, send any mail to
>> "freebsd-net-unsubscribe at freebsd.org
>> <mailto:freebsd-net-unsubscribe at freebsd.org>"
>>
>>
>
>
More information about the freebsd-net
mailing list