FreeBSD 10.1-REL - network unaccessible after high traffic

[Adrian Chadd]

hi,

I asked for the output of vmstat -z and vmstat -m in a loop. :)


-a


On 29 June 2015 at 02:02, Csaba Banhalmi <bimmer at field.hu> wrote:
> Hi All,
>
> "vmstat 5" output when system freezes:
>  procs      memory      page                    disks faults         cpu
>  r b w     avm    fre   flt  re  pi  po    fr  sr ad0 ad1   in sy   cs us sy
> id
>  0 0 0   8752M   126M  5663   0   0   0  4042 445  66   0 1219 7148 4870  3
> 2 95
>  0 0 0   8650M   145M  2167   0   0   0  3501 447  79   0  974 4042 3578  1
> 1 98
>  0 0 0   8374M   201M  3113   0   0   0  6790 441   5   0 1130 6670 3729  3
> 1 96
>  0 0 0   8252M   220M  2632   0   0   0  4014 435   4   0  726 11653 2401  2
> 1 97
>  0 0 0   8188M   224M  1625   0   0   0  2189 434   5   0  713 6714 2376  1
> 1 98
>  0 0 0   7992M   233M  1504   0   0   0  2254 433   2   0  867 2890 2868  1
> 1 98
>  4 0 0   8032M   216M  2145   0   0   0  1995 435  18   0  526 3769 2048  1
> 1 98
>  0 0 0   8180M   195M  1949   0   0   0  1741 435  50   0  593 3441 2363  1
> 1 98
>  0 0 0   8186M   178M  2859   0   0   0  2525 436   6   0  499 3313 1733  2
> 1 97
>  1 0 0   8410M   146M  2521   0   0   0  1764 440  11   0  736 67271 2121  4
> 2 94
>  0 0 0   8182M   205M  2910   0   0   0  6378 927   8   0  495 16043 1775  1
> 1 98
>  1 1 0   7944M   210M  3009   0   0   0  3696 438   8   0  522 4247 1963  2
> 1 97
>  0 0 0   8091M   169M  7529   0   0   0  3601 436 105   0 1359 75290 4400  9
> 3 88
>  0 0 0   8121M   141M  4607   0   0   0  3288 444  62   0  949 12169 3268  5
> 1 94
>  0 0 0   8044M   201M  1782   0   0   0  4954 1795   9   0  446 3025 1927  1
> 1 99
>  0 0 0   7916M   222M  1296   0   0   0  2671 438   5   0  525 2984 1920  1
> 1 98
>  1 0 0   7870M   230M   888   0   0   0  1677 432   8   0  473 6424 2126  1
> 1 99
>  0 0 0   7968M   228M  3375   0   0   0  2625 433  51   0  768 4100 2852  3
> 1 96
>  0 0 0   8238M   194M  7586   0   0   0  4758 436  88   0 1026 9631 3908  4
> 2 94
>  0 0 0   8293M   185M  3253   0   0   0  2362 437  52   0  747 4475 3105  2
> 1 97
>
> I increased the vm.v_free_min, but did not help. It was a different froze,
> the system was unreacheable even through IPMI, needed a hard reset.
>
> Regards,
> Csaba
>
>
>
> 2015.06.12. 20:17 keltezéssel, Adrian Chadd írta:
>>
>> On 12 June 2015 at 10:57, Christopher Forgeron <csforgeron at gmail.com>
>> wrote:
>>>
>>> I agree it shouldn't run out of memory. Here's what mine does under
>>> network
>>> load, or rsync load:
>>>
>>> 2 0 9   1822M  1834M     0   0   0   0    14   8   0   0 22750  724
>>> 136119
>>> 0 23 77
>>>
>>> 0 0 9   1822M  1823M     0   0   0   0     0   8   0   0 44317  347
>>> 138151
>>> 0 16 84
>>>
>>> 0 0 9   1822M  1761M     0   0   0   0    17   8   0   0 23818  820 92198
>>> 0
>>> 12 88
>>>
>>> 0 0 9   1822M  1727M     0   0   0   0    14   8   0   0 40768  634
>>> 126688
>>> 0 17 83
>>>
>>> 0 0 9   1822M  8192B     0   8   0   0    15   3   3   0 9236  305 57149
>>> 0
>>> 33 67
>>>
>>>
>>> That's with a 5 second vmstat output. After the 8KiB, the system is
>>> nearly
>>> completely brain-dead and needs a hard power-off.
>>>
>>>
>>> I've seen it go from 6 GiB free to 8KiB in 5 sec as well. Currently my
>>> large
>>> machines are set to 12 GiB free to keep them from crashing, from what I
>>> presume is just network load due to lots of iSCSI / NFS traffic on my
>>> 10GiB
>>> network.
>>>
>>>
>>> I haven't had time to type this up for the list yet, but I'm putting it
>>> here
>>> just to make sure people know it's real.
>>>
>> Hi,
>>
>> Then something is leaking or  holding onto memory when it shouldn't be.
>>
>> Try doing vmstat -z and vmstat -m in a one second loop, post the data
>> just before it falls over.
>>
>>
>> -adrian
>
>