Help with mbuf exhaustion
Josh Gitlin
jgitlin at goboomtown.com
Thu Sep 28 20:32:16 UTC 2017
My mistake, the "1" was cut off from my message. We are actually on FreeBSD 10.3-RELEASE-p21, _not_ p2
--
<http://www.goboomtown.com/>
Josh Gitlin
Senior Full Stack Developer
(415) 690-1610 x155
Stay up to date and join the conversation in Relay <http://relay.goboomtown.com/>.
> On Sep 28, 2017, at 4:30 PM, Alan Somers <asomers at freebsd.org> wrote:
>
> First of all, 10.3-RELEASE-p2 is very old and has known security
> vulnerabilities. Have you tried 10.3-RELEASE-p21 or even 10.4-RELEASE
> ?
>
> On Thu, Sep 28, 2017 at 1:30 PM, Josh Gitlin <jgitlin at goboomtown.com <mailto:jgitlin at goboomtown.com>> wrote:
>> Hi FreeBSD Gurus!
>>
>> We're having an issue with mbuf exhaustion on a FreeBSD server which was recently upgraded from 10.3-STABLE to 10.3-RELEASE-p2. Under the course of normal operation, we see mbuf usage steadily increasing until we reach kern.ipc.nmbufs limit, at which point the machine becomes unresponsive over the network (due to lack of mbufs for network access) and the console displays:
>>
>> cxl0: Interface stopped DISTRIBUTING, possible flapping
>> cxl1: Interface stopped DISTRIBUTING, possible flapping
>> [zone: mbuf] kern.ipc.nmbufs limit reached
>> [zone: mbuf] kern.ipc.nmbufs limit reached
>> The machine runs pf and acts as a packet filter, router, gateway and DHCP/DNS server. It has two Chelsio NICs in it, and is a CARP master with a secondary. The secondary has identical configuration of hardware and software and does not exhibit this issue.
>>
>> Given the downtime this causes, we set up our Nagios/Check_MK to graph the output of `netstat -m` and alert when mbufs in use approaches `kern.ipc.nmbufs` and we see a steady linear increase in mbuf usage until we reboot:
>>
>> https://i.stack.imgur.com/8bzAq.png <https://i.stack.imgur.com/8bzAq.png> <https://i.stack.imgur.com/8bzAq.png <https://i.stack.imgur.com/8bzAq.png>>
>>
>> mbuf *clusters* in use does not change when this happens and increasing mbuf cluster limits has no effect:
>>
>> https://i.stack.imgur.com/7OzdN.png <https://i.stack.imgur.com/7OzdN.png> <https://i.stack.imgur.com/7OzdN.png <https://i.stack.imgur.com/7OzdN.png>>
>>
>> This appears to be a kernel bug of some sort to me, looking for advice on further troubleshooting or assistance in resolving this!
>>
>> Helpful (maybe) information:
>>
>> netstat -m:
>>
>> 679270/3080/682350 mbufs in use (current/cache/total)
>> 10243/1657/11900/985360 mbuf clusters in use (current/cache/total/max)
>> 10243/1648 mbuf+clusters out of packet secondary zone in use (current/cache)
>> 8128/482/8610/124025 4k (page size) jumbo clusters in use (current/cache/total/max)
>> 0/0/0/36748 9k jumbo clusters in use (current/cache/total/max)
>> 128/0/128/20670 16k jumbo clusters in use (current/cache/total/max)
>> 224863K/6012K/230875K bytes allocated to network (current/cache/total)
>> 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters)
>> 0/0/0 requests for mbufs delayed (mbufs/clusters/mbuf+clusters)
>> 0/0/0 requests for jumbo clusters delayed (4k/9k/16k)
>> 0/0/0 requests for jumbo clusters denied (4k/9k/16k)
>> 0 requests for sfbufs denied
>> 0 requests for sfbufs delayed
>> 0 requests for I/O initiated by sendfile
>>
>> vmstat -z|grep -E '^ITEM|mbuf':
>>
>> ITEM SIZE LIMIT USED FREE REQ FAIL SLEEP
>> mbuf_packet: 256, 1587540, 10239, 1652,84058893, 0, 0
>> mbuf: 256, 1587540, 671533, 1206,914478880, 0, 0
>> mbuf_cluster: 2048, 985360, 11891, 9, 11891, 0, 0
>> mbuf_jumbo_page: 4096, 124025, 8128, 512,15011847, 0, 0
>> mbuf_jumbo_9k: 9216, 36748, 0, 0, 0, 0, 0
>> mbuf_jumbo_16k: 16384, 20670, 128, 0, 128, 0, 0
>> mbuf_ext_refcnt: 4, 0, 0, 0, 0, 0, 0
>>
>> vmstat -m:
>>
>> Type InUse MemUse HighUse Requests Size(s)
>> NFSD lckfile 1 1K - 1 256
>> filedesc 103 383K - 1134731 16,32,128,2048,4096,8192,16384,65536
>> sigio 1 1K - 1 64
>> filecaps 0 0K - 973 64
>> kdtrace 292 59K - 1099386 64,256
>> kenv 121 13K - 125 16,32,64,128,8192
>> kqueue 14 22K - 5374 256,2048,8192
>> proc-args 54 5K - 578448 16,32,64,128,256
>> hhook 2 1K - 2 256
>> ithread 146 24K - 146 32,128,256
>> KTRACE 100 13K - 100 128
>> NFS fh 1 1K - 584 32
>> linker 207 1052K - 234 16,32,64,128,256,512,1024,2048,4096,8192,16384,65536
>> lockf 29 3K - 20042 64,128
>> loginclass 2 1K - 1192 64
>> devbuf 17205 36362K - 17523 16,32,64,128,256,512,1024,2048,4096,8192,65536
>> temp 149 51K - 1280113 16,32,64,128,256,512,1024,2048,4096,8192,16384,65536
>> ip6opt 5 2K - 6 256
>> ip6ndp 27 2K - 27 64,128
>> module 230 29K - 230 128
>> mtx_pool 2 16K - 2 8192
>> osd 3 1K - 5 16,32,64
>> pmchooks 1 1K - 1 128
>> pgrp 30 4K - 2222 128
>> session 29 4K - 2187 128
>> proc 2 32K - 2 16384
>> subproc 211 368K - 1099014 512,4096
>> cred 204 32K - 6025704 64,256
>> plimit 19 5K - 3985 256
>> uidinfo 9 5K - 11892 128,4096
>> NFSD session 1 1K - 1 1024
>> sysctl 0 0K - 63851 16,32,64
>> sysctloid 7196 365K - 7369 16,32,64,128
>> sysctltmp 0 0K - 17834 16,32,64,128
>> tidhash 1 32K - 1 32768
>> callout 5 2184K - 5
>> umtx 522 66K - 522 128
>> p1003.1b 1 1K - 1 16
>> SWAP 2 549K - 2 64
>> bus 802 86K - 6536 16,32,64,128,256,1024
>> bus-sc 57 1671K - 2431 16,32,64,128,256,512,1024,2048,4096,8192,16384,65536
>> newnfsmnt 1 1K - 1 1024
>> devstat 8 17K - 8 32,4096
>> eventhandler 116 10K - 116 64,128
>> kobj 124 496K - 296 4096
>> acpiintr 1 1K - 1 64
>> Per-cpu 1 1K - 1 32
>> acpica 14355 1420K - 216546 16,32,64,128,256,512,1024,2048,4096
>> pci_link 16 2K - 16 64,128
>> pfs_nodes 21 6K - 21 256
>> rman 316 37K - 716 16,32,128
>> sbuf 1 1K - 41375 16,32,64,128,256,512,1024,2048,4096,8192,16384
>> sglist 8 8K - 8 1024
>> GEOM 88 15K - 1871 16,32,64,128,256,512,1024,2048,8192,16384
>> acpipwr 5 1K - 5 64
>> taskqueue 43 7K - 43 16,32,256
>> Unitno 22 2K - 1208250 32,64
>> vmem 3 144K - 6 1024,4096,8192
>> ioctlops 0 0K - 185700 256,512,1024,2048,4096
>> select 89 12K - 89 128
>> iov 0 0K - 19808992 16,64,128,256,512,1024
>> msg 4 30K - 4 2048,4096,8192,16384
>> sem 4 106K - 4 2048,4096
>> shm 1 32K - 1 32768
>> tty 20 20K - 499 1024
>> pts 1 1K - 480 256
>> accf 2 1K - 2 64
>> mbuf_tag 0 0K - 291472282 32,64,128
>> shmfd 1 8K - 1 8192
>> soname 32 4K - 1210442 16,32,128
>> pcb 36 663K - 76872 16,32,64,128,1024,2048,8192
>> CAM CCB 0 0K - 182128 2048
>> acl 0 0K - 2 4096
>> vfscache 1 2048K - 1
>> cl_savebuf 0 0K - 480 64
>> vfs_hash 1 1024K - 1
>> vnodes 1 1K - 1 256
>> entropy 1026 65K - 49107 32,64,4096
>> mount 64 3K - 140 16,32,64,128,256
>> vnodemarker 0 0K - 4212 512
>> BPF 112 20504K - 131 16,64,128,512,4096
>> CAM path 11 1K - 63 32
>> ifnet 29 57K - 30 128,256,2048
>> ifaddr 315 105K - 315 32,64,128,256,512,2048,4096
>> ether_multi 232 13K - 282 16,32,64
>> clone 10 2K - 10 128
>> arpcom 23 1K - 23 16
>> gif 4 1K - 4 32,256
>> lltable 155 53K - 551 256,512
>> UART 6 5K - 6 16,1024
>> vlan 56 5K - 74 64,128
>> acpitask 1 16K - 1 16384
>> acpisem 110 14K - 110 128
>> raid_data 0 0K - 108 32,128,256
>> routetbl 516 136K - 101735 32,64,128,256,512
>> igmp 28 7K - 28 256
>> CARP 76 30K - 83 16,32,64,128,256,512,1024
>> ipid 2 24K - 2 8192,16384
>> in_mfilter 112 112K - 112 1024
>> in_multi 43 11K - 43 256
>> ip_moptions 224 35K - 224 64,256
>> CAM periph 7 2K - 19 16,32,64,128,256
>> acpidev 128 8K - 128 64
>> CAM queue 15 5K - 39 16,32,512
>> encap_export_host 4 4K - 4 1024
>> sctp_a_it 0 0K - 36 16
>> sctp_vrf 1 1K - 1 64
>> sctp_ifa 115 15K - 204 128
>> sctp_ifn 21 3K - 23 128
>> sctp_iter 0 0K - 36 256
>> hostcache 1 32K - 1 32768
>> syncache 1 64K - 1 65536
>> in6_mfilter 1 1K - 1 1024
>> in6_multi 15 2K - 15 32,256
>> ip6_moptions 2 1K - 2 32,256
>> CAM dev queue 6 1K - 6 64
>> kbdmux 6 22K - 6 16,512,1024,2048,16384
>> mld 26 4K - 26 128
>> LED 20 2K - 20 16,128
>> inpcbpolicy 365 12K - 119277 32
>> secasvar 7 2K - 214 256
>> sahead 10 3K - 10 256
>> ipsecpolicy 748 187K - 241562 256
>> ipsecrequest 18 3K - 72 128
>> ipsec-misc 56 2K - 1712 16,32,64
>> ipsec-saq 0 0K - 24 128
>> ipsec-reg 3 1K - 3 32
>> pfsync 2 2K - 893 32,256,1024
>> pf_temp 0 0K - 78 128
>> pf_hash 3 2880K - 3
>> pf_ifnet 36 11K - 9510 256,2048
>> pf_tag 7 1K - 7 128
>> pf_altq 5 2K - 125 256
>> pf_rule 964 904K - 17500 128,1024
>> pf_osfp 1130 115K - 28250 64,128
>> pf_table 49 98K - 948 2048
>> crypto 37 11K - 1072 64,128,256,512,1024
>> xform 7 1K - 1530156 16,32,64,128,256
>> rpc 12 20K - 304 64,128,512,1024,8192
>> audit_evclass 187 6K - 231 32
>> ufs_dirhash 93 18K - 93 16,32,64,128,256,512
>> ufs_quota 1 1024K - 1
>> ufs_mount 3 13K - 3 512,4096,8192
>> vm_pgdata 2 513K - 2 128
>> UMAHash 5 6K - 10 512,1024,2048
>> CAM SIM 6 2K - 6 256
>> CAM XPT 30 3K - 1850 16,32,64,128,256,512,1024,2048,65536
>> CAM DEV 9 18K - 16 2048
>> fpukern_ctx 3 6K - 3 2048
>> memdesc 1 4K - 1 4096
>> USB 23 33K - 24 16,128,256,512,1024,2048,4096
>> DEVFS3 136 34K - 2027 256
>> DEVFS1 108 54K - 594 512
>> apmdev 1 1K - 1 128
>> madt_table 0 0K - 1 4096
>> DEVFS_RULE 55 26K - 55 64,512
>> DEVFS 12 1K - 13 16,128
>> DEVFSP 22 2K - 167 64
>> io_apic 1 2K - 1 2048
>> isadev 8 1K - 8 128
>> MCA 15 2K - 15 32,128
>> msi 30 4K - 30 128
>> nexusdev 5 1K - 5 16
>> USBdev 21 8K - 21 32,64,128,256,512,1024,4096
>> NFSD V4client 1 1K - 1 256
>> cdev 5 2K - 5 256
>> cxgbe 41 956K - 44 128,256,512,1024,2048,4096,8192,16384
>> ipmi 0 0K - 20155 128,2048
>> htcp data 127 4K - 13675 32
>> aesni_data 3 3K - 3 1024
>> solaris 142 12302K - 3189 16,32,64,128,512,1024,8192
>> kstat_data 6 1K - 6 64
>>
>> TCP States:
>>
>> https://i.stack.imgur.com/G7850.png <https://i.stack.imgur.com/G7850.png>
>>
>>
>> --
>> <http://www.goboomtown.com/ <http://www.goboomtown.com/>>
>> Josh Gitlin
>> Senior Full Stack Developer
>> (415) 690-1610 x155
>>
>> Stay up to date and join the conversation in Relay <http://relay.goboomtown.com/ <http://relay.goboomtown.com/>>.
>>
>> _______________________________________________
>> freebsd-net at freebsd.org <mailto:freebsd-net at freebsd.org> mailing list
>> https://lists.freebsd.org/mailman/listinfo/freebsd-net <https://lists.freebsd.org/mailman/listinfo/freebsd-net>
>> To unsubscribe, send any mail to "freebsd-net-unsubscribe at freebsd.org <mailto:freebsd-net-unsubscribe at freebsd.org>"
More information about the freebsd-net
mailing list