FreeBSD 7-STABLE mbuf corruption

Mon Sep 5 06:59:53 UTC 2011

Hi folks,

We have been trying to track down a bad mbuf management for about two
weeks on a customized 7.1 base. I have finally been able to reproduce
it with a stock FreeBSD 7-STABLE (kernel from r225276, userland from
7.4).

With the help of the attached patches, I have just been able to
trigger the following panic:

panic: Corrupted unused flags, expected 0xffffffff00000000, got 0x0, flags 0x3
cpuid = 1
Uptime: 3d10h5m3s
Cannot dump. No dump device defined

The patches taints high order 32bits of m_flags (extended to 64bits;
those are thus unused) right before the mbuf is referenced in the TX
ring of the igb(4) driver, I expect this value to never change until
right before the mbuf is freed igb_txeof(). [About this, Jack, am I
correct expecting the mbuf's flags not to be touched between
igb_xmit() and igb_txeof() ?]

I have strong suspicions that this is the cause of PR/155597,
eventually a few others PR. My current assumption about the root cause
of this behavior is that the same mbuf ends up being queued for TX
twice. After the first TX, it gets released, eventually reused in
socket's buffer, but ends up being freed again after the second TX,
screwing the chains at the same time, leading to crashes of the box.

On the crashes happens in multiple locations. We have seen crashes
(both clean panic() and NULL-pointer dereference) in various places
over the last weeks, first an almost daily panic() in sbdrop() or
sbsndptr(), and NULL-pointer dereferences in hfsc_dequeue(),
m_ext_free(), m_copym, and the list goes on.

On the driver p.o.v, crashes happened with igb(4) end of last year. At
the time, dropping the number of queue to 1 mitigated the problem...
so far. Now, the daily crashes happens with em(4) (single queue by
default on 7). We also have records of crashes in sbsndptr() on
vr(4)-based devices. Crashes seen on em(4) configuration are almost
always preceded by one or many:

emX: discard frame w/o packet header

which we agree, should not happen.

On the traffic p.o.v., crashes happens on a 24h basis with a box
handling about 30Mbps over a couple of thousands TCP connections. I
have been able to get consistent crashes in about 1h with ALTQ
enabled, proxying about 200 TCP connection over 200Mbps of traffic
(unidirectional, sub-ms RTT). Crashes becomes a lot faster with ALTQ
disabled, down to a reliable crash within 5min. The box is running a
few hundreds ipfw rules. Without any ipfw rules loaded, crashes
happens within 30min.

Now, the FreeBSD 7-STABLE machine has been able to handle about
900Mbps of traffic, over 2*500 TCP connections (500 receiving, 500
sending), over 24h without crashing. It crashed almost instantly when
I restarted the test today. The kernel has been built with INVARIANTS
and INVARIANT_SUPPORT. Hardware enumerate as follow:

CPU: Intel(R) Xeon(R) CPU           E5420  @ 2.50GHz (2493.76-MHz 686-class CPU)
  Origin = "GenuineIntel"  Id = 0x1067a  Family = 6  Model = 17  Stepping = 10
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x40ce3bd<SSE3,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,SSE4.1,XSAVE>
  AMD Features=0x20100000<NX,LM>
  AMD Features2=0x1<LAHF>
  Cores per package: 4
real memory  = 3757834240 (3583 MB)
avail memory = 3678064640 (3507 MB)
ACPI APIC Table: <100509 APIC1714>
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
 cpu0 (BSP): APIC ID:  0
 cpu1 (AP): APIC ID:  1
 cpu2 (AP): APIC ID:  2
 cpu3 (AP): APIC ID:  3
  cpu (AP): APIC ID:  4 (disabled)
  cpu (AP): APIC ID:  5 (disabled)
  cpu (AP): APIC ID:  6 (disabled)
  cpu (AP): APIC ID:  7 (disabled)
[...]
igb0: <Intel(R) PRO/1000 Network Connection version - 2.2.5> port
0xec00-0xec1f mem
0xfdf60000-0xfdf7ffff,0xfdf40000-0xfdf5ffff,0xfdfb8000-0xfdfbbfff irq
16 at device 0.0 on pci7
igb0: Using MSIX interrupts with 5 vectors
igb0: [ITHREAD]
igb0: [ITHREAD]
igb0: [ITHREAD]
igb0: [ITHREAD]
igb0: [ITHREAD]
igb0: Ethernet address: 00:15:b2:xx:xx:xx
igb1: <Intel(R) PRO/1000 Network Connection version - 2.2.5> port
0xec80-0xec9f mem
0xfdfe0000-0xfdffffff,0xfdfc0000-0xfdfdffff,0xfdfbc000-0xfdfbffff irq
17 at device 0.1 on pci7
igb1: Using MSIX interrupts with 5 vectors
igb1: [ITHREAD]
igb1: [ITHREAD]
igb1: [ITHREAD]
igb1: [ITHREAD]
igb1: [ITHREAD]
igb1: Ethernet address: 00:15:b2:xx:xx:xx

em(4) and igb(4) are a direct backport from HEAD, plus the build fix I
posted a few days ago.

Custom mbuf debugging is attached, as well as the config of the
kernel. The goal of the changes was first to enforce mbuf trashing,
then locate the bogus m_freem()[0], thus modifying the mbuf free path
to taint the mbuf not with the static 0xdeadc0de, but with the IP of
the call-site. Then add some consistency check.

At this point, any help is appreciated!

Thanks in advance,
 - Arnaud

[0]: the reason behind that is that I first got tons of crashes
related 0xdeadc0de, in particular, an mbuf being tagged M_PROMISC,
while the interface was not in promiscuous mode crashing on an
unwanted call to m_freem() in ether_demub()
-------------- next part --------------
A non-text attachment was scrubbed...
Name: GENERIC
Type: application/octet-stream
Size: 13418 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-net/attachments/20110905/c1507f26/GENERIC.obj
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-inlinize-m_freem.patch
Type: text/x-patch
Size: 2228 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-net/attachments/20110905/c1507f26/0001-inlinize-m_freem.bin
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0002-mbuf-use-after-free-marking.patch
Type: text/x-patch
Size: 3682 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-net/attachments/20110905/c1507f26/0002-mbuf-use-after-free-marking.bin
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0003-64bits-mbuf-flags.patch
Type: text/x-patch
Size: 5044 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-net/attachments/20110905/c1507f26/0003-64bits-mbuf-flags.bin
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0004-verify-mbuf-s-flags-consistency-after-tx.patch
Type: text/x-patch
Size: 3523 bytes
Desc: not available
Url : http://lists.freebsd.org/pipermail/freebsd-net/attachments/20110905/c1507f26/0004-verify-mbuf-s-flags-consistency-after-tx.bin