[Bug 166724] if_re(4): watchdog timeout

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=166724

Richard Russo <freebsd@ruka.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |freebsd@ruka.org

--- Comment #101 from Richard Russo <freebsd@ruka.org> ---
I spent some time trying to debug this, without significant results, but I
wanted to share what I learned, mostly things that didn't work.

My hardware is class=0x020000 rev=0x0c hdr=0x00 vendor=0x10ec device=0x8168
subvendor=0x1462 subdevice=0x7850; and I was able to fairly reliably trigger
the condition with iperf3 -s on the re0 (in a vnet jail), and iperf3 -c --bidir
on another interface. My hardware is fairly meager, Intel G3470, using a
multiport Intel em card for the otherside of the iperf3 testing.

Using the kernel driver (13.1-RELEASE), I would tend to see traffic stall and
timeouts be reported within one minute, although sometimes within a few
seconds. Using the vendor driver from net/realtek-re-kmod, I was unable to
reproduce the error condition.

There's a fair amount of differences between the two drivers, and I tried a
bunch of things, but could never pass my test. Sending more transmit requests
didn't help; the vendor driver sends it twice, but that doesn't seem to help;
sending TX requests in the watchdog handler didn't help either.

I noticed that the Tx queue would usually be stuck on the 2nd segment of a two
or three segment packet, so I tried adding m_defrag to make all the packets a
single segment. That didn't work either. There didn't seem to be a pattern of
which Tx segment the NIC would get stuck on, or anything obvious about the data
addresses.

I also tried messing with the reset to maybe make that more reliable. The
vendor driver does reset a little bit differently, but nothing there made
things reliable for me either. After a few resets, the NIC just doesn't seem to
start sending again; until a reboot (at least with my NIC; I've seen some
reports that a power cycle is required). Reading the tx/rx registers before
setting them, I saw that those are retained across reboots, so the device
doesn't seem to be getting fully reset, which might explain some reporter's
need to power cycle.

I suspect there's something in the mostly opaque vendor initialization code,
that puts the device in a better mode where it doesn't get stuck (at least with
my test loads). I do notice that the vendor initialization causes the device to
emit ethernet pause frames, which doesn't happen with the kernel code.

One thing I was able to make a positive difference with though, the kernel
reset doesn't clear the RX/TX buffers, although it frees the associated mbufs.
Sometimes during a reset or shortly afterwards, the NIC is still using those
descriptor arrays; so I would see weird packets apparently coming in on re0 via
tcpdump, many of them looking like recently used mbufs on other interfaces. I
also saw some bizarre packets RXed by the em NIC connected to re0, and some
evidence of other NICs receiving corrupted mbufs.

Using explicit_bzero during descriptor setup seemed to help, as well as turning
off the CMD_OWN flag on the descriptors during re_stop. It's obviously a bit
tricky if the device is acknowledging a command reset, but not actually fully
resetting. I don't have an IOMMU system, but I'd guess if you had that, you
could get more information about what's going on.

Running with INVARIANTS showed some use after free errors, which I believe are
related to the device using the mbufs, although it was hard to trigger and
difficult to debug.

-- 
You are receiving this mail because:
You are the assignee for the bug.
You are on the CC list for the bug.