Re: Chelsio cards, jumbo frames, memory fragmentation and performance in FreeBSD 13.x?

Ryan Stone <rysto32@gmail.com> wrote:
>
> Set it to 4k and move on.  I'm not aware of any efforts in the VM
> layer or network stack to make >PAGE_SIZE clusters usable in practical
> situations.  Honestly, at this point I wish that we'd just kill them
> entirely.
I feel about the same. However, I'm guessing that there are some net
drivers that can't work without them (at least for jumbo frames) and
no one available to fix those drivers.

At the least, we should strongly encourage net device driver authors
to avoid their use wherever practicable.

I also think that "don't use them" should be the default for device
drivers that can optionally not use them.

I'll admit I cannot think of a good way to fix jumbo clusters, so avoiding
them is about all we can do, I think?
- Separate allocation pools would avoid the fragmentation problem,
  but waste a lot of space unless the mix of cluster sizes needed for the
  network load on the machine was somehow known.

rick

On Fri, Dec 3, 2021 at 12:52 PM John Jasen <jjasen@gmail.com> wrote:
>
> Pretty close to two years ago, we tripped across conditions where heavily
> used FreeBSD 11.x packet-filter firewalls would slow to a crawl and load
> would go crazy.
>
> In a fit of hopefulness with our upgrade to FreeBSD 13.0, I removed the
> hw.cxgbe.largest_rx_cluster settings we put in place -- only to have our
> hardest hit firewall experience high load and plummeting bandwidth within a
> few hours.
>
> So, I have two questions -- is there a better fix for this? Or, is the
> approach to set largest_rx_cluster to 4k and move on?
>
> Thanks in advance!