Client Networking Issues / NIC Lab

[Chris]

On 2021-04-22 22:22, Kevin Bowling wrote:
> Greetings,
> 
> I have been looking into client networking issues in FreeBSD lately.
> To summarize the situation, common NICs like Intel gigabit (e1000 aka
> lem(4)/em(4)/igb(4)), Realtek (re(4)), Aquantia, and Tehuti Networks
> are unmaintained or not present on FreeBSD.  The purpose of this
> thread is to gauge whether that matters, and if it does what to do.  I
> believe it is important because we are losing out on a pipeline of new
> contributors by not supporting client hardware well.  We risk losing
> NAS, firewall, and other embedded users which may not be large enough
> to negotiate with these vendors for support or have the volume to do
> custom BOMs to avoid risky parts.  My opinion has been developed after
> researching the drivers, Bugzilla, and various internet forums where
> end users exchange advice or ask for help where FreeBSD is the
> underlying cause.
> 
> e1000 is in the best shape, with recent vendor involvement, but covers
> 20 years of silicon with over 100 chipsets (of which at least 60 are
> significant variations).  Datasheets are readily available for most of
> them, as well as "specification updates" which list errata.  There are
> chipsets which have been completely broken for several years.  More
> common, there are cases that lead to user frustration, including with
> the most recent hardware.  All of the silicon tends to have
> significant bugs around PCIe, TSO, NC-SI (IPMI sideband), arbitration
> conflicts with ME and more.  Intel doesn't patch the microcode on
> these, but many of the issues can be worked around in software.
> Performing an audit of the driver will take quite a while, and making
> and testing changes gives me concern.  When we (my previous employer
> and team) converted these drivers to iflib, we fixed some of the
> common cases for PCIe and TSO issues but only had a handful of chips
> to test against, so the driver works better for some and worse or not
> at all for others.  I have started fixing some of the bugs in
> Bugzilla, but I only have a few e1000 variants on hand to test, and I
> have an unrelated full time job so this is just occupying limited
> spare time as a hobby.
> 
> re(4) is in pretty abhorrent state.  All of these chips require
> runtime patching of the phy (which I believe is a DSP algorithm that
> gets improved over time) and mcu code.  That is totally absent in
> FreeBSD.  A vendor driver exists in net/realtek-re-kmod which contains
> the fixups and works alright for many users.  This driver cannot be
> imported into FreeBSD as is.  There is a strange use of the C
> PreProcessor which blows up compile time and driver size needlessly.
> The out of tree driver has a different set of supported adapters, so
> some kind of meld is necessary.  Realtek does not provide public chip
> documentation, I am trying to see if they will grant NDA access to
> contributors.
> 
> Aquantia has an out of tree driver in net/aquantia-atlantic-kmod.  The
> code is not currently in a place where I'd like to see it in the tree.
> I am not really sure how common these are, the company was acquired by
> Marvell which is still producing them as a client networking option
> while they have other IP for higher end/speed.
> 
> Tehuti Networks seems to have gone out of business.  Probably not
> worth worrying about.
> 
> 1) Do nothing.  This situation has gone on for a while.  Users are
> somewhat accustomed to purchasing FreeBSD-specific hardware for things
> like SOHO gateways and NAS.  A lot of people just revert back to Linux
> for client use.  OpenBSD seems to have more active contribution around
> this kind of thing and works better for common cases so that may be
> another exit ramp.
> 
> 2) Quantify usage data and beg the vendors for help.  This might work
> for Intel, however these devices have transferred to a client team at
> intel that does not plan to support FreeBSD, and intel does not keep
> test systems around long enough to meet FreeBSD user's needs. Realtek
> is a similar story, I am unsure how long they hold on to test systems
> and would probably need technical guidance to work with the FreeBSD
> community.  Unsure about Marvell, I've never worked with them.
> 
> 3) Build a NIC lab and focus on building community support.  It would
> also give the vendors a place to test hardware their labs have purged
> (due to IT asset management policies or other bureaucratic blunders).
> Set some boundaries like a 15 year window of chipsets which should
> cover practical embedded use cases.  There are backplane systems
> and/or external PCI(e) expansion systems that could be assembled to
> house a large number of NICs.  It would probably be cheaper than this,
> but say a budget of $15000USD is enough to purchase some expansions, a
> couple managed switches, and a few dozen common NICs.  Community
> members may also send in NICs they wish to see supported or undergo
> testing.  For this to work out long term, there needs to be a quorum
> of people interested in collaborating on the issue.  There are some
> risks around simply setting this up, depending on the configuration,
> the bus topology may introduce problems unrelated to the NICs and we'd
> probably need some semi-automated device.hints or devctl stuff to keep
> from over provisioning system resources (work on a subset of cards at
> a time).  An interesting extension of this would be a semi-automated
> validation setup for subsystem changes (significant driver changes,
> iflib, lro, etc).
> 
> 4) ???
Thank you for your efforts here, Kevin.
I had intended to shoot a similar note regarding the Realtek nics earlier
this week. But got sidelined.
I see a great deal of "bad press" on the internet regarding Realtek nics,
largely *bsd related, but also from may others as well. We ($work) have
had a pretty good experience with them. So much so, that we use them
almost exclusively (on FreeBSD). We've found them to be real performers
for the dollar. So I was going to send out a message that given our stock
of re(4) we could make ourselves available for testing against any work
being done on re(4).
IOW if we can help in any way. We'll make ourselves available.

Thanks again.

--Chris
> 
> Regards,
> Kevin
> _______________________________________________
> freebsd-net at freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "freebsd-net-unsubscribe at freebsd.org"