ixgbe, mdio, and marvell ethernet switches

A few weeks ago I started poking at some netgate hardware to use as freebsd
wifi test boxes in my little test lab here. The two I'm using are ADI (now
Silicom) boxes that needed some fun deep dives into BIOS versions,
interrupt storms and compatible mini-PCIe cards (some just don't actually
get probed!)

Anyway.

I ended up stumbling across one of the Silicom OEM boxes that got turned
into an AT&T open compute platform box back in 2018/2019 that has an
integrated Marvell 88E6190X switch hanging off of the intel X553 backplane
(the ixgbe driver.) However, it wasn't supported in FreeBSD, and the
instructions to get the switch up in linux were hit and miss.

People had gotten the etherswitch driver patched to work, but only against
pfsense plus, which has the MDIO support for ixgbe. But, I couldn't find
any diffs online to implement it.

So I did what I normally do, I got distracted from wifi stuff and decided
to get it going. Having a working amd64 etherswitch platform would be nice
for general etherswitch tinkering outside of the arm/mips/ppc world, and
then no-one would have any excuses getting involved in improving it!

It took a few days, but I figured it out. Here's the fun deep dive.

Firstly, DPDK introduced some support for the MDIO bus being exposed on the
X553, but with a couple caveats:

* it started as clause 22 (which is PHYs pre-2000) diff , but it eventually
got turned into a clause 45 (PHYs post-2000) diff, and they .. just deleted
the clause 22 code
* the MDIO support is really focused on talking MDIO to the PHYs internally
inside the X550 series hardware - clause 45, fixed PHY number, etc

So it wasn't enough to really bring up the marvell switch, as these all
seem to want clause 22 MDIO commands.

Then i went digging into the X550 and X553 datasheets.

* the X550 datasheet says that it only supports clause 45
* the X553 datasheet says it supports both clause 22 and clause 45, but
makes a point of (a) telling you if you need to, you should speak into
intel customer engineering, and (b) deleted the MDIO registers from their
register programming guide

Ok, so the X553 does do Clause 22, but .. how.

Then I went through the DPDK and linux driver history and lo and behold,
linux actually HAS the correct mdio bus support! specifically, they have a
/separate/ mdio bus driver hiding in their X550/X553 support, which:

* handles both clause 22 and clause 45 IO
* does the correct semaphore handshaking, since MDIO is shared between
multiple ethernet MACs, hardware and firmware
* exposes only one MDIO bus on the X553, as it only has a single MDIO bus
shared between the 1 to 4 MACs the chipset can expose

Armed with that I committed some driver sins and got it working. The in
progress absolutely disgusting hack diff is attached, so it won't get lost
/ forgotten.

Anyway, on the etherswitch side I did a bunch of cleanup to support
etherswitch/miiproxy/e6000sw as modules, and fix a bunch of e6000sw bugs,
and they're in -HEAD:

* https://reviews.freebsd.org/D50024 to https://reviews.freebsd.org/D50031
are all cleanups
* https://reviews.freebsd.org/D50044 adds the 88E6190X support

But then the next fun problem - none of the ports came up. Well, I lie.
Only one of them came up. I went digging into the PHY registers and found
that the marvell gige PHY code didn't actually power up the PHY if it was
powered down, so I fixed that too:

* https://reviews.freebsd.org/D50045

Then with some hints, I could see and use all of the switch ports on the
AT&T ATT-150 CPE device:

hint.mdio.0.at="ix0"
hint.e6000sw.0.addr="0x10"
hint.e6000sw.0.port0disabled="1"
hint.e6000sw.0.is6190x=1
hint.e6000sw.0.port9cpu=1
hint.e6000sw.0.port10cpu=1
hint.e6000sw.0.port9speed=2500
hint.e6000sw.0.port10speed=2500

Then, I also acquired a wireguard M270 box, as it also apprently has a
marvell switch attached to an X553 backplane, and none of the ethernet
ports work at all out of the box.
The above hints worked on that too - with addr set to "0x0" instead of
"0x10".

So, now i have two Atom C3558 boxes w/ X553 2.5gbit backplane ports hooked
into an 8 port marvell gige switch working on -HEAD (save the ixgbe mdio
diff that I attached; I'll sort that out soon). All it took was like $250
in hardware, staring at intel documentation and linux/dpdk driver code, and
three evenings thinking to myself "surely this can't be THAT hard.