Re: connect a "loose" device to a OFW/FDT node (and then use mii_fdt)?

On Sun, Mar 17, 2024 at 5:56 AM Dmitry Salychev <dsl@freebsd.org> wrote:

>
> Hi,
>
> I'd like to suggest a real world example of the case described by bz@.
> (please, excuse its verbosity)
>
> ----- dmesg -----
> dpaa2_mac3: <DPAA2 MAC> dpmcp (id=27) at dpmac (id=7) on dpaa2_rc0
> dpaa2_mac3: max_rate=1000, eth_if=QSGMII, link_type=PHY
> ...
> paa2_mac_fdt6: <DPAA2 MAC DEV> on dpaa2_mc0
> dpaa2_mac_fdt6: node 0x49f4 'ethernet@7': reg 0x7 sfp 0 \
> pcs-handle 0x3f7c phy-handle 0x3984 managed 'in-band-status' \
> phy-conn-type 'qsgmii'
> ...
> dpaa2_ni0: <DPAA2 Network Interface> ... on dpaa2_rc0
> dpaa2_ni0: connected to dpmac (id=7)
> dpaa2_ni0: connected DPMAC is in PHY mode
> dpaa2_mc0: dpmac_id 7 mdev 0xffffa0800203c800 (dpaa2_mac_fdt6)
>         pdev 0xffffa08000fc8300 (memacphy_fdt0)
> miibus0: <MII bus> on memacphy_fdt0
> vscphy0: <Vitesse VSC8514 10/100/1000TX PHY> PHY 28 on miibus0
> ...
> ----- dmesg -----
> ----- devinfo -----
> dpaa2_mac3 /* DPAA2 MAC abstraction */
>  dpaa2_rc0 /* DPAA2 resource container, firmware bus */
>  dpaa2_mc0 pnpinfo name=fsl-mc@80c000000 compat=fsl,qoriq-mc
>  simplebus0 pnpinfo name=soc compat=simple-bus
>  ofwbus0
>  nexus0
>
> dpaa2_mac_fdt6 /* compat=fsl,qoriq-mc-dpmac */
>  dpaa2_mc0 pnpinfo name=fsl-mc@80c000000 compat=fsl,qoriq-mc
>  simplebus0 pnpinfo name=soc compat=simple-bus
>  ofwbus0
>  nexus0
>
> vscphy0 pnpinfo oui=0x8083 model=0x27 rev=0x0 at phyno=28
>  miibus0
>  memacphy_fdt0 /* implements MII interface */
>  memac_mdio_fdt0 pnpinfo name=mdio@8b96000 compat=fsl,fman-memac-mdio
>  simplebus0 pnpinfo name=soc compat=simple-bus
>  ofwbus0
>  nexus0
> ----- devinfo -----
> ----- FDT -----
> fsl_mc: fsl-mc@80c000000 {
>         compatible = "fsl,qoriq-mc";
>         ...
>         dpmacs {
>                 ...
>                 dpmac7: ethernet@7 {
>                         compatible = "fsl,qoriq-mc-dpmac";
>                         reg = <7>;
>                         phy-handle = <&mdio1_phy1>;
>                         phy-connection-type = "qsgmii";
>                         pcs-handle = <&pcs7_0>;
>                         managed = "in-band-status";
>                 };
>                 ...
>         };
> };
> emdio1: mdio@8b96000 {
>         compatible = "fsl,fman-memac-mdio";
>         reg = <0x0 0x8b96000 0x0 0x1000>;
>         little-endian;
>         #address-cells = <1>;
>         #size-cells = <0>;
>         clock-frequency = <2500000>;
>         clocks = <&clockgen QORIQ_CLK_PLATFORM_PLL
>                             QORIQ_CLK_PLL_DIV(1)>;
>         ...
>         mdio1_phy1: ethernet-phy@1c {
>                 reg = <0x1c>;
>         };
> };
> ----- FDT -----
>
> dpaa2_rc (resource container) acts like a firmware bus and allows to
> discover DPAA2 devices (like dpaa2_mac) via commands implemented in the
> MC (management complex) firmware without traversing FDT. However,
> the firmware itself doesn't implement commands to retrieve a PHY link
> topology[1] yet.
>
> FDT describes the PHY topology in this case and we're only creating
> auxiliary drivers (like dpaa2_mac_fdt, memac_mdio_fdt and memacphy_fdt)
> in order to gather all of the necessary pieces of information and call
> mii_attach() somewhere in the dpaa2_ni (network interface).
>
> Personally, I don't think our current approach is good one because:
> (a) it isn't clear who is responsible to extract and form a description
> of the PHY topology and (b) auxiliary drivers provide no unified
> interface to obtain information extracted from FDT.
>
> dpaa2_mc (or dpaa2_rc) could be responsible for the information
> extraction in order to provide dpaa2_mac with everything needed to
> create a PHY link topology in its softc[2], I think.
>

So FDT name dpmac7 (FreeBSD device dpaa2_mac3) has a pointer to the PHY
mdio1_phy1 (FreeBSD name dpaa2_ni0). Neither looks like a detached device
at all. Both are parts of parallel device trees, with points from one tree
to the other.  The problem, though, becomes one of ordering: you'd need to
at least probe all the devices before you could start to attach, or you'd
need a callback when your target device attaches. Something needs to know
when all the parts are there to allow the mii_attach to be called.

This isn't a generic 'what to do with detached devices' problem in so much
as a 'what do we do in the embedded world where a FreeBSD logical device is
made up of several physical devices in the FDT tree, which effectively
attach asynchronously' since the order of attach is implemented as in DTB
order, but is actually undefined.  If the dependencies aren't there, how do
you, in a generic way, register for a callback when they are (or
alternatively, how do you create a 'set' of FDT devices (which you can know
at any time) that has a way to know when all the devices are present so
that the rest of the device's attach can happen (and you can do mii_attach,
etc). And how do you make that robust enough to inform the user when device
sets are incomplete. It's the opposite problem of refcounting a resource to
free it, in many ways.

Right now, there's nothing generic. It's all ad-hoc. I'd agree that this is
a less than ideal approach since you'd also want to be able to handle
situations where the driver isn't there initially, but was loaded after
mountroot (though this is likely uncommon, it's not unknown to have drivers
that work when loaded from userland, but not the loader due to ordering
bugs in the driver itself that aren't yet solved).

The traditional 'we have all our dependencies in attach because all we care
about is decoding (so just our parents) and everything is on the 'card'
that we've written drivers for' isn't such a good fit here, I'll grant. And
unfortunately, the current answer is that cooperative devices figure it out
themselves, which is an unsatisfying answer.  That's the basis for the
inquiry, right? If we had a callback that waited for all components to be
present (or other components to be present), then we could drive this via a
state machine that would have clear ordering and responsibilities (though
the exact nature of those would be device dependent). It's quite unlikely
that we'll ever support randomly attaching one part of the FDT to another
because that causes other problems... IIRC, Ian and I did experiments in
that area years ago and nothing but trouble came from it for arbitrary
topologies (though we did get some specific ones working).

Warner


> [1] https://www.kernel.org/doc/html/next/networking/phy-link-topology.html
> [2] https://www.kernel.org/doc/html/latest/networking/sfp-phylink.html
>
> Regards,
> Dmitry
>
> Warner Losh <imp@bsdimp.com> writes:
>
> > On Tue, Feb 27, 2024 at 1:40 PM Bjoern A. Zeeb <
> bzeeb-lists@lists.zabbadoz.net> wrote:
> >
> >  Hi,
> >
> >  I have a device which has its own ivars and is not created from OFW/FDT.
> >
> > How does it get this knowledge then?
> >
> >  The parent(parent(dev)) was but FDT does not fully represent the logic
> >  of it and we need to create the intermediate device *sigh*
> >
> > FDT just describes hardware, which is often non-linear where a driver
> would need to speak to two different devices to
> > accomplish its needs. A common example of this has been NIC drivers that
> don't have any access to the mii bus to talk to the
> > PHY that controls it, so has to do what's normally a parent/child
> relationship in add-in cards via a proxy. From what you
> > described below, it looks like that's what's going on here, but I could
> be mistaken.
> >
> >  I tried to put ASCII art below (try a plain text view if needed).
> >
> >  Is there any way to connect this "mac" device to its relevant node in
> FDT
> >  (continuing the "tree" there) and then use mii_fdt to attach (doing
> >  all the PHY stuff and references)?
> >
> > You'd likely have to proxy the miibus that the phy is connected to so
> that the NIC can talk to its PHY via this intermediary.
> >
> >  So looks something like this:
> >
> >    X (ofw/fdt/simplebus node/device)
> >    |
> >    +--- interim dev (not in FDT created by X)
> >
> > If there's no hardware, why is this device created?
> >
> > However, assuming that there's a good reason, this X device is just
> another bus. It will have whatever children are proper for
> > that bus to have. It's up to it to make that determination.
> >
> >         |
> >         +----- mac dev (in FDT but created as child of 'interim dev' and
> not out of ofw/fdt/simplebus)
> >
> > Wouldn't there be a PHANDLE or similar reference then pointing to this
> node in the tree?
> >
> > But regardless, you could create any device you want as a child of X,
> though in this case it is a little weird if the mac doesn't
> > belong to something else that would be a simple bus or a simple bus
> super class.
> >
> >                  |
> >                  +---- miibus(_fdt if possible)
> >                           |
> >                           +------ PHY
> >
> > Usually the non-linear references to PHYs that live on a MIIBUS not
> directly connected to the mac are done through cooperation
> > between the PHY driver and the MAC driver. One could model it the way
> you are saying, but that's not a requirement. We've
> > had several of these come up over the years since many SoCs have direct
> control of the MII bus not through resources owned
> > by the NIC as is often the case with add-in cards.
> >
> > So the question would become, is that the real situation here (in which
> case you'd need to proxie messages between the NIC
> > and PHY) or is it something else?
> >
> > I can't recall if we have drivers in the tree that do this today. I did
> this years ago with some stuff that never made it into the
> > tree on a consulting basis and it was relatively simple to write the
> wrappers that forwarded all the relevant newbus methods
> > called on behalf of driver A to driver B and vice versa. Though I do
> recall it was a pain in some spots since simple operations
> > couldn't be directly coded.
> >
> > Warner
>
> --
> https://wiki.freebsd.org/DmitrySalychev
>