Re: connect a "loose" device to a OFW/FDT node (and then use mii_fdt)?

On Sun, 17 Mar 2024, Warner Losh wrote:

> On Sun, Mar 17, 2024 at 5:56 AM Dmitry Salychev <dsl@freebsd.org> wrote:
>
>>
>> Hi,
>>
>> I'd like to suggest a real world example of the case described by bz@.
>> (please, excuse its verbosity)
>>
>> ----- dmesg -----
>> dpaa2_mac3: <DPAA2 MAC> dpmcp (id=27) at dpmac (id=7) on dpaa2_rc0
>> dpaa2_mac3: max_rate=1000, eth_if=QSGMII, link_type=PHY
>> ...
>> paa2_mac_fdt6: <DPAA2 MAC DEV> on dpaa2_mc0
>> dpaa2_mac_fdt6: node 0x49f4 'ethernet@7': reg 0x7 sfp 0 \
>> pcs-handle 0x3f7c phy-handle 0x3984 managed 'in-band-status' \
>> phy-conn-type 'qsgmii'
>> ...
>> dpaa2_ni0: <DPAA2 Network Interface> ... on dpaa2_rc0
>> dpaa2_ni0: connected to dpmac (id=7)
>> dpaa2_ni0: connected DPMAC is in PHY mode
>> dpaa2_mc0: dpmac_id 7 mdev 0xffffa0800203c800 (dpaa2_mac_fdt6)
>>         pdev 0xffffa08000fc8300 (memacphy_fdt0)
>> miibus0: <MII bus> on memacphy_fdt0
>> vscphy0: <Vitesse VSC8514 10/100/1000TX PHY> PHY 28 on miibus0
>> ...
>> ----- dmesg -----
>> ----- devinfo -----
>> dpaa2_mac3 /* DPAA2 MAC abstraction */
>>  dpaa2_rc0 /* DPAA2 resource container, firmware bus */
>>  dpaa2_mc0 pnpinfo name=fsl-mc@80c000000 compat=fsl,qoriq-mc
>>  simplebus0 pnpinfo name=soc compat=simple-bus
>>  ofwbus0
>>  nexus0
>>
>> dpaa2_mac_fdt6 /* compat=fsl,qoriq-mc-dpmac */
>>  dpaa2_mc0 pnpinfo name=fsl-mc@80c000000 compat=fsl,qoriq-mc
>>  simplebus0 pnpinfo name=soc compat=simple-bus
>>  ofwbus0
>>  nexus0
>>
>> vscphy0 pnpinfo oui=0x8083 model=0x27 rev=0x0 at phyno=28
>>  miibus0
>>  memacphy_fdt0 /* implements MII interface */
>>  memac_mdio_fdt0 pnpinfo name=mdio@8b96000 compat=fsl,fman-memac-mdio
>>  simplebus0 pnpinfo name=soc compat=simple-bus
>>  ofwbus0
>>  nexus0
>> ----- devinfo -----
>> ----- FDT -----
>> fsl_mc: fsl-mc@80c000000 {
>>         compatible = "fsl,qoriq-mc";
>>         ...
>>         dpmacs {
>>                 ...
>>                 dpmac7: ethernet@7 {
>>                         compatible = "fsl,qoriq-mc-dpmac";
>>                         reg = <7>;
>>                         phy-handle = <&mdio1_phy1>;
>>                         phy-connection-type = "qsgmii";
>>                         pcs-handle = <&pcs7_0>;
>>                         managed = "in-band-status";
>>                 };
>>                 ...
>>         };
>> };
>> emdio1: mdio@8b96000 {
>>         compatible = "fsl,fman-memac-mdio";
>>         reg = <0x0 0x8b96000 0x0 0x1000>;
>>         little-endian;
>>         #address-cells = <1>;
>>         #size-cells = <0>;
>>         clock-frequency = <2500000>;
>>         clocks = <&clockgen QORIQ_CLK_PLATFORM_PLL
>>                             QORIQ_CLK_PLL_DIV(1)>;
>>         ...
>>         mdio1_phy1: ethernet-phy@1c {
>>                 reg = <0x1c>;
>>         };
>> };
>> ----- FDT -----
>>
>> dpaa2_rc (resource container) acts like a firmware bus and allows to
>> discover DPAA2 devices (like dpaa2_mac) via commands implemented in the
>> MC (management complex) firmware without traversing FDT. However,
>> the firmware itself doesn't implement commands to retrieve a PHY link
>> topology[1] yet.
>>
>> FDT describes the PHY topology in this case and we're only creating
>> auxiliary drivers (like dpaa2_mac_fdt, memac_mdio_fdt and memacphy_fdt)
>> in order to gather all of the necessary pieces of information and call
>> mii_attach() somewhere in the dpaa2_ni (network interface).
>>
>> Personally, I don't think our current approach is good one because:
>> (a) it isn't clear who is responsible to extract and form a description
>> of the PHY topology and (b) auxiliary drivers provide no unified
>> interface to obtain information extracted from FDT.
>>
>> dpaa2_mc (or dpaa2_rc) could be responsible for the information
>> extraction in order to provide dpaa2_mac with everything needed to
>> create a PHY link topology in its softc[2], I think.
>>
>
> So FDT name dpmac7 (FreeBSD device dpaa2_mac3) has a pointer to the PHY
> mdio1_phy1 (FreeBSD name dpaa2_ni0). Neither looks like a detached device
> at all. Both are parts of parallel device trees, with points from one tree
> to the other.  The problem, though, becomes one of ordering: you'd need to
> at least probe all the devices before you could start to attach, or you'd
> need a callback when your target device attaches. Something needs to know
> when all the parts are there to allow the mii_attach to be called.
>
> This isn't a generic 'what to do with detached devices' problem in so much
> as a 'what do we do in the embedded world where a FreeBSD logical device is
> made up of several physical devices in the FDT tree, which effectively
> attach asynchronously' since the order of attach is implemented as in DTB
> order, but is actually undefined.  If the dependencies aren't there, how do
> you, in a generic way, register for a callback when they are (or
> alternatively, how do you create a 'set' of FDT devices (which you can know
> at any time) that has a way to know when all the devices are present so
> that the rest of the device's attach can happen (and you can do mii_attach,
> etc). And how do you make that robust enough to inform the user when device
> sets are incomplete. It's the opposite problem of refcounting a resource to
> free it, in many ways.
>
> Right now, there's nothing generic. It's all ad-hoc. I'd agree that this is
> a less than ideal approach since you'd also want to be able to handle
> situations where the driver isn't there initially, but was loaded after
> mountroot (though this is likely uncommon, it's not unknown to have drivers
> that work when loaded from userland, but not the loader due to ordering
> bugs in the driver itself that aren't yet solved).
>
> The traditional 'we have all our dependencies in attach because all we care
> about is decoding (so just our parents) and everything is on the 'card'
> that we've written drivers for' isn't such a good fit here, I'll grant. And
> unfortunately, the current answer is that cooperative devices figure it out
> themselves, which is an unsatisfying answer.  That's the basis for the
> inquiry, right? If we had a callback that waited for all components to be
> present (or other components to be present), then we could drive this via a
> state machine that would have clear ordering and responsibilities (though
> the exact nature of those would be device dependent). It's quite unlikely
> that we'll ever support randomly attaching one part of the FDT to another
> because that causes other problems... IIRC, Ian and I did experiments in
> that area years ago and nothing but trouble came from it for arbitrary
> topologies (though we did get some specific ones working).

And FYI (@dsl) I have completely rewritten parts of that code locally to
better match things but I have yet to test ACPI which I why it's nowhere
yet.

May immediate need was actually to make mii_fdt work (as I have further
logic for PHYs to set in FDT) and for that I had to attach the miibus
to dpmac (and so I do for sfp now).
dpni knows little to nothing about miibus but it's device_t which is
currently needed because some ifnet callbacks are tied to the ifp which
seems unhelpful but I haen't gone down the road to start investigting the
entire tree to avoid massive code duplications.  That's for another day....

Anyway, this cleans up some of the initial uncleanliness as well... but
yes, as warner said in his earlier response, we do have to proxy things
through somehow and that's what I ended up doing...

/bz

-- 
Bjoern A. Zeeb                                                     r15:7