Re: connect a "loose" device to a OFW/FDT node (and then use mii_fdt)?

"Bjoern A. Zeeb" <bzeeb-lists@lists.zabbadoz.net> writes:

> On Sun, 17 Mar 2024, Warner Losh wrote:
>
>> On Sun, Mar 17, 2024 at 5:56 AM Dmitry Salychev <dsl@freebsd.org> wrote:
>>
>>>
>>> Hi,
>>>
>>> I'd like to suggest a real world example of the case described by bz@.
>>> (please, excuse its verbosity)
>>>
>>> ----- dmesg -----
>>> dpaa2_mac3: <DPAA2 MAC> dpmcp (id=27) at dpmac (id=7) on dpaa2_rc0
>>> dpaa2_mac3: max_rate=1000, eth_if=QSGMII, link_type=PHY
>>> ...
>>> paa2_mac_fdt6: <DPAA2 MAC DEV> on dpaa2_mc0
>>> dpaa2_mac_fdt6: node 0x49f4 'ethernet@7': reg 0x7 sfp 0 \
>>> pcs-handle 0x3f7c phy-handle 0x3984 managed 'in-band-status' \
>>> phy-conn-type 'qsgmii'
>>> ...
>>> dpaa2_ni0: <DPAA2 Network Interface> ... on dpaa2_rc0
>>> dpaa2_ni0: connected to dpmac (id=7)
>>> dpaa2_ni0: connected DPMAC is in PHY mode
>>> dpaa2_mc0: dpmac_id 7 mdev 0xffffa0800203c800 (dpaa2_mac_fdt6)
>>>         pdev 0xffffa08000fc8300 (memacphy_fdt0)
>>> miibus0: <MII bus> on memacphy_fdt0
>>> vscphy0: <Vitesse VSC8514 10/100/1000TX PHY> PHY 28 on miibus0
>>> ...
>>> ----- dmesg -----
>>> ----- devinfo -----
>>> dpaa2_mac3 /* DPAA2 MAC abstraction */
>>>  dpaa2_rc0 /* DPAA2 resource container, firmware bus */
>>>  dpaa2_mc0 pnpinfo name=fsl-mc@80c000000 compat=fsl,qoriq-mc
>>>  simplebus0 pnpinfo name=soc compat=simple-bus
>>>  ofwbus0
>>>  nexus0
>>>
>>> dpaa2_mac_fdt6 /* compat=fsl,qoriq-mc-dpmac */
>>>  dpaa2_mc0 pnpinfo name=fsl-mc@80c000000 compat=fsl,qoriq-mc
>>>  simplebus0 pnpinfo name=soc compat=simple-bus
>>>  ofwbus0
>>>  nexus0
>>>
>>> vscphy0 pnpinfo oui=0x8083 model=0x27 rev=0x0 at phyno=28
>>>  miibus0
>>>  memacphy_fdt0 /* implements MII interface */
>>>  memac_mdio_fdt0 pnpinfo name=mdio@8b96000 compat=fsl,fman-memac-mdio
>>>  simplebus0 pnpinfo name=soc compat=simple-bus
>>>  ofwbus0
>>>  nexus0
>>> ----- devinfo -----
>>> ----- FDT -----
>>> fsl_mc: fsl-mc@80c000000 {
>>>         compatible = "fsl,qoriq-mc";
>>>         ...
>>>         dpmacs {
>>>                 ...
>>>                 dpmac7: ethernet@7 {
>>>                         compatible = "fsl,qoriq-mc-dpmac";
>>>                         reg = <7>;
>>>                         phy-handle = <&mdio1_phy1>;
>>>                         phy-connection-type = "qsgmii";
>>>                         pcs-handle = <&pcs7_0>;
>>>                         managed = "in-band-status";
>>>                 };
>>>                 ...
>>>         };
>>> };
>>> emdio1: mdio@8b96000 {
>>>         compatible = "fsl,fman-memac-mdio";
>>>         reg = <0x0 0x8b96000 0x0 0x1000>;
>>>         little-endian;
>>>         #address-cells = <1>;
>>>         #size-cells = <0>;
>>>         clock-frequency = <2500000>;
>>>         clocks = <&clockgen QORIQ_CLK_PLATFORM_PLL
>>>                             QORIQ_CLK_PLL_DIV(1)>;
>>>         ...
>>>         mdio1_phy1: ethernet-phy@1c {
>>>                 reg = <0x1c>;
>>>         };
>>> };
>>> ----- FDT -----
>>>
>>> dpaa2_rc (resource container) acts like a firmware bus and allows to
>>> discover DPAA2 devices (like dpaa2_mac) via commands implemented in the
>>> MC (management complex) firmware without traversing FDT. However,
>>> the firmware itself doesn't implement commands to retrieve a PHY link
>>> topology[1] yet.
>>>
>>> FDT describes the PHY topology in this case and we're only creating
>>> auxiliary drivers (like dpaa2_mac_fdt, memac_mdio_fdt and memacphy_fdt)
>>> in order to gather all of the necessary pieces of information and call
>>> mii_attach() somewhere in the dpaa2_ni (network interface).
>>>
>>> Personally, I don't think our current approach is good one because:
>>> (a) it isn't clear who is responsible to extract and form a description
>>> of the PHY topology and (b) auxiliary drivers provide no unified
>>> interface to obtain information extracted from FDT.
>>>
>>> dpaa2_mc (or dpaa2_rc) could be responsible for the information
>>> extraction in order to provide dpaa2_mac with everything needed to
>>> create a PHY link topology in its softc[2], I think.
>>>
>>
>> So FDT name dpmac7 (FreeBSD device dpaa2_mac3) has a pointer to the PHY
>> mdio1_phy1 (FreeBSD name dpaa2_ni0). Neither looks like a detached device
>> at all. Both are parts of parallel device trees, with points from one tree
>> to the other.  The problem, though, becomes one of ordering: you'd need to
>> at least probe all the devices before you could start to attach, or you'd
>> need a callback when your target device attaches. Something needs to know
>> when all the parts are there to allow the mii_attach to be called.
>>
>> This isn't a generic 'what to do with detached devices' problem in so much
>> as a 'what do we do in the embedded world where a FreeBSD logical device is
>> made up of several physical devices in the FDT tree, which effectively
>> attach asynchronously' since the order of attach is implemented as in DTB
>> order, but is actually undefined.  If the dependencies aren't there, how do
>> you, in a generic way, register for a callback when they are (or
>> alternatively, how do you create a 'set' of FDT devices (which you can know
>> at any time) that has a way to know when all the devices are present so
>> that the rest of the device's attach can happen (and you can do mii_attach,
>> etc). And how do you make that robust enough to inform the user when device
>> sets are incomplete. It's the opposite problem of refcounting a resource to
>> free it, in many ways.
>>
>> Right now, there's nothing generic. It's all ad-hoc. I'd agree that this is
>> a less than ideal approach since you'd also want to be able to handle
>> situations where the driver isn't there initially, but was loaded after
>> mountroot (though this is likely uncommon, it's not unknown to have drivers
>> that work when loaded from userland, but not the loader due to ordering
>> bugs in the driver itself that aren't yet solved).
>>
>> The traditional 'we have all our dependencies in attach because all we care
>> about is decoding (so just our parents) and everything is on the 'card'
>> that we've written drivers for' isn't such a good fit here, I'll grant. And
>> unfortunately, the current answer is that cooperative devices figure it out
>> themselves, which is an unsatisfying answer.  That's the basis for the
>> inquiry, right? If we had a callback that waited for all components to be
>> present (or other components to be present), then we could drive this via a
>> state machine that would have clear ordering and responsibilities (though
>> the exact nature of those would be device dependent). It's quite unlikely
>> that we'll ever support randomly attaching one part of the FDT to another
>> because that causes other problems... IIRC, Ian and I did experiments in
>> that area years ago and nothing but trouble came from it for arbitrary
>> topologies (though we did get some specific ones working).
>
> And FYI (@dsl) I have completely rewritten parts of that code locally to
> better match things but I have yet to test ACPI which I why it's nowhere
> yet.
>
> May immediate need was actually to make mii_fdt work (as I have further
> logic for PHYs to set in FDT) and for that I had to attach the miibus
> to dpmac (and so I do for sfp now).

Does it mean that you're attaching PCS PHY to dpaa2_mac?
(see attached picture)

> dpni knows little to nothing about miibus but it's device_t which is
> currently needed because some ifnet callbacks are tied to the ifp which
> seems unhelpful but I haen't gone down the road to start investigting the
> entire tree to avoid massive code duplications.  That's for another day....
>
> Anyway, this cleans up some of the initial uncleanliness as well... but
> yes, as warner said in his earlier response, we do have to proxy things
> through somehow and that's what I ended up doing...
>
> /bz


-- 
https://wiki.freebsd.org/DmitrySalychev