How's bus-space stuff supposed to work with superscalar MIPS?

[Adrian Chadd]

Hi,


On 5 October 2013 21:22, Warner Losh <imp at bsdimp.com> wrote:

>
> On Oct 5, 2013, at 5:51 PM, Adrian Chadd wrote:
>
> > On 5 October 2013 16:06, Stanislav Sedov <stas at freebsd.org> wrote:
> >
> >>
> >> On Oct 5, 2013, at 10:18 AM, Adrian Chadd <adrian at freebsd.org> wrote:
> >>
> >>> Hi all,
> >>>
> >>> I've been bringing up the AR9344 PHY and after a lot of digging, I
> >>> discovered that I can fix things by changing ARGE_WRITE() (ie, write to
> >> the
> >>> ethernet space registers) to:
> >>>
> >>> bus_write_4();
> >>> bus_read_4();
> >>>
> >>> .. to (what I'm guessing here) flush the write out before the next
> >>> instruction is run.
> >>>
> >>> So, given this particular hilarity has shown up, what's the story with
> >>> doing IO accesses on a superscalar MIPS CPU? If it's going to kseg1, is
> >> it
> >>> somehow going to magically enforce ordering? Or am I right in thinking
> we
> >>> will need explicit barriers here?
> >>>
> >>
> >> I don't know specifics of mips74k, but usually one indeed needs memory
> >> barriers
> >> when performing read of write operation sequences that require ordering
> on
> >> device I/O (e.g changing the ring and writing the new ring index
> >> afterwards).  I would
> >> not be surprised if the cpu reorders i/o bus memory access, especially a
> >> multi-issue
> >> one.
> >>
> >> It is a good idea to have barriers where needed regardless.  We have
> >> special macros
> >> for them which are defined to nothing on the in-order platforms.
> >
> >
> > Right. I know this stuff. I really though want to know this kind of
> stuff:
> >
> > * What the specifics are for superscalar MIPS CPUs;
>
> I believe they document that writes can be reordered unless there's an
> intervening read or memory barrier. I've not looked it up.
>

Would you mind helping me find where this is documented?

> * What the bus space stuff should be be providing by default (and I've
> been
> > down this path once, with ath(4) bugs, PPC, and the bus space macros not
> > enforcing flushes after IO operations, even though the API requires
> drivers
> > do it themselves..);
>
> It isn't so much flushes as barriers to prevent reordering. By doing the
> read after write, you are forcing an expensive memory barrier. Drivers that
> depend on a particular write ordering need to have explicit barriers.
>
> > * Whether it should be enough to map space COHERENT - then it's up to the
> > underlying bus implementation to implement enforcing ordering.
>
> The question here is whether there should be an implied barrier in write
> operations. On x86 there is, but as you are discovering on other
> architectures there isn't. While it would be convenient to force a memory
> barrier between every write (something trivial to do with an explicit
> barrier in your driver), it is not very performant to do so, since most
> writes don't have an explicit ordering...
>

I'm happy to treat MIPS as the "platform we try to properly fix this on"
versus just doing what we did over in PPC land when this came up.

So, what should it look like? Is the barrier() busdma instruction the right
method to implement this as and use in drivers?

Thanks!



-adrian