Kernelspace C11 atomics for MIPS

[Warner Losh]

On Jun 4, 2013, at 6:22 AM, Andrew Duane wrote:

> 
>> -----Original Message-----
>> From: owner-freebsd-mips at freebsd.org [mailto:owner-freebsd-
>> mips at freebsd.org] On Behalf Of Juli Mallett
>> Sent: Monday, June 03, 2013 11:55 PM
>> To: Adrian Chadd
>> Cc: Ed Schouten; freebsd-mips at FreeBSD.org; FreeBSD-arch
>> Subject: Re: Kernelspace C11 atomics for MIPS
>> 
>> On Mon, Jun 3, 2013 at 7:45 PM, Adrian Chadd <adrian at freebsd.org> wrote:
>> 
>>> Speaking of this; any idea why the SYNC operators have 8 NOPs following
>>> them?
>>> 
>>> I noticed that when going through disassemblies of various mips24k .o
>>> files.
>>> 
>> 
>> To drain the pipeline on certain deficient (and mostly older) CPUs by way
>> of guesswork and a little vague magic.  Most CPUs we support, I would
>> guess, do not need this, and it continues to exist solely for
>> hysterical reasons.
>> 
>> I've certainly gotten rid of them and some other cargo cult synchronization
>> on Octeon for testing and had it survive under considerable load, and
>> occasionally with some slight speedups (for some more commonly-used or
>> slower things than Just a Bunch Of NOPs.)
>> 
>> The trouble is that proving they aren't necessary requires being rigorous
>> and careful in understanding documentation and errata, and FUD about their
>> possible necessity is somewhat-intimidating.  It's not an easy kind of
>> corruption/unreliability/etc., to prove the lack of empirically.
> 
> The various CPU types are supposed to specify exactly how many NOPs are needed, what kind of barrier is needed and where, and which type of NOP is needed (Alpha had at least two). The barriers are designed to insure correct operation ordering across the memory architectures including write buffers, DMA hardware, L1/L2[/L3] caches and their connections to the cores. The CPU should specify exactly what is needed where, and why. It should never be "superstition". There is an exact hardware reason for every sync/barrier operation, and every NOP needed, just like the COP0 hazards.

Except that none of the examples in the ISA manual have them, and there's no mention of them at all, unlike COP0 hazards. I know of only one case in the Linux tree where it is done (for the au1xxxx cores).  There's another place where it is defined in a function, but that function is never called for older Broadcom MIPS.

In NetBSD, extra NOPs are not inserted at all. They do do two syncs for the SB1250 PASS 1.

None of the docs I've seen for latter-day MIPS CPUs document the need for NOPs. In fact, the only place I think that I've seen them was on one or two of the older R8000, R10000 and R120000 errata that stated they were needed there. Since these were the first complicated multi-issue designs, it wasn't surprising that the NOPs were needed as work arounds. I can't find these errata with a google search now, so I can't confirm this dim memory that I have.

The reason the NOPs are there today likely is superstition. A cargo-cult workaround from the past whose days have come and gone, leaving nothing but an echo in the code.

> Given that, Juli's last paragraph is right on point. The documentation can be dense and difficult to understand, since it's usually written by hardware engineers :-) And since getting it wrong can make for some really subtle, intermittent, and incredibly hard to diagnose problems, it's easier to err on the side of caution. It also happens that different CPUs included in a certain compile switch may have different requirements, so you have to use worst case.

I'll agree about the dense documentation. But usually that's around all the crazy cache effects that one must understand to cope with the design that puts half of the cache management in software.

I'm all for ditching them unless a specific reason for keeping them can be found.

>> Juli.
>> _______________________________________________
>> freebsd-mips at freebsd.org mailing list
>> http://lists.freebsd.org/mailman/listinfo/freebsd-mips
>> To unsubscribe, send any mail to "freebsd-mips-unsubscribe at freebsd.org"
>> 
> 
> 
> ....................................
> Andrew L. Duane
> Resident Architect - AT&T Technical Lead
> m   +1 603.770.7088
> o   +1 408.933.6944 (2-6944)
> skype: andrewlduane
> aduane at juniper.net
> 
> 
> 
> _______________________________________________
> freebsd-arch at freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-arch
> To unsubscribe, send any mail to "freebsd-arch-unsubscribe at freebsd.org"