CACHE_LINE_SIZE on x86

[Jim Harris]

On Thu, Oct 25, 2012 at 2:32 PM, John Baldwin <jhb at freebsd.org> wrote:
> On Thursday, October 25, 2012 12:30:02 pm Andre Oppermann wrote:
>> On 25.10.2012 15:18, John Baldwin wrote:
>> > On Wednesday, October 24, 2012 3:13:38 pm Jim Harris wrote:
>> >> While investigating padding of the ULE scheduler locks (r242014), I
>> >> recently discovered that CACHE_LINE_SIZE on x86 is defined as 128 (not
>> >> 64).  From what I can tell from svn logs, this was to account for 128
>> >> byte cache "sectors" that existed on the NetBurst micro architecture
>> >> CPUs.
>> >>
>> >> I'm curious if there's been consideration in changing this back to 64?
>> >>   With maybe a kernel config option to modify it?  On 2S systems (but
>> >> not on 1S systems), I see a benefit using CACHE_LINE_SIZE=128 for the
>> >> scheduler locks.  I suspect this is related to data prefetching but am
>> >> still running experiments to verify.
>> >
>> > All the i7 and later systems I've seen (maybe even Penryn?) have a BIOS option
>> > (typically enabled by default) to enable adjacent cache line prefetching (my
>> > understanding is that this only affects the LLC, and it seems to always fetch
>> > an aligned 128 bytes, so if your miss is in the "second" line it fetches N-1
>> > and N, not always fetching N and N+1).  That is why I thought we still use 128
>> > bytes on x86.
>>
>> As long as the additionally prefetched cache line has its own MOESI
>> state and gets marked as shared there is not problem with using only
>> 64B alignment and padding.
>
> It would be good to know though if there are performance benefits from
> avoiding sharing across paired lines in this manner.  Even if it has
> its own MOESI state, there might still be negative effects from sharing
> the pair.

On 2S, I do see further benefits by using 128 byte padding instead of
64.  On 1S, I see no difference.  I've been meaning to turn off
prefetching on my system to see if it has any effect in the 2S case -
I can give that a shot tomorrow.

> --
> John Baldwin