LOCAL_CPUS vs INTR_CPUS

[K. Macy]

>> In reference to:
>> https://github.com/freebsd/freebsd/compare/master...bsdjhb:numa_bus_get_cpus
>>
>>
>> The LOCAL_CPUS option will return the cpuset corresponding to the NUMA
>> (_pxm) domain id of a given device. INTR_CPUS can narrow that further.
>> However, in this particular branch INTR_CPUS will only ever return the
>> value given by LOCAL_CPUS. When and where would this come in to
>> effect? Do the new Haswells have cores in the same memory domain but
>> with non-uniform access to the PCIe bus?
>
> On x86 this means that INTR_CPUS only includes one thread per core in a
> the NUMA domain a device belongs to whereas LOCAL_CPUS will include all
> threads in the domain.  So if you have a dual-socket 8-core SB/IB system
> for example, a device on the IO hub for the first processor would use
> CPUs 0-15 for LOCAL_CPUS and 0,2,4,6,8,10,12,14 for INTR_CPUS if HTT is
> enabled.  If it is disabled, both requests would return CPUs 0-7.
>

I was confused by the fact that it was propagating it up the chain
only to have the nexus return intr_cpus and was in turn overlooking
intr_cpus being initialized in set_interrupt_apic_ids . The appearance
being that it was leaving the door open for some intermediate node to
narrow the cpuset - and nexus was just the default. As it stands now
just directly OR'ing LOCAL_CPUS  with intr_cpus now that you've made
it a global would make more sense. But perhaps the added flexibility
will prove useful in the future.

Thanks.