Permit init(8) use its own cpuset group.

[Alexander V. Chernikov]

On 06.06.2014 01:04, Alexander V. Chernikov wrote:
> On 05.06.2014 23:59, John Baldwin wrote:
>> On Thursday, June 05, 2014 3:46:15 pm Alexander V. Chernikov wrote:
>>> On 05.06.2014 18:09, John Baldwin wrote:
>>>> On Wednesday, June 04, 2014 3:16:59 pm Alexander V. Chernikov wrote:
>>>>> On 04.06.2014 19:06, John Baldwin wrote:
>>>>>> On Monday, June 02, 2014 12:48:50 pm Konstantin Belousov wrote:
>>>>>>> On Mon, Jun 02, 2014 at 06:52:10PM +0400, Alexander V. Chernikov wrote:
>>>>>>>> Hello list!
>>>>>>>>
>>>>>>>> Currently init(8) uses group 1 which is root group.
>>>>>>>> Modifications of this group affects both kernel and userland threads.
>>>>>>>> Additionally, such modifications are impossible, for example, in 
>>>> presence
>>>>>>>> of multi-queue NIC drivers (like igb or ixgbe) which binds their 
>> threads
>>>>>> to
>>>>>>>> particular cpus.
>>>>>>>>
>>>>>>>> Proposed change ("init_cpuset" loader tunable) permits changing cpu
>>>>>>>> masks for
>>>>>>>> userland more easily. Restricting user processes to migrate to/from 
>> CPU
>>>>>>>> cores
>>>>>>>> used for network traffic processing is one of the cases.
>>>>>>>>
>>>>>>>> Phabricator: https://phabric.freebsd.org/D141 (the same version 
>> attached
>>>>>>>> inline)
>>>>>>>>
>>>>>>>> If there are no objections, I'll commit this next week.
>>>>>>> Why is the tunable needed ?
>>>>>> Because some people already depend on doing 'cpuset -l 0 -s 1'.  It is 
>>>> also
>>>>>> documented in our manpages that processes start in cpuset 1 by default 
>> so
>>>>>> that you can use 'cpuset -l 0 -s 1' to move all processes, etc.
>>>>>>
>>>>>> For the stated problem (bound ithreads in drivers), I would actually 
>> like 
>>>> to
>>>>>> fix ithreads that are bound to a specific CPU to create a different 
>> cpuset
>>>>>> instead so they don't conflict with set 1.
>>>>> Yes, this seem to be much better approach.
>>>>> Please take a look on the new patch (here or in the phabricator).
>>>>> Comments:
>>>>>
>>>>> Use different approach for modifyable root set:
>>>>>
>>>>>   * Make sets in 0..15 internal
>>>>>   * Define CPUSET_SET1 & CPUSET_ITHREAD in that range
>>>>>   * Add cpuset_lookup_builtin() to retrieve such cpu sets by id
>>>>>   * Create additional root set for ithreads
>>>>>   * Use this set in ithread_create()
>>>>>   * Change intr_setaffinity() to use cpuset_iroot (do we really need 
>> this)?
>>>>>
>>>>> We can probably do the same for kprocs, but I'm unsure if we really need 
>> it.
>>>>
>>>> I imagined something a bit simpler.  Just create a new set in 
>> intr_event_bind
>>>> and bind the ithread to the new set.  No need to have more magic set ids, 
>> etc. 
>>> Well, we also have userland which can modify given changes via `cpuset
>>> -x', so we need to be able to add some more logic on set
>>> allocation/keeping. Additionally, we can try to do the same via `cpuset
>>> -t', so introducing something like cpuset_setIthread() and hooking into
>>> intr_event_bind() won't probably be enough. At least I can't think out a
>>> quick and easy way to do this.
>>
>> cpuset -x calls intr_event_bind().  If you just do it there you fix both 
>> places.
Yup. I was wrong. This version seems to be simpler while doing the right
thing.

> 1:04 [0] ra# procstat -t 12 | grep irq275
>    12 100121 intr             irq275: ix0:qu1    2  127 wait    -
> 1:04 [0] ra# cpuset -g -x 275
> irq 275 mask: 2
> 1:04 [0] ra# cpuset -g -t 100121
> tid 100121 mask: 2
> 1:04 [0] ra# cpuset -l 3 -t 100121
> -------------------------^^^------
> 1:05 [0] ra# cpuset -g -t 100121
> tid 100121 mask: 3
> 
>>
>>>> That also means that an ithread that isn't bound to a specific CPU via 
>> either 
>>>> 'cpuset -x' or BUS_BIND_INTR() will honor 'cpuset -s 1' like other
>>>> kernel processes.  I think that's probably fine and sensible.  The issue 
>> is
>>> Well, it is questionable. Kernel threads are a bit different in terms of
>>> TLB changes, memory working set and so on. (Personally I'd prefer to
>>> separate user / kthreads / ithreads to different sets in HEAD but that's
>>> another story).
>>>
>>> Anyway, we probably can (and should) MFC a bit different version which
>>> tries to change several sets at once if user supplied set 1 as argument.
>>
>> No, I don't think we need umpteen special sets.  I think we just need to fix
>> this one specific case of bound ithreads and everything else will work fine.  
>> If someone wants to move kprocs out of set 1, they can already do that with 
>> the existing tools via cpuset -C, etc.
>>
> 

-------------- next part --------------
A non-text attachment was scrubbed...
Name: D141_6.diff
Type: text/x-patch
Size: 2940 bytes
Desc: not available
URL: <http://lists.freebsd.org/pipermail/freebsd-hackers/attachments/20140607/ae89b8f9/attachment.bin>