SMP support for XLR processors.

[C. Jayachandran]

On Tue, Apr 20, 2010 at 4:26 PM, Rui Paulo <rpaulo at freebsd.org> wrote:
> On 20 Apr 2010, at 11:49, C. Jayachandran wrote:
>
>> On Tue, Apr 20, 2010 at 4:03 PM, Rui Paulo <rpaulo at freebsd.org> wrote:
>>> On 20 Apr 2010, at 11:05, Rui Paulo wrote:
>>>
>>>> On 20 Apr 2010, at 10:52, C. Jayachandran wrote:
>>>>
>>>>> On Mon, Apr 19, 2010 at 7:27 PM, C. Jayachandran
>>>>> <c.jayachandran at gmail.com> wrote:
>>>>>> I have a possible cause for the panic with invariants - we should not
>>>>>> schedule the msgring threads unless the smp is completely up. I guess
>>>>>> we start getting message ring interrupts on before the message ring
>>>>>> threads can be scheduled.  I am trying out some changes for this -
>>>>>> will send you a patch if this fixes it.
>>>>>
>>>>> I've attached a patch that should fix the issue. The cause was the way
>>>>> message ring threads are started on individual cores and the way
>>>>> interrupts are enabled in the core.  I've moved starting message ring
>>>>> threads on other cpus to be a SYSINIT after SMP is started.  I'd
>>>>> thought originally that it was due to some clash with the changes in
>>>>> HEAD - but looks like I was completely off-track there.
>>>>>
>>>>> Please let me know if you don't get multi-user with 32 cpus with this
>>>>> patch. There is still the original hang in buildworld, but that should
>>>>> be a bug elsewhere
>>>>>
>>>>> I have a copy at http://sites.google.com/site/cjayachandran/files too
>>>>
>>>> This works perfectly, thanks!
>>>
>>> On further inspection, I noticed that the load avg is now 7.
>>>
>>> last pid:  1613;  load averages:  6.99,  6.97,  6.08    up 0+00:30:11  10:32:48
>>> 108 processes: 40 running, 24 sleeping, 44 waiting
>>> CPU:  0.0% user,  0.0% nice, 21.9% system,  0.0% interrupt, 78.1% idle
>>> Mem: 8444K Active, 6028K Inact, 37M Wired, 308K Cache, 6800K Buf, 3190M Free
>>> Swap:
>>>
>>>  PID USERNAME  THR PRI NICE   SIZE    RES STATE   C   TIME   WCPU COMMAND
>>>   10 root       32 171 ki31     0G     0G CPU0    0 263:26 2500.00% idle
>>>   17 root        1 -16    -     0K     0G CPU12   2   0:00 100.00% msg_intr12
>>>   15 root        1 -16    -     0K     0G CPU4    2   0:00 100.00% msg_intr4
>>>   16 root        1 -16    -     0K     0G CPU8    2   0:00 100.00% msg_intr8
>>>   20 root        1 -16    -     0K     0G CPU24   1   0:00 100.00% msg_intr24
>>>   19 root        1 -16    -     0K     0G CPU20   1   0:00 100.00% msg_intr20
>>>   21 root        1 -16    -     0K     0G CPU28   1   0:00 100.00% msg_intr28
>>>   18 root        1 -16    -     0K     0G CPU16   1   0:00 100.00% msg_intr16
>>>
>>> What are these msg_intrXX kprocs doing?
>>
>> They should really be sleeping unless there is a lot of network
>> traffic :)  The msg_intr threads are interrupt handlers which we run
>> one per core, in the first thread of each core.  They were modelled
>> after interrupt threads (in FreeBSD 6). This should be sleeping until
>> there is a message ring interrupt (which tells us that an IO has send
>> data to our core over the message ring).
>>
>> Thanks for the report - I will look at the sleep logic.


I'm not seeing the issue here(my output for ref below).  The rge patch
should not really make a difference - but it will be good to try with
that.  The only other difference I can think of between our configs is
MFS root/NFS root and rge0/rge1 - but none of these should affect the
message ring threads.  Can you send me the config you use?

regards,
JC.


114 processes: 33 running, 29 sleeping, 52 waiting
CPU:  0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle
Mem: 7380K Active, 9252K Inact, 38M Wired, 8K Cache, 9200K Buf, 3160M Free
Swap:

  PID USERNAME  THR PRI NICE   SIZE    RES STATE   C   TIME   WCPU COMMAND
   10 root       32 171 ki31     0G     0G CPU0    0  61:46 3199.46% idle
 1802 rmi         1  44    0     0G     0G CPU1    1   0:00  0.29% top
   11 root       44 -68    -     0K     0G WAIT    0   0:00  0.00% intr
    0 root        3   8    0     0G     0G -       0   0:00  0.00% kernel
 1789 root        1  55    0     0G     0G sbwait  0   0:00  0.00% sshd
 1784 root        1  70    0     0G     0G ttyin   1   0:00  0.00% csh
    1 root        1  44    0     0G     0G wait   26   0:00  0.00% init
 1783 root        1  50    0     0G     0G wait    1   0:00  0.00% login
    6 root        1  -8    -     0K     0G mdwait  3   0:00  0.00% md0
 1792 rmi         1  44    0     0G     0G select  0   0:00  0.00% sshd
   13 root        1 -16    -     0K     0G WAIT    0   0:00  0.00% msg_intr0
   22 root        1 -16    -     0K     0G WAIT   28   0:00  0.00% msg_intr28
   12 root        1 -16    -     0K     0G -       2   0:00  0.00% yarrow
 1424 root        1  44    0     0G     0G select  3   0:00  0.00% syslogd
   19 root        1 -16    -     0K     0G WAIT   16   0:00  0.00% msg_intr16
   18 root        1 -16    -     0K     0G WAIT   12   0:00  0.00% msg_intr12
   21 root        1 -16    -     0K     0G WAIT   24   0:00  0.00% msg_intr24
   17 root        1 -16    -     0K     0G WAIT    8   0:00  0.00% msg_intr8
   20 root        1 -16    -     0K     0G WAIT   20   0:00  0.00% msg_intr20
 1793 rmi         1  44    0     0G     0G wait    0   0:00  0.00% sh
    2 root        1  -8    -     0K     0G -       4   0:00  0.00% g_event
    3 root        1  -8    -     0K     0G -       4   0:00  0.00% g_up
   16 root        1 -16    -     0K     0G WAIT    4   0:00  0.00% msg_intr4