Re: Hang ast / pipelk / piperd

From: Paul Floyd <paulf2718_at_gmail.com>
Date: Mon, 30 May 2022 18:00:40 UTC
> "procstat -kk <valgrind PID>" might help to reveal what's going on,
> since it sounds like the hand/livelock is happening somewhere in the
> kernel.

Hi

Here is the output

paulf@freebsd:~ $ procstat -kk 864
   PID    TID COMM                TDNAME              KSTACK 

   864 100075 none-amd64-freebsd  -                   mi_switch+0xc2 
sleepq_catch_signals+0x2e6 sleepq_wait_sig+0x9 _sleep+0x1f2 
umtxq_sleep+0x143 do_wait+0x3e5 __umtx_op_wait+0x53 sys__umtx_op+0x7e 
amd64_syscall+0x10c fast_syscall_common+0xf8

   864 100175 none-amd64-freebsd  -                   mi_switch+0xc2 
intr_event_handle+0x167 intr_execute_handlers+0x4b Xapic_isr1+0xdc 
setrunnable+0x31 wakeup_one+0x1f pipe_read+0x38f dofileread+0x81 
sys_read+0xbc amd64_syscall+0x10c fast_syscall_common+0xf8

   864 100176 none-amd64-freebsd  -                   mi_switch+0xc2 
sleepq_catch_signals+0x2e6 sleepq_wait_sig+0x9 _sleep+0x1f2 
pipe_read+0xb3 dofileread+0x81 sys_read+0xbc amd64_syscall+0x10c 
fast_syscall_common+0xf8

   864 100177 none-amd64-freebsd  -                   mi_switch+0xc2 
sleepq_catch_signals+0x2e6 sleepq_wait_sig+0x9 _sleep+0x1f2 
pipe_read+0xb3 dofileread+0x81 sys_read+0xbc amd64_syscall+0x10c 
fast_syscall_common+0xf8

   864 100178 none-amd64-freebsd  -                   mi_switch+0xc2 
sleepq_catch_signals+0x2e6 sleepq_wait_sig+0x9 _sleep+0x1f2 
pipe_read+0x3d6 dofileread+0x81 sys_read+0xbc amd64_syscall+0x10c 
fast_syscall_common+0xf8

   864 100179 none-amd64-freebsd  -                   mi_switch+0xc2 
sleepq_catch_signals+0x2e6 sleepq_wait_sig+0x9 _sleep+0x1f2 
pipe_read+0x3d6 dofileread+0x81 sys_read+0xbc amd64_syscall+0x10c 
fast_syscall_common+0xf8

   864 100180 none-amd64-freebsd  -                   mi_switch+0xc2 
sleepq_catch_signals+0x2e6 sleepq_wait_sig+0x9 _sleep+0x1f2 
pipe_read+0x3d6 dofileread+0x81 sys_read+0xbc amd64_syscall+0x10c 
fast_syscall_common+0xf8

   864 100181 none-amd64-freebsd  -                   mi_switch+0xc2 
sleepq_catch_signals+0x2e6 sleepq_wait_sig+0x9 _sleep+0x1f2 
pipe_read+0x3d6 dofileread+0x81 sys_read+0xbc amd64_syscall+0x10c 
fast_syscall_common+0xf8

   864 100182 none-amd64-freebsd  -                   mi_switch+0xc2 
sleepq_catch_signals+0x2e6 sleepq_wait_sig+0x9 _sleep+0x1f2 
pipe_read+0x3d6 dofileread+0x81 sys_read+0xbc amd64_syscall+0x10c 
fast_syscall_common+0xf8

   864 100183 none-amd64-freebsd  -                   mi_switch+0xc2 
sleepq_catch_signals+0x2e6 sleepq_wait_sig+0x9 _sleep+0x1f2 
pipe_read+0x3d6 dofileread+0x81 sys_read+0xbc amd64_syscall+0x10c 
fast_syscall_common+0xf8

   864 100184 none-amd64-freebsd  -                   mi_switch+0xc2 
sleepq_catch_signals+0x2e6 sleepq_wait_sig+0x9 _sleep+0x1f2 
pipe_read+0xb3 dofileread+0x81 sys_read+0xbc amd64_syscall+0x10c 
fast_syscall_common+0xf8

   864 100185 none-amd64-freebsd  -                   mi_switch+0xc2 
ast+0x1e6 doreti_ast+0x1f

   864 100186 none-amd64-freebsd  -                   mi_switch+0xc2 
sleepq_catch_signals+0x2e6 sleepq_wait_sig+0x9 _sleep+0x1f2 
pipe_read+0xb3 dofileread+0x81 sys_read+0xbc amd64_syscall+0x10c 
fast_syscall_common+0xf8

   864 100187 none-amd64-freebsd  -                   mi_switch+0xc2 
sleepq_catch_signals+0x2e6 sleepq_wait_sig+0x9 _sleep+0x1f2 
pipe_read+0xb3 dofileread+0x81 sys_read+0xbc amd64_syscall+0x10c 
fast_syscall_common+0xf8

   864 100188 none-amd64-freebsd  -                   mi_switch+0xc2 
sleepq_catch_signals+0x2e6 sleepq_wait_sig+0x9 _sleep+0x1f2 
pipe_read+0xb3 dofileread+0x81 sys_read+0xbc amd64_syscall+0x10c 
fast_syscall_common+0xf8

It doesn't seem to be totally hung. If I repeatedly sample I do see 
activity changing between the threads other than main (in _umtx_op) and 
12 (stuck in ast / doreti_ast. But it looks like none of the calls to 
read is completing, meaning the Valgrind scheduler is blocked.


A+
Paul