panic: Assertion pgrp->pg_jobc > 0 failed at kern_proc.c:816

[John Baldwin]

On 12/28/20 12:24 PM, John Baldwin wrote:
> I got this panic again today in a VM when quitting a gdb
> session after killing a child process via 'kill'.
> 
> panic: Assertion pgrp->pg_jobc > 0 failed at /git/bhyve/sys/kern/kern_proc.c:816
> cpuid = 1
> time = 1609185862
> KDB: stack backtrace:
> db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe00946547f0
> vpanic() at vpanic+0x181/frame 0xfffffe0094654840
> panic() at panic+0x43/frame 0xfffffe00946548a0
> _refcount_update_saturated() at _refcount_update_saturated/frame 0xfffffe00946548e0
> killjobc() at killjobc+0x6a6/frame 0xfffffe0094654940
> exit1() at exit1+0x6af/frame 0xfffffe00946549b0
> sys_sys_exit() at sys_sys_exit+0xd/frame 0xfffffe00946549c0
> amd64_syscall() at amd64_syscall+0x12e/frame 0xfffffe0094654af0
> fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe0094654af0
> --- syscall (1, FreeBSD ELF64, sys_sys_exit), rip = 0x8024c8f0a, rsp = 0x7fffffffe358, rbp = 0x7fffffffe370 ---
> KDB: enter: panic
> [ thread pid 44034 tid 102484 ]
> Stopped at      kdb_enter+0x37: movq    $0,0x10ab066(%rip)
> 
> From what I can tell, the child process that was killed via 'kill' has
> not yet exited and is stuck in ptracestop() from fork():
> 
> (kgdb) where
> #0  sched_switch (td=0xfffffe0094001a00, flags=<optimized out>) at /git/bhyve/sys/kern/sched_ule.c:2147
> #1  0xffffffff80bf4015 in mi_switch (flags=266) at /git/bhyve/sys/kern/kern_synch.c:542
> #2  0xffffffff80bfeba5 in thread_suspend_switch (td=<optimized out>, p=<optimized out>)
>     at /git/bhyve/sys/kern/kern_thread.c:1477
> #3  0xffffffff80bef04b in ptracestop (td=0xfffffe0094001a00, sig=17, si=0x0)
>     at /git/bhyve/sys/kern/kern_sig.c:2642
> #4  0xffffffff80ba1a54 in fork_return (td=0xfffffe0094001a00, frame=0xfffffe0094671b00)
>     at /git/bhyve/sys/kern/kern_fork.c:1106
> #5  0xffffffff80ba18b0 in fork_exit (callout=0xffffffff80ba1950 <fork_return>, arg=0xfffffe0094001a00, 
>     frame=0xfffffe0094671b00) at /git/bhyve/sys/kern/kern_fork.c:1069
> #6  <signal handler called>
> #7  0x00000008007b71aa in ?? ()
> 
> kgdb can't find the panicking process due to the zombproc removal, so I will
> have to go work on kgdb to recover from that change. :(

I've come up with a shorter reproducer (original was trying to debug a perl script
in OpenSSL's test suite).

Compile this program:

#include <sys/types.h>
#include <sys/wait.h>
#include <err.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>

int
main(void)
{
	pid_t pid, wpid;

	pid = fork();
	if (pid == -1)
		err(1, "fork");
	if (pid == 0) {
		printf("I'm in the child\n");
		exit(1);
	}
	printf("I'm in the parent\n");
	wpid = waitpid(pid, NULL, 0);
	if (wpid < 0)
		err(1, "waitpid");

	return (0);
}

Then in gdb do the following:

# gdb101 ./forktest
...
(gdb) catch fork
Catchpoint 1 (fork)
(gdb) r
Starting program: /mnt/forktest/forktest 

Catchpoint 1 (forked process 830), _fork () at _fork.S:4
4       _fork.S: No such file or directory.
(gdb) kill
Kill the program being debugged? (y or n) y
[Inferior 1 (process 828) killed]
(gdb) quit
<panic>

-- 
John Baldwin