ps -axl during textdumps occasionally segfaults with a HUGE ps.core

[Bruce Cran]

On Wed, 29 Jul 2009 22:19:47 +0200
Thomas Backman <serenity at exscape.org> wrote:

> All the info I happen to have:
> 
> (from core.txt.X)
> "ps -axl
> 
> Segmentation fault (core dumped)"
> 
> The last core I got (/ps.core) was 1076211712 bytes (1026 MiB).
> 
> Anyone else with this problem?
> Unfortunately, I deleted the most recent core and so can't gdb it,
> at least not right now. I did try it on the first one, but got a
> very broken backtrace.

Can you try the patches at
http://www.cran.org.uk/~brucec/libkvm_20090822.diff and
http://www.cran.org.uk/~brucec/ps_20090822.diff please?  I've
tested them on both amd64 and i386 PCs and it seems to work. It turned
out there were 3 bugs:

1. The call to kvm_nlist on line 558 of lib/libkvm/kvm_proc.c was
failing with -1, but the code assumed it was returning a positive
number and so ended up walking off the end of the array.  gavin@
created the patch - a standalone version is at
http://people.freebsd.org/~gavin/PRs/137890.2.diff but has been
integrated into libkvm_20090822.diff.  There may be more calls to
kvm_nlist that don't have the correct error checking in kvm_proc.c

2. kvm_open(3) states that execfile can be NULL, but line 215 of
bin/ps/ps.c initializes it to _PATH_DEVNULL.  That was why kvm_nlist was
failing.

3. On line 154 of kvm_proc.c bcopy is called with the
address in ucred.cr_groups. It appears that it's a kernel address and I
guess that an extra call to KREAD needs to be made.

At the same time as fixing those bugs I bumped WARNS up and fixed the
resulting errors: invalid formatting strings, casts and unused
variables - and converted some functions from K&R to ANSI to try and get
better warnings of any potential problems from gcc. I can split out the
actual bug fixes into separate patches if needed.

-- 
Bruce