Unable to kill a non-zombie process with -9

[John-Mark Gurney]

elof2 at sentor.se wrote this message on Wed, Oct 08, 2014 at 13:30 +0200:
> 
> I guess this is a bug report for FreeBSD 10.0.
> 
> 
> 
> Sometimes I can't kill my snort process on FreeBSD 10.0.
> It won't die, even with kill -9.
> 
> I'm not talking about a zombie process. Snort is a process that should 
> die normally.
> I've run snort on over 100 nodes since FreeBSD v6.x and I've never seen 
> this behavior until now in FreeBSD 10.0.
> 
> 
> Example:
> 
> #ps faxuw
> USER      PID  %CPU %MEM    VSZ    RSS TT  STAT STARTED        TIME 
> COMMAND
> root    49222  53.4  2.2 492648 183012  -  Rs   11:46AM     7:05.59 
> /usr/local/bin/snort -q -D -c snort.conf
> root    47937   0.0  2.2 488552 182864  -  Ts   10:56AM    29:35.98 
> /usr/local/bin/snort -q -D -c snort.conf

What is the MWCHAN?  add l to the ps command...

> The pid 47937 has been killed (repeatedly) with -9.
> Its status is "Ts" meaning it is Stopped.

have you tried to kill -CONT <pid> to resume it?

> But it won't actually die and disappear. The only way to get rid of it 
> seem to be to reboot the machine. :-(
> 
> (pid 49222 is the new process that was started after 47937 was killed)
> 
> 
> The problem doesn't happen all the time and I haven't found any patterns 
> as to when it does. :-(
> If I restart snort once every day, it fails to die approximately 2-4 times 
> per month.
> Even though the problem doesn't happen on every kill, it is a definately a 
> recurring event.

Can you run kgdb on the machine? (yes, it works on a live machine), use
info threads to find the thread id, and then use thread <threadid> to
switch to it, and run bt to get a back trace...

> I began to see it on a heavily loaded 10GE sensor, so I thought it could 
> have something to do with the ix driver, or the heavy load.
> But now another FreeBSD 10.0-sensor had the exact same problem, and this 
> sensor don't have any 10GE NICs. In fact, this sensor has been running 
> just fine with both FreeBSD 9.1 and 9.3 for the past years. Snort has 
> always terminated correctly! After I reinstalled this machine with FreeBSD 
> 10.0 last friday, snort has then terminated correctly every day until 
> today, when it failed with the above pid 47937. (this sensor use the 'em' 
> driver, not 'ixgbe')
> 
> I'm running snort with the same configuration, settings, version, daq, 
> libs, etc on 10.0 as I do on 9.3.
> None of the 9.3 sensors have this problem, so it has to be something new 
> in FreeBSD 10.0.

-- 
  John-Mark Gurney				Voice: +1 415 225 5579

     "All that I will do, has been done, All that I have, has not."