Unable to kill a non-zombie process with -9

[elof2 at sentor.se]

I guess this is a bug report for FreeBSD 10.0.



Sometimes I can't kill my snort process on FreeBSD 10.0.
It won't die, even with kill -9.

I'm not talking about a zombie process. Snort is a process that should 
die normally.
I've run snort on over 100 nodes since FreeBSD v6.x and I've never seen 
this behavior until now in FreeBSD 10.0.


Example:

#ps faxuw
USER      PID  %CPU %MEM    VSZ    RSS TT  STAT STARTED        TIME 
COMMAND
root    49222  53.4  2.2 492648 183012  -  Rs   11:46AM     7:05.59 
/usr/local/bin/snort -q -D -c snort.conf
root    47937   0.0  2.2 488552 182864  -  Ts   10:56AM    29:35.98 
/usr/local/bin/snort -q -D -c snort.conf

The pid 47937 has been killed (repeatedly) with -9.
Its status is "Ts" meaning it is Stopped.
But it won't actually die and disappear. The only way to get rid of it 
seem to be to reboot the machine. :-(

(pid 49222 is the new process that was started after 47937 was killed)


The problem doesn't happen all the time and I haven't found any patterns 
as to when it does. :-(
If I restart snort once every day, it fails to die approximately 2-4 times 
per month.
Even though the problem doesn't happen on every kill, it is a definately a 
recurring event.

I began to see it on a heavily loaded 10GE sensor, so I thought it could 
have something to do with the ix driver, or the heavy load.
But now another FreeBSD 10.0-sensor had the exact same problem, and this 
sensor don't have any 10GE NICs. In fact, this sensor has been running 
just fine with both FreeBSD 9.1 and 9.3 for the past years. Snort has 
always terminated correctly! After I reinstalled this machine with FreeBSD 
10.0 last friday, snort has then terminated correctly every day until 
today, when it failed with the above pid 47937. (this sensor use the 'em' 
driver, not 'ixgbe')

I'm running snort with the same configuration, settings, version, daq, 
libs, etc on 10.0 as I do on 9.3.
None of the 9.3 sensors have this problem, so it has to be something new 
in FreeBSD 10.0.



Q1:
Has anyone seen anything simillar, or have any clues as to what is going 
on and why?


Q2:
Is there any other way to kill and purge the stopped process? I don't want 
it laying around.
('kill -HUP 1' didn't help)




(
The closest thing I've come across myself is last year, when I 
tested enabling zerocopy-bpf in FreeBSD 9.1. Then I couldn't kill snort 
if the sniffer-interface was completely silent.
The above problem is not like this though. I haven't enabled zerocopy and 
there are lots of mirrored traffic on the sniffer interface.
)

/Elof