istgt does not appear to shut down cleanly

[John]

Hi Folks,

   I've been putting together a zfs server which exports zvols
via istgt:

#istgt -V
istgt version 0.4
istgt extra version 20111008

   and about a 1 month old 9-stable:

9.0-STABLE FreeBSD 9.0-STABLE #0 r231864M: Fri Feb 17 23:21:34 UTC 2012

   When adding a new LUC/LUN combination, I want to terminate and then
restart the daemon. However, istgt does not appear to completely exit.

+ /bin/ps -Ho pid,lwp,inblk,oublk,%cpu,%mem,stat,time,mwchan,xstat,command -p 18261
  PID    LWP INBLK OUBLK  %CPU %MEM STAT    TIME MWCHAN XSTAT COMMAND
18261 102021    68     5   0.0  0.0 S    0:00.01 kqread     0 istgt -c /config/pool0/istgt/test1.istgt.conf -p /config/pool0/istgt/test1.pid -D -t all
18261 102065     0     0   0.0  0.0 I    0:00.00 sigwai     0 istgt -c /config/pool0/istgt/test1.istgt.conf -p /config/pool0/istgt/test1.pid -D -t all
18261 102066     0     0   0.0  0.0 I    0:00.00 uwait      0 istgt -c /config/pool0/istgt/test1.istgt.conf -p /config/pool0/istgt/test1.pid -D -t all
18261 102067     0     0   0.0  0.0 I    0:00.00 uwait      0 istgt -c /config/pool0/istgt/test1.istgt.conf -p /config/pool0/istgt/test1.pid -D -t all
18261 102068     0     0   0.0  0.0 I    0:00.00 uwait      0 istgt -c /config/pool0/istgt/test1.istgt.conf -p /config/pool0/istgt/test1.pid -D -t all
18261 102069     0     0   0.0  0.0 I    0:00.00 uwait      0 istgt -c /config/pool0/istgt/test1.istgt.conf -p /config/pool0/istgt/test1.pid -D -t all
+ kill -INT 18261

I then loop waiting for the pidfile to be removed:

+ [ -s /config/pool0/istgt/test1.pid ]
+ sleep 0.1
+ [ -s /config/pool0/istgt/test1.pid ]
+ sleep 0.1
+ [ -s /config/pool0/istgt/test1.pid ]
+ sleep 0.1
+ [ -s /config/pool0/istgt/test1.pid ]
+ sleep 0.1
+ [ -s /config/pool0/istgt/test1.pid ]

and then finally for the process to exit (it doesn't matter how long
I wait), but it never does:

+ jot 10
+ sleep 0.1
+ kill -0 18261
...
+ kill -0 18261
+ sleep 0.1
+ kill -0 18261
+ /bin/ps -Ho pid,lwp,inblk,oublk,%cpu,%mem,stat,time,mwchan,xstat,command -p 18261
  PID    LWP INBLK OUBLK  %CPU %MEM STAT    TIME MWCHAN XSTAT COMMAND
18261 102021    68     5   0.0  0.0 S    0:00.01 uwait      0 istgt -c /config/pool0/istgt/test1.istgt.conf -p /config/pool0/istgt/test1.pid -D -t all
18261 102065     0     0   0.0  0.0 I    0:00.00 sigwai     0 istgt -c /config/pool0/istgt/test1.istgt.conf -p /config/pool0/istgt/test1.pid -D -t all

Finally I just forcefully kill it..

+ kill -9 18261

Above, the LUC threads appear to exiting, but that's all. From looking
at the source:

        istgt_remove_pidfile(istgt);
        istgt_close_log();
        istgt->config = NULL;
        istgt_free_config(config);
        istgt_set_state(istgt, ISTGT_STATE_SHUTDOWN);

        /* stop signal thread */
        rc = pthread_join(sigthread, NULL);

I think it's hanging on the pthread_join().

Does anyone else have any experience with this issue? Or a better way
to do this?

Many Thanks!!

John