netgraph modules won't unload after use

[Ash Gokhale]

  I'm developing a kernel module that will be doing inspection and  
needed access to raw network frames,
  so I turned to netgraph  as the solution.However it seems that  
netgraph will not permit a module
  to unload once it's participated in a mkpeer/connect operation.  
Rebooting to remove a module is
  angrymaking (not like mtx/sleep crashes).

	This going into the kernel because my bpf based userland stuff is  
probably not going to hold up to the packet rate.

Should I file a PR? Or is there magic in the documentation I havn't  
found?


I've observed the trouble in 7.0 release, and tonight's  7_RELENG,  
with GENERIC + WITNESS/INVARIANTS

The module code  ( cobbled together from netgraph/ng_sample.c /  
ng_echo.c)
http://pastebin.com/m31b6ece6

The module loads and unloads fine until connected to a netgraph hook:

> #make load                                          :ro:~/tmp/ 
> food.ko:3:06:42:20
> /sbin/kldload -v /root/tmp/food.ko/food.ko
> Loaded /root/tmp/food.ko/food.ko, id=3
> #Mar 18 03:06:44  kernel: foodmod loaded            :ro:~/tmp/ 
> food.ko:3:06:44:21
> #make unload                                        :ro:~/tmp/ 
> food.ko:3:10:16:21
> /sbin/kldunload -v food.ko
> Unloading food.ko, id=3
> #Mar 18 03:10:19  kernel: quiesced                  :ro:~/tmp/ 
> food.ko:3:10:19:22
> Mar 18 03:10:19  kernel: foodmod unloaded


Fine.. so let's connect it:
> #make ath                                           :ro:~/tmp/ 
> food.ko:3:10:59:23
> kldload ng_ether
> ifconfig ath0 ssid ')(allmightytallest'
> ifconfig ath0 promisc
>>
> #Mar 18 03:11:04  kernel: ath0: permanently promiscuous mode  
> enabledo:3:11:04:24
> make connect
> #make load                                          :ro:~/tmp/ 
> food.ko:3:11:13:26
> /sbin/kldload -v /root/tmp/food.ko/food.ko
> Loaded /root/tmp/food.ko/food.ko, id=7
> #Mar 18 03:12:05  kernel: foodmod loaded
> make connect
> ngctl mkpeer ath0: food lower fredhook
> ngctl name ath0:lower eater
> sleep 1
> Mar 18 03:12:15  kernel: ngconstruct
> ifconfig ath0 down
> ifconfig ath0 up
> #Mar 18 03:12:18  kernel: not our messageunrecognized command/ 
> food.ko:3:12:16:28
> Mar 18 03:12:18  kernel: ath0: link state changed to UP
> #                                                   :ro:~/tmp/ 
> food.ko:3:12:18:28
> #Mar 18 03:12:21  kernel: ....

Fine; we get packets to the ng_rx_data procedure .. now we pull the  
plug on the netgraph

>  #make discon                                        :ro:~/tmp/ 
> food.ko:3:12:35:28
> ngctl shutdown eater:
> #Mar 18 03:12:37  kernel: discondiconnecting last ng  
> nodengshutdownko:3:12:36:29
>
> #ngctl list                                         :ro:~/tmp/ 
> food.ko:3:12:39:29
> There are 3 total nodes:
>   Name: bge0            Type: ether           ID: 00000001   Num  
> hooks: 0
>   Name: ngctl1058       Type: socket          ID: 00000008   Num  
> hooks: 0
>   Name: ath0            Type: ether           ID: 00000002   Num  
> hooks: 0
>


But the module will never unload: ( but the UNLOAD  and QUIESE event  
handlers are invoked)
> #make unload                                        :ro:~/tmp/ 
> food.ko:3:14:28:31
> /sbin/kldunload -vf food.ko
> Unloading food.ko, id=7
> kldunload: can't unload file: Device busy
> *** Error code 1
>
> Stop in /root/tmp/food.ko.
> Exit 1
> #Mar 18 03:14:31  kernel: quiesced                  :ro:~/tmp/ 
> food.ko:3:14:31:32
> Mar 18 03:14:31  kernel: foodmod unloaded


Seems that I can't unload some of the other netgraph types either  
( it's not just me):

> #kldunload ng_ether                                 :ro:~/tmp/ 
> food.ko:3:24:07:41
> kldunload: can't unload file: Device busy
> Exit 1