netgraph modules won't unload after use

[Ermal Luçi]

On Wed, Mar 18, 2009 at 5:16 PM, Julian Elischer <julian at elischer.org> wrote:
> Ash Gokhale wrote:
>>
>>  I'm developing a kernel module that will be doing inspection and needed
>> access to raw network frames,
>>  so I turned to netgraph  as the solution.However it seems that netgraph
>> will not permit a module
>>  to unload once it's participated in a mkpeer/connect operation. Rebooting
>> to remove a module is
>>  angrymaking (not like mtx/sleep crashes).
>>
>>    This going into the kernel because my bpf based userland stuff is
>> probably not going to hold up to the packet rate.
>>
>> Should I file a PR? Or is there magic in the documentation I havn't found?
>>
>>
>> I've observed the trouble in 7.0 release, and tonight's  7_RELENG, with
>> GENERIC + WITNESS/INVARIANTS
>>
>> The module code  ( cobbled together from netgraph/ng_sample.c / ng_echo.c)
>> http://pastebin.com/m31b6ece6
>>
>> The module loads and unloads fine until connected to a netgraph hook:
>
> hmm they are supposed to, and they did in the past..
> let me check...
>
> root at trafmon1:kldload ng_ether
> root at trafmon1:ifconfig
> bge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
>        options=9b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
>        ether 00:11:43:30:fb:8a
>        inet 10.7.2.3 netmask 0xffffff00 broadcast 10.7.2.255
>        media: Ethernet autoselect (100baseTX <full-duplex>)
>        status: active
> bge1: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
>        options=9b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
>        ether 00:11:43:30:fb:8b
>        media: Ethernet autoselect (none)
>        status: no carrier
> fxp0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
>        options=b<RXCSUM,TXCSUM,VLAN_MTU>
>        ether 00:0e:0c:62:aa:14
>        inet 10.7.0.101 netmask 0xffffff00 broadcast 10.7.0.255
>        media: Ethernet autoselect (100baseTX <full-duplex>)
>        status: active
> lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
>        inet 127.0.0.1 netmask 0xff000000
> root at trafmon1:ngctl
> + list
> There are 4 total nodes:
>  Name: bge0            Type: ether           ID: 00000002   Num hooks: 0
>  Name: bge1            Type: ether           ID: 00000003   Num hooks: 0
>  Name: ngctl4252       Type: socket          ID: 00000005   Num hooks: 0
>  Name: fxp0            Type: ether           ID: 00000004   Num hooks: 0
> + mkpeer bge0: hole lower hook
> + list
> There are 5 total nodes:
>  Name: <unnamed>       Type: hole            ID: 00000006   Num hooks: 1
>  Name: bge0            Type: ether           ID: 00000002   Num hooks: 1
>  Name: bge1            Type: ether           ID: 00000003   Num hooks: 0
>  Name: ngctl4252       Type: socket          ID: 00000005   Num hooks: 0
>  Name: fxp0            Type: ether           ID: 00000004   Num hooks: 0
> + shutdown [6]:
> + list
> There are 4 total nodes:
>  Name: bge0            Type: ether           ID: 00000002   Num hooks: 0
>  Name: bge1            Type: ether           ID: 00000003   Num hooks: 0
>  Name: ngctl4252       Type: socket          ID: 00000005   Num hooks: 0
>  Name: fxp0            Type: ether           ID: 00000004   Num hooks: 0
> + quit
> root at trafmon1:kldstat -v
> Id Refs Address    Size     Name
>  1   36 0xc0400000 6a9c28   kernel (/boot/kernel/kernel)
>
> [...]
>
>  7    1 0xccb16000 4000     ng_ether.ko (/boot/kernel/ng_ether.ko)
>        Contains modules:
>                Id Name
>                246 ng_ether
>  8    1 0xccb1b000 2000     ng_hole.ko (/boot/kernel/ng_hole.ko)
>        Contains modules:
>                Id Name
>                247 ng_hole
> root at trafmon1:klunload ng_hole
> klunload: Command not found.
> root at trafmon1:kldunload ng_hole
> root at trafmon1:kldunload ng_ether
> kldunload: can't unload file: Device busy
> root at trafmon1:kldstat -v
> Id Refs Address    Size     Name
>  1   36 0xc0400000 6a9c28   kernel (/boot/kernel/kernel)
>
> [...]
>
>  7    1 0xccb16000 4000     ng_ether.ko (/boot/kernel/ng_ether.ko)
>        Contains modules:
>                Id Name
>                246 ng_ether
> root at trafmon1:
>
>
> this is expected.  ng-ether is not unloadable as the connections are too
> complicated to unwind easily.. one day....
>
> root at trafmon1:
>
>>> Stop in /root/tmp/food.ko.
>>> Exit 1
>>> #Mar 18 03:14:31  kernel: quiesced
>>>  :ro:~/tmp/food.ko:3:14:31:32
>>> Mar 18 03:14:31  kernel: foodmod unloaded
>>
>>
>> Seems that I can't unload some of the other netgraph types either ( it's
>> not just me):
>>
>>> #kldunload ng_ether
>>> :ro:~/tmp/food.ko:3:24:07:41
>>> kldunload: can't unload file: Device busy
>>> Exit 1
>
> try unloading a node that IS unloadable..
>
> see the following comment in ng_ehter.c:
>         /*
>          * Note that the base code won't try to unload us until
>          * all nodes have been removed, and that can't happen
>          * until all Ethernet interfaces are removed. In any
>          * case, we know there are no nodes left if the action
>          * is MOD_UNLOAD, so there's no need to detach any nodes.
>          */
>
> so we can't run this unless you have removeable ethernet interfaces, and
> have removed them.
>
>
>
Just saw the message.
Actually ng_ether can be unloaded but this is left at your own risk
and there is a comment in code about this.
for each $if in `ifconfig -l`
ngctl shutdown $if:
kldunload ng_ether

This brings the question that should ng_ether be taught about ifnet
events(arrival/departure)?!

-- 
Ermal