netgraph modules won't unload after use

[Ermal Luçi]

On Sat, Mar 28, 2009 at 8:10 PM, Ermal Luçi <eri at freebsd.org> wrote:
> On Wed, Mar 18, 2009 at 5:16 PM, Julian Elischer <julian at elischer.org> wrote:
>> Ash Gokhale wrote:
>>>
>>>  I'm developing a kernel module that will be doing inspection and needed
>>> access to raw network frames,
>>>  so I turned to netgraph  as the solution.However it seems that netgraph
>>> will not permit a module
>>>  to unload once it's participated in a mkpeer/connect operation. Rebooting
>>> to remove a module is
>>>  angrymaking (not like mtx/sleep crashes).
>>>
>>>    This going into the kernel because my bpf based userland stuff is
>>> probably not going to hold up to the packet rate.
>>>
>>> Should I file a PR? Or is there magic in the documentation I havn't found?
>>>
>>>
>>> I've observed the trouble in 7.0 release, and tonight's  7_RELENG, with
>>> GENERIC + WITNESS/INVARIANTS
>>>
>>> The module code  ( cobbled together from netgraph/ng_sample.c / ng_echo.c)
>>> http://pastebin.com/m31b6ece6
>>>
>>> The module loads and unloads fine until connected to a netgraph hook:
>>
>> hmm they are supposed to, and they did in the past..
>> let me check...
>>
>> root at trafmon1:kldload ng_ether
>> root at trafmon1:ifconfig
>> bge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
>>        options=9b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
>>        ether 00:11:43:30:fb:8a
>>        inet 10.7.2.3 netmask 0xffffff00 broadcast 10.7.2.255
>>        media: Ethernet autoselect (100baseTX <full-duplex>)
>>        status: active
>> bge1: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
>>        options=9b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
>>        ether 00:11:43:30:fb:8b
>>        media: Ethernet autoselect (none)
>>        status: no carrier
>> fxp0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
>>        options=b<RXCSUM,TXCSUM,VLAN_MTU>
>>        ether 00:0e:0c:62:aa:14
>>        inet 10.7.0.101 netmask 0xffffff00 broadcast 10.7.0.255
>>        media: Ethernet autoselect (100baseTX <full-duplex>)
>>        status: active
>> lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
>>        inet 127.0.0.1 netmask 0xff000000
>> root at trafmon1:ngctl
>> + list
>> There are 4 total nodes:
>>  Name: bge0            Type: ether           ID: 00000002   Num hooks: 0
>>  Name: bge1            Type: ether           ID: 00000003   Num hooks: 0
>>  Name: ngctl4252       Type: socket          ID: 00000005   Num hooks: 0
>>  Name: fxp0            Type: ether           ID: 00000004   Num hooks: 0
>> + mkpeer bge0: hole lower hook
>> + list
>> There are 5 total nodes:
>>  Name: <unnamed>       Type: hole            ID: 00000006   Num hooks: 1
>>  Name: bge0            Type: ether           ID: 00000002   Num hooks: 1
>>  Name: bge1            Type: ether           ID: 00000003   Num hooks: 0
>>  Name: ngctl4252       Type: socket          ID: 00000005   Num hooks: 0
>>  Name: fxp0            Type: ether           ID: 00000004   Num hooks: 0
>> + shutdown [6]:
>> + list
>> There are 4 total nodes:
>>  Name: bge0            Type: ether           ID: 00000002   Num hooks: 0
>>  Name: bge1            Type: ether           ID: 00000003   Num hooks: 0
>>  Name: ngctl4252       Type: socket          ID: 00000005   Num hooks: 0
>>  Name: fxp0            Type: ether           ID: 00000004   Num hooks: 0
>> + quit
>> root at trafmon1:kldstat -v
>> Id Refs Address    Size     Name
>>  1   36 0xc0400000 6a9c28   kernel (/boot/kernel/kernel)
>>
>> [...]
>>
>>  7    1 0xccb16000 4000     ng_ether.ko (/boot/kernel/ng_ether.ko)
>>        Contains modules:
>>                Id Name
>>                246 ng_ether
>>  8    1 0xccb1b000 2000     ng_hole.ko (/boot/kernel/ng_hole.ko)
>>        Contains modules:
>>                Id Name
>>                247 ng_hole
>> root at trafmon1:klunload ng_hole
>> klunload: Command not found.
>> root at trafmon1:kldunload ng_hole
>> root at trafmon1:kldunload ng_ether
>> kldunload: can't unload file: Device busy
>> root at trafmon1:kldstat -v
>> Id Refs Address    Size     Name
>>  1   36 0xc0400000 6a9c28   kernel (/boot/kernel/kernel)
>>
>> [...]
>>
>>  7    1 0xccb16000 4000     ng_ether.ko (/boot/kernel/ng_ether.ko)
>>        Contains modules:
>>                Id Name
>>                246 ng_ether
>> root at trafmon1:
>>
>>
>> this is expected.  ng-ether is not unloadable as the connections are too
>> complicated to unwind easily.. one day....
>>
>> root at trafmon1:
>>
>>>> Stop in /root/tmp/food.ko.
>>>> Exit 1
>>>> #Mar 18 03:14:31  kernel: quiesced
>>>>  :ro:~/tmp/food.ko:3:14:31:32
>>>> Mar 18 03:14:31  kernel: foodmod unloaded
>>>
>>>
>>> Seems that I can't unload some of the other netgraph types either ( it's
>>> not just me):
>>>
>>>> #kldunload ng_ether
>>>> :ro:~/tmp/food.ko:3:24:07:41
>>>> kldunload: can't unload file: Device busy
>>>> Exit 1
>>
>> try unloading a node that IS unloadable..
>>
>> see the following comment in ng_ehter.c:
>>         /*
>>          * Note that the base code won't try to unload us until
>>          * all nodes have been removed, and that can't happen
>>          * until all Ethernet interfaces are removed. In any
>>          * case, we know there are no nodes left if the action
>>          * is MOD_UNLOAD, so there's no need to detach any nodes.
>>          */
>>
>> so we can't run this unless you have removeable ethernet interfaces, and
>> have removed them.
>>
>>
>>
> Just saw the message.
> Actually ng_ether can be unloaded but this is left at your own risk
> and there is a comment in code about this.
> for each $if in `ifconfig -l`
> ngctl shutdown $if:
actually ngctl msg $if: disconnect

> kldunload ng_ether
>
> This brings the question that should ng_ether be taught about ifnet
> events(arrival/departure)?!
>
> --
> Ermal
>



-- 
Ermal