Application layer classifier for ipfw

[Mike Makonnen]

Mike Makonnen wrote:
> Patrick Tracanelli wrote:
>>
>> To let you know of my current (real world) tests:
>>
>> - Wireless Internet Provider 1:
>>     - 4Mbit/s of Internet Traffic
>>     - Classifying default protocols + soulseek + ssh
>>     - Classifying 100Mbit/s of dump over ssh
>>
>> Results in:
>>     No latency added, very low CPU usage, no packets dropping.
>>
>> - Wireless ISP 2:
>>     - 21 Mbit/s of Internet Traffic
>>     - Classifying default protocols + soulseek + ssh
>>
>> Results in:
>>     No tcp or udp traffic at all; everything that gets diverted never 
>> comes out of the divert socket, and ipfw-classifyd logs
>>
>> Aug  1 12:07:35 ourofino last message repeated 58 times
>> Aug  1 12:17:54 ourofino ipfw-classifyd: Loaded Protocol: bittorrent 
>> (rule 50000)
>> Aug  1 12:17:54 ourofino ipfw-classifyd: Loaded Protocol: edonkey 
>> (rule 50000)
>> Aug  1 12:17:54 ourofino ipfw-classifyd: Loaded Protocol: fasttrack 
>> (rule 50000)
>> Aug  1 12:17:54 ourofino ipfw-classifyd: Loaded Protocol: gnutella 
>> (rule 1000)
>> Aug  1 12:17:54 ourofino ipfw-classifyd: Loaded Protocol: soulseek 
>> (rule 50000)
>> Aug  1 12:17:54 ourofino ipfw-classifyd: Loaded Protocol: ssh   (rule 
>> 50000)
>> Aug  1 12:18:28 ourofino ipfw-classifyd: unable to write to divert 
>> socket: Operation not permitted
>> Aug  1 12:18:50 ourofino last message repeated 90 times
>
> Hmmm... this part means that the call to sendto(2) to write the packet 
> back into network stack failed.  This explains why you are not seein g 
> any traffic comming back out of the divert socket, but I don't see why 
> it would suddenly fail with a permission error. Could this be a kernel 
> bug?
>> Aug  1 12:18:51 ourofino ipfw-classifyd: packet dropped: input queue 
>> full
>> Aug  1 12:19:11 ourofino last message repeated 94 times
>>
>> Raised queue len a lot (up to 40960), when the application starts it 
>> uses up to 25% CPU and a second after that, CPU usage gets lower the 
>> 0.1%.
>
> This looks like a deadlock. If it weren't able to process packets fast 
> enough the cpu usage should be high even as it's spewing "packet 
> dropped" messages. Can you send me some more information like memory 
> usage and the firewall script you are using? How much of the 21Mbits/s 
> of traffic is P2P? If you reduce the number of protocols you are 
> trying to match against does the behavior change? Using netstat -w1 
> -I<interface> can you tell me how many packets per second we're 
> talking about for 4Mbits/s and 21Mbit/s? Also, the timestamps from the 
> log file seem to show that the daemon is running for approx. 34 sec. 
> before the first "unable to write to write to divert socket" message. 
> Is it passing traffic during this time? Thanks.
>
> I've uploaded a newer version. Can you try that also please. It includes:
>   o SIGHUP forces it to re-read its configuration file
>   o rc.d script
>   o minor optimization (calls pthread_cond_signal with the mutex 
> unlocked)
>   o code cleanup
>
> Also, for your convenience I have attached a patch against the earlier 
> version that removes a debugging printf that should remove spammage to 
> your log files (the current version has it removed already).
>

Ooops, a few minutes after I sent this email I found a couple of bugs 
(one major, and one minor). They were in the original tarball as well as 
the newer one I uploaded earlier today. I've uploaded a fixed version of 
the code. Can you try that instead please.

Also, to help track down performance issues I've modified the Makefile 
to build a profiled version of the application so you can use gprof(1) 
to figure out where any problems lie.

Cheers.

-- 
Mike Makonnen       | GPG-KEY: http://people.freebsd.org/~mtm/mtm.asc
mtm @ FreeBSD.Org   | AC7B 5672 2D11 F4D0 EBF8  5279 5359 2B82 7CD4 1F55
FreeBSD             | http://www.freebsd.org