Packet loss with traffic shaper and routing

[tpeixoto at widesoft.com.br]

Hello all!

We have a machine working as a router and bandwidth limiter for our
network. It routes the traffic through two 'bge' interfaces utilizing only
public IPs. No NAT is used. We only have IPFW rules for traffic shaping by
MAC addresses.
It works fine, but we have been experiencing high latency and packet loss.
There is no other major services in this machine, cpu utilization is low,
memory is fine, no disk activity, but load average is always around 1.5 or
2, even if I disable IPFW layer 2 filtering.
Network traffic is not greater than 12 Mbit/s.

The system is a FreeBSD 5.4-RELEASE, running on a dual CPU: Intel(R)
Xeon(TM) CPU 3.06GHz (3051.47-MHz 686-class CPU) compiled with SMP kernel:
 cpu0 (BSP): APIC ID:  0
 cpu1 (AP): APIC ID:  1
 cpu2 (AP): APIC ID:  6
 cpu3 (AP): APIC ID:  7


top output:

last pid: 14039;  load averages:  1.61,  1.62,  1.56
37 processes:  1 running, 35 sleeping, 1 zombie
CPU states:  0.0% user,  0.0% nice,  1.4% system, 30.8% interrupt, 67.8% idle
Mem: 16M Active, 421M Inact, 149M Wired, 688K Cache, 112M Buf, 1417M Free
Swap: 1024M Total, 1024M Free

  PID USERNAME PRI NICE   SIZE    RES STATE  C   TIME   WCPU    CPU COMMAND
 6245 root      96    0  2672K  2236K select 2   3:48  0.00%  0.00% dhcpd
  329 root      96    0  1328K   892K select 0   2:48  0.00%  0.00% syslogd
  453 root      96    0  3384K  2508K select 2   0:33  0.00%  0.00% sshd




systat -iostat output:

                    /0   /1   /2   /3   /4   /5   /6   /7   /8   /9   /10
     Load Average   ||||||||||

          /0   /10  /20  /30  /40  /50  /60  /70  /80  /90  /100
cpu  user|
     nice|
   system|
interrupt|XXXXXXXXXXXXXXXXX
     idle|XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

          /0   /10  /20  /30  /40  /50  /60  /70  /80  /90  /100
da0   MB/s
      tps|X
pass0 MB/s
      tps|


Can anyone give me some advice?
Thank you in advance!