tap device poor performance

[Oleg]

Hello,

I wrote a simple program to test tap device performance.
It open 2 tap devices, configure them (with aliases):

tap64: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 00:13:ec:00:10:00
        inet 192.168.64.1 netmask 0xffffff00 broadcast 192.168.64.255
        inet 192.168.64.64 netmask 0xffffffff broadcast 192.168.64.64
        Opened by PID 1353
tap32: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 00:13:ec:00:20:00
        inet 192.168.32.1 netmask 0xffffff00 broadcast 192.168.32.255
        inet 192.168.32.32 netmask 0xffffffff broadcast 192.168.32.32
        Opened by PID 1353

Then read packet from one tap device, change mask to opposite
(192.168.64.0/24 <--> 192.168.32.0/24)
in the IP headers (src_addr, dst_addr), recalculate checksum and push
to opposite tap. (tap64 <--> tap32).

How to reproduce:
run tapbridge:
#./tapbridge

Then, 2 choise:
1) with help of TTCP utilitty.

TTCP (need patched version, look ports/139112,
http://www.freebsd.org/cgi/query-pr.cgi?pr=139112)

Run ttcp in listen mode:
# ttcp -r -s -n 131000
(output skipped)

On second console run ttcp in sender mode:
# ttcp -t -s -n 131000 -S 192.168.64.64 192.168.64.32
ttcp-t: buflen=8192, nbuf=131000, align=16384/0, port=5001  tcp  ->
192.168.64.32
ttcp-t: socket
ttcp-t: connect
ttcp-t: 1073152000 bytes in 25.70 real seconds = 40778.27 KB/sec +++
ttcp-t: 131000 I/O calls, msec/call = 0.20, calls/sec = 5097.28
ttcp-t: 0.0user 4.5sys 0:25real 17% 26i+214d 494maxrss 0+2pf 493201+173csw

SPEED is appr. 40 MB/sec.

Now lets run ttcp in sender mode via lo0:

# ttcp -t -s -n 131000 192.168.64.64
ttcp-t: buflen=8192, nbuf=131000, align=16384/0, port=5001  tcp  ->
192.168.64.64
ttcp-t: socket
ttcp-t: connect
ttcp-t: 1073152000 bytes in 2.06 real seconds = 508079.09 KB/sec +++
ttcp-t: 131000 I/O calls, msec/call = 0.02, calls/sec = 63509.89
ttcp-t: 0.0user 1.3sys 0:02real 64% 21i+180d 494maxrss 0+2pf 33559+41csw

SPEED is appr. 500 MB/sec.

2) with httpd, wget
configure httpd and run it.

Download some file:
# wget --bind-address=192.168.64.64 -O /dev/null http://192.168.64.32:81/8G
--2009-09-28 18:00:45--  http://192.168.64.32:81/8G
Connecting to 192.168.64.32:81... connected.
HTTP request sent, awaiting response... 200 OK
Length: 8589934592 (8.0G) [text/plain]
Saving to: `/dev/null'

100%[=========================>] 8,589,934,592 40.5M/s   in 3m 8s

2009-09-28 18:03:53 (43.6 MB/s) - `/dev/null' saved [8589934592/8589934592]

SPEED is appr. 43 MB/sec.

The test is gone on Intel Xeon 8 core machine with 16 GB of memory
running FreeBSD 7.0-RELEASE-p12 amd64.
The SAME SPEED results measuring on virtual machines
(FreeBSD 7.0 amd64, FreeBSD 7.2 amd64, FreeBSD 8.0 RC1 amd64).

So, where is bottleneck?
Also, I make a LOCK_PROFILING measure.

# sysctl debug.lock.prof.stats | grep if_tap
   max        total   wait_total       count   avg wait_avg
cnt_hold     cnt_lock name
   252      1163161           64      317424     3     0           28
         10 /usr/src/sys/modules/if_tap/../../net/if_tap.c:681 (sleep
mutex:tap_mtx)
   109       783634          761      317426     2     0           67
         74 /usr/src/sys/modules/if_tap/../../net/if_tap.c:848 (sleep
mutex:tap_mtx)
   103       173087            0         69617     2     0
1            0 /usr/src/sys/modules/if_tap/../../net/if_tap.c:872
(sleep mutex:tap_mtx)
   204       460785          303      190427     2     0           24
         67 /usr/src/sys/modules/if_tap/../../net/if_tap.c:865 (sleep
mutex:tap32)
    81        487945             0      196614     2     0
1            0 /usr/src/sys/modules/if_tap/../../net/if_tap.c:865
(sleep mutex:tap64)
      1                 3             0              2     1     0
       0            0
/usr/src/sys/modules/if_tap/../../net/if_tap.c:493 (sleep
mutex:tap_mtx)
   319       471162          509      317428     1     0           34
        122 /usr/src/sys/modules/if_tap/../../net/if_tap.c:653 (sleep
mutex:tap_mtx)

Seems, that this is not a locking problem...

Who can help me?

p.s. if your downloading will stuck - you need to patch your tap
device with kern/123892
(http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/123892)

p.p.s. btw on Linux similar test give me about 70 MB/sec...

Regards,
   Oleg Dolgov.