Much improved sendfile(2) kernel implementation

[Andrew Gallatin]

Between TSO and your sendfile changes, things are looking up! 

Here are some Myri10GbE 1500 byte results from a 1.8GHz UP
FreeBSD/amd64 machine (AMD Athlon(tm) 64 Processor 3000+) sending to a
2.0GHz SMP Linux/x86_64 machine (AMD Athlon(tm) 64 X2 Dual Core Processor
3800+) running 26.17.7smp and our 1.1.0 Myri10GE driver (with LRO).
I used a linux receiver because LRO is the only way to receive
standard frames at line rate (without a TOE).

These tests are all for sendfile of a 10MB file in /var/tmp: 

  % netperf242 -Hrome-my -tTCP_SENDFILE -F /var//tmp/zot -T,1 -c -C  -- -s393216

The -T,1 is required to force the netserver to use a different core
than the interrupt handler is bound to on the linux machine.  BTW,
it would be really nice if FreeBSD supported CPU affinity for processes
and interrupt handlers..

I did a number of runs with TSO and the patch applied and found that
setting the send-side socket buffer size to 393216 gave the best
performance in that case.  I used this size for all tests, but it is
possible there is a different sweet spot for other configurations.
Note that linux auto-tunes socket buffer sizes, so I omitted the --
-s393216 for linux.


Recv   Send    Send                          Utilization       Service Demand
Socket Socket  Message  Elapsed              Send     Recv     Send    Recv
Size   Size    Size     Time     Throughput  local    remote   local   remote
bytes  bytes   bytes    secs.    10^6bits/s  % S      % S      us/KB   us/KB

Without patch:
 87380 393216 393216    10.00      2163.08   100.00   19.35    3.787   1.466 
Without patch + TSO:
 87380 393216 393216    10.00      4367.18   71.54    42.07    1.342   1.578 
With patch:
 87380 393216 393216    10.01      1882.73   86.15    18.43    3.749   1.604 
With patch + TSO:
 87380 393216 393216    10.00      6961.08   47.69    60.11    0.561   1.415 

For comparision, if I reboot the sender into RHEL (Linux 2.6.9-11.EL x86_64):
 87380  65536  65536    10.01      9333.00   28.98    75.23    0.254   1.321 



The above results are the median result for 5 runs at each setting.


Drew