Much improved sendfile(2) kernel implementation

[Andrew Gallatin]

Andre Oppermann writes:
 > Andrew Gallatin wrote:
 > > 
 > > Between TSO and your sendfile changes, things are looking up! 
 > > 
 > > Here are some Myri10GbE 1500 byte results from a 1.8GHz UP
 > > FreeBSD/amd64 machine (AMD Athlon(tm) 64 Processor 3000+) sending to a
 > > 2.0GHz SMP Linux/x86_64 machine (AMD Athlon(tm) 64 X2 Dual Core Processor
 > > 3800+) running 26.17.7smp and our 1.1.0 Myri10GE driver (with LRO).
 > > I used a linux receiver because LRO is the only way to receive
 > > standard frames at line rate (without a TOE).
 > > 
 > > These tests are all for sendfile of a 10MB file in /var/tmp: 
 > > 
 > >   % netperf242 -Hrome-my -tTCP_SENDFILE -F /var//tmp/zot -T,1 -c -C  -- -s393216
 > 
 > You should use -m5M as well.  netperf is kinda dumb and does only

I will try this.

 > > The -T,1 is required to force the netserver to use a different core
 > > than the interrupt handler is bound to on the linux machine.  BTW,
 > > it would be really nice if FreeBSD supported CPU affinity for processes
 > > and interrupt handlers..
 > 
 > I have a gross version of that in my tree.  The kernel itself supports
 > it but it's not yet exposed to userland for manual intervention.

Sweet.

 > Be a bit careful with the CPU usage figures.  The numbers netperf reports
 > differ quite a bit from those reported by time(1) on the high side.  And
 > there are some differences in the approach how FreeBSD and Linux do their
 > statistical measurements of user and system time.  This doesn't change the

Netperf essentially subtracts idle time from all other time.  The
numbers I see reported from netperf have historically matched what
vmstat says.   BTW, I use netperf from svn.  I don't use the
FreeBSD port.

 > throughput number though.  But see the -m5M option.  New sendfile is really
 > optimized to chew on a large file (larger than the socket buffer size) as
 > it normally happens in reality.
 > 
 > > For comparision, if I reboot the sender into RHEL (Linux 2.6.9-11.EL x86_64):
 > >  87380  65536  65536    10.01      9333.00   28.98    75.23    0.254   1.321 
 > > 
 > > 
 > > The above results are the median result for 5 runs at each setting.
 > 
 > How large is the variance between the runs?

Much bigger for FreeBSD than for Linux.  I think I had one 9.1 and 4
9.3s for linux.  I had a 6.2 a 6.5 and 3 around 6.9 for FreeBSD,

Drew