sendfile(2) SF_NOPUSH flag proposal

[Richard Sharpe]

On Tue, 27 May 2003, Igor Sysoev wrote:

> On Tue, 27 May 2003, Peter Jeremy wrote:
> 
> > 2) The new feature provides significant performance benefit.   In this
> >    case, I believe the overhead of calling setsockopt(2) is negligible
> >    so the performance gain would be negligible.
> 
> I think the calling setsockopt(TCP_NOPUSH, 1) syscall has huge overhead
> as compared to several C operators inside sendfile(2).
> 
> The turing TF_NOPUSH off has almost the same overhead as
> setsockopt(TCP_NOPUSH, 0) if you need to call tcp_output(tp) inside
> sendfile(2) and has no overhead at all if you do not need to call it.
> 
> > At this stage, I would suggest that you need to do better than "the
> > change is cheap" to justify adding this feature.  Can you quantify
> > the performance benefits, or provide some other justification?
> 
> My point is not "the cheap change" but "the cheap overhead".

While I was chasing down a performance problem with Samba using sendfile 
on FreeBSD 4.6.2, I changed sendfile to:

1. Use sosend for the header, and
2. Not to push the header out if there was data following (by passing 
MSG_MORE to sosend, and maybe frobbing sosend to do the right things)..

I was also using TCP_NODELAY, and sendfile was being used to to handle SMB 
Read&X calls only.

The performance impact of doing this as measured by tests like NetBench 
was negligible.
 
I did not test raw throughput (as NetBench is not really 
about raw throughput), but I suspect that it would not make much 
difference either.

I also modified sendfile so that it uses VOP_GETPAGES rather than 
VOP_READ, and this had more impact, I believe.

Regards
-----
Richard Sharpe, rsharpe[at]ns.aus.com, rsharpe[at]samba.org, 
sharpe[at]ethereal.com, http://www.richardsharpe.com