TCP Performance advice needed [long!]

[Andre Oppermann]

Ivan Voras wrote:
> 
> Ragnar Lonn wrote:
> > I don't know what tcp.inflight does but I know that this type of
> > application
> > protocol, that expects a reply before sending the next piece of data, will
> > always be completely dependent upon roundtrip times for its throughput -
> > the roundtrip time for the exchange "transmit-reply" will limit the
> > possible throughput you can get so if you want higher performance, either
> 
> I understand this bottleneck, and know (at least in theory :) ) how it
> could be solved, but my problems are not directly related to that:
> 
> - For small (but consistent in size) packet sizes, I get randomly
> varying round-trip times, and much lower packets-per-second ratio then
> with big packets (consistent in size) with the exact same lock-step
> protocol. Packet generation and processing are not CPU intensive.

This is because of the Nagle algo which tries to accumulate some more
data in packets and to piggyback ACKs to replies.  Enable TCP_NODELAY
on the socket and it should go away.  Disabling delayed ACKs may help
in your case too.

> - When using big packets (actually, when switching back and forth from
> small packets to big packets), the PPS performance starts low and climbs
> to "normal" levels, and I'd like to avoid this. This is a local network
> with 0 errors.
> 
> (if you like it, replace the word "packets" with "messages" in the above
> explanation :) )
> 
> I think the above problems are not directly related to the protocol
> (which could be better, I agree, but it won't happen at least until I
> understand what is happening with this version) but on fine-tuning of
> the network or socket options.

This is 'normal' TCP socket behaviour which is obviously not good with
your application.

-- 
Andre