TCP Performance advice needed [long!]

[Ivan Voras]

Ragnar Lonn wrote:
> I don't know what tcp.inflight does but I know that this type of 
> application
> protocol, that expects a reply before sending the next piece of data, will
> always be completely dependent upon roundtrip times for its throughput -
> the roundtrip time for the exchange "transmit-reply" will limit the
> possible throughput you can get so if you want higher performance, either

I understand this bottleneck, and know (at least in theory :) ) how it 
could be solved, but my problems are not directly related to that:

- For small (but consistent in size) packet sizes, I get randomly 
varying round-trip times, and much lower packets-per-second ratio then 
with big packets (consistent in size) with the exact same lock-step 
protocol. Packet generation and processing are not CPU intensive.

- When using big packets (actually, when switching back and forth from 
small packets to big packets), the PPS performance starts low and climbs 
to "normal" levels, and I'd like to avoid this. This is a local network 
with 0 errors.

(if you like it, replace the word "packets" with "messages" in the above 
explanation :) )

I think the above problems are not directly related to the protocol 
(which could be better, I agree, but it won't happen at least until I 
understand what is happening with this version) but on fine-tuning of 
the network or socket options.