read() returns ETIMEDOUT on steady TCP connection

[Mark Hills]

Hello,

I'm are having a trouble with TCP connections being dropped with "read: 
Operation timed out". What is unusual is that this is happening right in 
the middle of sending a steady stream of data with no network congestion.

The system is FreeBSD 7 and a bespoke streaming server with 1Gbit 
connection. The server receives a 192kbps inbound stream over TCP, and 
broadcasts it over a large number of TCP streams.

With no visible or obvious pattern, the inbound read() fails with 
ETIMEDOUT. The likelihood of this happening seems to increase as the 
number of audience connections increases. It's happens every few minutes 
even with a small audience (eg. 300 outbound connections and about 
60mbit).

It doesn't cough and splutter -- steady data is coming in, then it just 
drops the connection.

systat doesn't show problems inbound; all packets received are delivered 
to the upper layer. But on outbound, there is consistent 'output drops':

     IP Output
7028 total packets sent
7028 - generated locally
  314 - output drops

As the number of outbound connections increases, the 'output drops' 
increases to around 10% of the total packets sent and maintains that 
ratio. There's no problems with network capacity.

I've tried different servers, different network interfaces (bge, em), 
different kernel (7-RELEASE, 7-STABLE). Have also checked dev.bge.0.stats 
and dev.em.0.stats for CRC errors etc. which show no problems. 'netstat 
-m' doesn't show any reaching of mbuf and sbuf limits. The problem is seen 
in a dedicated, uncontended test environment.

Can anyone explain why the packets are being dropped outbound, and how 
this could affect inbound TCP data in such an abrupt way? What can I do to 
solve this?

Thanks,

Mark