Sockets stuck in FIN_WAIT_1

[Robert Blayzor]

On May 29, 2008, at 3:30 PM, Matthew Dillon wrote:
>    It is quite possible that the other ends of the connection are  
> still
>    live and that the issue could very well be a timeout setting in the
>    server config file instead of something in the TCP stack.



I think we're onto something here, but for some reason it doesn't make  
any sense.  I have keepalives turned OFF in Apache:

In the OS I have:

net.inet.tcp.keepidle: 900000
net.inet.tcp.keepintvl: 30000
net.inet.tcp.keepinit: 75000
net.inet.tcp.always_keepalive: 1


I see FIN_WAIT_1 connections collecting like crazy again.  I managed  
to investigate one that's been hung around for several minutes.

1.1.1.1 = our server ip
2.2.2.2 = remote client ip

[mirror0:/usr/local/etc/apache22] netstat -an  | grep 2.2.2.2
tcp4       0  25861  1.1.1.1.80      2.2.2.2.33379   FIN_WAIT_1



When I tcpdump this, I see something sending ack's back and forth  
every 60 seconds, but what?  Apache?  I'm not sure why.   I don't see  
any timeouts in Apache for ~60 seconds.  As you can see, sometimes we  
send an ack, but never see a reply.  I'm gathering the OS level  
keepalives don't come into play because this session is not considered  
idle?


0:13:07.640426 IP 1.1.1.1.80 > 2.2.2.2.33379: .  
4208136508:4208136509(1) ack 1471446041 win 520 <nop,nop,timestamp  
3019088951 5004131>
20:13:07.736505 IP 2.2.2.2.33379 > 1.1.1.1.80: . ack 0 win 0  
<nop,nop,timestamp 5022148 3019088951>
20:14:07.702647 IP 1.1.1.1.80 > 2.2.2.2.33379: . 0:1(1) ack 1 win 520  
<nop,nop,timestamp 3019148951 5022148>
20:15:07.764920 IP 1.1.1.1.80 > 2.2.2.2.33379: . 0:1(1) ack 1 win 520  
<nop,nop,timestamp 3019208951 5022148>
20:15:07.860988 IP 2.2.2.2.33379 > 1.1.1.1.80: . ack 0 win 0  
<nop,nop,timestamp 5058183 3019208951>
20:16:07.827262 IP 1.1.1.1.80 > 2.2.2.2.33379: . 0:1(1) ack 1 win 520  
<nop,nop,timestamp 3019268951 5058183>
20:16:07.923341 IP 2.2.2.2.33379 > 1.1.1.1.80: . ack 0 win 0  
<nop,nop,timestamp 5076200 3019268951>
20:17:07.889690 IP 1.1.1.1.80 > 2.2.2.2.33379: . 0:1(1) ack 1 win 520  
<nop,nop,timestamp 3019328951 5076200>
20:17:07.984770 IP 2.2.2.2.33379 > 1.1.1.1.80: . ack 0 win 0  
<nop,nop,timestamp 5094217 3019328951>
20:18:07.952167 IP 1.1.1.1.80 > 2.2.2.2.33379: . 0:1(1) ack 1 win 520  
<nop,nop,timestamp 3019388951 5094217>
20:18:08.048249 IP 2.2.2.2.33379 > 1.1.1.1.80: . ack 0 win 0  
<nop,nop,timestamp 5112234 3019388951>
20:19:08.014715 IP 1.1.1.1.80 > 2.2.2.2.33379: . 0:1(1) ack 1 win 520  
<nop,nop,timestamp 3019448951 5112234>
20:19:08.110803 IP 2.2.2.2.33379 > 1.1.1.1.80: . ack 0 win 0  
<nop,nop,timestamp 5130252 3019448951>
20:20:08.077321 IP 1.1.1.1.80 > 2.2.2.2.33379: . 0:1(1) ack 1 win 520  
<nop,nop,timestamp 3019508951 5130252>
20:21:08.139995 IP 1.1.1.1.80 > 2.2.2.2.33379: . 0:1(1) ack 1 win 520  
<nop,nop,timestamp 3019568951 5130252>
20:21:08.236063 IP 2.2.2.2.33379 > 1.1.1.1.80: . ack 0 win 0  
<nop,nop,timestamp 5166286 3019568951>
20:22:08.202435 IP 1.1.1.1.80 > 2.2.2.2.33379: . 0:1(1) ack 1 win 520  
<nop,nop,timestamp 3019628951 5166286>
20:22:08.297499 IP 2.2.2.2.33379 > 1.1.1.1.80: . ack 0 win 0  
<nop,nop,timestamp 5184303 3019628951>
20:23:08.264631 IP 1.1.1.1.80 > 2.2.2.2.33379: . 0:1(1) ack 1 win 520  
<nop,nop,timestamp 3019688951 5184303>
20:23:08.360700 IP 2.2.2.2.33379 > 1.1.1.1.80: . ack 0 win 0  
<nop,nop,timestamp 5202321 3019688951>



I'm finding several of these sessions doing the same exact thing....

-- 
Robert Blayzor, BOFH
INOC, LLC
rblayzor at inoc.net
http://www.inoc.net/~rblayzor/