kern/127528: [icmp]: icmp socket receives icmp replies not owned by the process.

Wed May 13 20:20:05 UTC 2009

The following reply was made to PR kern/127528; it has been noted by GNATS.

From: "Seth Mos" <seth.mos at xs4all.nl>
To: bug-followup at FreeBSD.org, seth.mos at xs4all.nl
Cc:  
Subject: kern/127528: [icmp]: icmp socket receives icmp replies not owned 
     by the process.
Date: Wed, 13 May 2009 21:58:40 +0200 (CEST)

 Hi there again, it's been a while.

 We are currently on a pfSense 1.2.3-RC1 release and we are now getting a
 host of people on the forum and in the mailing lists that are complaining
 about the load balancer flapping.

 Some debugging turned out some very weird behaviour.

 Every now and then, depending on the concurrenty of icmp traffic
 originating from the FreeBSD host, ping will miss icmp replies.

 Here are the steps performed to debug this issue.

 For example the output
 tcpdump -ni vlan0 -t icmp
 Where vlan0 is the external interface. The IP address being pinged is a
 local gateway connected by ethernet so there is nothing but the switch in
 between.
 http://pastebin.com/f6608c875

 The ping command issued is
 /sbin/ping -t 4 -oqc 5 -i 0.7 213.23.199.249

 This command line will ping the target and exit with a return status of 0
 when it receives a reply which can be any of the 5 attempts within the
 maximum of 4 seconds.

 The output clearly shows that all attempts have unique process IDs, which
 is good. It also shows that all attempt have a sequence number of 0 which
 is the 1st attempt to ping.

 There is however one attempt where this is different, on attempt with pid
 46060 you will see that it attempted 4 times to ping the target, it got
 valid responses to all 5 requests!.

 However, ping exited with a return status of 1 which would imply a timeout.

 So now it is worse then the originally reported issue we had on 7.0. We
 can now make the FreeBSD distributed ping fail.

 Specifically, it fails to see the valid replies.

 For your reference I also pasted the tcpdump below. I can also reproduce
 this exact same behaviour by using fping. So I would not consider this a
 bug in the ping binary itself.

 Since the original PR had this issue with apinger, which is different from
 both ping and fping I am quite confident there is a serious regression in
 here.

 # uname -a
 FreeBSD noord.coltex.nl 7.1-RELEASE-p4 FreeBSD 7.1-RELEASE-p4 #0: Tue Mar
 24 01:18:19 EDT 2009    
 sullrich at RELENG_1_2-snapshots.pfsense.org:/usr/obj.pfSense/usr/pfSensesrc/src/sys/pfSense_SMP.7
  i386

 Kind regards,

 Seth Mos
 pfSense Developer

 --------------
    1.
       IP 213.23.199.251 > 213.23.199.249: ICMP echo request, id 25068, seq
 0, length 64
    2.
       IP 213.23.199.249 > 213.23.199.251: ICMP echo reply, id 25068, seq
 0, length 64
    3.
       IP 213.23.199.251 > 213.23.199.249: ICMP echo request, id 27372, seq
 0, length 64
    4.
       IP 213.23.199.249 > 213.23.199.251: ICMP echo reply, id 27372, seq
 0, length 64
    5.
       IP 213.23.199.251 > 213.23.199.249: ICMP echo request, id 29676, seq
 0, length 64
    6.
       IP 213.23.199.249 > 213.23.199.251: ICMP echo reply, id 29676, seq
 0, length 64
    7.
       IP 213.23.199.251 > 213.23.199.249: ICMP echo request, id 31468, seq
 0, length 64
    8.
       IP 213.23.199.249 > 213.23.199.251: ICMP echo reply, id 31468, seq
 0, length 64
    9.
       IP 213.23.199.251 > 213.23.199.249: ICMP echo request, id 33772, seq
 0, length 64
   10.
       IP 213.23.199.249 > 213.23.199.251: ICMP echo reply, id 33772, seq
 0, length 64
   11.
       IP 213.23.199.251 > 213.23.199.249: ICMP echo request, id 36076, seq
 0, length 64
   12.
       IP 213.23.199.249 > 213.23.199.251: ICMP echo reply, id 36076, seq
 0, length 64
   13.
       IP 213.23.199.251 > 213.23.199.249: ICMP echo request, id 46060, seq
 0, length 64
   14.
       IP 213.23.199.249 > 213.23.199.251: ICMP echo reply, id 46060, seq
 0, length 64
   15.
       IP 213.23.199.251 > 213.23.199.249: ICMP echo request, id 46060, seq
 1, length 64
   16.
       IP 213.23.199.249 > 213.23.199.251: ICMP echo reply, id 46060, seq
 1, length 64
   17.
       IP 213.23.199.251 > 213.23.199.249: ICMP echo request, id 12525, seq
 0, length 64
   18.
       IP 213.23.199.249 > 213.23.199.251: ICMP echo reply, id 12525, seq
 0, length 64
   19.
       IP 213.23.199.251 > 213.23.199.249: ICMP echo request, id 14829, seq
 0, length 64
   20.
       IP 213.23.199.249 > 213.23.199.251: ICMP echo reply, id 14829, seq
 0, length 64
   21.
       IP 213.23.199.249 > 213.23.199.250: ICMP host 192.168.120.254
 unreachable - admin prohibited filter, length 60
   22.
       IP 213.23.199.249 > 213.23.199.250: ICMP host 192.168.148.1
 unreachable - admin prohibited filter, length 60
   23.
       IP 213.23.199.249 > 213.23.199.250: ICMP host 192.168.112.1
 unreachable - admin prohibited filter, length 60
   24.
       IP 213.23.199.251 > 213.23.199.249: ICMP echo request, id 46060, seq
 2, length 64
   25.
       IP 213.23.199.249 > 213.23.199.251: ICMP echo reply, id 46060, seq
 2, length 64
   26.
       IP 213.23.199.251 > 213.23.199.249: ICMP echo request, id 17645, seq
 0, length 64
   27.
       IP 213.23.199.249 > 213.23.199.251: ICMP echo reply, id 17645, seq
 0, length 64
   28.
       IP 213.23.199.251 > 213.23.199.249: ICMP echo request, id 46060, seq
 3, length 64
   29.
       IP 213.23.199.249 > 213.23.199.251: ICMP echo reply, id 46060, seq
 3, length 64
   30.
       IP 213.23.199.251 > 213.23.199.249: ICMP echo request, id 46060, seq
 4, length 64
   31.
       IP 213.23.199.249 > 213.23.199.251: ICMP echo reply, id 46060, seq
 4, length 64