net.inet.udp.blackhole issue

[Robert Watson]

On Wed, 24 Dec 2008, Yony Yossef wrote:

> I'm facing lots of UDP "Connection refused" errors while running multistream 
> iperf test. Analyzing it with wireshark showed several "ICMP Port 
> Unreachable" problems.
>
> I've overriden it with "sysctl net.inet.udp.blackhole=1", but I'm not sure 
> this is the correct thing to do, I feel like I've sweeped the problem under 
> the carpet.
>
> PS - I see similar failures with TCP bidirectional iperf test, it can also 
> be overriden by "sysctl net.inet.tcp.blackhole=1".
>
> My question is - can it be a NIC problem? If so, how can a driver problem 
> cause an iperf UDP socket to be in a "non listening state"?

Hi Yony:

This is fairly unlikely to be a NIC problem, although anything is possible in 
software.  I'm not familiar with iperf, but generally speaking ICMP port 
unreachable is a result of packets arriving at a closed socket; 
net.inet.udp.blackhole suppresses that ICMP:

                 if (udp_blackhole)
                         goto badheadlocked;
                 if (badport_bandlim(BANDLIM_ICMP_UNREACH) < 0)
                         goto badheadlocked;
                 *ip = save_ip;
                 ip->ip_len += iphlen;
                 icmp_error(m, ICMP_UNREACH, ICMP_UNREACH_PORT, 0, 0);

I think I'd suspect an application bug/feature, in which socket gets closed 
and opened during execution and once in a while a datagram is delivered in 
that window.  Perhaps packets are being delivered with a non-trivial delay 
causing them to arrive after the application has timed out waiting for it?

Robert N M Watson
Computer Laboratory
University of Cambridge