ab: No buffer space available

[Ivan Voras]

"ab" is "apache benchmark", a trivial tool for testing web servers:

> ab -n 100000 -c 10 http://localhost/file
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
Completed 10000 requests
Send request failed!
socket: No buffer space available (55)
Total of 16479 requests completed

Each request "ab" makes is a full TCP connection and HTTP "GET" request, 
a round-trip over localhost. The "file" is a static local file, fully 
cached, probably mmaped by Apache and ready to be served in any case.

When I force "ab" to reuse connections (keepalive), the test finishes 
cleanly (with ~~ 19,000 RPS).

The fact that it doesn't finish when always establishing new connections 
looks a bit strange.

After digging around it looked like it's not about buffer space but 
maybe about ephemaral port exhaustion. If I set 
net.inet.ip.portrange.hifirst to 10,000 the ab tool proceeds to more 
requests (25,635) before it fails. This is more than before but still 
not the 65535-10000=55535 requests I'd expect if that theory was true. I 
have also disabled net.inet.ip.portrange.randomized but without effect.

This number (25,000) looked suspiciously like kern.ipc.maxsockets so 
I've increased that to 60,000 and ab made more requests (~~57,000) 
before dieing from "apr_socket_connect(): Can't assign requested address".

All this kind of makes sense but:

1) Why is this an issue at all? Since concurrency is 10, this means that 
ab establishes at most 10 concurrent TCP sessions, not tens of thousands 
of them (as far as I can tell, this is true judging from the number of 
active web server threads).

2) In the context of #1, why does the performance goes down as more 
requests are made?

3) What is the next barrier? After the test is run I see a lot of 
connections in netstat in LAST_ACK state - how do I reduce those?