Alternate port randomization approaches

[Mike Silbersack]

There have been a few reports by users of front end web proxies and other 
systems under FreeBSD that port randomization causes them problems under 
load.  This seems to be due to a combination of port randomization and 
rapid connections to the same host causing ports to be recycled before the 
ISN has advanced past the end of the previous connection, thereby causing 
the TIME_WAIT socket on the receiving end to ignore the new SYN.

I haven't proven it, but I imagine that part of the problem is due to the 
interaction of established and finished connections.  Say that ports 1500 
-> 1600 have established connections, and that arc4random picks port 1504 
as the next one to try.  Rather than picking another random port, we 
simply look sequentially through the rest of the ports, and will pick 
1601.  I had considered continuing to do random port guesses until an 
unused one was found, but that has the nasty property of looping an 
undetermined amount of time, and it would most likely not help anything 
anyway.

The solution I've come up with that should provide a decent tradeoff 
between the increased security of port randomization and the realities of 
the problem of quick port recycling is attached.  What it does is only 
randomize the first net.inet.ip.portrange.randomized ports assigned each 
second.  Beyond that, it uses sequentual allocation.  The default I 
picked, 20, is arbitrary.  Initial testing by a front-end web proxy admin 
shows that 20 is too high to avoid problems in his case, but 1 works 
decently.

Although this isn't a perfect fix, I think that it should be acceptable 
for the vast majority of systems, and I'd like to get it in before 
4.11-release ships.  To be conservative, I'll probably choose a value like 
5, which should be fine for most systems out there.  Super specialized 
users will always be able to lower it to 0.

Since this is an important issue, I was wondering if anyone had any 
suggestions on how to improve this method, or how to solve the problem in 
another way.

Remember that 4.11 ships in the next few weeks, so any solution which gets 
added to it must be relatively simple, we can't be doing anything like 
changing the TCP state machine.  (Yes, I've considered removing the ISN > 
window requirement from the TIME_WAIT state, but that's not something to 
be changing on a whim.)

Also note that 5.x and 6.x handle TIME_WAIT recycling in the exact same 
way 4.x does; the sequence number check is just now in the function 
tcp_timewait rather than in tcp_input.  So, the same problems plagues all 
versions.  (Despite my other modifications to sequence number generation 
which were attempts to fix it.)

Comments appreciated,

Mike "Silby" Silbersack
-------------- next part --------------
--- /usr/src/sys.old/netinet/in_pcb.c	Thu Dec 16 03:26:11 2004
+++ in_pcb.c	Thu Dec 16 03:27:29 2004
@@ -95,8 +95,10 @@
 int	ipport_hifirstauto = IPPORT_HIFIRSTAUTO;	/* 49152 */
 int	ipport_hilastauto  = IPPORT_HILASTAUTO;		/* 65535 */
 
-/* Shall we allocate ephemeral ports in random order? */
-int	ipport_randomized = 0;
+/* How many ephemeral port randomizations should occur each second? */
+int		ipport_maxrps = 20;
+struct timeval	ipport_lastrandom;
+int		ipport_currps;
 
 #define RANGECHK(var, min, max) \
 	if ((var) < (min)) { (var) = (min); } \
@@ -135,7 +137,7 @@
 SYSCTL_PROC(_net_inet_ip_portrange, OID_AUTO, hilast, CTLTYPE_INT|CTLFLAG_RW,
 	   &ipport_hilastauto, 0, &sysctl_net_ipport_check, "I", "");
 SYSCTL_INT(_net_inet_ip_portrange, OID_AUTO, randomized, CTLFLAG_RW,
-	   &ipport_randomized, 0, "");
+	   &ipport_maxrps, 0, "");
 
 /*
  * in_pcb.c: manage the Protocol Control Blocks.
@@ -323,7 +325,8 @@
 			/*
 			 * counting down
 			 */
-			if (ipport_randomized)
+			if (ipport_maxrps &&
+	 ppsratecheck(&ipport_lastrandom, &ipport_currps, ipport_maxrps))
 				*lastport = first -
 					    (arc4random() % (first - last));
 			count = first - last;
@@ -343,7 +346,8 @@
 			/*
 			 * counting up
 			 */
-			if (ipport_randomized)
+			if (ipport_maxrps &&
+	 ppsratecheck(&ipport_lastrandom, &ipport_currps, ipport_maxrps))
 				*lastport = first +
 					    (arc4random() % (last - first));
 			count = last - first;
-------------- next part --------------
--- /usr/src/sys.old/netinet/in_pcb.c	Thu Nov 25 00:43:21 2004
+++ /usr/src/sys/netinet/in_pcb.c	Thu Dec 16 02:55:46 2004
@@ -97,8 +97,10 @@
 int	ipport_reservedhigh = IPPORT_RESERVED - 1;	/* 1023 */
 int	ipport_reservedlow = 0;
 
-/* Shall we allocate ephemeral ports in random order? */
-int	ipport_randomized = 1;
+/* How many ephemeral port randomizations should occur each second? */
+int		ipport_maxrps = 20;
+struct timeval	ipport_lastrandom;
+int		ipport_currps;
 
 #define RANGECHK(var, min, max) \
 	if ((var) < (min)) { (var) = (min); } \
@@ -142,7 +144,7 @@
 SYSCTL_INT(_net_inet_ip_portrange, OID_AUTO, reservedlow,
 	   CTLFLAG_RW|CTLFLAG_SECURE, &ipport_reservedlow, 0, "");
 SYSCTL_INT(_net_inet_ip_portrange, OID_AUTO, randomized,
-	   CTLFLAG_RW, &ipport_randomized, 0, "");
+	   CTLFLAG_RW, &ipport_maxrps, 0, "");
 
 /*
  * in_pcb.c: manage the Protocol Control Blocks.
@@ -404,7 +406,8 @@
 			/*
 			 * counting down
 			 */
-			if (ipport_randomized)
+			if (ipport_maxrps &&
+	 ppsratecheck(&ipport_lastrandom, &ipport_currps, ipport_maxrps))
 				*lastport = first -
 					    (arc4random() % (first - last));
 			count = first - last;
@@ -422,7 +425,8 @@
 			/*
 			 * counting up
 			 */
-			if (ipport_randomized)
+			if (ipport_maxrps &&
+	 ppsratecheck(&ipport_lastrandom, &ipport_currps, ipport_maxrps))
 				*lastport = first +
 					    (arc4random() % (last - first));
 			count = last - first;