Issues with TCP Timestamps allocation

Mon Jul 8 13:24:32 UTC 2019

Hi Michael,

8 July 2019, 15:53:15, by "Michael Tuexen" <tuexen at freebsd.org>:

> > On 8. Jul 2019, at 12:37, Paul <devgs at ukr.net> wrote:
> > 
> > Hi team,
> > 
> > Recently we had an upgrade to 12 Stable. Immediately after, we have started 
> > seeing some strange connection establishment timeouts to some fixed number
> > of external (world) hosts. The issue was persistent and easy to reproduce.
> > Thanks to a patience and dedication of our system engineer we have tracked  
> > this issue down to a specific commit:
> > 
> > https://svnweb.freebsd.org/base?view=revision&revision=338053
> > 
> > This patch was also back-ported into 11 Stable:
> > 
> > https://svnweb.freebsd.org/base?view=revision&revision=348435
> > 
> > Among other things this patch changes the timestamp allocation strategy,
> > by introducing a deterministic randomness via a hash function that takes
> > into account a random key as well as source address, source port, dest
> > address and dest port. As the result, timestamp offsets of different
> > tuples (SA,SP,DA,DP) will be wildly different and will jump from small 
> > to large numbers and back, as long as something in the tuple changes.
> Hi Paul,
> 
> this is correct.
> 
> Please note that the same happens with the old method, if two hosts with
> different uptimes are bind a consumer grade NAT.

If NAT does not replace timestamps then yes, it should be the case.

> > 
> > After performing various tests of hosts that produce the above mentioned 
> > issue we came to conclusion that there are some interesting implementations 
> > that drop SYN packets with timestamps smaller  than the largest timestamp 
> > value from streams of all recent or current connections from a specific 
> > address. This looks as some kind of SYN flood protection.
> This also breaks multiple hosts with different uptimes behind a consumer
> level NAT talking to such a server.
> > 
> > To ensure that each external host is not going to see a wild jumps of 
> > timestamp values I propose a patch that removes ports from the equation
> > all together, when calculating the timestamp offset:
> > 
> > Index: sys/netinet/tcp_subr.c
> > ===================================================================
> > --- sys/netinet/tcp_subr.c	(revision 348435)
> > +++ sys/netinet/tcp_subr.c	(working copy)
> > @@ -2224,7 +2224,22 @@
> > uint32_t
> > tcp_new_ts_offset(struct in_conninfo *inc)
> > {
> > -	return (tcp_keyed_hash(inc, V_ts_offset_secret));
> > +        /* 
> > +         * Some implementations show a strange behaviour when a wildly random 
> > +         * timestamps allocated for different streams. It seems that only the
> > +         * SYN packets are affected. Observed implementations drop SYN packets
> > +         * with timestamps smaller than the largest timestamp value of all 
> > +         * recent or current connections from specific a address. To mitigate 
> > +         * this we are going to ensure that each host will always observe 
> > +         * timestamps as increasing no matter the stream: by dropping ports
> > +         * from the equation.
> > +         */ 
> > +        struct in_conninfo inc_copy = *inc;
> > +
> > +        inc_copy.inc_fport = 0;
> > +        inc_copy.inc_lport = 0;
> > +
> > +	return (tcp_keyed_hash(&inc_copy, V_ts_offset_secret));
> > }
> > 
> > /*
> > 
> > In any case, the solution of the uptime leak, implemented in rev338053 is 
> > not going to suffer, because a supposed attacker is currently able to use 
> > any fixed values of SP and DP, albeit not 0, anyway, to remove them out 
> > of the equation.
> Can you describe how a peer can compute the uptime from two observed timestamps?
> I don't see how you can do that...

Supposed attacker could run a script that continuously monitors timestamps,
for example via a periodic TCP connection from a fixed local port (eg 12345) 
and a fixed local address to the fixed victim's address and port (eg 80).
Whenever large discrepancy is observed, attacker can assume that reboot has 
happened (due to V_ts_offset_secret re-generation), hence the received 
timestamp is considered an approximate point of reboot from which the uptime
can be calculated, until the next reboot and so on.

> > 
> > There is the list of example hosts that we were able to reproduce the 
> > issue with:
> > 
> > curl -v http://88.99.60.171:80
> > curl -v http://163.172.71.252:80
> > curl -v http://5.9.242.150:80
> > curl -v https://185.134.205.105:443
> > curl -v https://136.243.1.231:443
> > curl -v https://144.76.196.4:443
> > curl -v http://94.127.191.194:80
> > 
> > To reproduce, call curl repeatedly with a same URL some number of times. 
> > You are going  to see some of the requests stuck in 
> > `*    Trying XXX.XXX.XXX.XXX...`
> > 
> > For some reason, the easiest way to reproduce the issue is with nc:
> > 
> > $ echo "foooooo" | nc -v 88.99.60.171 80
> > 
> > Only a few such calls are required until one of them is stuck on connect():
> > issuing SYN packets with an exponential backoff.
> Thanks for providing an end-point to test with. I'll take a look.
> Just to be clear: You are running a FreeBSD client against one of the above
> servers and experience the problem with the new timestamp computations.
> 
> You are not running arbitrary clients against a FreeBSD server...

We are talking about FreeBSD being the client. Peers that yield this unwanted
behaviour are unknown. Little bit of tinkering showed that some of them run 
Debian:

telnet 88.99.60.171 22
Trying 88.99.60.171...
Connected to 88.99.60.171.
Escape character is '^]'.
SSH-2.0-OpenSSH_6.7p1 Debian-5+deb8u3

> 
> Best regards
> Michael
> 
> 