Network anomalies after update from 11.2 STABLE to 12.1 STABLE

Sat Oct 19 06:39:14 UTC 2019

> On 18. Oct 2019, at 14:57, Paul <devgs at ukr.net> wrote:
> 
> Our current version is:
> 
>    FreeBSD 11.2-STABLE #0 r340725
> 
> New version that we have problems with:
> 
>    FreeBSD 12.1-STABLE #5 r352893
> 
> 
> After update to new version we have started to observe an incredible number of 
> errors in HTTP requests in between various services in our system. This problem
> appeared on all the servers that were upgraded, and seems to not be specific to
> concrete network card: we use different models, all are affected.
> 
> During various tests, we observed a lot of spontaneous TCP stream abortions, 
> including at the establishment stage (SYN) in cases that were 100% issue free
> on 11.2-STABLE. Concrete test cases will be shown below.
> 
> We also want to highlight that, on numerous occasions, we have observed random,
> huge ACK indices in a first response to a SYN packet, instead of 1, as expected.
> This forces client to abort connection via RST.
> 
> On the fist glance it looks like races in the kernel, because problem disappears when:
>   * we use `dev.ixl.0.iflib.override_nrxqs=1` and `dev.ixl.0.iflib.override_ntxqs=1`
>   * we use `dev.ixl.0.iflib.override_nrxqs=0` and `dev.ixl.0.iflib.override_ntxqs=0`, but don't issue concurrent TCP streams
> 
> These are some debug log messages, emitted by 12.1-STABLE:
> 
> Oct 18 14:59:01 test kernel: TCP: [10.10.10.39]:16304 to [10.10.10.92]:80 tcpflags 0x4<RST>; syncache_chkrst: Spurious RST without matching syncache entry (possibly syncookie only), segment ignored
> Oct 18 14:59:01 test kernel: TCP: [10.10.10.39]:16326 to [10.10.10.92]:80 tcpflags 0x4<RST>; syncache_chkrst: Spurious RST without matching syncache entry (possibly syncookie only), segment ignored
> Oct 18 14:59:01 test kernel: TCP: [10.10.10.39]:16402 to [10.10.10.92]:80 tcpflags 0x4<RST>; syncache_chkrst: Spurious RST without matching syncache entry (possibly syncookie only), segment ignored
> Oct 18 14:59:01 test kernel: TCP: [10.10.10.39]:16652 to [10.10.10.92]:80 tcpflags 0x4<RST>; syncache_chkrst: Spurious RST without matching syncache entry (possibly syncookie only), segment ignored
> Oct 18 14:59:01 test kernel: TCP: [10.10.10.39]:16686 to [10.10.10.92]:80 tcpflags 0x4<RST>; syncache_chkrst: Spurious RST without matching syncache entry (possibly syncookie only), segment ignored
> Oct 18 14:59:01 test kernel: TCP: [10.10.10.39]:18562 to [10.10.10.92]:80 tcpflags 0x4<RST>; tcp_do_segment: Timestamp missing, no action
> Oct 18 14:59:01 test kernel: TCP: [10.10.10.39]:18918 to [10.10.10.92]:80 tcpflags 0x4<RST>; syncache_chkrst: Spurious RST without matching syncache entry (possibly syncookie only), segment ignored
> Oct 18 14:59:01 test kernel: TCP: [10.10.10.39]:19331 to [10.10.10.92]:80 tcpflags 0x4<RST>; syncache_chkrst: Spurious RST without matching syncache entry (possibly syncookie only), segment ignored
> Oct 18 14:59:01 test kernel: TCP: [10.10.10.39]:19340 to [10.10.10.92]:80 tcpflags 0x4<RST>; tcp_do_segment: Timestamp missing, no action
> Oct 18 14:59:01 test kernel: TCP: [10.10.10.39]:19340 to [10.10.10.92]:80 tcpflags 0x4<RST>; syncache_chkrst: Spurious RST without matching syncache entry (possibly syncookie only), segment ignored
> Oct 18 14:59:01 test kernel: TCP: [10.10.10.39]:19340 to [10.10.10.92]:80 tcpflags 0x4<RST>; syncache_chkrst: Spurious RST without matching syncache entry (possibly syncookie only), segment ignored
> Oct 18 14:59:01 test kernel: TCP: [10.10.10.39]:19489 to [10.10.10.92]:80 tcpflags 0x4<RST>; syncache_chkrst: Spurious RST without matching syncache entry (possibly syncookie only), segment ignored
> Oct 18 14:59:01 test kernel: TCP: [10.10.10.39]:19580 to [10.10.10.92]:80 tcpflags 0x4<RST>; tcp_do_segment: Timestamp missing, no action
> Oct 18 14:59:01 test kernel: TCP: [10.10.10.39]:19580 to [10.10.10.92]:80 tcpflags 0x4<RST>; syncache_chkrst: Spurious RST without matching syncache entry (possibly syncookie only), segment ignored
> Oct 18 14:59:01 test kernel: TCP: [10.10.10.39]:19580 to [10.10.10.92]:80 tcpflags 0x4<RST>; syncache_chkrst: Spurious RST without matching syncache entry (possibly syncookie only), segment ignored
> Oct 18 14:59:02 test kernel: TCP: [10.10.10.39]:17705 to [10.10.10.92]:80; syncache_timer: Response timeout, retransmitting (1) SYN|ACK
> Oct 18 14:59:02 test kernel: TCP: [10.10.10.39]:18066 to [10.10.10.92]:80; syncache_timer: Response timeout, retransmitting (1) SYN|ACK
> Oct 18 14:59:02 test kernel: TCP: [10.10.10.39]:18066 to [10.10.10.92]:80 tcpflags 0x4<RST>; syncache_chkrst: Our SYN|ACK was rejected, connection attempt aborted by remote endpoint
> Oct 18 14:59:02 test kernel: TCP: [10.10.10.39]:17705 to [10.10.10.92]:80 tcpflags 0x4<RST>; syncache_chkrst: Our SYN|ACK was rejected, connection attempt aborted by remote endpoint
> 
> Here, 10.10.10.92 runs 12.1-STABLE, while 10.10.10.39 is a client that runs 11.2-STABLE.
> 
> 
> In our test case we use nginx and wrk , with a minimal config, where nginx always returns 
> error page 404. nginx is on the 12.1-STABLE, while wrk is on 11.2-STABLE.
> 
> We run wrk like so:
> 
>    wrk -c 10 --header "Connection: close" -d 10 -t 1 --latency http://10.10.10.92:80/missing
> 
> and often see errors like these:
> 
>    Socket errors: connect 12, read 4, write 4, timeout 0
> 
> If we reverse the test, by switching two servers places, ie 12.1-STABLE becomes a client and 
> issues requests via wrk, we see no problems at all. Same is true between two between two
> 11.2-STABLE machines.
> 
> 
> It seems like issue appears only when the same local port is used for multiple connections 
> on 12.1-STABLE. Currently this is possible only when  12.1-STABLE is a server and accepts 
> connections on port, say 80, as in our case. To confirm, this we made  another test. We've 
> configured nginx to listen on 10 different ports, 80 through 89, and then launched 10 
> different wrk processes, each using only one concurrent connection, meaning that we will 
> have only 10 TCP streams, each having its own unique port on the 12.1-STABLE's side:
> 
>    for I in {0..9}; do wrk -c 1 --header "Connection: close" -d 10 -t 1 --latency http://10.10.10.92:8${I}/missing & ; done
> 
> Socket errors stopped appearing. We ran this test many many times, errors just don't appear.
> 
> Though, whenever we repeat a previous test, using a single port:
> 
>    wrk -c 10 --header "Connection: close" -d 10 -t 1 --latency http://10.10.10.92:80/missing
> 
> errors start appearing again and again:
> 
>    Socket errors: connect 8, read 14, write 9, timeout 0
> 
> 
> We've tested different drivers with the same outcome:
> 
> em driver:
> em0 at pci0:10:0:0:        class=0x020000 card=0x000015d9 chip=0x10d38086 rev=0x00 hdr=0x00
>    vendor     = 'Intel Corporation'
>    device     = '82574L Gigabit Network Connection'
> 
> ixl driver:
> ixl0 at pci0:4:0:0:        class=0x020000 card=0x00078086 chip=0x15728086 rev=0x01 hdr=0x00
>    vendor     = 'Intel Corporation'
>    device     = 'Ethernet Controller X710 for 10GbE SFP+'
> 
> Even the driver from ports (/usr/ports/net/intel-ixl-kmod): ixl-1.11.9
> 
> 
> Help with this matter would be really appreciated.
I would like to reproduce this locally.

Could you send me (privately) the config of nginx such that I can setup two machines?
Are your client/server physical machines or virtual machines?  Are there any middleboxes
(NAT/Firewall/whatever) involved?

One thing (no idea if it is relevant or not):
Could you set
sudo sysctl -w net.inet.tcp.ts_offset_per_conn=0
on the 12.1 machine and test and report if it helps?

Best regards
Michael
> 
> Best regards,
> -Paul
> 
> _______________________________________________
> freebsd-net at freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "freebsd-net-unsubscribe at freebsd.org"