From nobody Thu Jun 03 13:09:06 2021 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id ECDD513A7E39 for ; Thu, 3 Jun 2021 13:09:16 +0000 (UTC) (envelope-from freebsd@grem.de) Received: from mail.evolve.de (mail.evolve.de [213.239.217.29]) (using TLSv1.3 with cipher TLS_CHACHA20_POLY1305_SHA256 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA512 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mail.evolve.de", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4FwmSW4Bgxz3GnH for ; Thu, 3 Jun 2021 13:09:15 +0000 (UTC) (envelope-from freebsd@grem.de) Received: by mail.evolve.de (OpenSMTPD) with ESMTP id 8b571cb2 for ; Thu, 3 Jun 2021 13:09:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=grem.de; h=date:from:to :subject:message-id:in-reply-to:references:mime-version :content-type:content-transfer-encoding; s=20180501; bh=3FYN8fFq hYP+TMR5Mu4g6jOtooI=; b=emcvStVSZsVOl7HapdcfzUWVJoO4pvX55nIjUsFW hPY+eWMn22GO9NBa18I+4pOxBtxFOdqqVBWS2WTd2/4SwMFNkQkdj09/XVjLDLlL KzxpWk92LDrt8TjvO7KUqkoPazDztm4j2FSwparmg5QCyUWR7EbaoVNWDzKFi3LL zqBRtIqTMs4MRhtHjukEYdmDv40+BNGg0rTRVJksyZThYGZ6aZJ/uvnAZ6AAaM5A qv8TSY38FZHy9dbr7bRsuXEsX4WuhMsS9BM7noBTEaGse4xiTi4L+FssFA3HWjBD f3bGgrXG5UUMgK5lm3up4vZ7ITKJwgeYaIWDmbhg3yH2rA== DomainKey-Signature: a=rsa-sha1; c=nofws; d=grem.de; h=date:from:to :subject:message-id:in-reply-to:references:mime-version :content-type:content-transfer-encoding; q=dns; s=20180501; b=tU LAEYqkEndMQ1FJQu7V/xEOvDTXPPCl7Eat8acC43Xa7SusIIteKiEm4jMSmV5AFw I771PEFzEEkw9GFHQRbMVxGqxee0wYiYytuzpGyFL6aQVF1PjX1oMYj8DCf1Ld/x J3Q9Ury2hKqvO70YSnmEuRVcFvF1GTDJO86EQdlQf/wU4ur0wJ8SlMWDXyCtdKWV F0YV7CQP8oJ/N3wdBH6Owg6UsJrmBPKLDECTYESNhvZqD4mw9ZV6RcvusJTquEhd tll1cldUiqdF6eKT9Xl54ed0Uhkjs1bT+MJHtURIr/4zbw0lrLVXDjJ4Px5HFdoo rH1s8J3UoVO2W9nzIXBQ== Received: by mail.evolve.de (OpenSMTPD) with ESMTPSA id 6c779dbb (TLSv1.3:AEAD-CHACHA20-POLY1305-SHA256:256:NO) for ; Thu, 3 Jun 2021 13:09:09 +0000 (UTC) Date: Thu, 3 Jun 2021 15:09:06 +0200 From: Michael Gmelin To: "freebsd-current@freebsd.org" Subject: Re: ssh connections break with "Fssh_packet_write_wait" on 13 Message-ID: <20210603150906.48cbd638@bsd64.grem.de> In-Reply-To: <20210601134747.40920d51@bsd64.grem.de> References: <20210601134747.40920d51@bsd64.grem.de> X-Face: $wrgCtfdVw_H9WAY?S&9+/F"!41z'L$uo*WzT8miX?kZ~W~Lr5W7v?j0Sde\mwB&/ypo^}> +a'4xMc^^KroE~+v^&^#[B">soBo1y6(TW6#UZiC]o>C6`ej+i Face: iVBORw0KGgoAAAANSUhEUgAAADAAAAAwBAMAAAClLOS0AAAAJFBMVEWJBwe5BQDl LASZU0/LTEWEfHbyj0Txi32+sKrp1Mv944X8/fm1rS+cAAAACXBIWXMAAAsTAAAL EwEAmpwYAAAAB3RJTUUH3wESCxwC7OBhbgAAACFpVFh0Q29tbWVudAAAAAAAQ3Jl YXRlZCB3aXRoIFRoZSBHSU1QbbCXAAAAAghJREFUOMu11DFvEzEUAGCfEhBVFzuq AKkLd0O6VrIQsLXVSZXoWE5N1K3DobBBA9fQpRWc8OkWouaIjedWKiyREOKs+3PY fvalCNjgLVHeF7/3bMtBzV8C/VsQ8tecEgCcDgrzjekwKZ7TwsJZd/ywEKwwP+ZM 8P3drTsAwWn2mpWuDDuYiK1bFs6De0KUUFw0tWxm+D4AIhuuvZqtyWYeO7jQ4Aea 7jUqI+ixhQoHex4WshEvSXdood7stlv4oSuFOC4tqGcr0NjEqXgV4mMJO38nld4+ xKNxRDon7khyKVqY7YR4d+Cg0OMrkWXZOM7YDkEfKiilCn1qYv4mighZiynuHHOA Wq9QJq+BIES7lMFUtcikMnkDGHUoncA+uHgrP0ctIEqfwLHzeSo+eUA66AqzwN6n 2ZHJhw6Qh/PoyC/QENyEyC/AyNjq74Bs+3UH0xYwzDUC4B97HgLocg1QLYgDDO1v f3UX9Y307Ew4AHh67YAFFsxEpkXwpXY3eIgMhAAE3R19L919nNnuD2wlPcDE3UeT L2ytEICQib9BXgS2fU8PrD82ToYO1OEmMSnYTjSqSv9wdC0tPYC+rQRQD9ESnldF CyqfmiYW+tlALt8gH2xrMdC/youbjzPXEun+/ReXsMCDyve3dZc09fn2Oas8oXGc Jj6/fOeK5UmSMPmf/jL+GD8BEj0k/Fn6IO4AAAAASUVORK5CYII= List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 4FwmSW4Bgxz3GnH X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=grem.de header.s=20180501 header.b=emcvStVS; dmarc=none; spf=pass (mx1.freebsd.org: domain of freebsd@grem.de designates 213.239.217.29 as permitted sender) smtp.mailfrom=freebsd@grem.de X-Spamd-Result: default: False [-3.50 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[grem.de:s=20180501]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:213.239.217.29/32]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-current@freebsd.org]; DMARC_NA(0.00)[grem.de]; RCPT_COUNT_ONE(0.00)[1]; SPAMHAUS_ZRD(0.00)[213.239.217.29:from:127.0.2.255]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[grem.de:+]; NEURAL_HAM_SHORT(-1.00)[-0.997]; TO_DN_EQ_ADDR_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RBL_DBL_DONT_QUERY_IPS(0.00)[213.239.217.29:from]; ASN(0.00)[asn:24940, ipnet:213.239.192.0/18, country:DE]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[freebsd-current] X-ThisMailContainsUnwantedMimeParts: N On Tue, 1 Jun 2021 13:47:47 +0200 Michael Gmelin wrote: > Hi, > > Since upgrading servers from 12.2 to 13.0, I get > > Fssh_packet_write_wait: Connection to 1.2.3.4 port 22: Broken pipe > > consistently, usually after about 11 idle minutes, that's with and > without pf enabled. Client (11.4 in a VM) wasn't altered. > > Verbose logging (client and server side) doesn't show anything special > when the connection breaks. In the past, QoS problems caused these > disconnects, but I didn't see anything apparent changing between 12.2 > and 13 in this respect. > > I did a test on a newly commissioned server to rule out other factors > (so, same client connections, some routes, same everything). On 12.2 > before the update: Connection stays open for hours. After the update > (same server): connections breaks consistently after < 15 minutes > (this is with unaltered configurations, no *AliveInterval configured > on either side of the connection). > I did a little bit more testing and realized that the problem goes away when I disable "Proportional Rate Reduction per RFC 6937" on the server side: sysctl net.inet.tcp.do_prr=0 Keeping it on and enabling net.inet.tcp.do_prr_conservative doesn't fix the problem. This seems to be specific to Parallels. After some more digging, I realized that Parallels Desktop's NAT daemon (prl_naptd) handles keep-alive between the VM and the external server on its own. There is no direct communication between the client and the server. This means: - The NAT daemon starts sending keep-alive packages right away (not after the VM's net.inet.tcp.keepidle), every 75 seconds. - Keep-alive packages originating in the VM never reach the server. - Keep-alive originating on the server never reaches the VM. - Client and server basically do keep-alive with the nat daemon, not with each other. It also seems like Parallels is filtering the tos field (so it's always 0x00), but that's unrelated. I configured a bhyve VM running FreeBSD 11.4 on a separate laptop on the same network for comparison and is has no such issues. Looking at TCP dump output on the server, this is what a keep-alive package sent by Parallels looks like: 10:14:42.449681 IP (tos 0x0, ttl 64, id 15689, offset 0, flags [none], proto TCP (6), length 40) 192.168.1.1.58222 > 192.168.1.2.22: Flags [.], cksum x (correct), seq 2534, ack 3851, win 4096, length 0 While those originating from the bhyve VM (after lowering net.inet.tcp.keepidle) look like this: 12:18:43.105460 IP (tos 0x0, ttl 62, id 0, offset 0, flags [DF], proto TCP (6), length 52) 192.168.1.3.57555 > 192.168.1.2.22: Flags [.], cksum x (correct), seq 1780337696, ack 45831723, win 1026, options [nop,nop,TS val 3003646737 ecr 3331923346], length 0 Like written above, once net.inet.tcp.do_prr is disabled, keepalive seems to be working just fine. Otherwise, Parallel's NAT daemon kills the connection, as its keep-alive requests are not answered (well, that's what I think is happening): 10:19:43.614803 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 40) 192.168.1.1.58222 > 192.168.1.2.22: Flags [R.], cksum x (correct), seq 2535, ack 3851, win 4096, length 0 The easiest way to work around the problem Client side is to configure ServerAliveInterval in ~/.ssh/config in the Client VM. I'm curious though if this is basically a Parallels problem that has only been exposed by PRR being more correct (which is what I suspect), or if this is actually a FreeBSD problem. Michael -- Michael Gmelin