From nobody Wed Jun 22 21:53:03 2022 X-Original-To: freebsd-arm@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id D35EE872D36 for ; Wed, 22 Jun 2022 21:53:03 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4LSxvg3v0Hz4YpT for ; Wed, 22 Jun 2022 21:53:03 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 659A46713 for ; Wed, 22 Jun 2022 21:53:03 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 25MLr3Rq067893 for ; Wed, 22 Jun 2022 21:53:03 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 25MLr3AA067892 for freebsd-arm@FreeBSD.org; Wed, 22 Jun 2022 21:53:03 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: freebsd-arm@FreeBSD.org Subject: [Bug 264836] arm/arm/busdma_machdep-v6.c: bounce page accounting leak (noticed with high traffic ftdi usb serial devices) Date: Wed, 22 Jun 2022 21:53:03 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: new X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: arm X-Bugzilla-Version: 13.1-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: jcfyecrayz@liamekaens.com X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-arm@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_id short_desc product version rep_platform op_sys bug_status bug_severity priority component assigned_to reporter Message-ID: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Porting FreeBSD to ARM processors List-Archive: https://lists.freebsd.org/archives/freebsd-arm List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-arm@freebsd.org MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1655934783; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=u6w6rUBZ1LBlMAZDbhBLHw01QlRb0Zv4m1gdX0zcN0c=; b=tHOKRRKe/MVljzBfuttphb3RIiUPuIS0D7YM4sHGr6ewTF/F1FNPT83M1YDM4X0tYg+HRL 0qPNVZZhNkUvqDctY9aBm5kicPmNlgcflep0GKsYEe8il5Ot7hZqEFf0utilpuIcU03blN ctcd6reCw02BpuBg7GqYCh1LhA0NKH1Sd8lurKVlL7LRTaHhZ9LsT8ddpzTwMcyiJVWT5g AR7vdXOE0uyb6hCtfxROA3Mhbsjw34SpTEQnz0rLjU3BIgjUnovAFX31pji5zD9AKHEZy6 1/4xWLtaD8wkNv/3MyoexX68iwg8Z9IFbb2O33woHqrEHVYHyeTxh/2N1M16Uw== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1655934783; a=rsa-sha256; cv=none; b=FGTCrvVizLkb1g99pwZnQsBLChKgwaNjRjuZzbIMmPNC/Mq+Jtq43rlnoy++f5WkRsaa/w bwYAffbcBaQR1eAy9EcoKDFcE+zP932jGAF4qdCsN5zHXX9ZLbmkA+y2FnKETHW1uvL1n4 gNonJJm7IbFcA4AuzStCAyYXNPVv2bX3XQGrM4fMIAWY7MUh2QQCClrFEsW8HIZTvrqBEp arFaJg9RB8Y+YZqQR7kKIA0jv9hPZZcJ0OvjAtfN+3oYqEGE+RTkDj9RC16IOpVg8c8hf2 b6rX5KAMneUqZYqBmZBgSYXrhaKwvQbPd4G1VklwqRn3jqCahBy0p8UvC3P7tw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D264836 Bug ID: 264836 Summary: arm/arm/busdma_machdep-v6.c: bounce page accounting leak (noticed with high traffic ftdi usb serial devices) Product: Base System Version: 13.1-STABLE Hardware: Any OS: Any Status: New Severity: Affects Some People Priority: --- Component: arm Assignee: freebsd-arm@FreeBSD.org Reporter: jcfyecrayz@liamekaens.com In bus_dmamap_unload(), the counters for free_bpages and reserved_bpages ap= pear to be vulnerable to unprotected read-modify-write operations that result in accounting that looks like a page leak. This was noticed on a 2GB quad core i.MX6 system that has more than one dev= ice attached via FTDI based USB serial connection. This system happens to be u= sing FTDI US4232H quad port chips, but the problem is more general. There is a latency timer setting in FTDI chips that is used to set the inte= rval at which short packets of data are flushed from the USB endpoint by the FTDI chip (which has some internal buffer memory). The default latency is 16 ms= .=20 We had set the latency to 4 ms to get data more quickly. We started noticing problems with slower USB responses and eventually the network stack would be affected as well. In the system in question, it fai= rly reliably "locked up" (couldn't ssh any more, trouble spawning processes when logged in on the serial port). In the locked up state, the usb/usbus0.xplr thread of the usb system process was hung and the system could not process usb messages (this i.MX6 system h= as an ehci USB controller). The typical stack dump for usbus0.xplr when things were hung is: 13 100029 usb usbus0.xplr sched_switch+0x9d4 mi_switch+0x184 sleepq_wait+0x2c _cv_wait+0x1bc usbd_do_request_flags+0x4bc usbd_req_get_port_status+0x44 uhub_explore+0xc4 uhub_explore+0x8f8 uhub_explore+0x8f8 usb_bus_explore+0x150 usb_process+0x124 fork_exit+0xc0 swi_exit+0 Once we noticed that hw.busdma.zone0.free_pages was steadily decrementing - eventually down to zero - we started investigating (dtrace was helpful here) why there appeared to be a leak of bounce pages. That's when we found what appears to be the vulnerability in bus_dmamap_unload(). This code has been this way for more than a decade, b= ut it takes a lot of transactions for this to occur and was particularly hard = to find. For a long time, we would work around the problem by detecting the symptoms and just reboot this system to recover - hardly ideal. It would t= ake weeks to months depending on USB traffic load. Adjusting the FTDI latency timer to 0 ms (force packet delivery on every high speed microframe) finally made this happen more quickly. Even if someone else has enough traffic to experience the same problem,=20 I will submit a patch for review. Early results seem promising (in particu= lar the free bounce page accounting now does not show what looks like a leak). This was originally noticed quite a while ago on 11.x, but it has been confirmed on 13.x as well. As an indication of system load, with the 0 ms latency timer we see more th= an 100 bounce pages per second (based on hw.busdma.zone0.total_bounced), and t= he load due to interrupts is about 15%. The high rate of bounce page and interrupt activity gives lots of good opportunity for preemption at just the right time to trigger the accounting leak. Work sponsored by: Microchip Technology, Inc. --=20 You are receiving this mail because: You are the assignee for the bug.=