From nobody Thu Sep 05 14:05:42 2024 X-Original-To: dev-commits-src-all@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4X01NQ6JFKz5W6SG; Thu, 05 Sep 2024 14:05:42 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R11" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4X01NQ5cNcz4Tfc; Thu, 5 Sep 2024 14:05:42 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1725545142; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=827YhUn5LPJSOgfoPSPDPjoYPxOR6KfhVUD1/nyHcbA=; b=YGBRRZI25QSezk9SYcUswaiRbBrwQ0aJvoTKJolESlhKWiOOL4pHBH0SjqUYO3YnxfoS8j UteJpyECW9NaUMSCPM8E9WivcOuAEnwDWg9MduJGRDg2wWwrC8O3JtGEG/rIJ4gzeWGpoX EblHgYALJXwC7XQx9c+OkrLi1b/lQWTSMTaze3Dp9L+az6Ed3+P0rBJ7ck6bS1fccZnRX8 PncNlcG+6zo3QxWUWuLdJiYjINfF5oclVjXCESqtg5wpj1c/asfmEgQp+HV/pXF53ra2MC ldMFmaQRpwrGoGE3fDzx7gCZz9Wnnj4cAx1IxskmWk/RlRijaVmcyGOGSgkTwg== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1725545142; a=rsa-sha256; cv=none; b=Ilp9BqMhoNSz8YEZRdMEMAbFtDR4IfC2KYz9HeZ2STi8sPrdUZzg9FCk7j1hue9DH/PogT T/cF9ajpcmGmG2IKqfVtwF7WyNv9VwW8SWlA4GGjcGJ+rj3Q9N7XBM/COyCip4C/b4q9Fw DDRKK8nboyDMPDQJhkiQbI2XH2m3DZtK14ZZu60x0RjgLWVwftupKhSiyNzonsLXAseE4F DCsJru0WeM1g7gYfCpLqztB6KsQwqVwVq0BeoBGLtPEwekc+HL1f/EcMuJXCq69TTQaSf0 v7kDX6ayyQh9d0alXmSeA0kGUFgP0H+FzORRsKF6O0mox7RoTZkdFRV30lq8Bg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1725545142; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=827YhUn5LPJSOgfoPSPDPjoYPxOR6KfhVUD1/nyHcbA=; b=j3LoAhKixvzFNt4vzOWlBvpU1YknolWUwqGwj1CRthd0H/lQgY+ZliSRi+SwrGSZyVbG6c hOqDmFjplv0zsDyQz5ILWDiZlsM1W7xXej3m8gxFFqcY2LEFzM2luhSnaCW6Wl2arX31b4 1ltf1b9Dl0htWAheMO73T9P+zhQcYZe71EfY/U8gCz6HjLIfuL803gTWh9II5WXCQVwC+s rFf8LWo50OuS71zZhh9Mogfu0CKb6ExreDdCX10ht6G/DaZ1Xsi1jc8ib6fhmCEdttf30z P3kKsj3wPb4gvGjT3aUhExzi5sLW3ZcX0/8ugJm+M8iPWheNFMnejDRSXXRIYQ== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4X01NQ58dFz1B8q; Thu, 5 Sep 2024 14:05:42 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.18.1/8.18.1) with ESMTP id 485E5gXA099351; Thu, 5 Sep 2024 14:05:42 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.18.1/8.18.1/Submit) id 485E5g0F099348; Thu, 5 Sep 2024 14:05:42 GMT (envelope-from git) Date: Thu, 5 Sep 2024 14:05:42 GMT Message-Id: <202409051405.485E5g0F099348@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-branches@FreeBSD.org From: Pietro Cerutti Subject: git: ab7a79806e31 - stable/14 - libfetch: don't include fragments in HTTP requests List-Id: Commit messages for all branches of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-all List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: dev-commits-src-all@freebsd.org Sender: owner-dev-commits-src-all@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: gahr X-Git-Repository: src X-Git-Refname: refs/heads/stable/14 X-Git-Reftype: branch X-Git-Commit: ab7a79806e3103accb1ba2d89ba51e977704a2e3 Auto-Submitted: auto-generated The branch stable/14 has been updated by gahr: URL: https://cgit.FreeBSD.org/src/commit/?id=ab7a79806e3103accb1ba2d89ba51e977704a2e3 commit ab7a79806e3103accb1ba2d89ba51e977704a2e3 Author: Pietro Cerutti AuthorDate: 2024-08-21 12:35:27 +0000 Commit: Pietro Cerutti CommitDate: 2024-09-05 14:05:15 +0000 libfetch: don't include fragments in HTTP requests Fragments are reserved for client-side processing, see https://www.rfc-editor.org/rfc/rfc9110.html#section-7.1 Also, some servers don't like to receive HTTP requests with fragments. ``` $ fetch 'https://dropbox.com/a/b' fetch: https://dropbox.com/a/b: Not Found $ fetch 'https://dropbox.com/a/b#' fetch: https://dropbox.com/a/b#: Bad Request ``` This is a real-world scenario, where some download link from dropbox (eventually) redirects to an URL with a fragment: ``` $ fetch -v 'https://www.dropbox.com/sh//?dl=1' 2>&1 | grep requesting requesting https://www.dropbox.com/sh//?dl=1 requesting https://www.dropbox.com/scl/fo//?rlkey=&dl=1 requesting https://.dl.dropboxusercontent.com/zip_download_get/# ``` See how the last redirect ends with a `#`. Currently, libfetch includes the ending fragment and makes it impossible to download the file. Differential Revision: https://reviews.freebsd.org/D46318 MFC after: 2 weeks (cherry picked from commit 1af7d5f389536a2f391153513d95d92ffdf360e4) --- lib/libfetch/fetch.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/lib/libfetch/fetch.c b/lib/libfetch/fetch.c index 8cb3536a46c5..6e14a62ca596 100644 --- a/lib/libfetch/fetch.c +++ b/lib/libfetch/fetch.c @@ -448,7 +448,10 @@ nohost: goto ouch; } u->doc = doc; - while (*p != '\0') { + /* fragments are reserved for client-side processing, see + * https://www.rfc-editor.org/rfc/rfc9110.html#section-7.1 + */ + while (*p != '\0' && *p != '#') { if (!isspace((unsigned char)*p)) { *doc++ = *p++; } else {