From nobody Wed Aug 21 12:35:33 2024 X-Original-To: dev-commits-src-all@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Wpm5K420Jz5TpBs; Wed, 21 Aug 2024 12:35:33 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R11" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Wpm5K3Mcdz43fy; Wed, 21 Aug 2024 12:35:33 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1724243733; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=utHkc/pAA8C4vyL1MK+WU+Oj6B0g/7tUgWoDqhOijZA=; b=KU6ITXG5L4gHdEzAMWDyNe2zlH2W+ykYsyJUT8XA6L4WTJAN/fsTLsjK9fAkmmYsfHTLbC HlvbZsbaqOseKavTEN0XpTKAShYeiUBMPIZaSJUiXdM0dGQQ00uc4G4HsEcGBUpGXZ4c9v EbtyyfGwXy/q0MzA7F7DQoj+Pz9BkaC4bPypKpuXExvqm5UQfDa5VQ0+GoH6qryMVJnaRR plBr5wF7FUQqap3zit+JoQn/PqR77ziBr2Q1iGy6amQIDqEKyNJtg0UosjpHuYYPZJnRJw bA9UO2loNIDipLiIivLoYQ78+HYAeVL9Z+7SInKluBPSDb0iaSbUsg+OnhvMMw== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1724243733; a=rsa-sha256; cv=none; b=hMIzy2SigsaL1ZWpe54iQMBqIJ7tk0MwDYVR9pQTx2BPyeES2uNiGaIdpRETaafW1S4Ck3 2YNgb7sawF3AjudBR1IGiJkokRKgdiKirbMthcqJG/qGMeqaaA+cq0ytxOtjmM/PH5/VdZ 9Unc/4H+4CWYlICSg9SvJQR8qvQED+etVuhiETMJSXczuPpm/wDE3ZocMDympbrsaoUJTx 9JkNmvFjFNe8xqblrfOqeoIBg5fwETOg/ADDnK9Th5jYqFQ2rCneePd7wSYDf6v+DcX3sN fJJzSkMw91VI+auiMchyNCz+8s2E4QeJ69JW5XkNhYWTShX67BkJdWuggcADRA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1724243733; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=utHkc/pAA8C4vyL1MK+WU+Oj6B0g/7tUgWoDqhOijZA=; b=ZT1Xc/cdSiqG5DZ/pdEN0V1bHIcw8/q8s9K5pmhA9ABh97pZdesEh3MeWTxrys3Lz6TFa3 hEvSMQ5n/7ZvcDlHvozn37HONxPrXw1H7X3Rd86tHNVkuZu1sZosJI+aLBd4zBdnCiyld/ SXREfskL1p8ssI+r4aaZQmiZFVYyWIDyQBGQWpUOx6MeD4FyuTCYmzNiNfWrrFD96hYueF +e2CAHBy4kkT2Vje+yZrSTNXxxuMmEuF42LMcPi3GzWZVvSCK/ydmKCvUFBalItbMGwqjP 7DLktu6Cj0D0hRPtEeDzsRC9hBnQIeoR3jMNtTNoBShgNYPdYLqZ0mhDhlivcg== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4Wpm5K2yg2zl1G; Wed, 21 Aug 2024 12:35:33 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.18.1/8.18.1) with ESMTP id 47LCZXmH081247; Wed, 21 Aug 2024 12:35:33 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.18.1/8.18.1/Submit) id 47LCZXBS081244; Wed, 21 Aug 2024 12:35:33 GMT (envelope-from git) Date: Wed, 21 Aug 2024 12:35:33 GMT Message-Id: <202408211235.47LCZXBS081244@gitrepo.freebsd.org> To: src-committers@FreeBSD.org, dev-commits-src-all@FreeBSD.org, dev-commits-src-main@FreeBSD.org From: Pietro Cerutti Subject: git: 1af7d5f38953 - main - libfetch: don't include fragments in HTTP requests List-Id: Commit messages for all branches of the src repository List-Archive: https://lists.freebsd.org/archives/dev-commits-src-all List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: dev-commits-src-all@freebsd.org Sender: owner-dev-commits-src-all@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: gahr X-Git-Repository: src X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: 1af7d5f389536a2f391153513d95d92ffdf360e4 Auto-Submitted: auto-generated The branch main has been updated by gahr: URL: https://cgit.FreeBSD.org/src/commit/?id=1af7d5f389536a2f391153513d95d92ffdf360e4 commit 1af7d5f389536a2f391153513d95d92ffdf360e4 Author: Pietro Cerutti AuthorDate: 2024-08-21 12:35:27 +0000 Commit: Pietro Cerutti CommitDate: 2024-08-21 12:35:27 +0000 libfetch: don't include fragments in HTTP requests Summary: Fragments are reserved for client-side processing, see https://www.rfc-editor.org/rfc/rfc9110.html#section-7.1 Also, some servers don't like to receive HTTP requests with fragments. ``` $ fetch 'https://dropbox.com/a/b' fetch: https://dropbox.com/a/b: Not Found $ fetch 'https://dropbox.com/a/b#' fetch: https://dropbox.com/a/b#: Bad Request ``` This is a real-world scenario, where some download link from dropbox (eventually) redirects to an URL with a fragment: ``` $ fetch -v 'https://www.dropbox.com/sh//?dl=1' 2>&1 | grep requesting requesting https://www.dropbox.com/sh//?dl=1 requesting https://www.dropbox.com/scl/fo//?rlkey=&dl=1 requesting https://.dl.dropboxusercontent.com/zip_download_get/# ``` See how the last redirect ends with a `#`. Currently, libfetch includes the ending fragment and makes it impossible to download the file. Differential Revision: https://reviews.freebsd.org/D46318 MFC after: 2 weeks --- lib/libfetch/fetch.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/lib/libfetch/fetch.c b/lib/libfetch/fetch.c index 12cbd0fb746f..97fc04bb09a6 100644 --- a/lib/libfetch/fetch.c +++ b/lib/libfetch/fetch.c @@ -447,7 +447,10 @@ nohost: goto ouch; } u->doc = doc; - while (*p != '\0') { + /* fragments are reserved for client-side processing, see + * https://www.rfc-editor.org/rfc/rfc9110.html#section-7.1 + */ + while (*p != '\0' && *p != '#') { if (!isspace((unsigned char)*p)) { *doc++ = *p++; } else {