From nobody Tue Oct 03 12:26:20 2023 X-Original-To: dev-commits-doc-all@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4S0H9n1Ml6z4wCDl for ; Tue, 3 Oct 2023 12:26:21 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4S0H9n0rSjz4V2Z; Tue, 3 Oct 2023 12:26:21 +0000 (UTC) (envelope-from git@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1696335981; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=tCK1NAdKyn8nwTsxXPS4nR10KbuCGp95RSijzHYUlOQ=; b=dlnPyaAG3uf/e2o8rDzJtjj9KxuUbVO/1yG+7q/zflVRmzrgynffBIayEfmniSALqDKEtf d9gpxxI1Bv270J9dxKSAeF685K7doS+xKgQtUJo2bRQsOUp7smIGMjoE/EBJ0606L9h8O+ 3QmAdKdUpC8/wKQILKPSSRSZB7x7qEPM1UlmVVrm5GnbqaLwzvLVCPjNOlyr+vUyvX10p0 AlBh3OqXR1YG50+ieFNfqdYgO4A7jKjfpOC9ghGU03RAxi9zT4d5W9c6KykkyHAFNkQ56e x/UH//FVqGv5xe9bUi7nRGRNolqzQCtv7vTaxilivn5wXcFl7UB/hOtMGMBZnw== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1696335981; a=rsa-sha256; cv=none; b=Ak3XjViVT66nYPNNXhzcUFLGZbCgfw/bUgPXfCXBfk/txkuJuVbIzIo3v3V2Ki634AqdQ/ N5G3a0CpJtDRdwh0xqTkuP1Z4W9eYC75SP8Kdz78+gtKt/9uLc2tO/ycGqhOBerVwdI+gM DLJeAl7ekOOcJoar9b/+aKGv6flsRKYUByAtTnjYEvdfUJclkt2LZm0zy6KlpwOJ185kue S8aMf60xKnf/9JkarKrS/fr5BeRhi7cSSvPllwinIbs5ATY4v1G9yDYkKx+ratqWdBE+Aj KjF5H385q/aCVyDkwQWUaEosLYRATdxbgvt+5cSxyGkvR7GKWfN8p+8G055l/A== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1696335981; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=tCK1NAdKyn8nwTsxXPS4nR10KbuCGp95RSijzHYUlOQ=; b=fKfdGrwcAPVGixTcv0E4RL0MC+10mlWEVXNGEcKCAUznHNknYbia1pfXRGBUdYtZK5cWKV N4MYA6h+Ni5MjE2D+pSkXWUdhsfIkpkvhzH9lwP8QXhRbaK5NHJ+ujHBOUz6iaQts4Zyfq prTe/rIN8rjMOHs+x0WxyqQLgpfJX03geQgnhi7XTSlRoelqBdRytImjsqLjQElIg4OrAu CJfg6Opa+XJ7Lec7hQECqlY/tC1B7mVfDFgWhIdA8vkrFq7SHVU8/9EuFbK9Xt4mKgL0FW kQsl2ujJdjvkpVgl+Sc2dgfnNsbHUTCrlHWbxjABL503gCakDlwDcEs8/fOSeA== Received: from gitrepo.freebsd.org (gitrepo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4S0H9m6zVXzvQn; Tue, 3 Oct 2023 12:26:20 +0000 (UTC) (envelope-from git@FreeBSD.org) Received: from gitrepo.freebsd.org ([127.0.1.44]) by gitrepo.freebsd.org (8.17.1/8.17.1) with ESMTP id 393CQKwJ086639; Tue, 3 Oct 2023 12:26:20 GMT (envelope-from git@gitrepo.freebsd.org) Received: (from git@localhost) by gitrepo.freebsd.org (8.17.1/8.17.1/Submit) id 393CQKOn086636; Tue, 3 Oct 2023 12:26:20 GMT (envelope-from git) Date: Tue, 3 Oct 2023 12:26:20 GMT Message-Id: <202310031226.393CQKOn086636@gitrepo.freebsd.org> To: doc-committers@FreeBSD.org, dev-commits-doc-all@FreeBSD.org From: Wolfram Schneider Subject: git: 67b05e8601 - main - block SEO bots from accessing cgi scripts List-Id: Commit messages for all branches of the doc repository List-Archive: https://lists.freebsd.org/archives/dev-commits-doc-all List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-dev-commits-doc-all@freebsd.org X-BeenThere: dev-commits-doc-all@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Git-Committer: wosch X-Git-Repository: doc X-Git-Refname: refs/heads/main X-Git-Reftype: branch X-Git-Commit: 67b05e86012db7e1be22ab28203f8bc9f6a8148a Auto-Submitted: auto-generated The branch main has been updated by wosch: URL: https://cgit.FreeBSD.org/doc/commit/?id=67b05e86012db7e1be22ab28203f8bc9f6a8148a commit 67b05e86012db7e1be22ab28203f8bc9f6a8148a Author: Wolfram Schneider AuthorDate: 2023-10-03 12:26:09 +0000 Commit: Wolfram Schneider CommitDate: 2023-10-03 12:26:09 +0000 block SEO bots from accessing cgi scripts --- website/themes/beastie/layouts/robots.txt | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/website/themes/beastie/layouts/robots.txt b/website/themes/beastie/layouts/robots.txt index b3bb9e880b..70cc4362be 100644 --- a/website/themes/beastie/layouts/robots.txt +++ b/website/themes/beastie/layouts/robots.txt @@ -11,3 +11,21 @@ Disallow: /cgi User-agent: CCBot Disallow: /cgi +# SEO +User-agent: SemrushBot +Disallow: /cgi + +User-agent: MJ12bot +Disallow: /cgi + +User-agent: BLEXBot +Disallow: /cgi + +User-agent: dotbot +Disallow: /cgi + +User-agent: SeekportBot +Disallow: /cgi + +User-agent: AhrefsBot +Disallow: /cgi