From nobody Thu May 04 16:07:47 2023 X-Original-To: freebsd-arch@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4QBzHY1GlHz49GBj for ; Thu, 4 May 2023 16:07:53 +0000 (UTC) (envelope-from bofh@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4QBzHY0lFKz3mqh; Thu, 4 May 2023 16:07:53 +0000 (UTC) (envelope-from bofh@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1683216473; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=jc0Aurw8ZjUOvCNKaUZZzs5pydXz4vcwfX9umvf3RAE=; b=YTAfckdcRyrqAORdXwx7KWSUvKBKYugQEEosNzjr6RLjAdIFE47SiwdhqwiO+U9FIZ7Ky1 I7V6f6wEQFhuNJijo6wWnSHDUxAS2F9e6yvBBJ9Or+G7/eaTk2FZTl7qOu3TokgBk40+b7 gT+arDFxFIJCEF+cJNnBRaGGYs6iWlRYPiKV34tv5OdF9fMJgx+ONIHoyi5aZk1T8GOOEm b+0Jn7/szNYs3eJ+ycaUd2SDyx3lV8lLaUVaG+xZINn6vveLpfMeVXS4EmIWLB/nAygnnH +sjzAGm33zhMZ/bqqS5sobwagNq6WSDJQnda3JUvdD3FNJhZje5xo2Aux30U+g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1683216473; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=jc0Aurw8ZjUOvCNKaUZZzs5pydXz4vcwfX9umvf3RAE=; b=BFS2ADMR4C8hQuLJeRyNsV1h6xnkKoNsy6Yz3ZRgGHUaCAjgqv+OuwZcMOaBieLFKddeXC AOSwK+v/H9BnwOs4kDL8sEwqzlM5IkKNnyHWVEcucw8R3CBjEkzvWAVHPir8PNjhsq72kC D7RHdOpTkSY5QpX3ZSiy2MLduezAHQ/OIBZnTw5yS18f/ZPz0sAhybapgbfJt8xKqS7UNj yhKBjPgpNuhCIj5vpYWPeLK5au3weDqC3yAxp/znftG5nNgb57cCuS8uXNhILqbZc6JzFz VR62Ja44DRyrDIWIJQyiv/RtjolsbBt3QD2Udq3Qdn9lkKwHtdWe67W1hGM/sw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1683216473; a=rsa-sha256; cv=none; b=B1kaljDmFXgPD7UuYnh/PhLH9XndYpGgmaUfNCrKL2qKAC8b6+x6lP+mWF3zEGZwAnT9Vl LV7tRvNl2ciSJ37pVSq9tULmJrXLZbKJyHi0/WSDY6/SSiiyooN1Oh1UBr+KG5EYlzZVHW i6GyLi/YGZ4pokudprGAjmUdNjINwQdrmG2ubAfaz25YRC4s+UuX9lbsS2iCS4GFXPUdBI 0nM/BiUOfwu//n4F8TC9hgH+FkGY5p8Z5v+24yBB0zHKS8Bjm9p27GKjZx6EFytD0gEOsT 4b9b/Hssl3VGJMrgk6Tkcd3bcz/E7owoE18BnShlYt271jiLC1y+mtPsZDQcgQ== Received: from mx.bofh.network (mx.bofh.network [IPv6:2a01:4f8:261:25de::227]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature ECDSA (P-256) client-digest SHA256) (Client CN "mx.bofh.network", Issuer "R3" (verified OK)) (Authenticated sender: bofh/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id 4QBzHX1Bqrz1DNk; Thu, 4 May 2023 16:07:52 +0000 (UTC) (envelope-from bofh@freebsd.org) Received: from smtpclient.apple (gw.office.cyso.net [95.97.78.194]) by mx.bofh.network (OpenSMTPD) with ESMTPSA id 7202cc9a (TLSv1.2:ECDHE-ECDSA-AES256-GCM-SHA384:256:NO); Thu, 4 May 2023 16:07:50 +0000 (UTC) From: Moin Rahman Message-Id: <4733AD1A-7954-4B79-9789-A57A0B46C71B@freebsd.org> Content-Type: multipart/alternative; boundary="Apple-Mail=_9D0700B7-AE03-4465-AE0D-777B38B5EADB" List-Id: Discussion related to FreeBSD architecture List-Archive: https://lists.freebsd.org/archives/freebsd-arch List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-arch@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3696.120.41.1.1\)) Subject: Re: OpenSSL 3.0 for 14.0-RELEASE: issues with 1.x/3.x symbol clashing, ports linking against base OpenSSL, ports that don't compile/link against OpenSSL 3, etc Date: Thu, 4 May 2023 18:07:47 +0200 In-Reply-To: Cc: FreeBSD-arch list , Bernard Spil , Cy Schubert , Ed Maste , vishwin@freebsd.org, portmgr To: Enji Cooper References: X-Mailer: Apple Mail (2.3696.120.41.1.1) X-ThisMailContainsUnwantedMimeParts: N --Apple-Mail=_9D0700B7-AE03-4465-AE0D-777B38B5EADB Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 > On May 3, 2023, at 3:14 AM, Moin Rahman wrote: >=20 >=20 >=20 >> On May 2, 2023, at 3:55 AM, Enji Cooper = wrote: >>=20 >> Hello, >> One of the must-haves for 14.0-RELEASE is the introduction of = OpenSSL 3.0 into the base system. This is a must because, in short, = OpenSSL 1.1 is no longer supported as of 09/26/2023 [1]. >>=20 >> I am proposing OpenSSL be made private along with all dependent = libraries, for the following reasons: >> 1. More than a handful of core ports, e.g., = security/py-cryptography [2] [3], still do not support OpenSSL 3.0. >> i. If other dependent ports (like lang/python38, etc) = move to OpenSSL 3, the distributed modules would break on load due to = clashing symbols if the right mix of modules were dlopen=E2=80=99ed in a = specific order (importing ssl, then importing hazmat=E2=80=99s crypto = would fail). >> ii. Such ports should be deprecated/marked broken as = I=E2=80=99ve recommended on the 3.0 exp-run PR [4]. >> 2. OpenSSL 1.1 and 3.0 have clashing symbols, which makes = linking in both libraries at runtime impossible without resorting to a = number of linker tricks hiding the namespaces using symbol prefixing of = public symbols, etc. >>=20 >> The libraries which would need to be made private are as = follows: >> - kerberos >> - libarchive >> - libbsnmp >> - libfetch [5] >> - libgeli >> - libldns >> - libmp >> - libradius >> - libunbound >>=20 >> I realize I=E2=80=99m jumping to a prescribed solution without = additional discussion, but I=E2=80=99ve been doing offline analysis = related to uplifting code from OpenSSL 1.x to 3.x over the last several = months and this is the general prescribed solution I=E2=80=99ve come to = which is needed for $work. My perspective might have some blind spots = and some of the discussion done over IRC and might need to be rehashed = here for historical reference/to widen the discussion for alternate = solutions that don=E2=80=99t have the degree of tunnel vision which the = solution I=E2=80=99m employing at $work requires. >> I=E2=80=99ve tried to include some of the previously involved = parties so they can chime in. >> Thank you, >> -Enji >>=20 >> 1. https://www.openssl.org/blog/blog/2023/03/28/1.1.1-EOL/ >> 2. https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D254853 . >> 3. The reason why it hasn=E2=80=99t been upgraded is because newer = versions require rustc to build, which apparently doesn=E2=80=99t work = on QEMU builders due to missing emulation support: = https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D254853 . >> 4. https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D258413#c15 >> 5. If I remember correctly, some folks suggested that making libfetch = private wasn=E2=80=99t required since the only port that required it was = ports-mgmt/pkg, but I haven=E2=80=99t validated this claim. >=20 > Hi Enji, >=20 > I appreciate your work creating the bugs but please hold on a moment = before you create the bugs. It will slow me down. >=20 > While you were wasting your time creating the ticket for nrpe3 I have = already updated the port to 4.1.0 to unbreak. So until I have the final = list which you will have by end of this week please do not create = tickets. >=20 > And I have not exactly described the process too what I was doing. The = list you are getting in my poudriere might have two possible failure = reason. OpenSSL 3 or LLVM15; and some might be fixed with little = intervention and testing. And as it's not possible to ask poudriere not = to try BROKEN ports so I have marked some port as blacklisted which are = unfixable or broken for other reasons. If you really would like to = create tickets and chase upstream please do: > find /usr/local/poudriere/ports/default -name Makefile -type f -d 3 = -exec grep -E '(BROKEN_SSL\=3D|IGNORE_SSL\=3D).*openssl3' {} \+ >=20 > Thanks for your cooperation. Hi, I have completed my final partial exp-run with ports that has USES=3Dssl = in any form and the result is available from here: = https://pkg.bofh.network/build.html?mastername=3DMAIN-default-openssl3&bui= ld=3D2023-05-04_16h25m58s You will see some blacklisted ports which fails to build on all = supported RELEASE or HEAD. Apart from that I have tried to fix up as = much as possible ports by updating/changing. I will not run anymore = exp-run until we have a proper OSVERSION and OpenSSL 3.0.0 vendor branch = merged into base and fix those ports that has BROKEN_SSL=3Dopenssl30 to = conditionally mark BROKEN with ssl=3Dbase. If someone wants to chase the upstreams of the ports feel free to do so. During fixing these issues I had to upgrade some of the ports with = portmgr(blanket) approval and in case if you think that I have = overstepped on your plannings I apologies for that. However desperate = time needs desperate measures. Now I will go backup fixing LLVM15 issues with 14.0-RELEASE and ports. Kind regards, Moin= --Apple-Mail=_9D0700B7-AE03-4465-AE0D-777B38B5EADB Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8

On May 3, 2023, at 3:14 AM, Moin Rahman <bofh@freebsd.org> = wrote:



On May 2, 2023, at 3:55 = AM, Enji Cooper <yaneurabeya@gmail.com> wrote:

Hello,
One of the must-haves for = 14.0-RELEASE is the introduction of OpenSSL 3.0 into the base system. = This is a must because, in short, OpenSSL 1.1 is no longer supported as = of 09/26/2023 [1].

I am = proposing OpenSSL be made private along with all dependent libraries, = for the following reasons:
1. More than a handful of core = ports, e.g., security/py-cryptography [2] [3], still do not support = OpenSSL 3.0.
i. If other dependent ports (like = lang/python38, etc) move to OpenSSL 3, the distributed modules would = break on load due to clashing symbols if the right mix of modules were = dlopen=E2=80=99ed in a specific order (importing ssl, then importing = hazmat=E2=80=99s crypto would fail).
ii. Such = ports should be deprecated/marked broken as I=E2=80=99ve recommended on = the 3.0 exp-run PR [4].
2. OpenSSL 1.1 and 3.0 have = clashing symbols, which makes linking in both libraries at runtime = impossible without resorting to a number of linker tricks hiding the = namespaces using symbol prefixing of public symbols, etc.

The libraries which would need to = be made private are as follows:
- = kerberos
- libarchive
- = libbsnmp
- libfetch [5]
- = libgeli
- libldns
- = libmp
- libradius
- = libunbound

I realize I=E2=80=99m jumping to = a prescribed solution without additional discussion, but I=E2=80=99ve = been doing offline analysis related to uplifting code from OpenSSL 1.x = to 3.x over the last several months and this is the general prescribed = solution I=E2=80=99ve come to which is needed for $work. My perspective = might have some blind spots and some of the discussion done over IRC and = might need to be rehashed here for historical reference/to widen the = discussion for alternate solutions that don=E2=80=99t have the degree of = tunnel vision which the solution I=E2=80=99m employing at $work = requires.
I=E2=80=99ve tried to include = some of the previously involved parties so they can chime in.
Thank you,
-Enji

1. = https://www.openssl.org/blog/blog/2023/03/28/1.1.1-EOL/
2. https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D254853 = .
3. The reason why it hasn=E2=80=99t been upgraded is = because newer versions require rustc to build, which apparently = doesn=E2=80=99t work on QEMU builders due to missing emulation support: = https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D254853 = .
4. https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D258413#c15<= /a>
5. If I remember correctly, some folks suggested that = making libfetch private wasn=E2=80=99t required since the only port that = required it was ports-mgmt/pkg, but I haven=E2=80=99t validated this = claim.

Hi Enji,

I appreciate your work creating the bugs but = please hold on a moment before you create the bugs. It will slow me = down.

While you were wasting your time = creating the ticket for nrpe3 I have already updated the port to 4.1.0 = to unbreak. So until I have the final list which you will have by end of = this week please do not create tickets.

And = I have not exactly described the process too what I was doing. The list = you are getting in my poudriere might have two possible failure reason. = OpenSSL 3 or LLVM15; and some might be fixed with little intervention = and testing. And as it's not possible to ask poudriere not to try BROKEN = ports so I have marked some port as blacklisted which are unfixable or = broken for other reasons. If you really would like to create tickets and = chase upstream please do:
find = /usr/local/poudriere/ports/default -name Makefile -type f -d 3 -exec = grep -E '(BROKEN_SSL\=3D|IGNORE_SSL\=3D).*openssl3' {} \+

Thanks for your cooperation.

Hi,

I have completed my final partial = exp-run with ports that has USES=3Dssl in any form and the result is = available from here:
https://pkg.bofh.network/build.html?mastername=3DMAIN-default-o= penssl3&build=3D2023-05-04_16h25m58s

You will see some blacklisted ports = which fails to build on all supported RELEASE or HEAD. Apart from that I = have tried to fix up as much as possible ports by updating/changing. I = will not run anymore exp-run until we have a proper OSVERSION and = OpenSSL 3.0.0 vendor branch merged into base and fix those ports that = has BROKEN_SSL=3Dopenssl30 to conditionally mark BROKEN with = ssl=3Dbase.

If = someone wants to chase the upstreams of the ports feel free to do = so.

During = fixing these issues I had to upgrade some of the ports with = portmgr(blanket) approval and in case if you think that I have = overstepped on your plannings I apologies for that. However desperate = time needs desperate measures.

Now I will go backup fixing LLVM15 = issues with 14.0-RELEASE and ports.

Kind regards,
Moin
= --Apple-Mail=_9D0700B7-AE03-4465-AE0D-777B38B5EADB--