From nobody Wed Oct 11 15:44:10 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4S5HBf1yVjz4wJmH for ; Wed, 11 Oct 2023 15:44:26 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-ed1-x52c.google.com (mail-ed1-x52c.google.com [IPv6:2a00:1450:4864:20::52c]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4S5HBd3md4z3dBV for ; Wed, 11 Oct 2023 15:44:25 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-ed1-x52c.google.com with SMTP id 4fb4d7f45d1cf-5230a22cfd1so4002a12.1 for ; Wed, 11 Oct 2023 08:44:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20230601.gappssmtp.com; s=20230601; t=1697039062; x=1697643862; darn=freebsd.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=MgJCt797c6zDcxkIMPWcioHp1rW1/yxzo9DdNbkPc70=; b=pcDKJwsELQ+EByz5ToGXJ26zcBiR7Ozaob0hAw4/pldKj6G34TyNhlBv6q+BLX7Dux FI8cmSk9Kj6bJqQx1TgTW5FS2Ykf5YcwYIvm547SbeH6qsA45Et6POfiPwVKuN/ntfU3 PbSfIlysfYmH0dYyHZfsfZPUXM/xx1RIsrUQ1wW9rI8sNzj0UET9DgRV0my+Mn0IKzDB 1ZFEKPR2p1U1GNQ1IGnk8ler2o+76na5GpXm1s1JDIyTmx65nymeCi/mibKjXSu7NZlF pCL7pmJR8TBrDLoy12sphjJtfp0/sDcDS+hDD/jlALQApwyo8SK2bQbzkfubyShzjo+1 +vdw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1697039062; x=1697643862; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=MgJCt797c6zDcxkIMPWcioHp1rW1/yxzo9DdNbkPc70=; b=Re1SLP7WEIP4OAVQ+O3k7TEDGL9t3yjcQWC/5JQZ/Vv5CEFmVRw27EBq297hXjtGcK 9iAG7vcD8iP9GIwX+iNBIbJUO//18b2uj5MGN47HWrlObojSa6h9xPDXCUx67Zbe3Ah5 bYPnisyXC+VVVux2b26WFRWNX3bIuF5wNhIcCLa6AkcO7UDjRPjn6G7OET+S99kUQl+e 5w81d3wvs81DNZiBe6WP8d2Z9169ht3Ieq9GXEOdcxDFuE3PQMJ+TovtJZT3P+4KfIBb LFeRcB0YXN+++GazEAWPqoJMKbQ43LL3Vda1MqhO5TEMINVTCvbfJjmlnIC/Lm9Z1zfn XMzw== X-Gm-Message-State: AOJu0YybmvuX2vKlXU+KUX0JMcWDGwZRgP8x7shlEFY0f7kRpR0GfH2O 3IOHVANoFCba4BA0qePJdavP3WK8+GVYg370ZxAVP6tf2Rg+OXGOiiJn/w== X-Google-Smtp-Source: AGHT+IHysJbQPIhrZiJvRCh1gXMc9Qs90+2HkIkqymlNRRk7U9wjIlEONSQeErFJ292JWYh8M0QfXezcSaGVhwG0ATA= X-Received: by 2002:a05:6402:5d82:b0:53d:ea81:d021 with SMTP id if2-20020a0564025d8200b0053dea81d021mr1270341edb.37.1697039061582; Wed, 11 Oct 2023 08:44:21 -0700 (PDT) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 References: In-Reply-To: From: Warner Losh Date: Wed, 11 Oct 2023 09:44:10 -0600 Message-ID: Subject: Re: Question of installing over SAS controller and USB flash To: "azathanai@icloud.com" Cc: "freebsd-hackers@FreeBSD.org" Content-Type: multipart/alternative; boundary="0000000000006fa25e060772b2de" X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US] X-Rspamd-Queue-Id: 4S5HBd3md4z3dBV --0000000000006fa25e060772b2de Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable There's not a FreeBSD port/pkg for lsiutil, but I'ved used it at $WORK for things. https://github.com/thomaslovell/LSIUtil has an archive of sources. I've not used this at all, and it hasn't been ported to FreeBSD in this repo. But I know it allows one to turn on/off the POST / BIOS in the card, which is what you need. It runs under Linux for sure (again, don't know what packages are available there). I don't think I can share what I have for $WORK with the public since I think we got it in a way that doesn't let me disclose the sources. One might also be able move the code lsiutil that enables it to mpsutil. It's going to be sending a raw command to the device. It shouldn't be too hard.... Warner On Wed, Oct 11, 2023 at 6:02=E2=80=AFAM azathanai@icloud.com wrote: > Hey Warner, > the box is very featureLESS (an old Xyratex HS-1235T box, as far as I > researched it). > T2029_QR v1.3.ai (hackspace.org.uk) > > (5) Xyratex 0944037-03 Motherboard Mystery SOLVED : homelab (reddit.com) > > > Through the POST process and the BIOS there is no option to get into the > SAS controller and enable its BIOS. I would love to install FreeBSD > directly on the drives and not to deal with the flash card, but I suppose > the lack of controller BIOS is the reason the box comes with reader and > card directly. > > As I'm new to FreeBSD and ZFS I'm struggling to set up the partitioning. = I > was thinking to: > > 1. Partition the entire FLASH card (4gigs) on MBR with freebsd-ufs and > mount /boot. As I've read there should not be ESP (efi) partition as t= his > is flash, thus MBR, which is "legacy" and without ESP. > 2. Partition one of the drives (3tera) with GPT, freebsd-zfs stripe > and mount / > 3. When the system is up and running, add the rest of the drives (11 > more) to the pool and change the pool to RAIDZ2. However I'm not sure = this > is possible, to change the type of the pool without erasing everything= . > Logical thinking says it is not possible. > 4. After all of these comes the mounting of filesystems and creating > fstab to continue the installation, which is full darkness at the mome= nt, > as I'm still struggling with the initial start and not have read anyth= ing > on that. > > > All of these should be done in shell with the gpart utility during instal= l > and I cannot figure out, how to make points 2 and 3 on one go having > directly the RAIDZ2. > > > ------------------------------ > *From:* owner-freebsd-hackers@freebsd.org < > owner-freebsd-hackers@freebsd.org> on behalf of Warner Losh < > imp@bsdimp.com> > *Sent:* Wednesday, October 11, 2023 00:20 > *To:* azathanai@icloud.com > *Cc:* freebsd-hackers@FreeBSD.org > *Subject:* Re: Question of installing over SAS controller and USB flash > > > > On Tue, Oct 10, 2023 at 2:36=E2=80=AFPM azathanai@icloud.com > wrote: > > Hi all, > a newbie in FreeBSD here trying to install a RAIDZ2 on a server with mps > SAS controller with drives which are not visible in the boot order in BIO= S. > So, I cannot install directly on the drives, despite that the installer > "sees" them, because the BIOS does not see the controller and its drives = as > option. > > > Have you enabled the BIOS on the mps card itself? Otherwise it won't > install a BIOS fragment to run at boot, and the drives won't be seen. > > On the other hand, I have a card reader with a flash card on it (4gigs) > from which I can boot and where I can have my /boot partition with the > kernel. > I've a lot of questions and I still cannot find the answers to most of > them, despite going through the handbook, the boot process explanation, t= he > installation process handbook and a lot of more reading. > > > - How should the flash card be partitioned and with what mountpoints? > The server is BIOS capable. > - How should the rest of the drives be partitioned with ZFS and the > mountpoints? The standard configuration from the installer on RAIDZ2 i= s > good for me. > > I've experience with Linux but is not helping a lot here. The setup looks > quite different .... probably because of the ZFS. > > Any help, links, ideas and comments will be much appreciated! > > > That's a workaround, but you should be able to enable the BIOS part of th= e > mps/mpr cards and boot off them. I've done that before. > > Warner > > > > Cyberly yours, > deragoth > > > --0000000000006fa25e060772b2de Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
There's not a FreeBSD port/pkg for lsiutil, but I'= ved used it at $WORK for things.


has an archive of sources. I've not used thi= s at all, and it hasn't been ported to FreeBSD in this repo.
= But I know it allows one to turn on/off the POST / BIOS in the card, which = is what you need.=C2=A0 It runs
under Linux for sure (again, don&= #39;t know what packages are available there). I don't think I can shar= e
what I have for $WORK with the public since I think we got it i= n a way that doesn't let me disclose the
sources.
<= br>
One might also be able move the code lsiutil that enables it = to mpsutil. It's going to be sending a raw
command to the dev= ice. It shouldn't be too hard....

Warner
=

= On Wed, Oct 11, 2023 at 6:02=E2=80=AFAM azathanai@icloud.com <a= zathanai@icloud.com> wrote:
Hey Warner,
the box is very featureLESS (an old Xyratex HS-1235T box, as far as I resea= rched it).=C2=A0

Through the POST process and the BIOS there is no option to get into the SA= S controller and enable its BIOS. I would love to install FreeBSD directly = on the drives and not to deal with the flash card, but I suppose the lack o= f controller BIOS is the reason the box comes with reader and card directly.

As I'm new to FreeBSD and ZFS I'm struggling to set up the partitio= ning. I was thinking to:
  1. Partition the entire FL= ASH card (4gigs) on MBR with freebsd-ufs and mount /boot. As I've read = there should not be ESP (efi) partition as this is flash, thus MBR, which i= s "legacy" and without ESP.
  2. Partition one of the drives (3tera) with GPT, fre= ebsd-zfs stripe and mount /
  3. When the system is up and=C2=A0running, add the rest of the= drives (11 more) to the pool and change the pool to RAIDZ2. However I'= m not sure this is possible, to change the type of the pool without erasing= everything. Logical thinking says it is not possible.
  4. After all of these comes the mounting of filesystems and c= reating fstab to continue the installation, which is full darkness at the m= oment, as I'm still struggling with the initial start and not have read= anything on that.

All of these should be done in shell with the gpart utility duri= ng install and I cannot figure out, how to make points 2 and 3 on one go ha= ving directly the RAIDZ2.


From: = owne= r-freebsd-hackers@freebsd.org <owner-freebsd-hackers@freebsd.org>= on behalf of Warner Losh <imp@bsdimp.com>
Sent: Wednesday, October 11, 2023 00:20
To: azatha= nai@icloud.com <azathanai@icloud.com>
Cc: freebsd-hackers@FreeBSD.org <freebsd-hackers@freebsd.org>
Subject: Re: Question of installing over SAS controller and USB flas= h
=C2=A0


On Tue, Oct 10, 2023 at 2:36=E2=80=AFPM azathanai@icloud.com <azathanai@icloud.com> wrote:
Hi all,
a newbie in FreeBSD here trying to install a RAIDZ2 on a server with mps SA= S controller with drives which are not visible in the boot order in BIOS.
So, I cannot install directly on the drives, despite that the installer &qu= ot;sees" them, because the BIOS does not see the controller and its dr= ives as option.

Have you enabled the BIOS on the mps card itself? Otherwise it won'= ;t install a BIOS fragment to run at boot, and the drives won't be seen= .

On the other hand, I have a card reader with a flash card on it (4gigs) fro= m which I can boot and where I can have my /boot partition with the kernel.= =C2=A0
I've a lot of questions and I still cannot find the answers to most of = them, despite going through the handbook, the boot process explanation, the= installation process handbook and a lot of more reading.

  • How should the flash car= d be partitioned and with what mountpoints? The server is BIOS capable.
  • How should t= he rest of the drives be partitioned with ZFS and the mountpoints? The stan= dard configuration from the installer on RAIDZ2 is good for me.
    • =
I've experience with Linux but is not helping a lot here. Th= e setup looks quite different .... probably because of the ZFS.

Any help, links, ideas and comments will be much appreciated!

That's a workaround, but you should be able to enable the BIOS par= t of the mps/mpr cards and boot off them. I've done that before.

Warner
=C2=A0

Cyberly yours,
deragoth


--0000000000006fa25e060772b2de--