From nobody Tue Mar 12 21:03:11 2024 X-Original-To: freebsd-questions@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TvR226lknz5FVDX for ; Tue, 12 Mar 2024 21:03:22 +0000 (UTC) (envelope-from robert@webtent.org) Received: from mx3.webtent.net (mx3.webtent.net [208.38.145.5]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4TvR2157Mhz4bxy for ; Tue, 12 Mar 2024 21:03:21 +0000 (UTC) (envelope-from robert@webtent.org) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=webtent.org header.s=201611 header.b=r3S1uws5; dmarc=pass (policy=reject) header.from=webtent.org; spf=pass (mx1.freebsd.org: domain of robert@webtent.org designates 208.38.145.5 as permitted sender) smtp.mailfrom=robert@webtent.org Received: from localhost (localhost [127.0.0.1]) by mx3.webtent.net (WebTent ESMTP Postfix Internet Mail Exchange) with ESMTP id 65985D79A5 for ; Tue, 12 Mar 2024 17:03:19 -0400 (EDT) Received: from mx3.webtent.net ([127.0.0.1]) by localhost (mx3.webtent.net [127.0.0.1]) (maiad, port 10024) with ESMTP id 80549-07 for ; Tue, 12 Mar 2024 17:03:19 -0400 (EDT) Received: from www1.webtent.net (www1.webtent.net [208.38.145.42]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx3.webtent.net (WebTent ESMTP Postfix Internet Mail Exchange) with ESMTPS id C2E1ED7997 for ; Tue, 12 Mar 2024 17:03:18 -0400 (EDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=webtent.org; s=201611; t=1710277398; bh=nn0QYYT/55mxHlxggQBjXp4qnzW43o7V9cx5ifr0jvM=; h=To:From:Subject:Date; b=r3S1uws5l2IAiRdQdH/MPQAx2/SRyOuj6jDsgFEzXF1b9dJLC2SLRjj1rfIa6ENHy QnMsohnMDgf1r0T4R5c/YpDjYw3uGDgTm0b1MDEtwtAE9LRcNZKvF5K2KdDJWLe6FL e7IPaHZq/XbTSYbODTfMhBcLY2AZGtSVSYiL0LQVQi340zg+2Yv0SuuZpV+bwRWpWv XTPCIlkySs7/KilanDo7IPwbeL11kWHbydXxJQJToTJt7qqf2hf0L+kIMw+fcdpjyt iGWtxgwB9OYBvFgGIik4U9h2oYqp8EgivfjOn0aZOcbier+CT8gsK377Zz6aB4WXSi 2EujQXAhM30jQ== Received: from [192.168.1.50] (ns2.webtent.net [144.129.73.34] (may be forged)) (authenticated bits=0) by www1.webtent.net (8.13.8/8.13.8) with ESMTP id 42CL3Hkk005145 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Tue, 12 Mar 2024 17:03:18 -0400 To: FreeBSD From: Robert Fitzpatrick Subject: HAST on ZFS with CARP failover Message-ID: <7fbdaa85-4ac4-3c3f-f655-5512c281b262@webtent.org> Date: Tue, 12 Mar 2024 17:03:11 -0400 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 PostboxApp/7.0.60 List-Id: User questions List-Archive: https://lists.freebsd.org/archives/freebsd-questions List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-questions@freebsd.org X-BeenThere: freebsd-questions@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Content-Language: en-US X-Virus-Scanned: WebTent Mailguard 1.0.4_3 X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.70 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; RCVD_DKIM_ARC_DNSWL_MED(-0.50)[]; DMARC_POLICY_ALLOW(-0.50)[webtent.org,reject]; R_DKIM_ALLOW(-0.20)[webtent.org:s=201611]; R_SPF_ALLOW(-0.20)[+mx]; RCVD_IN_DNSWL_MED(-0.20)[208.38.145.5:from]; MIME_GOOD(-0.10)[text/plain]; ARC_NA(0.00)[]; FREEFALL_USER(0.00)[robert]; MIME_TRACE(0.00)[0:+]; RCPT_COUNT_ONE(0.00)[1]; ASN(0.00)[asn:16724, ipnet:208.38.144.0/22, country:US]; RCVD_TLS_LAST(0.00)[]; MLMMJ_DEST(0.00)[freebsd-questions@freebsd.org]; FROM_EQ_ENVFROM(0.00)[]; FROM_HAS_DN(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; TO_DN_ALL(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-questions@freebsd.org]; RCVD_COUNT_THREE(0.00)[4]; DKIM_TRACE(0.00)[webtent.org:+] X-Rspamd-Queue-Id: 4TvR2157Mhz4bxy I was looking for suggestions for this scenario. I have the following two server to test this with in the lab: Lenovo x3550 M5 using M5210 RAID controller 16 cores total from dual CPU and 32GB IBM x3550 M4 using M5110 RAID controller 12 cores total from dual CPU and 32GB I have some of these M4 servers in production for years now on Linux with mdadm used only for KVM virtualization, needing to replace drives maybe 4 or 5 times in 2 of 3 servers over 3-5 years. Linux RAID always allowed me to hot-swap by disk clone of another drive structure and re-insert without issue except once, which turned out to be a server issue. However, I have a lot of FreeBSD VM and db server using ZFS. I'm a huge fan and want to use ZFS more. I am planning to use these two to both test bhyve as an alternative VM host as well as HAST high availability storage. The hardware RAID would be disabled in favor of JBOD, I believe that will be the only option with these controllers. I'll also use emulated NVMe vs VirtIO as well as raw files versus zvols. These are the docs I've based my setup on... https://klarasystems.com/articles/virtualization-showdown-freebsd-bhyve-linux-kvm/ https://forums.freebsd.org/threads/hast-and-zfs-with-carp-failover.29639/ So, my questions... Would there be any reason not to use virtual machines as the HAST hosts, instead using direct ZFS pools? Would this setup using JBOD disks be OK for production? If I use ZFS for storage, is it more beneficial to use one pool for each VM raw file or one pool for all raw files? I would assume separate to enable snapshots per VM raw disk? Thanks for any suggestions, recommendation or guidance. -- Robert