From nobody Wed Feb 07 15:19:44 2024 X-Original-To: freebsd-fs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4TVP1P3Y52z59YCq for ; Wed, 7 Feb 2024 15:19:53 +0000 (UTC) (envelope-from shuriku@shurik.kiev.ua) Received: from mail.flex-it.com.ua (mail.flex-it.com.ua [193.239.74.7]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4TVP1P1ghhz4K7n for ; Wed, 7 Feb 2024 15:19:53 +0000 (UTC) (envelope-from shuriku@shurik.kiev.ua) Authentication-Results: mx1.freebsd.org; none Received: from [188.231.181.61] (helo=[10.2.1.104]) by mail.flex-it.com.ua with esmtpsa (TLS1.3) tls TLS_AES_128_GCM_SHA256 (Exim 4.97.1 (FreeBSD)) (envelope-from ) id 1rXjiH-00000000K7O-2Ihr; Wed, 07 Feb 2024 17:19:49 +0200 Message-ID: Date: Wed, 7 Feb 2024 17:19:44 +0200 List-Id: Filesystems List-Archive: https://lists.freebsd.org/archives/freebsd-fs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-fs@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: ZFS on a shared iSCSI Content-Language: uk-UA To: Julien Cigar Cc: freebsd-fs@freebsd.org References: <761c8030-5de9-4114-9313-22ec860fb257@shurik.kiev.ua> From: Oleksandr Kryvulia In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-ACL-Warn: SPF failed. 188.231.181.61 is not allowed to send mail from shurik.kiev.ua. X-Rspamd-Queue-Id: 4TVP1P1ghhz4K7n X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:35297, ipnet:193.239.72.0/22, country:UA] 07.02.24 16:24, Julien Cigar: > On Wed, Feb 07, 2024 at 04:12:21PM +0200, Oleksandr Kryvulia wrote: >> 07.02.24 14:24, Julien Cigar: >>> On Wed, Feb 07, 2024 at 11:55:54AM +0100, Andrea Brancatelli wrote: >>>> Hello guys, I'm not 100% this is the correct list to ask this question, if >>>> not please feel free to point me in the right direction. >>>> >>>> I was wondering what could be the best recipe to have an HA cluster sharing >>>> an external ZFS storage. >>>> >>>> Let's say I have two servers running a bunch of Jails and, thus, I'd like >>>> to use ZFS as the underlying storage layer and I have an external (iSCSI) >>>> storage connected. >>>> >>>> Would it be "easily possible" to have some (2?) iSCSI LUN exposed to both >>>> servers and then activate the pool on one or the other server? >>>> >>>> The idea would be to reactivate the filesystem from server A on server B if >>>> the server A fails. >>>> >>>> Would it be "easier" to replicate everything and zfs send datas back and >>>> forth? Clearly that would mean doubling datas and havin a scheduled replica >>>> with a possible delay in data replication, so I'd like to avoid this. >>>> >>>> Any thoughts? >>> I asked something similar a few years ago, the whole thread may be an interesting >>> read: https://lists.freebsd.org/pipermail/freebsd-fs/2016-June/023456.html >>> >>> Just to say that I tried a lot of different things, and came up to the >>> conclusion that for critical production usage all those things are >>> fragile and with hidden dragons everywhere. >>> >>> I love FreeBSD (I use it exclusively everywhere, also at work), but >>> there aren't any supported open-source solution for something like an >>> "highly available ZFS cluster". On the commercial side there is RSF-1 >>> and beasts like Pure storage ($$$) but ... >>> >>> I would really like to see CEPH ported to FreeBSD, that would be >>> extremely useful >> What about www/minio? Does anyone have any experience of production use? We >> have a lot of bhyve vm's backened by local zfs storage, but I am interesting >> in some kind of hyperconverged solution. > I'm not sure Minio offers block storage..? No, but maybe md(4) can be used if needed. For bhyve vm's in my case we can move from zvol to img files.