From nobody Wed Apr 17 09:16:47 2024 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4VKFfP2Gjyz5JCfl for ; Wed, 17 Apr 2024 09:17:01 +0000 (UTC) (envelope-from aimass@yabarana.com) Received: from mail-il1-x12b.google.com (mail-il1-x12b.google.com [IPv6:2607:f8b0:4864:20::12b]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4VKFfN3DSxz4MBp for ; Wed, 17 Apr 2024 09:17:00 +0000 (UTC) (envelope-from aimass@yabarana.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yabarana-com.20230601.gappssmtp.com header.s=20230601 header.b=YJACTDNo; dmarc=none; spf=pass (mx1.freebsd.org: domain of aimass@yabarana.com designates 2607:f8b0:4864:20::12b as permitted sender) smtp.mailfrom=aimass@yabarana.com Received: by mail-il1-x12b.google.com with SMTP id e9e14a558f8ab-36aff02f998so19718385ab.3 for ; Wed, 17 Apr 2024 02:17:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yabarana-com.20230601.gappssmtp.com; s=20230601; t=1713345418; x=1713950218; darn=freebsd.org; h=to:subject:message-id:date:from:mime-version:from:to:cc:subject :date:message-id:reply-to; bh=eiV5Hk8ChBBKJs39T0nDwCLsAZLcSEsHolfuc3HqWfk=; b=YJACTDNognsya0oszg6GJLHP12+rAOSKLBxW2Pn3cGYkvG5JWxuJr4FWaOOHBUq6OC f1EeOA052Owasp9TWz+jcuQkvFxYxT4gPraj8IJ5nGXtJCdMgVcHnDl0GXRyywDTGjgV oiJ4JZ03e5m2mp12Q3lX1Fpr8woC04yfj6Qd2zdeL5hJwPUpYUcX8sCkFUjOXZn3Dth3 Su2tzFgrdrFO8KUWSWkcABrkPZZu1ono3unkUsk+UK3cpDxQGuWXLMdxvtmw6kOhYqeX kwVVB3/yo7TFuW5JZrAv7YLzM+GsvcTpIvvW+LxOL0CEVMH8rtRhDilVZPy2PR1dn5sf wHgw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1713345418; x=1713950218; h=to:subject:message-id:date:from:mime-version:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=eiV5Hk8ChBBKJs39T0nDwCLsAZLcSEsHolfuc3HqWfk=; b=vojFUAm/JOQp+PvWD3XwSnuN6eBuFK8eBHEaley5oppFyIh9LMBf/BhRVufBshXZMj pLP5KAlF34/69yPoJdQ1uo/mISQjK30ivL4eqXSfKnxiT5SzdqdllD4C0sr31KQg3ORN TJx2rij6i3xjR4siqWwp5PV6wChdtiNpRmeYjUWj2xLyap6wTOB5l+sNCQqsrK0e6iJv wJvg0xpQHt77Akobe+HvuEo7H5ficIr/MlXEpRcuSzEYqrK/1VE7bhYxskMDFa/J23Er vLOzoolv4boFJqvSk7LRNxXWaXzzyusjqq27WTK6ddcpaeMitgp6uCb+7pnU4oXGU3cR MZMQ== X-Gm-Message-State: AOJu0YxLJuNI9nItlucRAawWK24p8OfRPPcjUF6QvPWkw6JX2fyPLN5+ bXutNBVEpqCHi33yrsSoEVbKVzUCluHS3vLgYqcFL9EY6oHM/3I3VKsEyGEozt8yNeocgy6Fdh0 NiX72DPq+F9L9s3aRRHYtvQ5tJyg8+NCj2ZXB9i6BFFLdCyfXISk= X-Google-Smtp-Source: AGHT+IHy13rah3u89KtYgmTBJxvhDJySWWJ1wmdqsAC/CZOdGxFHmufRy0fQAud8d5yLw320nOf9pG2LBXP32bJx1BQ= X-Received: by 2002:a05:6e02:1fc1:b0:36a:a6c4:fa7d with SMTP id dj1-20020a056e021fc100b0036aa6c4fa7dmr19315299ilb.0.1713345418257; Wed, 17 Apr 2024 02:16:58 -0700 (PDT) List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org MIME-Version: 1.0 From: Alejandro Imass Date: Wed, 17 Apr 2024 11:16:47 +0200 Message-ID: Subject: Corosync and Pacemaker with Jails. Is it worth the effort? To: FreeBSD virtualization Content-Type: multipart/alternative; boundary="00000000000008889e061647515e" X-Spamd-Bar: -- X-Spamd-Result: default: False [-2.48 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; SUBJECT_ENDS_QUESTION(1.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.98)[-0.982]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; R_DKIM_ALLOW(-0.20)[yabarana-com.20230601.gappssmtp.com:s=20230601]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; MISSING_XM_UA(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; RCVD_COUNT_ONE(0.00)[1]; ARC_NA(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FROM_HAS_DN(0.00)[]; DMARC_NA(0.00)[yabarana.com]; RCVD_TLS_LAST(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::12b:from]; PREVIOUSLY_DELIVERED(0.00)[freebsd-virtualization@freebsd.org]; MLMMJ_DEST(0.00)[freebsd-virtualization@freebsd.org]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[yabarana-com.20230601.gappssmtp.com:+] X-Rspamd-Queue-Id: 4VKFfN3DSxz4MBp --00000000000008889e061647515e Content-Type: text/plain; charset="UTF-8" Hello, Are these cluster tools actually usable on FBSD and in Jails? I got a 3 way PostgreSQL cluster with floating IPs to _mostly_ work across 3 Bastille-based jails but the whole process has been very painful, so I'm wondering if this stuff is actually usable on FBSD or am I just wasting my time, and this port is just a curiosity ? I have noticed that several of the resource agents are very Linux oriented, for example, requiring the ip command, and some of the very basic clustering logic seem to depend on things like systemd. So far I have discovered that you have to use VNET jail and I have had to add: allow.mlock = 1; sysvmsg = new; sysvsem = new; sysvshm = new; As well as this to corosync: system { qb_ipc_type: shm sched_rr: no stonith-enabled: false } Not sure if there are other things I haven't noticed because even though I finally got the cluster to run, the promotion and demotion seems very unstable still. Also, I cannot get rid of these warnings in every command: WARNING: could not get the pacemaker version, bad installation? WARNING: list index out of range I am happy to continue the calvary if it's worthwhile and there are other people using and/or interested in this, but the lack of information seems to suggest that maybe this is just a waste of time, and these tools are nowhere near a usable state without heavy lifting. If the latter is the case, are there any other stable HA/Clustering solutions that are mature for FBSD+Jails, and that include resource managers such as databases (namely Pg and MariaDB) and floating IPs ? This is the environment: FreeBSD 14.0-RELEASE Bastille: 0.10.20231125 Pacemaker 2.1.6 Corosync 3.1.7 Many thanks in advance, for any pointers or comments. -- Alex --00000000000008889e061647515e Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hello,

Are these cluster tools ac= tually usable on FBSD and in Jails?

I got a 3 way Postgr= eSQL cluster with floating IPs to _mostly_ work across 3 Bastille-based jai= ls but the whole process has been very painful, so I'm wondering if thi= s stuff is actually usable on FBSD or am I just wasting my time, and this p= ort is just a curiosity ?

I have noticed that = several of the resource agents are very Linux oriented, for example, requir= ing the ip command, and some of the very basic clustering logic seem=C2=A0t= o depend on things like systemd.

So far I have discovered that you have to use VNET jail and I have= had to add:

allow.mlock =3D 1;
sysvmsg = =3D new;
sysvsem =3D new;
sysvshm =3D new;

As well as this to corosync:
system {
=C2=A0 =C2=A0 =C2=A0 = =C2=A0qb_ipc_type: shm
=C2=A0 =C2=A0 =C2=A0 =C2=A0sched_rr: no
=C2=A0= =C2=A0 =C2=A0 =C2=A0stonith-enabled: false
}

Not= sure if there are other things I haven't noticed because even though I= finally got the cluster to run, the promotion and demotion seems very unst= able still.

Also, I cannot get rid of these warnin= gs in every command:
WARNING: could not get the pacemaker version= , bad installation?
WARNING: list index out of range

<= /div>
I am happy to continue the calvary if it's worthwhile an= d there are other people using and/or interested in this, but the lack of i= nformation seems to suggest that maybe this is just a waste of time, and th= ese tools are nowhere near a usable state without heavy lifting.
<= div>
If the latter is the case, are there any other stable HA= /Clustering solutions that are mature for FBSD+Jails, and that include reso= urce managers such as databases (namely Pg and MariaDB) and floating IPs ?<= /div>

This is the environment:
FreeBSD 14= .0-RELEASE
Bastille:=C2=A00.10.20231125
Pacemaker 2= .1.6
Corosync=C2=A03.1.7

Many = thanks in advance, for any pointers or comments.

-= -=C2=A0
Alex




--00000000000008889e061647515e--