Cannot connect to master IPs from CARP backup

From: Alejandro Imass <aimass_at_yabarana.com>
Date: Tue, 28 May 2024 08:28:53 UTC
Hi,

I have a CARP setup which works as expected except that from the backup
server, it is impossible to connect to VIPs on the master. I have read
several similar threads:

https://www.reddit.com/r/PFSENSE/comments/15s6e8j/carp_backup_node_unable_to_ping_vips/
https://forums.freebsd.org/threads/carp-problem-ping-only-works-from-the-master-itself.14986/
https://forums.freebsd.org/threads/pf-carp-cant-ping-vip.6039/

My situation is specifically:

   - CARP works, and failover works
   - I can ping the master VIP from any other server on the LAN
   - Ping only fails from inside the backup server, if and only if, the
   backup IPs are up in backup state

My configuration is based on jails:

  ip6 = disable;
  interface = em1;
  ip4.addr = 10.77.3.11/22;
  ip4.addr += "10.77.0.100/22 vhid 10 advbase 1 advskew 100 pass yuca";
  ip4.addr += "10.77.0.101/22 vhid 138 advbase 1 advskew 100 pass yuca";

I do notice a warning each time saying this:
invalid netmask '/22 vhid 138 advbase 1 advskew 0 pass yuca'

I'm using Bastille jails and haven't looked deeply into the error above
because since ifconfig shows the interface properly configured. But perhaps
this is an important clue?

I have tried /32 netmask, broadcast to the specific CARP IPs, promiscuous
mode and almost anything I can think of, and in every combination CARP
seems to work fine except for this one issue where I cannot connect to the
VIPs from the backup whilst in backup mode.

I have tcp dumped arp traffic and the arp tables everywhere seem correct,
even in the backup server, with and without the VIPs active on the backup.

Here is one additional clue and not sure if this is relevant. When I use
CARP alias in the jails as specified above, and I run PostgreSQL with
listen * then I notice that it binds to *:5432 on the base host and not on
the specific jail IPs like a normal jail would do if I did not use the
alias. It has made me wonder if something Bastille does is conflicting with
the CARP aliases I am setting up above. But since everything else works as
expected I really haven't delved deep into Bastille does for NIC aliases.

Version is 14.0-RELEASE standard kernel and everything else is pretty
standard.

Any help or ideas greatly appreciated !!

Best,

-- 
Alex