Public IPv6s fail on KVM bridge with "No buffer space available"

[William Gathoye]

Hello everyone,

I've already asked this question on the #networking and #freebsd IRC
channels on Freenode but nobody was able to answer my question and
forwarded me over here as it seems this issue is kicky to solve.

I have a Proxmox hypervisor where are hosted LXC containers and KVM
machines going from Debian 9, Arch Linux and Windows. All of them are
bridged to the hypervisor and the IPv6 stack is working flawlessly with
public IP addresses given by my ISP and routed directly to the VMs.

One of the VM I have is an OPNsense firewall appliance (based on FreeBSD
11.0-RELEASE-p8).

Contrary to the other LXC/KVM, the latter cannot ping any IPv6 machines
outside those accessible directly from the bridge. As soon I try to ping
either the gateway of my hypervisor (still in IPv6), or any other far
away IPv6 hosts (e.g. google.com), I got the following error message:

    [...]
    ping6: sendmsg: No buffer space available
    [...]

- At first, I thought the issue was due to issues with VirtIO drivers
(bundled with FreeBSD). So I switched to emulated Intel E1000 NIC, but
the problem persists. (I'm back with VirtIO now).
- A netstat -m reports the buffers as empty, so the problem doesn't come
from here either.
- Putting the interface down and up again or rebooting doesn't fix the
issue.
- I tested with a fresh FreeBSD and OpenBSD install (to avoid the
OPNsense overlay), but the problem persists as well.

Pinging the VM either from the bridge or from a machine completely
outside of the infrastructure doesn't respond, nor connect (i.e. I had
started sshd on 2222, but weren't able to connect).

The FreeBSD host is configured like this:

    ifconfig vtnet0 <ipv4>/32
    route add <gw ipv4> -iface vtnet0
    route add default <gw ipv4>

    ifconfig vtnet0 inet6 <ipv6> prefixlen 64
    route add -inet6 <gw ipv6> -iface vtnet0
    route add -inet6 default <gw ipv6>

Please note all my GWs are outside of my IP subnets.

After applying these lines, the routes reported by netstat -rn are
sensible to me. Nothing wrong.

I precise the pf firewall is completely disabled (pfctl -d). I want to
make sure this is working flawlessly before enabling yet another level
of failures. :)

Is there a bug somewhere in the BSD IPv6 stack as Linux is not
complaining at all? This sounds weird as I think I'm not the only one in
this situation and not the only one having GW outside their IP ranges.

Thanks in advance for your time / help.

Regards.



--
William Gathoye
<william at gathoye.be>

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: OpenPGP digital signature
URL: <http://lists.freebsd.org/pipermail/freebsd-net/attachments/20170516/37f34257/attachment.sig>