Re: Diagnosing virtual machine network issues

> On Aug 14, 2024, at 11:29 AM, Rodney W. Grimes <freebsd-rwg@gndrsh.dnsmgr.net> wrote:
> 
>>> 
>>> On Aug 13, 2024, at 9:15?AM, Bakul Shah <bakul@iitbombay.org> wrote:
>>> 
>>> This weird 127. address seems like a systemd feature/bug thing: https://unix.stackexchange.com/questions/612416/why-does-etc-resolv-conf-point-at-127-0-0-53
>>> 
>>> This behavior seems like some strange interaction between systemd assumptions and freebsd?s, or something not being set up quite right on the linux side when the vm is running freebsd. 
>> 
>> Could libvirt be a factor here, do you think? For example, perhaps the
>> network should be configured differently than the default when the host
>> is using systemd-resolved and/or when the guest is FreeBSD. In the network
>> XML format for libvirt (https://libvirt.org/formatnetwork.html), there is
>> a `domain` element with a `localOnly` attribute that I have seen set by
>> some virtualization projects. As far as I can tell, our setup isn't using
>> the `domain` element at all.
> 
> Having a /etc/resolv.conf entry of 127.0.0.53 is indeed something
> out of the normal on a freebsd box.  You need to find where that
> is coming from and why that value is used.

The 127.0.0.53 entry in /etc/resolv.conf is on the Linux host machine,
not on the FreeBSD VM. The host is using `systemd-resolved` for managing
its /etc/resolv.conf. In the VM, /etc/resolv.conf has the host IP by
default, and we added 8.8.8.8 so that it wouldn't take a full 30 seconds
to report a domain resolution failure.

>> 
>>> 
>>>> On Aug 13, 2024, at 8:46 AM, Alex Arslan <ararslan@comcast.net> wrote:
>>>> 
>>>> ?
>>>> Hi Rodney,
>>>> 
>>>>> On Aug 10, 2024, at 9:11?AM, Rodney W. Grimes <freebsd-rwg@gndrsh.dnsmgr.net> wrote:
>>>>> 
>>>>>> 
>>>>>> 
>>>>>>> On Aug 2, 2024, at 5:58?PM, Bakul Shah <bakul@iitbombay.org> wrote:
>>>>>>> 
>>>>>>> On Aug 2, 2024, at 3:52?PM, Alex Arslan <ararslan@comcast.net> wrote:
>>>>>>>> 
>>>>>>>>> Just a comment and a name server line:
>>>>>>>>> 
>>>>>>>>> $ cat /etc/resolv.conf
>>>>>>>>> # Generated by resolvconf
>>>>>>>>> nameserver 192.168.122.1
>>>>>>>> 
>>>>>>>> I believe that is the host IP, so I guess the VM is using the host for DNS
>>>>>>>> resolution? Interestingly, if I add `nameserver 8.8.8.8` below the line
>>>>>>>> with the host IP, it takes 10 seconds rather than 30 to reach the expected
>>>>>>>> domain resolution failure. If I put 8.8.8.8 above the host IP, the domain
>>>>>>>> resolution failure is instantaneous.
>>>>>>> 
>>>>>>> What does your host use as a namesever?
>>>>>> 
>>>>>> The nameserver is 127.0.0.53. It sets options edns0 and trust-ad, and
>>>>>> includes a search entry as well.
>>>>> 
>>>>> First, is that a typo and you mean 127.0.0.1:53?
>>>> 
>>>> No, the host's /etc/resolv.conf has `nameserver 127.0.0.53`, I just went
>>>> back and rechecked to be sure.
>>>> 
>>>>> Second, is that name server locked to 127.0.0.1, or is it
>>>>> actually listinging on *:53?  If it is LOCKED you have no name server
>>>>> running on 192.168.122.1 to be reached by the VM, if it is NOT locked
>>>>> can the guest ping 192.168.122.1, and can it reach dns at that IP on
>>>>> port 53?   Can the host send a packet BACK to the guest?
>>>> 
>>>> I apologize but I don't really know enough about these things to know how
>>>> to answer your question. I did post the output of tcpdump on the VM and
>>>> the host a while back but that was for the invalid request, so that
>>>> probably doesn't capture what you're describing.
>>>> 
>>>>> Third you can "fix" the "nameserver 192.168.122.1" entry in /etc/resolv.conf
>>>>> by configuring the DHCP server that handed out the lease to the VM to send
>>>>> a namserver entry of 8.8.8.8.
>>>> 
>>>> If I understand correctly, that is indeed what we've done as a Band-Aid fix
>>>> for the time being: I added the line `prepend_nameservers=8.8.8.8` to
>>>> /etc/resolvconf.conf.
>>>> 
>>>>>> 
>>>>>>> 
>>>>>>>> Not a particularly satisfying conclusion to this saga as I don't understand
>>>>>>>> why it's happening but at least I have a workaround that should hopefully
>>>>>>>> do the job. I really appreciate everyone's help and input thus far!
>>>>>>>> 
>>>>>>>> What's the best way to add `nameserver 8.8.8.8` to /etc/resolv.conf as
>>>>>>>> part of the VM's configuration?
>>>>>>> 
>>>>>>> You should diagnose the problem of the nameserver at 192.168.122.1
>>>>>>> and fix it to act properly. I don't use vm (just bhyve) so can't help
>>>>>>> you with its config.
>>>>>> 
>>>>>> I do still plan to try to figure out what the actual issue is, but I also
>>>>>> now have a path forward in the meantime. :)
>>>>>> 
>>>>>> 
>>>>> 
>>>>> -- 
>>>>> Rod Grimes                                                 rgrimes@freebsd.org <mailto:rgrimes@freebsd.org><mailto:rgrimes@freebsd.org>
>> 
> 
> -- 
> Rod Grimes                                                 rgrimes@freebsd.org <mailto:rgrimes@freebsd.org>