From nobody Wed Aug 14 23:38:04 2024 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Wkl7p0QBzz5T4gj for ; Wed, 14 Aug 2024 23:38:46 +0000 (UTC) (envelope-from ararslan@comcast.net) Received: from resqmta-c2p-570503.sys.comcast.net (resqmta-c2p-570503.sys.comcast.net [IPv6:2001:558:fd00:56::5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Wkl7n5SnJz4f2x for ; Wed, 14 Aug 2024 23:38:45 +0000 (UTC) (envelope-from ararslan@comcast.net) Authentication-Results: mx1.freebsd.org; none Received: from resomta-c2p-555661.sys.comcast.net ([96.102.18.228]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 256/256 bits) (Client did not present a certificate) by resqmta-c2p-570503.sys.comcast.net with ESMTPS id eMxrswRSUmPvHeNZdsf6Cp; Wed, 14 Aug 2024 23:38:37 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=comcast.net; s=20190202a; t=1723678717; bh=JzX2l69YiRufzqm9tF0ohVwah1lUdZsWpD/OK8AMT3Y=; h=Received:Received:From:Message-Id:Content-Type:Mime-Version: Subject:Date:To:Xfinity-Spam-Result; b=3jm1l8ZrKTVDBu+Py0ruGPABtpVa3lzWrYc7ItKFnyEqlona1t5PkANnG0m69bqTc x4je081+3b7a43uIYbwL018ofyvDGbpAYiX4oYN3UtDICv9fUDFRPyg9+j4i5VFwmp KIGkX6IHKfkELJm/OWgTzIh6MG8mQT7ijozwQEHH57dfSaU9KVB6ij95FFendwqhql n9AYZfTLvPUSTzgdUYgdfjDgbbSHdf3A65daisDLtt2lOmVveP4JVkmcCZYnFVIdg5 zNhHiyNahTkVlSq2wXO4neLEa998jOj7TyB1ySvuRcolL/TWdhQ/Ilu31L+KB5k6eT dDX1+YOnhgT2Q== Received: from smtpclient.apple ([67.160.29.205]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 256/256 bits) (Client did not present a certificate) by resomta-c2p-555661.sys.comcast.net with ESMTPSA id eNZGslHGEdWKEeNZHsZwzj; Wed, 14 Aug 2024 23:38:16 +0000 From: Alex Arslan Message-Id: <08AA87E3-D631-4EA1-AA30-37B4709630CB@comcast.net> Content-Type: multipart/alternative; boundary="Apple-Mail=_92F98D39-DE38-40D3-865E-5B5E24DB9BDC" List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@FreeBSD.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3774.600.62\)) Subject: Re: Diagnosing virtual machine network issues Date: Wed, 14 Aug 2024 16:38:04 -0700 In-Reply-To: <202408141829.47EITc7B080532@gndrsh.dnsmgr.net> Cc: Bakul Shah , FreeBSD Hackers To: "Rodney W. Grimes" References: <202408141829.47EITc7B080532@gndrsh.dnsmgr.net> X-Mailer: Apple Mail (2.3774.600.62) X-CMAE-Envelope: MS4xfI0Vy0hqOAtwPfsMvmRhotPu+h1ujSoY81sZARM9mZKx/By2GxFFrb+3Hcr3rmPKZbBa99FmQGFHlGHimL2GD/1AB6kFjNYRofDOBXVwWCPCzxaTn/vi WmonsqqBZobYXGPtnhpqVbiNIA8B+9m8COpWYd0VeDem4f+8kzY1ANfpa0z1AGA+pXoV4Y5vh5CJUqBA0Pqd/jvXTF1ronPSOWLtEXpHJypdgMQbEZhxcp+S anQE8NCh/3MfdOBuAWn6iM5+B5Jck+DlJbnhjWK8FyQ= X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:7922, ipnet:2001:558::/29, country:US] X-Rspamd-Queue-Id: 4Wkl7n5SnJz4f2x --Apple-Mail=_92F98D39-DE38-40D3-865E-5B5E24DB9BDC Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 > On Aug 14, 2024, at 11:29=E2=80=AFAM, Rodney W. Grimes = wrote: >=20 >>>=20 >>> On Aug 13, 2024, at 9:15?AM, Bakul Shah wrote: >>>=20 >>> This weird 127. address seems like a systemd feature/bug thing: = https://unix.stackexchange.com/questions/612416/why-does-etc-resolv-conf-p= oint-at-127-0-0-53 >>>=20 >>> This behavior seems like some strange interaction between systemd = assumptions and freebsd?s, or something not being set up quite right on = the linux side when the vm is running freebsd.=20 >>=20 >> Could libvirt be a factor here, do you think? For example, perhaps = the >> network should be configured differently than the default when the = host >> is using systemd-resolved and/or when the guest is FreeBSD. In the = network >> XML format for libvirt (https://libvirt.org/formatnetwork.html), = there is >> a `domain` element with a `localOnly` attribute that I have seen set = by >> some virtualization projects. As far as I can tell, our setup isn't = using >> the `domain` element at all. >=20 > Having a /etc/resolv.conf entry of 127.0.0.53 is indeed something > out of the normal on a freebsd box. You need to find where that > is coming from and why that value is used. The 127.0.0.53 entry in /etc/resolv.conf is on the Linux host machine, not on the FreeBSD VM. The host is using `systemd-resolved` for managing its /etc/resolv.conf. In the VM, /etc/resolv.conf has the host IP by default, and we added 8.8.8.8 so that it wouldn't take a full 30 seconds to report a domain resolution failure. >>=20 >>>=20 >>>> On Aug 13, 2024, at 8:46 AM, Alex Arslan = wrote: >>>>=20 >>>> ? >>>> Hi Rodney, >>>>=20 >>>>> On Aug 10, 2024, at 9:11?AM, Rodney W. Grimes = wrote: >>>>>=20 >>>>>>=20 >>>>>>=20 >>>>>>> On Aug 2, 2024, at 5:58?PM, Bakul Shah = wrote: >>>>>>>=20 >>>>>>> On Aug 2, 2024, at 3:52?PM, Alex Arslan = wrote: >>>>>>>>=20 >>>>>>>>> Just a comment and a name server line: >>>>>>>>>=20 >>>>>>>>> $ cat /etc/resolv.conf >>>>>>>>> # Generated by resolvconf >>>>>>>>> nameserver 192.168.122.1 >>>>>>>>=20 >>>>>>>> I believe that is the host IP, so I guess the VM is using the = host for DNS >>>>>>>> resolution? Interestingly, if I add `nameserver 8.8.8.8` below = the line >>>>>>>> with the host IP, it takes 10 seconds rather than 30 to reach = the expected >>>>>>>> domain resolution failure. If I put 8.8.8.8 above the host IP, = the domain >>>>>>>> resolution failure is instantaneous. >>>>>>>=20 >>>>>>> What does your host use as a namesever? >>>>>>=20 >>>>>> The nameserver is 127.0.0.53. It sets options edns0 and trust-ad, = and >>>>>> includes a search entry as well. >>>>>=20 >>>>> First, is that a typo and you mean 127.0.0.1:53? >>>>=20 >>>> No, the host's /etc/resolv.conf has `nameserver 127.0.0.53`, I just = went >>>> back and rechecked to be sure. >>>>=20 >>>>> Second, is that name server locked to 127.0.0.1, or is it >>>>> actually listinging on *:53? If it is LOCKED you have no name = server >>>>> running on 192.168.122.1 to be reached by the VM, if it is NOT = locked >>>>> can the guest ping 192.168.122.1, and can it reach dns at that IP = on >>>>> port 53? Can the host send a packet BACK to the guest? >>>>=20 >>>> I apologize but I don't really know enough about these things to = know how >>>> to answer your question. I did post the output of tcpdump on the VM = and >>>> the host a while back but that was for the invalid request, so that >>>> probably doesn't capture what you're describing. >>>>=20 >>>>> Third you can "fix" the "nameserver 192.168.122.1" entry in = /etc/resolv.conf >>>>> by configuring the DHCP server that handed out the lease to the VM = to send >>>>> a namserver entry of 8.8.8.8. >>>>=20 >>>> If I understand correctly, that is indeed what we've done as a = Band-Aid fix >>>> for the time being: I added the line `prepend_nameservers=3D8.8.8.8` = to >>>> /etc/resolvconf.conf. >>>>=20 >>>>>>=20 >>>>>>>=20 >>>>>>>> Not a particularly satisfying conclusion to this saga as I = don't understand >>>>>>>> why it's happening but at least I have a workaround that should = hopefully >>>>>>>> do the job. I really appreciate everyone's help and input thus = far! >>>>>>>>=20 >>>>>>>> What's the best way to add `nameserver 8.8.8.8` to = /etc/resolv.conf as >>>>>>>> part of the VM's configuration? >>>>>>>=20 >>>>>>> You should diagnose the problem of the nameserver at = 192.168.122.1 >>>>>>> and fix it to act properly. I don't use vm (just bhyve) so can't = help >>>>>>> you with its config. >>>>>>=20 >>>>>> I do still plan to try to figure out what the actual issue is, = but I also >>>>>> now have a path forward in the meantime. :) >>>>>>=20 >>>>>>=20 >>>>>=20 >>>>> --=20 >>>>> Rod Grimes = rgrimes@freebsd.org = >>=20 >=20 > --=20 > Rod Grimes = rgrimes@freebsd.org --Apple-Mail=_92F98D39-DE38-40D3-865E-5B5E24DB9BDC Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8
On Aug 14, = 2024, at 11:29=E2=80=AFAM, Rodney W. Grimes = <freebsd-rwg@gndrsh.dnsmgr.net> wrote:


On Aug 13, 2024, at 9:15?AM, Bakul = Shah <bakul@iitbombay.org> wrote:

This weird 127. address = seems like a systemd feature/bug thing: = https://unix.stackexchange.com/questions/612416/why-does-etc-resolv-conf-p= oint-at-127-0-0-53

This behavior seems like some strange = interaction between systemd assumptions and freebsd?s, or something not = being set up quite right on the linux side when the vm is running = freebsd. 

Could = libvirt be a factor here, do you think? For example, perhaps = the
network should be configured differently than the default when = the host
is using systemd-resolved and/or when the guest is FreeBSD. = In the network
XML format for libvirt = (https://libvirt.org/formatnetwork.html), there is
a `domain` element = with a `localOnly` attribute that I have seen set by
some = virtualization projects. As far as I can tell, our setup isn't = using
the `domain` element at all.

Having a /etc/resolv.conf entry of = 127.0.0.53 is indeed something
out of = the normal on a freebsd box.  You need to find where that
is coming from and why that value is = used.

The 127.0.0.53 = entry in /etc/resolv.conf is on the Linux host machine,
not on = the FreeBSD VM. The host is using `systemd-resolved` for = managing
its /etc/resolv.conf. In the VM, /etc/resolv.conf has = the host IP by
default, and we added 8.8.8.8 so that it = wouldn't take a full 30 seconds
to report a domain resolution = failure.



On Aug 13, 2024, at 8:46 AM, Alex Arslan = <ararslan@comcast.net> wrote:

?
Hi = Rodney,

On Aug 10, 2024, at 9:11?AM, = Rodney W. Grimes <freebsd-rwg@gndrsh.dnsmgr.net> = wrote:



On Aug 2, 2024, at 5:58?PM, Bakul Shah = <bakul@iitbombay.org> wrote:

On Aug 2, 2024, at 3:52?PM, = Alex Arslan <ararslan@comcast.net> wrote:

Just a comment and a name = server line:

$ cat /etc/resolv.conf
# Generated by = resolvconf
nameserver 192.168.122.1

I believe = that is the host IP, so I guess the VM is using the host for = DNS
resolution? Interestingly, if I add `nameserver 8.8.8.8` below = the line
with the host IP, it takes 10 seconds rather than 30 to = reach the expected
domain resolution failure. If I put 8.8.8.8 above = the host IP, the domain
resolution failure is = instantaneous.

What does your host use as a = namesever?

The nameserver is 127.0.0.53. It sets = options edns0 and trust-ad, and
includes a search entry as = well.

First, is that a typo and you mean = 127.0.0.1:53?

No, the host's /etc/resolv.conf has = `nameserver 127.0.0.53`, I just went
back and rechecked to be = sure.

Second, is that name server = locked to 127.0.0.1, or is it
actually listinging on *:53?  If = it is LOCKED you have no name server
running on 192.168.122.1 to be = reached by the VM, if it is NOT locked
can the guest ping = 192.168.122.1, and can it reach dns at that IP on
port 53? =   Can the host send a packet BACK to the = guest?

I apologize but I don't really know enough = about these things to know how
to answer your question. I did post = the output of tcpdump on the VM and
the host a while back but that = was for the invalid request, so that
probably doesn't capture what = you're describing.

Third you can "fix" = the "nameserver 192.168.122.1" entry in /etc/resolv.conf
by = configuring the DHCP server that handed out the lease to the VM to = send
a namserver entry of 8.8.8.8.

If I = understand correctly, that is indeed what we've done as a Band-Aid = fix
for the time being: I added the line = `prepend_nameservers=3D8.8.8.8` = to
/etc/resolvconf.conf.



Not a particularly satisfying conclusion to this saga as I = don't understand
why it's happening but at least I have a workaround = that should hopefully
do the job. I really appreciate everyone's help = and input thus far!

What's the best way to add `nameserver = 8.8.8.8` to /etc/resolv.conf as
part of the VM's = configuration?

You should diagnose the problem of = the nameserver at 192.168.122.1
and fix it to act properly. I don't = use vm (just bhyve) so can't help
you with its = config.

I do still plan to try to figure out what = the actual issue is, but I also
now have a path forward in the = meantime. :)



-- 
Rod Grimes =             &n= bsp;           &nbs= p;            =            rgrimes@freebsd.org<mailto:rgrimes@freebsd.org>
=


-- 
Rod Grimes =             &n= bsp;           &nbs= p;            =            <= a href=3D"mailto:rgrimes@freebsd.org" style=3D"font-family: Helvetica; = font-size: 12px; font-style: normal; font-variant-caps: normal; = font-weight: 400; letter-spacing: normal; orphans: auto; text-align: = start; text-indent: 0px; text-transform: none; white-space: normal; = widows: auto; word-spacing: 0px; -webkit-text-stroke-width: = 0px;">rgrimes@freebsd.org

= --Apple-Mail=_92F98D39-DE38-40D3-865E-5B5E24DB9BDC--