From nobody Wed Aug 14 16:15:22 2024 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4WkYK74ygdz5SPtf for ; Wed, 14 Aug 2024 16:16:11 +0000 (UTC) (envelope-from ararslan@comcast.net) Received: from resqmta-a2p-658764.sys.comcast.net (resqmta-a2p-658764.sys.comcast.net [IPv6:2001:558:fd01:2bb4::6]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4WkYK62G4cz536h for ; Wed, 14 Aug 2024 16:16:10 +0000 (UTC) (envelope-from ararslan@comcast.net) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=comcast.net header.s=20190202a header.b=MIuZbluw; dmarc=pass (policy=quarantine) header.from=comcast.net; spf=pass (mx1.freebsd.org: domain of ararslan@comcast.net designates 2001:558:fd01:2bb4::6 as permitted sender) smtp.mailfrom=ararslan@comcast.net Received: from resomta-a2p-646771.sys.comcast.net ([96.103.145.239]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 256/256 bits) (Client did not present a certificate) by resqmta-a2p-658764.sys.comcast.net with ESMTPS id eE5EsyXnnf5MeeGfFsmi0X; Wed, 14 Aug 2024 16:15:57 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=comcast.net; s=20190202a; t=1723652157; bh=89jqD3ADTrKc+xVbI1nsHbxlpUvtN5eI3ZxMRZwBANs=; h=Received:Received:From:Message-Id:Content-Type:Mime-Version: Subject:Date:To:Xfinity-Spam-Result; b=MIuZbluwA0WJiqwdII/Un90fSCCCjRq7Ufh/IdinUZuOA+VU95U1bt1x742FO5JXO TFLexAxiI55WZVTarnbsGCNZhAfft03Ilo6t+qujWyObsqL/6OoMBPfNx+sYPVblj4 4hCbKOS2mTVPB245jtLJF5EGY+5aDBEau6CHBV3ive6K011M2ni+qT86bg5NVx2HMi P41ST2DGqbted/TCUlqfJmbVvJ0XiUEHO8a/nAw4hku8qkqXe+U2OdRYle+pcEtCvx 3CqW/wgX1YpjyBoDqm3rf1VBmwYXSjXm2FJplkQY8KpU7+IWYTJ+FXfoFXM6emRlYt d8u1EOBuO6+2Q== Received: from smtpclient.apple ([67.160.29.205]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 256/256 bits) (Client did not present a certificate) by resomta-a2p-646771.sys.comcast.net with ESMTPSA id eGeqscEdAgZaxeGessbI3S; Wed, 14 Aug 2024 16:15:35 +0000 From: Alex Arslan Message-Id: Content-Type: multipart/alternative; boundary="Apple-Mail=_69A9E170-4852-4398-9205-07FFF645B079" List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@FreeBSD.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3774.600.62\)) Subject: Re: Diagnosing virtual machine network issues Date: Wed, 14 Aug 2024 09:15:22 -0700 In-Reply-To: Cc: "Rodney W. Grimes" , FreeBSD Hackers To: Bakul Shah References: <607068B0-E531-4D7F-8B61-923EE5DC443D@comcast.net> X-Mailer: Apple Mail (2.3774.600.62) X-CMAE-Envelope: MS4xfCH81BFMTMXI7RE+lrNthbtYTK9lfcH8jyHPMXipHNVQU8dVDOZiupSqCHMfclob+To1C2PXt/QAE0qSgqlKEpOaZRE2UoAJCiTkGsS7exxO/42yKNim OVe0Pc9nAp6SCTlXaxtb1ozUL5IZUnsHfQfrFmyI+EsD9CignyNO5QhLnuD44LksG/j9rqeiUJpZ1vxya/gfqrj4vWsIZCfW4YtFNUuC/l0F+U1zZW+X/5Eq DKUYLWxKlKSj0Lo/lMehj8ZEO8lPcZrqsTfaf2Vyq70= X-Spamd-Bar: - X-Spamd-Result: default: False [-1.20 / 15.00]; HFILTER_HELO_5(3.00)[resqmta-a2p-658764.sys.comcast.net]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; DMARC_POLICY_ALLOW(-0.50)[comcast.net,quarantine]; R_SPF_ALLOW(-0.20)[+ip6:2001:558:fd01:2bb4::/64]; RCVD_IN_DNSWL_LOW(-0.20)[2001:558:fd01:2bb4::6:from,96.103.145.239:received]; R_DKIM_ALLOW(-0.20)[comcast.net:s=20190202a]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; DKIM_TRACE(0.00)[comcast.net:+]; TO_DN_ALL(0.00)[]; RCVD_TLS_ALL(0.00)[]; FREEMAIL_FROM(0.00)[comcast.net]; MIME_TRACE(0.00)[0:+,1:+,2:~]; ARC_NA(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; FROM_HAS_DN(0.00)[]; ASN(0.00)[asn:7922, ipnet:2001:558::/29, country:US]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[]; FREEMAIL_ENVFROM(0.00)[comcast.net]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; APPLE_MAILER_COMMON(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; DWL_DNSWL_NONE(0.00)[comcast.net:dkim] X-Rspamd-Queue-Id: 4WkYK62G4cz536h --Apple-Mail=_69A9E170-4852-4398-9205-07FFF645B079 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 > On Aug 13, 2024, at 9:15=E2=80=AFAM, Bakul Shah = wrote: >=20 > This weird 127. address seems like a systemd feature/bug thing: = https://unix.stackexchange.com/questions/612416/why-does-etc-resolv-conf-p= oint-at-127-0-0-53 >=20 > This behavior seems like some strange interaction between systemd = assumptions and freebsd=E2=80=99s, or something not being set up quite = right on the linux side when the vm is running freebsd.=20 Could libvirt be a factor here, do you think? For example, perhaps the network should be configured differently than the default when the host is using systemd-resolved and/or when the guest is FreeBSD. In the = network XML format for libvirt (https://libvirt.org/formatnetwork.html), there = is a `domain` element with a `localOnly` attribute that I have seen set by some virtualization projects. As far as I can tell, our setup isn't = using the `domain` element at all. >=20 >> On Aug 13, 2024, at 8:46 AM, Alex Arslan = wrote: >>=20 >> =EF=BB=BF >> Hi Rodney, >>=20 >>> On Aug 10, 2024, at 9:11=E2=80=AFAM, Rodney W. Grimes = wrote: >>>=20 >>>>=20 >>>>=20 >>>>> On Aug 2, 2024, at 5:58?PM, Bakul Shah = wrote: >>>>>=20 >>>>> On Aug 2, 2024, at 3:52?PM, Alex Arslan = wrote: >>>>>>=20 >>>>>>> Just a comment and a name server line: >>>>>>>=20 >>>>>>> $ cat /etc/resolv.conf >>>>>>> # Generated by resolvconf >>>>>>> nameserver 192.168.122.1 >>>>>>=20 >>>>>> I believe that is the host IP, so I guess the VM is using the = host for DNS >>>>>> resolution? Interestingly, if I add `nameserver 8.8.8.8` below = the line >>>>>> with the host IP, it takes 10 seconds rather than 30 to reach the = expected >>>>>> domain resolution failure. If I put 8.8.8.8 above the host IP, = the domain >>>>>> resolution failure is instantaneous. >>>>>=20 >>>>> What does your host use as a namesever? >>>>=20 >>>> The nameserver is 127.0.0.53. It sets options edns0 and trust-ad, = and >>>> includes a search entry as well. >>>=20 >>> First, is that a typo and you mean 127.0.0.1:53? >>=20 >> No, the host's /etc/resolv.conf has `nameserver 127.0.0.53`, I just = went >> back and rechecked to be sure. >>=20 >>> Second, is that name server locked to 127.0.0.1, or is it >>> actually listinging on *:53? If it is LOCKED you have no name = server >>> running on 192.168.122.1 to be reached by the VM, if it is NOT = locked >>> can the guest ping 192.168.122.1, and can it reach dns at that IP on >>> port 53? Can the host send a packet BACK to the guest? >>=20 >> I apologize but I don't really know enough about these things to know = how >> to answer your question. I did post the output of tcpdump on the VM = and >> the host a while back but that was for the invalid request, so that >> probably doesn't capture what you're describing. >>=20 >>> Third you can "fix" the "nameserver 192.168.122.1" entry in = /etc/resolv.conf >>> by configuring the DHCP server that handed out the lease to the VM = to send >>> a namserver entry of 8.8.8.8. >>=20 >> If I understand correctly, that is indeed what we've done as a = Band-Aid fix >> for the time being: I added the line `prepend_nameservers=3D8.8.8.8` = to >> /etc/resolvconf.conf. >>=20 >>>>=20 >>>>>=20 >>>>>> Not a particularly satisfying conclusion to this saga as I don't = understand >>>>>> why it's happening but at least I have a workaround that should = hopefully >>>>>> do the job. I really appreciate everyone's help and input thus = far! >>>>>>=20 >>>>>> What's the best way to add `nameserver 8.8.8.8` to = /etc/resolv.conf as >>>>>> part of the VM's configuration? >>>>>=20 >>>>> You should diagnose the problem of the nameserver at 192.168.122.1 >>>>> and fix it to act properly. I don't use vm (just bhyve) so can't = help >>>>> you with its config. >>>>=20 >>>> I do still plan to try to figure out what the actual issue is, but = I also >>>> now have a path forward in the meantime. :) >>>>=20 >>>>=20 >>>=20 >>> --=20 >>> Rod Grimes = rgrimes@freebsd.org --Apple-Mail=_69A9E170-4852-4398-9205-07FFF645B079 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8
On Aug 13, = 2024, at 9:15=E2=80=AFAM, Bakul Shah <bakul@iitbombay.org> = wrote:

This weird 127. address seems like a systemd = feature/bug thing: https://unix.stackexchange.com/questions/61241= 6/why-does-etc-resolv-conf-point-at-127-0-0-53

This = behavior seems like some strange interaction between systemd assumptions = and freebsd=E2=80=99s, or something not being set up quite right on the = linux side when the vm is running = freebsd. 

Could libvirt be a factor here, do you think? For example, perhaps = the
network should be configured differently than the default = when the host
is using systemd-resolved and/or when the guest = is FreeBSD. In the network
XML format for libvirt = (https://libvirt.org/formatnetwork.html), there is
a `domain` = element with a `localOnly` attribute that I have seen set = by
some virtualization projects. As far as I can tell, our = setup isn't using
the `domain` element at = all.


On Aug 13, 2024, at 8:46 AM, Alex = Arslan <ararslan@comcast.net> = wrote:

=EF=BB=BF
Hi Rodney,

On Aug 10, 2024, at 9:11=E2=80=AFAM, Rodney W. Grimes = <freebsd-rwg@gndrsh.dnsmgr.net> wrote:



On Aug 2, = 2024, at 5:58?PM, Bakul Shah <bakul@iitbombay.org> = wrote:

On Aug 2, 2024, at 3:52?PM, Alex Arslan = <ararslan@comcast.net> wrote:

Just a comment and a name = server line:

$ cat /etc/resolv.conf
# Generated by = resolvconf
nameserver 192.168.122.1

I believe = that is the host IP, so I guess the VM is using the host for = DNS
resolution? Interestingly, if I add `nameserver 8.8.8.8` below = the line
with the host IP, it takes 10 seconds rather than 30 to = reach the expected
domain resolution failure. If I put 8.8.8.8 above = the host IP, the domain
resolution failure is = instantaneous.

What does your host use as a = namesever?

The nameserver is 127.0.0.53. It sets = options edns0 and trust-ad, and
includes a search entry as = well.

First, = is that a typo and you mean 127.0.0.1:53?

No, = the host's /etc/resolv.conf has `nameserver 127.0.0.53`, I just = went
back and rechecked to be sure.

Second, = is that name server locked to 127.0.0.1, or is it
actually listinging on *:53?  If it is = LOCKED you have no name server
running = on 192.168.122.1 to be reached by the VM, if it is NOT locked
can the guest ping 192.168.122.1, and can = it reach dns at that IP on
port = 53?   Can the host send a packet BACK to the guest?

I apologize but I = don't really know enough about these things to know how
to = answer your question. I did post the output of tcpdump on the VM = and
the host a while back but that was for the invalid = request, so that
probably doesn't capture what you're = describing.

Third you can "fix" the "nameserver = 192.168.122.1" entry in /etc/resolv.conf
by = configuring the DHCP server that handed out the lease to the VM to = send
a namserver entry of = 8.8.8.8.

If I understand correctly, = that is indeed what we've done as a Band-Aid fix
for the time = being: I added the line `prepend_nameservers=3D8.8.8.8` = to
/etc/resolvconf.conf.



Not a particularly satisfying conclusion to this saga as I = don't understand
why it's happening but at least I have a workaround = that should hopefully
do the job. I really appreciate everyone's help = and input thus far!

What's the best way to add `nameserver = 8.8.8.8` to /etc/resolv.conf as
part of the VM's = configuration?

You should diagnose the problem of = the nameserver at 192.168.122.1
and fix it to act properly. I don't = use vm (just bhyve) so can't help
you with its = config.

I do still plan to try to figure out what = the actual issue is, but I also
now have a path forward in the = meantime. :)



-- 
Rod Grimes =             &n= bsp;           &nbs= p;            =            <= a href=3D"mailto:rgrimes@freebsd.org" style=3D"font-family: Helvetica; = font-size: 12px; font-style: normal; font-variant-caps: normal; = font-weight: 400; letter-spacing: normal; orphans: auto; text-align: = start; text-indent: 0px; text-transform: none; white-space: normal; = widows: auto; word-spacing: 0px; -webkit-text-stroke-width: = 0px;">rgrimes@freebsd.org


= --Apple-Mail=_69A9E170-4852-4398-9205-07FFF645B079--