Hyper-V networking: problem after upgrade to 10.2

Sun Jan 31 12:24:46 UTC 2016

I had this issue with a Windows Server 2012 R2 Hyper-V host and FreeBSD
10.1 sometime last year, but chalked it up to a random fault during
upgrade. It wasn't a production VM so I didn't put much effort into
debugging it at the time.

I remember I was using SR-IOV on the virtual network switch, and had it
enabled on the VM as well. I had also enabled DHCP guard. Maybe some of
these settings are relevant to reproducing the bug?

Regards,
Marie Helene Kvello-Aune

On Sun, Jan 31, 2016 at 1:15 PM Dexuan Cui <decui at microsoft.com> wrote:

> Hi Jac,
> Good to know the information.
>
> Since I can't repro the issue, it's difficult for me to debug it. :(
> I'm guessing if it would help if you use a permanent ARP entry in the VM
> ("arp -s hostname ether_addr") for the other end - surely this is only for
> debug purpose.
>
> During the VM boot-up, can you keep pinging the VM from the other host. I
> mean: it looks the NIC never works since it becomes UP in the VM?
>
> BTW, I'm not sure if it's easy for you to do the same test as mine, i.e.,
> do a fresh installation of 10.1 VM and upgrade it to 10.2.
>
> @Sephe, any idea?
>
> Thanks,
> -- Dexuan
>
> From: Jac Backus [mailto:j.backus at bugworks.com]
> Sent: Sunday, January 31, 2016 3:42
> To: Dexuan Cui <decui at microsoft.com<mailto:decui at microsoft.com>>; Sephe
> Qiao (Wicresoft) <v-yanqia at microsoft.com<mailto:v-yanqia at microsoft.com>>;
> Kylie Liang <kyliel at microsoft.com<mailto:kyliel at microsoft.com>>; '
> freebsd-virtualization at freebsd.org' <freebsd-virtualization at freebsd.org
> <mailto:freebsd-virtualization at freebsd.org>>; BSD Integration Components
> for Hyper-V <bsdic at microsoft.com<mailto:bsdic at microsoft.com>>
> Subject: RE: Hyper-V networking: problem after upgrade to 10.2
>
> Good day Dexuan,
>
> There is something wrong with getting mac addresses for host on the lan,
> it seems.
>
> When I ping the 10.2 server from a host on the net, I see on that host
> arp requests  (Wireshark: who has ... Tell ...) for the 10.2 server.
>
> Arp -a on the 10.2 server itself says for the non-server entries
>  ? <address> at (imcomplete) on hn0 expired [ethernet]
>
> Tcpdump on the 10.2 server only shows arp requests:
> ARP, Request who-has ... tell ...
>
> Does this help?
>
> With kind regards,
>
> Jac
> Van: Dexuan Cui [mailto:decui at microsoft.com]
> Verzonden: vrijdag 29 januari 2016 9:59
> Aan: Jac Backus; Sephe Qiao (Wicresoft); Kylie Liang; '
> freebsd-virtualization at freebsd.org'; BSD Integration Components for
> Hyper-V
> Onderwerp: RE: Hyper-V networking: problem after upgrade to 10.2
>
> Hmm, it's strange we can't repro.
>
> I suppose you can't ping the netgate VM (or machine) 's IP address either?
> When this happens, can you check the arp table in both sides?
> Can you please run tcpdump in the VM and in the gateway to diagnose the
> issue?
>
> Thanks,
> -- Dexuan
>
> From: Jac Backus [mailto:j.backus at bugworks.com]
> Sent: Friday, January 29, 2016 15:36
> To: Dexuan Cui <decui at microsoft.com<mailto:decui at microsoft.com>>; Sephe
> Qiao (Wicresoft) <v-yanqia at microsoft.com<mailto:v-yanqia at microsoft.com>>;
> Kylie Liang <kyliel at microsoft.com<mailto:kyliel at microsoft.com>>; '
> freebsd-virtualization at freebsd.org' <freebsd-virtualization at freebsd.org
> <mailto:freebsd-virtualization at freebsd.org>>; BSD Integration Components
> for Hyper-V <bsdic at microsoft.com<mailto:bsdic at microsoft.com>>
> Subject: RE: Hyper-V networking: problem after upgrade to 10.2
>
> Hello Dexuan,
>
> That remarkable.
> My uname -a is indentical.
>
> I can ping local interfaces. Ping to other addresses in local subnet
> gives: Ping: sendto: Host is down. Ping to other addresses gives: Ping:
> sendto: No route to host.
> Routing tables (netstat -rn) for both versions look the same.
>
> Is there something I can test?
>
> With kind regards,
>
> Jac
>
> Van: Dexuan Cui [mailto:decui at microsoft.com]
> Verzonden: vrijdag 29 januari 2016 4:25
> Aan: Jac Backus; Sephe Qiao (Wicresoft); Kylie Liang; '
> freebsd-virtualization at freebsd.org'; BSD Integration Components for
> Hyper-V
> Onderwerp: RE: Hyper-V networking: problem after upgrade to 10.2
>
>
> Hi Jac,
>
> I installed a 10.1 VM with FreeBSD-10.1-RELEASE-amd64-dvd1.iso and
> upgraded it to 10.2 by running "freebsd-update upgrade -r 10.2-RELEASE".
> Everything worked just fine. With the new kernel (see the below), ssh and
> scp still works fine for me.
>
> # uname -a
> FreeBSD bsd101 10.2-RELEASE-p9 FreeBSD 10.2-RELEASE-p9 #0: Thu Jan 14
> 01:32:46 UTC 2016     root at amd64-builder.daemonology.net:
> /usr/obj/usr/src/sys/GENERIC<mailto:root at amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC>
> amd64
>
> What's the specific symptom for "networking does not work anymore" in your
> side(upgrading from 10.1 to 10.2)?
>
> Thanks,
> -- Dexuan
>
> From: Jac Backus [mailto:j.backus at bugworks.com]
> Sent: Wednesday, January 27, 2016 17:35
> To: Dexuan Cui <decui at microsoft.com<mailto:decui at microsoft.com>>; Sephe
> Qiao (Wicresoft) <v-yanqia at microsoft.com<mailto:v-yanqia at microsoft.com>>;
> Kylie Liang <kyliel at microsoft.com<mailto:kyliel at microsoft.com>>; '
> freebsd-virtualization at freebsd.org' <freebsd-virtualization at freebsd.org
> <mailto:freebsd-virtualization at freebsd.org>>; BSD Integration Components
> for Hyper-V <bsdic at microsoft.com<mailto:bsdic at microsoft.com>>
> Subject: RE: Hyper-V networking: problem after upgrade to 10.2
>
> Hello Dexuan,
>
> Unfortunetely, no OACTIVE flag:
>
> hn0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
>         options=31b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,TSO4,TSO6>
>
> With kind regards,
>
> Jac
>
> Van: Dexuan Cui [mailto:decui at microsoft.com]
> Verzonden: woensdag 27 januari 2016 4:09
> Aan: Sephe Qiao (Wicresoft); Jac Backus; Kylie Liang; '
> freebsd-virtualization at freebsd.org'; BSD Integration Components for
> Hyper-V
> Onderwerp: RE: Hyper-V networking: problem after upgrade to 10.2
>
> Hi Jac,
> Please show 'ifconfig -a' when the issue happens (when you upgrade 10.1
> from 10.2).
> We suspect it may be a known OACTIVE issue and "ifconfig -a' can confirm
> this, the output has the string "OACTIVE".
>
> It looks somehow the issue doesn't happen when we use a 10.2 fresh
> installation.
>
> Thanks,
> -- Dexuan
>
> From: Sephe Qiao (Wicresoft)
> Sent: Wednesday, January 27, 2016 9:13
> To: Jac Backus <j.backus at bugworks.com<mailto:j.backus at bugworks.com>>;
> Dexuan Cui <decui at microsoft.com<mailto:decui at microsoft.com>>; Kylie Liang
> <kyliel at microsoft.com<mailto:kyliel at microsoft.com>>; '
> freebsd-virtualization at freebsd.org' <freebsd-virtualization at freebsd.org
> <mailto:freebsd-virtualization at freebsd.org>>; BSD Integration Components
> for Hyper-V <bsdic at microsoft.com<mailto:bsdic at microsoft.com>>
> Subject: RE: Hyper-V networking: problem after upgrade to 10.2
>
> Oh, please ignore this, I think its solved :)
>
> From: Sephe Qiao (Wicresoft)
> Sent: Wednesday, January 27, 2016 9:10 AM
> To: Jac Backus <j.backus at bugworks.com<mailto:j.backus at bugworks.com>>;
> Dexuan Cui <decui at microsoft.com<mailto:decui at microsoft.com>>; Kylie Liang
> <kyliel at microsoft.com<mailto:kyliel at microsoft.com>>; '
> freebsd-virtualization at freebsd.org' <freebsd-virtualization at freebsd.org
> <mailto:freebsd-virtualization at freebsd.org>>; BSD Integration Components
> for Hyper-V <bsdic at microsoft.com<mailto:bsdic at microsoft.com>>
> Subject: RE: Hyper-V networking: problem after upgrade to 10.2
>
> Hi Jac,
>
> What's the output of 'ifconfig -a' when this happened?
>
> Thanks,
> sephe
>
> From: Jac Backus [mailto:j.backus at bugworks.com]
> Sent: Tuesday, January 26, 2016 6:37 PM
> To: Dexuan Cui <decui at microsoft.com<mailto:decui at microsoft.com>>; Kylie
> Liang <kyliel at microsoft.com<mailto:kyliel at microsoft.com>>; '
> freebsd-virtualization at freebsd.org' <freebsd-virtualization at freebsd.org
> <mailto:freebsd-virtualization at freebsd.org>>; BSD Integration Components
> for Hyper-V <bsdic at microsoft.com<mailto:bsdic at microsoft.com>>
> Subject: RE: Hyper-V networking: problem after upgrade to 10.2
>
>
> Hello Dexuan,
>
>
>
> It seems, it is not completely correct, although the effect is as if it is
> not working.
>
> Systat -ifstat 1 shows this:
>
> [cid:image001.png at 01D159DA.65A3A0E0]
>
>
>
> So something is happening. But I can not reach anything. And the server
> can not be reached from the lan (hn0) or internet (hn1 and hn2).
>
>
>
> I get a firewall message in /var/log/messages (first message from
> 11:18:55):
>
> [cid:image002.png at 01D159DA.65A3A0E0]
>
> But this is just caused by the problem?
>
>
>
> If I can help with further information, please let me know.
>
>
>
> Regarding Bug 187006, all interfaces have fixed addresses.
>
>
>
> With kind regards,
>
>
>
> Jac
>
>
>
>
>
>
>
> -----Oorspronkelijk bericht-----
> Van: Dexuan Cui [mailto:decui at microsoft.com]
> Verzonden: dinsdag 26 januari 2016 7:55
> Aan: Kylie Liang; Jac Backus; 'freebsd-virtualization at freebsd.org'; BSD
> Integration Components for Hyper-V
> Onderwerp: RE: Hyper-V networking: problem after upgrade to 10.2
>
>
>
> Hi Jac,
>
> BTW,  what do you mean by saying "networking does not work anymore" -- can
> you please check if your issue is the same as
>
>
>
> Bug 187006 - [hyper-v] dynamic address (dhcp) obtaining doesn't work on
> HYPER-V OS 2012 R2
>
> https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=187006<
> https://na01.safelinks.protection.outlook.com/?url=https%3a%2f%2fbugs.freebsd.org%2fbugzilla%2fshow_bug.cgi%3fid%3d187006&data=01%7c01%7cv-yanqia%40064d.mgd.microsoft.com%7cef82474449e745da88c908d3263e7e48%7c72f988bf86f141af91ab2d7cd011db47%7c1&sdata=1IyQb3x7ecm%2f6uESQmVAAyAyOQr4ZMJ3Fkawp93dZgQ%3d>
> ?
>
>
>
> Thanks,
>
> -- Dexuan
>
>
>
>
>
> > -----Original Message-----
>
> > From: Dexuan Cui
>
> > Sent: Tuesday, January 26, 2016 14:49
>
> > To: Kylie Liang <kyliel at microsoft.com<mailto:kyliel at microsoft.com>>;
> Jac Backus
>
> > <j.backus at bugworks.com<mailto:j.backus at bugworks.com>>; '
> freebsd-virtualization at freebsd.org'
>
> > <freebsd-virtualization at freebsd.org<mailto:
> freebsd-virtualization at freebsd.org>>; BSD Integration Components for
>
> > Hyper-V <bsdic at microsoft.com<mailto:bsdic at microsoft.com>>
>
> > Subject: RE: Hyper-V networking: problem after upgrade to 10.2
>
> >
>
> > Hi Jac,
>
> > Kylie meant disabling TSO. Please try this ("ifconfig hn0 -tso").
>
> >
>
> > The message " hn0: unknown status 1073872902 received" should be an
>
> > unnecessary warning only.
>
> > My 10.2 VM can work fine even if I see the message too.
>
> >
>
> > Can you please install a 10.2 VM from the 10.2 .ISO file directly as I
>
> > did and see if it works for you?
>
> >
>
> > I guess we never tried upgrading 10.1 from 10.2.
>
> > Can you please list the steps how you did the upgrading? We'll try the
>
> > same steps.
>
> >
>
> > Thanks,
>
> > -- Dexuan
>
> >
>
> > > -----Original Message-----
>
> > > From: Kylie Liang
>
> > > Sent: Tuesday, January 26, 2016 8:01
>
> > > To: Jac Backus <j.backus at bugworks.com<mailto:j.backus at bugworks.com>>;
> 'freebsd-
>
> > virtualization at freebsd.org<mailto:virtualization at freebsd.org>'
>
> > > <freebsd-virtualization at freebsd.org<mailto:
> freebsd-virtualization at freebsd.org>>; BSD Integration Components for
>
> > > Hyper-
>
> > V
>
> > > <bsdic at microsoft.com<mailto:bsdic at microsoft.com>>
>
> > > Subject: RE: Hyper-V networking: problem after upgrade to 10.2
>
> > >
>
> > > Hi Jac,
>
> > >
>
> > > Thank you for asking. To isolate your issue, could you please try
>
> > > disabling SO
>
> > on
>
> > > your 10.2 system first? Thank you.
>
> > >
>
> > > And I would like to confirm with you
>
> > > 1) You met issue for 10.2 kernel + 10.2 system
>
> > > 2) No issue for 10.1 kernel + 10.1 system
>
> > > 3) No issue for 10.1 kernel + 10.2 system
>
> > >
>
> > > Right? And add our engineers in the list.
>
> > >
>
> > > Thanks,
>
> > > Kylie Liang
>
> > >
>
> > > -----Original Message-----
>
> > > From: owner-freebsd-virtualization at freebsd.org<mailto:
> owner-freebsd-virtualization at freebsd.org>
>
> > > [mailto:owner-freebsd- virtualization at freebsd.org<mailto:
> virtualization at freebsd.org>] On Behalf Of Jac
>
> > > Backus
>
> > > Sent: Tuesday, January 26, 2016 5:56 AM
>
> > > To: 'freebsd-virtualization at freebsd.org'
>
> > > <freebsd-virtualization at freebsd.org<mailto:
> freebsd-virtualization at freebsd.org>>
>
> > > Subject: Hyper-V networking: problem after upgrade to 10.2
>
> > >
>
> > > Dear reader,
>
> > >
>
> > > Today, I did  upgrade  FreeBSD 10.1 to 10.2 running on Hyper-V on a
>
> > > full
>
> > patched
>
> > > Windows Server 2012 R2 x64 version.
>
> > >
>
> > > After the update, networking does not work anymore.
>
> > >
>
> > > In /var/log/messages is this:
>
> > >
>
> > > Jan 25 21:02:01 mercurius kernel: hn0: <Synthetic Network Interface>
>
> > > on
>
> > > vmbus0 Jan 25 21:02:01 mercurius kernel: hn0: unknown status
>
> > > 1073872902 received Jan 25 21:02:01 mercurius kernel: hn0: unknown
>
> > > status 1073872902 received Jan 25 21:02:01 mercurius kernel: hn0: hv
>
> > > send offload request succeeded Jan 25 21:02:01 mercurius kernel: hn0:
> Using defaults for TSO:
>
> > > 65518/35/2048 Jan 25 21:02:01 mercurius kernel: hn0: Ethernet address:
>
> > > 00:15:5d:ac:11:08 Jan 25 21:02:01 mercurius kernel: hn1: <Synthetic
>
> > > Network
>
> > > Interface> on vmbus0 Jan 25 21:02:01 mercurius kernel: hn1: unknown
>
> > > Interface> status
>
> > > 1073872902 received Jan 25 21:02:01 mercurius kernel: hn1: unknown
>
> > > status
>
> > > 1073872902 received Jan 25 21:02:01 mercurius kernel: hn1: hv send
>
> > > offload request succeeded Jan 25 21:02:01 mercurius kernel: hn1:
>
> > > Using defaults for
>
> > TSO:
>
> > > 65518/35/2048 Jan 25 21:02:01 mercurius kernel: hn1: Ethernet address:
>
> > > 00:15:5d:ac:11:09 Jan 25 21:02:01 mercurius kernel: hn2: <Synthetic
>
> > > Network
>
> > > Interface> on vmbus0 Jan 25 21:02:01 mercurius kernel: hn2: unknown
>
> > > Interface> status
>
> > > 1073872902 received Jan 25 21:02:01 mercurius kernel: hn2: unknown
>
> > > status
>
> > > 1073872902 received Jan 25 21:02:01 mercurius kernel: hn2: hv send
>
> > > offload request succeeded Jan 25 21:02:01 mercurius kernel: hn2:
>
> > > Using defaults for
>
> > TSO:
>
> > > 65518/35/2048 Jan 25 21:02:01 mercurius kernel: hn2: Ethernet address:
>
> > > 00:15:5d:ac:11:07
>
> > >
>
> > > It worked fine with the 10.1 kernel, and when I boot this kernel, it
> works again:
>
> > >
>
> > > Jan 25 22:20:02 mercurius kernel: hn0: <Synthetic Network Interface>
>
> > > on
>
> > > vmbus0 Jan 25 22:20:02 mercurius kernel: hn0: Ethernet address:
>
> > > 00:15:5d:ac:11:07 Jan 25 22:20:02 mercurius kernel: hn1: <Synthetic
>
> > > Network
>
> > > Interface> on vmbus0 Jan 25 22:20:02 mercurius kernel: hn1: Ethernet
> address:
>
> > > 00:15:5d:ac:11:08 Jan 25 22:20:02 mercurius kernel: hn2: <Synthetic
>
> > > Network
>
> > > Interface> on vmbus0 Jan 25 22:20:02 mercurius kernel: hn2: Ethernet
> address:
>
> > > 00:15:5d:ac:11:09
>
> > >
>
> > > So I am running  a 10.2 system on a 10.1 kernel at the moment.
>
> > >
>
> > > I found nothing in /usr/src/UPDATING and not really anything on the
> net.
>
> > >
>
> > > So, could you tell why does this happen, and how can I solve this?
>
> > >
>
> > > Thanks for the help!
>
> > >
>
> > > With kind regards,
>
> > >
>
> > > Jac Backus
>
> > >
>
> > >
>
> > >
>
> > > _______________________________________________
>
> > > freebsd-virtualization at freebsd.org<mailto:
> freebsd-virtualization at freebsd.org> mailing list
>
> > >
>
> > https://na01.safelinks.protection.outlook.com/?url=https%3a%2f%2flists
>
> > .freebs
>
> > > d.org%2fmailman%2flistinfo%2ffreebsd-
>
> > >
>
> > virtualization&data=01%7c01%7ckyliel%40064d.mgd.microsoft.com%7cc9ca2e
>
> > >
>
> > 0d0fef482b553f08d325d3aefb%7c72f988bf86f141af91ab2d7cd011db47%7c1&s
>
> > > data=o%2bMZGuBW0frrQhjAPkhrWlLgNEH8LJ7BiLUyiO4tvR0%3d
>
> > > To unsubscribe, send any mail to "freebsd-virtualization-
>
> > > unsubscribe at freebsd.org<mailto:unsubscribe at freebsd.org>"
> _______________________________________________
> freebsd-virtualization at freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization
> To unsubscribe, send any mail to "
> freebsd-virtualization-unsubscribe at freebsd.org"
>