Re: Suddenly unable to access VMs
- In reply to: Odhiambo Washington : "Re: Suddenly unable to access VMs"
- Go to: [ bottom of page ] [ top of archives ] [ this month ]
Date: Thu, 18 Jul 2024 16:07:35 UTC
On Thu, Jul 11, 2024 at 6:23 PM Odhiambo Washington <odhiambo@gmail.com>
wrote:
>
>
> On Thu, Jul 11, 2024 at 5:49 PM Rodney W. Grimes <
> freebsd-rwg@gndrsh.dnsmgr.net> wrote:
>
>> > My bhyve VMs have been all fine until now.
>> > I can't ping them and can't SSH into them. However, I can connect to
>> them
>> > with VNCViewer from a remote host (my PC from my house) :-(
>> >
>> > I haven't done any changes on the host at all.
>> > dnsmasq is running, but seems like the VMs aren't getting the IPs for
>> some
>> > reason.
>> >
>> > ```
>> > cloned_interfaces="bridge0 tap0 tap1 tap2 tap3 tap4 tap5"
>> > ifconfig_bridge0_name="vmbridge"
>> > ifconfig_vmbridge="addm em1 addm tap0 addm tap1 addm tap2 addm tap3 addm
>> > tap4 addm tap5 up"
>> > ifconfig_vmbridge_alias0="inet 172.16.0.1 netmask 255.255.255.0"
>> > ```
>> > What might have happened?
>> >
>> >
>> > root@gw:/home/wash # ifconfig vmbridge
>> > vmbridge: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP>
>> > metric 0 mtu 1500
>> > options=0
>> > ether 58:9c:fc:10:df:1d
>> > inet 172.16.0.1 netmask 0xffffff00 broadcast 172.16.0.255
>> > id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
>> > maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
>> > root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
>> > member: tap5 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
>> > ifmaxaddr 0 port 10 priority 128 path cost 2000000
>> > member: tap4 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
>> > ifmaxaddr 0 port 9 priority 128 path cost 2000000
>> > member: tap3 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
>> > ifmaxaddr 0 port 8 priority 128 path cost 2000000
>> > member: tap2 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
>> > ifmaxaddr 0 port 7 priority 128 path cost 2000000
>> > member: tap1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
>> > ifmaxaddr 0 port 6 priority 128 path cost 2000000
>> > member: tap0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
>> > ifmaxaddr 0 port 5 priority 128 path cost 2000000
>> > member: em1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
>> > ifmaxaddr 0 port 2 priority 128 path cost 55
>> > groups: bridge
>> > nd6 options=9<PERFORMNUD,IFDISABLED>
>> > root@gw:/home/wash # ssh 172.16.0.99
>> > ssh: connect to host 172.16.0.99 port 22: Permission denied
>> > root@gw:/home/wash # ssh 172.16.0.100
>> > ssh: connect to host 172.16.0.100 port 22: Permission denied
>> > root@gw:/home/wash # ping 172.16.0.100
>> > PING 172.16.0.100 (172.16.0.100): 56 data bytes
>> > ping: sendto: Permission denied
>> > ping: sendto: Permission denied
>> > ping: sendto: Permission denied
>> > ping: sendto: Permission denied
>> > ^C
>> > --- 172.16.0.100 ping statistics ---
>> > 4 packets transmitted, 0 packets received, 100.0% packet loss
>> > root@gw:/home/wash # ping 172.16.0.99
>> > PING 172.16.0.99 (172.16.0.99): 56 data bytes
>> > ping: sendto: Permission denied
>> > ping: sendto: Permission denied
>> > ping: sendto: Permission denied
>> > ^C
>> > --- 172.16.0.99 ping statistics ---
>> > 3 packets transmitted, 0 packets received, 100.0% packet loss
>> > root@gw:/home/wash # service dnsmasq status
>> > dnsmasq is running as pid 4190.
>> > root@gw:/home/wash #
>>
>> Permission denied is almost certainly coming from firewall,
>> either ipfw or pf.
>>
>
> I haven't changed anything in my pf.conf either.
> What also baffles me is that the VMs are not obtaining IP addresses from
> dnsmasq.
>
Is anyone able to spot something obvious from the following pf.conf that
could be causing the problem I am having?
Thanking you in advance.
```
#-------------------------------------------------------------------------------
# PF: List and Macros
#-------------------------------------------------------------------------------
# Interfaces
ext_if = "em0" # macro for external interface - use tun0 for PPPoE
int_if = "em1" # macro for internal interface
jail_if = "lo1" # the interface we chose for communication between jails
#
# bhyve
bhyve_net="172.16.0.0/24"
int_addr = "192.168.55.254" # Internal IPv4 address (i.e.,
gateway for private network)
int_network = "192.168.54.0/23" # Internal IPv4 network
# External services - port 25214 is used instead of 514 for syslog
ext_tcp_services = "{ 21 22 25 26 53 80 110 123 143 443 465 587 2222 2525
2587 25214 993 995 3000 5000><5100 6276 6277 8000 8069 8080 8081 8082 8083
8999 10000 5900><6000 30000><50000 1024><6277 8765 }"
ext_udp_services = "{ ntp, 1194 }"
# Internal services
int_tcp_services = "{ domain, bootps, dhcpv6-server, ntp, http, https,
http-alt, \
smtp, smtps, pop3, pop3s, imap, imaps, ftp-data, ftp, ssh, svn,
2222, 2232, 4444, \
3128, 3129, 13128, 13129, 23129, 9050, 8123, 8056, 2199, 8191, 82,
2087, 8333, 1157, \
2083, 8030, 10443, 3389, 8080, 8081, 9091, 81, 8086, 8000, 8001,
8002, 8005, 8006, 8090, 8100, 8800, 443, 465, 587, 8444, 8443, 9443, \
18082, 18087, 18092, 18093, 9447, 7005, 115, 8030, 18090, 18083,
18084, 15001, 15002, 15003, 2082 }"
int_udp_services = "{ domain, bootps, dhcpv6-server, ntp, 1194, svn, sip,
8056, 500, 1000, 10000 }"
# The martians table denotes the RFC 1918 addresses and a few other ranges
which
# are mandated by various RFCs not to be in circulation on the open
Internet.
martians = "{ 127.0.0.0/8, 192.168.0.0/16, 172.16.0.0/12, \
10.0.0.0/8, 169.254.0.0/16, 192.0.2.0/24, \
0.0.0.0/8, 240.0.0.0/4 }"
#-------------------------------------------------------------------------------
# PF: Tables
#-------------------------------------------------------------------------------
table <whatsapp-cidr> persist file "/etc/firewall/whatsapp-cidr.txt"
table <bruteforce> file "/etc/firewall/bruteforce_attackers" persist
table <f2b> persist file "/etc/firewall/f2b"
table <fail2ban> persist
table <sshguard> persist
#-------------------------------------------------------------------------------
# PF: Options
#-------------------------------------------------------------------------------
# No restrictions on jail network
set skip on $jail_if
set skip on lo
#-------------------------------------------------------------------------------
# PF: Scrub (Packet Normalization)
#-------------------------------------------------------------------------------
set limit { states 40000, frags 20000, src-nodes 20000 }
set timeout { adaptive.start 18000, adaptive.end 39000 }
#-------------------------------------------------------------------------------
# PF: Packet Queueing and Priorization
#-------------------------------------------------------------------------------
#-------------------------------------------------------------------------------
# PF: Netkwork Address Translation (NAT) and Packet Redirection
#-------------------------------------------------------------------------------
# Network address translation
# Internet
nat on $ext_if inet from any to any -> $ext_if
# Bhyve
nat on $ext_if from $bhyve_net to any -> ($ext_if)
# Traccar
rdr pass on $ext_if inet proto { tcp, udp } from any to port 5055 ->
127.0.0.1 port 5055
# Nominatim Test
rdr pass on $ext_if inet proto { tcp, udp } from any to port 8089 ->
127.0.0.1 port 8089
#-------------------------------------------------------------------------------
# PF: Packet Filtering
#-------------------------------------------------------------------------------
# Restrictive default rules
block all
# Block packets and reply with a TCP RST or ICMP Unreachable response
block return
# FTP-Proxy
# We need to have an anchor for ftp-proxy
anchor "ftp-proxy/*"
# Anchor for fail2ban
anchor "f2b/*"
# Anchor for blacklistd
# This makes sure that the rules within blacklistd are only used for
incoming data on ext_if
anchor "blacklistd/*" in on $ext_if
# Blocking Spoofed Packets
#-------------------------------------------------------------------------------
# Filter rules for $ext_if inbound
# Temporarily let go everything, do not leave active!
#pass in on $ext_if inet
#pass in on $ext_if inet6
# Allow ping
pass in on $ext_if inet proto icmp all
pass in on $ext_if inet6 proto icmp6 all
# External services
pass in on $ext_if inet proto tcp to port $ext_tcp_services
pass in on $ext_if inet proto udp to port $ext_udp_services
block drop in quick on $ext_if inet from <bruteforce> to any
block drop in quick on $ext_if inet from <f2b> to any
block in quick from <fail2ban>
# Custom blocks
block drop in quick on $ext_if inet from 31.130.184.0/24 to any
block in proto tcp from <sshguard> to any
# Now block the ssh bruteforce
block drop in quick on $ext_if inet from <ssh-bruteforce>
block drop in quick on $ext_if inet6 from <ssh-bruteforce>
# PF "Self-Protecting" an FTP Server (passive)
pass in on $ext_if inet proto tcp from any to any port { ftp, > 49151 }
pass in on $ext_if inet6 proto tcp from any to any port { ftp, > 49151 }
# Traccar
pass in on $ext_if inet proto tcp from any to any port { 5000><5100 }
pass in on $ext_if inet6 proto tcp from any to any port { 5000><5100 }
# Nominatim Test
pass in on $ext_if inet proto tcp from any to any port 8089
#-------------------------------------------------------------------------------
# Filter rules for $ext_if outbound
pass out on $ext_if inet
pass out on $ext_if inet6
#-------------------------------------------------------------------------------
# Filter rules for $int_if inbound
# block drop in quick on $int_if inet from 192.168.55.x to any port { 443
993 }
#block drop in quick on $int_if from 192.168.54.190 to any
#block drop in quick on $int_if from any to 13.107.4.50
# Temporarily let go everything, do not leave active!
#pass in on $int_if inet
#pass in on $int_if inet6
# Internal services
pass in on $int_if inet proto tcp to port $int_tcp_services
pass in on $int_if inet proto udp to port $int_udp_services
# Filter rules for $int_if outbound
pass out on $int_if inet
pass out on $int_if inet6
#-------------------------------------------------------------------------------
# DEBUG: RULES FOR VMM
pass in quick on bridge0 all keep state
pass in quick on tap0 all keep state
pass in quick on tap1 all keep state
pass in quick on tap2 all keep state
pass in quick on tap3 all keep state
pass out quick on bridge0 all keep state
pass out quick on tap0 all keep state
pass out quick on tap1 all keep state
pass out quick on tap2 all keep state
pass out quick on tap3 all keep state
# Bhyve hosts
pass in on tap0
pass in on tap1
pass in on tap2
pass in on tap3
```
--
Best regards,
Odhiambo WASHINGTON,
Nairobi,KE
+254 7 3200 0004/+254 7 2274 3223
In an Internet failure case, the #1 suspect is a constant: DNS.
"Oh, the cruft.", egrep -v '^$|^.*#' ¯\_(ツ)_/¯ :-)
[How to ask smart questions:
http://www.catb.org/~esr/faqs/smart-questions.html]