Re: Slow startup from D19488 (rtsol: sendmsg: Permission denied)

On Tue, Mar 29, 2022 at 5:10 PM Peter <pmc@citylink.dinoex.sub.org> wrote:

>
> Hello Bjoern,
>
>   thanks much for the quick reply!
>
> On Tue, Mar 29, 2022 at 10:04:11PM +0000, Bjoern A. Zeeb wrote:
> ! On Tue, 29 Mar 2022, Peter wrote:
> !
> ! Hi,
> !
> ! I am a bit puzzled as after two years you are the first one to report
> ! that problem to my knowledge for either base system or jails.
>
> This is what greatly wonders me, too. So I was stronly thinking
> that I am doing something wrong or unusual. But I cannot figure
> it out, it just seems that the detrimental effect of the change
> cannot be avoided (e.g. "service jail start" takes quite long now -
> there's a lot of them).
>
> ! >  after upgrading 12.3 to stable/13, I am seeing these
> ! > errors in all my jails:
> ! >
> ! > > Additional TCP/IP options: log_in_vain=1.
> ! > > ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib
> ! >     /usr/local/lib/c cmpat/pkg /usr/local/lib/compat/pkg
> ! > > 32-bit compatibility ldconfig path:
> ! > > rtsol: sendmsg on nrail1l: Permission denied
> ! > > rtsol: sendmsg on nrail1l: Permission denied
> ! > > rtsol: sendmsg on nrail1l: Permission denied
> ! > > Starting Network: lo0 nrail1l.
> !
> ! Can you give us a full startup log?
>
> It's the above, right from the beginning, and then follows:
>
> > lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
> >         options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
> >         inet 127.0.0.1 netmask 0xff000000
> >         inet6 ::1 prefixlen 128
> >         inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
> >         groups: lo
> >         nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
> > nrail1l: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu
> 1500
> >         options=28<VLAN_MTU,JUMBO_MTU>
> >         ether 06:1d:92:01:01:0a
> >         hwaddr 58:9c:fc:10:28:71
> >         inet ************* netmask ********** broadcast *************
> >         inet6 fe80::41d:92ff:fe01:10a%nrail1l prefixlen 64 scopeid 0x2
> >         inet6 fd00:************ prefixlen 120
> >         media: Ethernet autoselect (1000baseT <full-duplex>)
> >         status: active
> >         nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
> > Starting rtsold.
> > add host 127.0.0.1: gateway lo0 fib 0: route already in table
> > add net default: gateway *************
> > Additional inet routing options: log ICMP redirect=YES.
> > add host ::1: gateway lo0 fib 0: route already in table
> > add net fe80::: gateway ::1
> > add net ff02::: gateway ::1
> > add net ::ffff:0.0.0.0: gateway ::1
> > add net ::0.0.0.0: gateway ::1
> > add net default: gateway fd00:*************
> > Flushed all rules.
> > Firewall rules loaded.
> > Firewall logging pseudo-interface (ipfw0) created.
> > Creating and/or trimming log files.
> > Updating /var/run/os-release done.
> > Clearing /tmp (X related).
> > Updating motd:.
> > Starting syslogd.
> > Starting rapp.
> > Starting cron.
> > Starting sendmail.
> > Starting sendmail_msp_queue.
> > Performing sanity check on sshd configuration.
> > Starting sshd.
> >
> > Wed Mar 30 00:52:15 CEST 2022
>
> ! > Searching the cause I find change  1b5be7204eaeeaf  aka  D19488
> ! >
> ! > This doesn't work, because the firewall is not yet present. This is
> !
> ! Given you are talking firewall, I assume you are using vnet jails?
>
> Yes.
>
> ! And given you are talking ipfw I assume your default policy is deny
> ! and not accept?
>
> Yes.
>
> ! And given rtsol runs I assume you have IPv6 configured and in use?
>
> Yes. Here is how I do it:
> https://daemon.contact/ankh/articles/X3OyjgTpuv
>
> ! The same issue then should also happen in your base system on boot?
>
> No. The base system does (second level) prefix delegation and has
> ipv6_gateway_enable="YES" and rtsold_enable="NO" and is not affected.
>
> There is one vnet jail intended as VPN server, which also has these
> parameters in rc.conf and is also not affected.
>
> (I did not yet bother to figure out why, The shell code run from
> rc.d/netif is a bit lenghty...)
>
> ! > happening in rc.d/netif, and that must run before rc.d/ipfw in any
> ! > case, because the firewall needs to see the netifs.
> !
> ! I thought ipfw could log deal with interfaces coming and going?
>
> Maybe it can, but then modifying the rc.d logic so to get "ipfw" run
> before "netif" - that does likely open a box of worms.
>
> Furthermore, I do use ipfw as a genuine rerouting+filtering
> framework, and that logic is entirely based on the interfaces; all
> rules belong to exactly two interfaces. Here is a short abstract
> of the idea:
> https://forums.freebsd.org/threads/ipfw-or-pf.46706/post-561760
>
>
> cheerio,
> PMc
>
> This may be irrelevant, but updating to the stable branch is not
recommended as it is not regularly tested. Updating to 13.0-Release and
then to stable is less likely to be problematic.
-- 
Kevin Oberman, Part time kid herder and retired Network Engineer
E-mail: rkoberman@gmail.com
PGP Fingerprint: D03FB98AFA78E3B78C1694B318AB39EF1B055683