Re: Slow startup from D19488 (rtsol: sendmsg: Permission denied)

Hello Bjoern,

  thanks much for the quick reply!

On Tue, Mar 29, 2022 at 10:04:11PM +0000, Bjoern A. Zeeb wrote:
! On Tue, 29 Mar 2022, Peter wrote:
! 
! Hi,
! 
! I am a bit puzzled as after two years you are the first one to report
! that problem to my knowledge for either base system or jails.

This is what greatly wonders me, too. So I was stronly thinking
that I am doing something wrong or unusual. But I cannot figure
it out, it just seems that the detrimental effect of the change
cannot be avoided (e.g. "service jail start" takes quite long now -
there's a lot of them).

! >  after upgrading 12.3 to stable/13, I am seeing these
! > errors in all my jails:
! > 
! > > Additional TCP/IP options: log_in_vain=1.
! > > ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib
! >     /usr/local/lib/c cmpat/pkg /usr/local/lib/compat/pkg
! > > 32-bit compatibility ldconfig path:
! > > rtsol: sendmsg on nrail1l: Permission denied
! > > rtsol: sendmsg on nrail1l: Permission denied
! > > rtsol: sendmsg on nrail1l: Permission denied
! > > Starting Network: lo0 nrail1l.
! 
! Can you give us a full startup log?

It's the above, right from the beginning, and then follows:

> lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
>         options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
>         inet 127.0.0.1 netmask 0xff000000
>         inet6 ::1 prefixlen 128
>         inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
>         groups: lo
>         nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
> nrail1l: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
>         options=28<VLAN_MTU,JUMBO_MTU>
>         ether 06:1d:92:01:01:0a
>         hwaddr 58:9c:fc:10:28:71
>         inet ************* netmask ********** broadcast *************
>         inet6 fe80::41d:92ff:fe01:10a%nrail1l prefixlen 64 scopeid 0x2
>         inet6 fd00:************ prefixlen 120
>         media: Ethernet autoselect (1000baseT <full-duplex>)
>         status: active
>         nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
> Starting rtsold.
> add host 127.0.0.1: gateway lo0 fib 0: route already in table
> add net default: gateway *************
> Additional inet routing options: log ICMP redirect=YES.
> add host ::1: gateway lo0 fib 0: route already in table
> add net fe80::: gateway ::1
> add net ff02::: gateway ::1
> add net ::ffff:0.0.0.0: gateway ::1
> add net ::0.0.0.0: gateway ::1
> add net default: gateway fd00:*************
> Flushed all rules.
> Firewall rules loaded.
> Firewall logging pseudo-interface (ipfw0) created.
> Creating and/or trimming log files.
> Updating /var/run/os-release done.
> Clearing /tmp (X related).
> Updating motd:.
> Starting syslogd.
> Starting rapp.
> Starting cron.
> Starting sendmail.
> Starting sendmail_msp_queue.
> Performing sanity check on sshd configuration.
> Starting sshd.
> 
> Wed Mar 30 00:52:15 CEST 2022

! > Searching the cause I find change  1b5be7204eaeeaf  aka  D19488
! > 
! > This doesn't work, because the firewall is not yet present. This is
! 
! Given you are talking firewall, I assume you are using vnet jails?

Yes.

! And given you are talking ipfw I assume your default policy is deny
! and not accept?

Yes.
 
! And given rtsol runs I assume you have IPv6 configured and in use?

Yes. Here is how I do it:
https://daemon.contact/ankh/articles/X3OyjgTpuv

! The same issue then should also happen in your base system on boot?

No. The base system does (second level) prefix delegation and has
ipv6_gateway_enable="YES" and rtsold_enable="NO" and is not affected.

There is one vnet jail intended as VPN server, which also has these
parameters in rc.conf and is also not affected.

(I did not yet bother to figure out why, The shell code run from
rc.d/netif is a bit lenghty...)

! > happening in rc.d/netif, and that must run before rc.d/ipfw in any
! > case, because the firewall needs to see the netifs.
! 
! I thought ipfw could log deal with interfaces coming and going?

Maybe it can, but then modifying the rc.d logic so to get "ipfw" run
before "netif" - that does likely open a box of worms.

Furthermore, I do use ipfw as a genuine rerouting+filtering
framework, and that logic is entirely based on the interfaces; all
rules belong to exactly two interfaces. Here is a short abstract
of the idea:
https://forums.freebsd.org/threads/ipfw-or-pf.46706/post-561760


cheerio,
PMc