[Bug 252334] Regression for running jails with fibs

[bugzilla-noreply at freebsd.org]

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=252334

            Bug ID: 252334
           Summary: Regression for running jails with fibs
           Product: Base System
           Version: CURRENT
          Hardware: amd64
                OS: Any
            Status: New
          Severity: Affects Some People
          Priority: ---
         Component: misc
          Assignee: bugs at FreeBSD.org
          Reporter: freebsd at fizz.buzz

I've noticed that there appears to be a regression in FreeBSD 13 when trying to
run a jail with a fib. I've created a minimal example script:
https://gist.github.com/tomalexander/3ecde8933e482794ef808a37047b07a0

If you take that script and run it in a FreeBSD virtual machine with a clean
install of FreeBSD with git installed then it will update itself to a revision
you specify at the top of the script and build a jail using the same revision.
I strongly advise everyone to only run the script inside a VM because it will
overwrite system config files.

This process will involve multiple reboots, but the script installs itself as a
service to auto-launch itself on reboots so it should fully set itself up
without intervention. Regardless, the script is quite small and readable so you
should be able to follow the steps manually if you'd like (explanation at
bottom of email).

If you set REVISION="stable/12" at the top of the script then after it goes
through its full setup you can run "service jail onestart test" and you can
read /var/log/jail.log and see that no errors related to the fib are printed.
The jail itself won't have network connectivity because this setup is missing
other pieces like a firewall to do NAT since I wanted to cut this down to a
minimal example.

If instead you set REVISION="main" at the top of the script then if you had
KERNCONF="GENERIC" at the top of the script you will get dropped into a
debugger upon running "service jail onestart test":
https://user-images.githubusercontent.com/458272/103318144-75636800-49fb-11eb-92f8-66351c65ca7d.png
. If instead you set KERNCONF="GENERIC-NODEBUG" then you can read
/var/log/jail.log and see messages that say "route: fiboptlist_csv failed."

Through some printf debugging, I have found that in FreeBSD 12 the numfibs in
route.c inside the jail is 2 but in FreeBSD 13 the numfibs in route.c inside
the jail is 1. Thats as far as my debugging has reached so far. I've attempted
to git bisect my way there, but I've been unsuccessful (hitting a lot of
revision that fail to build between stable/12 and main)

Finally, a quick overview of what the script does for people who want to
recreate this on their own. I'll leave out the minutia of the script injecting
itself as a service and cleaning itself up:

1. Creates devfs rule for vnet
2. sets net.fibs=2 in the host /boot/loader.conf
3. sets net.add_addr_allfibs=1 in the host /etc/sysctl.conf
4. sets up a jail.conf using the example from /usr/share/examples/jails for
if_bridge but set the fib for the jail to 1
5. Checks out the revision of freebsd specified in the script. Builds and
installs that version to the host.
6. Also install that built version to /jail to be used as a jail
7. Configure the jail's rc.conf to use a static local ip address

-- 
You are receiving this mail because:
You are the assignee for the bug.