can't find routing entry for network routes

[remodeler]

On Tue, 06 Oct 2009 18:37:49 -0700, Julian Elischer wrote

> please recap with a script that fails

Thank you for your response Julian. I very much respect the work everyone has
done on netgraph / vimage / jails, and also the help extended to me.

Kernel options in addition to amd64 GENERIC are geom_journal, ufs_gjournal,
geom_mirror, geom_eli, vimage, netgraph, netgraph_bridge, netgraph_ether, and
netgraph_eiface. Additional devices are crypto. World and kernel are in sync.
I have been testing by csup'ing from head, but I have the same errors in 8.0
beta1, beta3, and rc1.

  # make a bridge and connect the physical ethernet interface to it
  ngctl mkpeer msk0: bridge lower link0
  ngctl name msk0:lower bridge0
  ngctl connect msk0: bridge0: upper link1

  # Start Name Server Jail
  jail -c -l -U root -n ns host.hostname=ns.my.org path=\
/jail/j/ns vnet persist
  mount -t procfs proc /jail/j/ns/proc
  mount -t devfs dev /jail/j/ns/dev
  devfs -m /jail/j/ns/dev rule -s 4 applyset
  mount -t fdescfs null /jail/j/ns/dev/fd
  ngctl mkpeer eiface ether ether
  ngctl connect ngeth0: bridge0: ether link2
  ifconfig ngeth0 vnet ns
  vimage ns ifconfig lo0 localhost
  vimage ns ifconfig ngeth0 link 02:0a:0b:0c:01:01
  vimage ns ifconfig ngeth0 172.26.64.10
  vimage ns route add default -link 00:23:54:08:2b:f7

This results in an error on applying the devfs ruleset, so I see all of root's
/dev in the jail. I receive an error on the route command. I get identical
errors when I specify / as the path and omit the mounts/devfs command. 

My expectation is that this would leave me with two network stacks, msk0 and
ngeth0, with msk0 connected to the ng_bridge by its upper and lower hooks and
ngeth0 by its ether hook. I would expect network connectivity over the bridge,
and to be able to manipulate the vnet jail's FIB from the host to add a
default route. Most of what I expect happens:

# ngctl list
There are 5 total nodes:
  Name: bridge0         Type: bridge          ID: 00000004   Num hooks: 3
  Name: ipfw            Type: ipfw            ID: 00000001   Num hooks: 0
  Name: ngeth0          Type: eiface          ID: 00000008   Num hooks: 1
  Name: ngctl1633       Type: socket          ID: 0000000a   Num hooks: 0
  Name: msk0            Type: ether           ID: 00000002   Num hooks: 2

# vimage -l
ns

# jls
   JID  IP Address      Hostname                      Path
     1  -               ns.my.org                  /jail/j/ns

but the devfs ruleset command fails (executed on the host):

# devfs -m /jail/j/ns/dev rule -s 4 applyset
devfs rule: ioctl DEVFSIO_SAPPLY: No such process

and in the jail:

ns# # devfs -m /jail/j/ns/dev rule -s 4 applyset
#: Command not found.
ns# fs rule: ioctl DEVFSIO_SAPPLY: No such processdevfs rule: ioctl
DEVFSIO_SAPPLY: No such process

adding the route from the host:

# vimage ns route add default -link 00:23:54:08:2b:f7
route: writing to routing socket: Network is unreachable
add net default: gateway 00:23:54:08:2b:f7: Network is unreachable

and from the jail:

ns# route add default -link 00:23:54:08:2b:f7
route: writing to routing socket: Network is unreachable
add net default: gateway 00:23:54:08:2b:f7: Network is unreachable

I get the same error for netstat -r from the host and the jail:

# netstat -r
netstat: kvm not available: /dev/mem: Permission denied
Routing tables
rt_tables: symbol not in namelist

Before I compiled in Bjorn's patches, netstat -r worked properly on the host.
The host has network connectivity. If I boot without starting the jail,
everything works as I expect on the host (haven't tested that very far since
the patches).

Thank you.