Two ISP lines

[Dinesh Nair]

On Wed, 21 Jan 2004, Dinesh Nair wrote:

> will do. i'm on 4.9-STABLE. will try the following in this order:
>
> 1. the multipath patches
> 2. security/pf port
> 3. rewriting ng_ether to plonk in the new ip address directly into the
>    mbuf. along with ng_one2many, this should provide the same
>    functionality.

i downloaded the multipath patches, and applied them to 4.9-STABLE. by
itself, the kernel didn't compile, and broke in if_ether.c due to the
added argument list into rtrequest(). this was easily remedied with an
#ifdef, and after a new kernel and new userland route and netstat binaries
followed by a reboot, i attempted to test out multipath routing over two
interfaces, fxp0 and aue0 (see below).

i added a default route using both gateways like so:

route add default -gateway 10.1.1.1 -gateway 192.168.0.1 and the routing
table (netstat -rn) was as follows:

Destination        Gateway            Flags/  Refs/      Use  Netif Expire
                                      metric  left
default                               UGSc        0        0   aue0
                   192.168.0.1            10               0     fxp0
                  *10.1.1.1               10      0        0     aue0
10.1/16            link#8             UC          1        0   aue0
10.1.1.1           00:00:1c:d5:03:63  UHLW        2        2   aue0   1115
127.0.0.1          127.0.0.1          UH          0      362    lo0
192.168.0          link#2             UC          1        0   fxp0
192.168.0.1        00:30:ab:10:6c:0d  UHLW        1        0   fxp0    981

i then attempted a tcp connection to an external machine as well as
snooping on both interfaces and came to this observation.

while this multipath patch does round robin the packets (according to
pathmetric) across the two gateways, it does not change the source ip
address of the packet to correspond with the interface it went out on.

as such, the round robin works this way. the first 10 packets are sent
with a src ip of 192.168.0.5 (fxp0's address) out via fxp0. the next 10
are sent still with a src ip of 192.168.0.5 out via aue0. then the cycle
repeats with 10 more being sent out via fxp0. this magic number 10 is
equivalent to the -pathmetric argument given when creating the route. the
default, obviously, is 10. but i digress.

this obviously doesn't work for me, since higher upstream from aue0 (ip
address 10.1.105.26), the src address of 192.168.0.5 is address translated
to a public IP address. the router/natd which does that will obviously not
have any address translation maps for 192.168.0.5, and thus silently drop
the packet. packets going out fxp0 work the way they should. after all,
why shouldn't they ? :)

it would however work if both the interfaces were assigned a public ip
address, though all this still would do is to round robin outgoing
packets, but incoming packets will still come down the same interface (due
to the src address not changing). i'm guessing the the picked source ip
address depends on the current active gateway for that route, as i've seen
the same happen in reverse with a source ip address of 10.1.105.26, the
address on aue0. which leads me to deduce that the src address is picked
on a per connection basis, depending on the current active gateway, and
held for the duration of the connection. thus this does not ensure proper
load balancing over both links, even if public ip addresses were used.

this does not do what i want to do, i.e. multipath routing in a round
robin fashion over x number of interfaces with src ip address changed to
the address of the interface the packet goes out on. this would ensure
that the reply packets from the destination are routed back in correctly.

i'm probably going to experiment with creating a new netgraph node type to
do this. looking thru /usr/src/sys/netgraph, and reading the
ng_one2many(4) man page, i think it would be possible to base a new node
type on one2many which takes input from one interface (hooked to the upper
hook of the ether node) and writes to the lower nodes of the interfaces we
want to round robin multipath over. in between reading from one and
writing to the lower interfaces, i'd need to figure out how to do the
following:

for outgoing packets:
1. modify the passed in mbuf to change the src ip address of the packet
from what it is to what is currently tagged on the chosen outgoing
interface.

2. modify the ethernet dest address of the mbuf to point to the next hop
router for the chosen outgoing interface

for incoming packets:
3. modify the destination ip address on incoming packets to what is
currently tagged on the interface hooked to the one hook.

conceptually, you'd have just one default route pointing to one interface,
and the netgraph nodes would take care of the round robin and packet
delivery.

i'm new to netgraph, though i think i understand how it works and have
read the source in /usr/src/sys/netgraph. i'm absolutely new to mbuf
manipulation however. any pointers in this endeavour will be much
appreciated, as well as any gotchas i'd need to watch out for. also, would
the method i'm proposing above be the way to do it ?

Regards,                           /\_/\   "All dogs go to heaven."
dinesh at alphaque.com                (0 0)    http://www.alphaque.com/
+==========================----oOO--(_)--OOo----==========================+
| for a in past present future; do                                        |
|   for b in clients employers associates relatives neighbours pets; do   |
|   echo "The opinions here in no way reflect the opinions of my $a $b."  |
| done; done                                                              |
+=========================================================================+