Beaglebone Black + FreeBSD + USB WiFi = WAP?

Wed Sep 6 17:32:01 UTC 2017

On Wed, Sep 6, 2017 at 3:44 AM, Chris Gordon <freebsd at theory14.net> wrote:
> No problem -- I figured as much.  Incidentally, I’m using a urtwn NIC and I have not seen any panics.
>
> Chris
>
>
>> On Sep 5, 2017, at 10:17 AM, Russell Haley <russ.haley at gmail.com> wrote:
>>
>> Hi Chris,
>>
>> Sorry, I was working late and this was supposed to go to the arm mailing list, not just you.
>>
>> Cheers,
>> Russ

(Please see https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=221845.
Note, I am going to add Chris' original message to the bug...)

Hi Chris,

I have some questions that may seem rather basic, but I need to get a
baseline understanding.

Have you monitored your system on a serial console or direct console
(i.e. via hdmi/keyboard)? Is the system still responding to other
commands after you run the speed test? My thought is that the really
really low bandwidth belies a kernel panic on the main terminal that
you are not seeing.

Regardless of your answer, both your low bandwidth and my kernel panic
indicate serious issues between the bridge interface and the device
drivers on bbb. The offending bridge_broadcast function in my kernel
panic uses m_dup with the NO_WAIT flag. My current working theory is
that something in the usb or other intermediate bus is holding a sleep
lock or a mutex that it shouldn't. If your wi-fi card is faster than
the one I am using, perhaps it avoids the kernel panic, but would
ultimately lead to really low bandwidth. (Again, all theory from a
dilettante, not a kernel developer).

If you would like to do some further testing, you could perhaps help
me answer these things:
- Can you find a command line way of measuring throughput and latency
separately that can be run on a host and on the bbb? I'm sure there
are lots of ways to do so. I will leave it up to you to decide and
will adopt the same tests so we can compare results.
- Can you run the bbb as a standard device (not an access point) and
test the performance of the wlan0 interface using the method of
measurement pointed above? I will do the same at some point with my
wi-fi dongle.

While the above won't provide too much detailed clarity, it will at
least allow us to perhaps remove the bridge driver from the suspect
list.

Some tests I would like to do:
- Get DTrace involved as a debugging tool. I have rudimentary DTrace
skills but will need to consult my books on how to measure throughput
and latency. There are some examples early in the DTrace book of
logging system calls made by a process and I will review that again
when time permits.
- Run the system through the kernel debugger. I think this is going to
be difficult though as pausing the kernel in the middle of TCP traffic
might invalidate any results I get. I know how difficult it can be to
debug threaded applications, I can't see a kernel being any easier. ;)

Cheers,

Russ

>> Sent from my BlackBerry 10 smartphone on the Virgin Mobile network.
>>   Original Message
>> From: Russell Haley
>> Sent: Monday, September 4, 2017 10:50 PM
>> To: Chris Gordon
>> Subject: Re: Beaglebone Black + FreeBSD + USB WiFi = WAP?
>>
>> On Sat, Aug 26, 2017 at 11:48 PM, Russell Haley <russ.haley at gmail.com> wrote:
>>> On Sat, Aug 26, 2017 at 11:35 PM, Russell Haley <russ.haley at gmail.com> wrote:
>>>> On Thu, Aug 24, 2017 at 11:26 PM, Russell Haley <russ.haley at gmail.com> wrote:
>>>>> On Tue, Aug 22, 2017 at 5:51 PM, Chris Gordon <freebsd at theory14.net> wrote:
>>>>>> Ilya,
>>>>>>
>>>>>> Thanks for the follow up.
>>>>>>
>>>>>>> On Aug 22, 2017, at 3:11 PM, Ilya Bakulin <ilya at bakulin.de> wrote:
>>>>>>>
>>>>>>> Hi Chris,
>>>>>>>
>>>>>>> have you found the issue already?
>>>>>>
>>>>>> I have not. See below for some theories...
>>>>>>
>>>>>>> If not: what does `top -Sa` show when you're running your speed test?
>>>>>>> Specifically what does "CPU:" line look like, and what are the top processes in the list?
>>>>>>
>>>>>> The system stays at >90% idle through the entire test (upload and download). I see 2-4% WCPU for interrupts and 1-2% for USB.
>>>>>>
>>>>>>> I've had an issue with FreeBSD acting as WAP (although using Atheros-based NIC) some years ago,
>>>>>>> the problem back then was that the machine CPU was just too slow to process the traffic.
>>>>>>
>>>>>> I had initially thought that maybe the little CPU in the BeagleBone wasn’t up to the WPA encryption or the interrupt rate + usb where just too much for it. Sometimes changing channels helps for a little bit. Since I’ve been tinkering with this little project, I’ve been paying a bit more attention to my overall WiFi performance and I’m beginning to think there are just too many WiFi signals nearby and congestion is just killing my overall WiFi performance.
>>>>>>
>>>>>> Any other ideas?
>>>>>
>>>>> Hi, I'm just trying to reproduce your setup with my BBB and an ASUS
>>>>> wi-fi stick. The chipset is Ralink RT3052. I just got the dongle
>>>>> working so I'll see if I can set it up as an access point this
>>>>> weekend. I can't make any promises on play time though. :)
>>>>
>>>>
>>>>
>>>> Hi!
>>>>
>>>> So I'm only partially successful repeating your test so far, but I can
>>>> cause a kernel panic! The following are my observations:
>>>>
>>>> Running BBB through ftdi cable.
>>>> Asus WiFi Adapter, RT3071 chipset
>>>> https://wikidevi.com/files/Ralink/RT307x%20product%20brief.pdf
>>>>
>>>> root at bbb:~ # uname -a
>>>> FreeBSD bbb.highfell.local 12.0-CURRENT FreeBSD 12.0-CURRENT #7
>>>> r321601M: Thu Aug 17 22:13:21 PDT 2017
>>>> russellh at prescott.highfell.local:/usr/home/russellh/FreeBSD/rh-armv6/obj/arm.armv6/usr/home/russellh/FreeBSD/rh-armv6/src/sys/BEAGLEBONE-MMCCAM
>>>> arm
>>>>
>>>> root at bbb:~ # cat /boot/loader.conf
>>>> if_run0_load="YES"
>>>> wlan_mac_load="YES"
>>>>
>>>> root at bbb:~ # cat /etc/rc.conf
>>>> hostname="bbb.highfell.local"
>>>> ifconfig_cpsw0="inet 192.168.2.101 netmask 255.255.255.0"
>>>> defaultrouter="192.168.2.1"
>>>> hostapd_enable="YES"
>>>> wlans_run0="wlan0"
>>>> create_args_wlan0="wlanmode hostap"
>>>> ifconfig_wlan0="up"
>>>> #gateway_enable="YES"
>>>> cloned_interfaces="bridge0"
>>>> ifconfig_bridge0="addm cpsw0 addm wlan0 up"
>>>>
>>>> sshd_enable="YES"
>>>> sendmail_enable="NONE"
>>>> sendmail_submit_enable="NO"
>>>> sendmail_outbound_enable="NO"
>>>> sendmail_msp_queue_enable="NO"
>>>> growfs_enable="YES"
>>>>
>>>>
>>>>
>>>> root at bbb:~ # cat /etc/hostapd.conf
>>>> interface=wlan0
>>>> debug=1
>>>> ctrl_interface=/var/run/hostapd
>>>> ctrl_interface_group=wheel
>>>> ssid=freebsd
>>>> wpa=2
>>>> wpa_passphrase=testing
>>>> wpa_key_mgmt=WPA-PSK
>>>> wpa_pairwise=CCMP
>>>>
>>>> root at bbb:~ # cat /etc/resolv.conf
>>>> # Generated by resolvconf
>>>> nameserver 192.168.2.1
>>>>
>>>>
>>>> 1) Before the kernel loads, loader give the following errors:
>>>>
>>>> can't find 'if_run'
>>>> can't find 'wlan_mac'
>>>>
>>>> 2) It seems the run0 usb wi-fi interface only comes up after the
>>>> bridge0 is already enabled. dmesg does NOT capture the output from the
>>>> failed attempt to add the non-existent wlan0 interface. However, I
>>>> grabbed it from the boot output in the serial console:
>>>>
>>>> #From dmesg:
>>>>
>>>> ugen1.2: <Ralink 802.11 n WLAN> at usbus1
>>>> random: unblocking device.
>>>> bridge0: Ethernet address: 02:94:dd:d7:a3:00
>>>> cpsw0: link state changed to DOWN
>>>> cpsw0: promiscuous mode enabled
>>>> bridge0: link state changed to DOWN
>>>> cpsw0: link state changed to UP
>>>> bridge0: link state changed to UP
>>>> run0 on uhub1
>>>> run0: <1.0> on usbus1
>>>> run0: MAC/BBP RT3572 (rev 0x0223), RF RT3052 (MIMO 2T2R), address
>>>> 60:a4:4c:ec:c9:a5
>>>> ieee80211_load_module: load the wlan_amrr module by hand for now.
>>>> wlan0: Ethernet address: 60:a4:4c:ec:c9:a5
>>>> run0: firmware RT3071 ver. 0.33 loaded
>>>>
>>>> #From console grab:
>>>>
>>>> eeding entropy: .
>>>> ifconfig: SIOCIFCREATE2: Invalid argument
>>>> bridge0: Ethernet address: 02:94:dd:d7:a3:00
>>>> Created clone interfaces: bridge0.
>>>> cpsw0: link state changed to DOWN
>>>> cpsw0: promiscuous mode enabled
>>>> bridge0: link state changed to DOWN
>>>> ifconfig: BRDGADD wlan0: No such file or directory
>>>> cpsw0: link state changed to UP
>>>> bridge0: link state changed to UP
>>>> Starting Network: lo0 cpsw0 bridge0.
>>>> lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
>>>> options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
>>>> inet6 ::1 prefixlen 128
>>>> inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
>>>> inet 127.0.0.1 netmask 0xff000000
>>>> groups: lo
>>>> nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
>>>> cpsw0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST>
>>>> metric 0 mtu 1500
>>>> options=8000b<RXCSUM,TXCSUM,VLAN_MTU,LINKSTATE>
>>>> ether a0:f6:fd:8a:c5:be
>>>> hwaddr a0:f6:fd:8a:c5:be
>>>> inet 192.168.2.101 netmask 0xffffff00 broadcast 192.168.2.255
>>>> media: Ethernet autoselect (100baseTX <full-duplex>)
>>>> status: active
>>>> nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
>>>> bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
>>>> ether 02:94:dd:d7:a3:00
>>>> id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
>>>> maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
>>>> root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
>>>> member: cpsw0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
>>>> ifmaxaddr 0 port 1 priority 128 path cost 55
>>>> groups: bridge
>>>> nd6 options=9<PERFORMNUD,IFDISABLED>
>>>> Starting devd.
>>>> run0 on uhub1
>>>> run0: <1.0> on usbus1
>>>> run0: MAC/BBP RT3572 (rev 0x0223), RF RT3052 (MIMO 2T2R), address
>>>> 60:a4:4c:ec:c9:a5
>>>> ieee80211_load_module: load the wlan_amrr module by hand for now.
>>>> wlan0: Ethernet address: 60:a4:4c:ec:c9:a5
>>>> Created wlan(4) interfaces: wlan0.
>>>> run0: firmware RT3071 ver. 0.33 loaded
>>>> Starting Network: wlan0.
>>>> wlan0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
>>>> ether 60:a4:4c:ec:c9:a5
>>>> hwaddr 60:a4:4c:ec:c9:a5
>>>> groups: wlan
>>>> ssid "" channel 11 (2462 MHz 11g)
>>>> regdomain FCC country US authmode OPEN privacy OFF txpower 30
>>>> scanvalid 60 protmode CTS wme dtimperiod 1 -dfs bintval 0
>>>> media: IEEE 802.11 Wireless Ethernet autoselect <hostap>
>>>> (autoselect <hostap>)
>>>> status: no carrier
>>>> nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
>>>> add host 127.0.0.1: gateway lo0 fib 0: route already in table
>>>> add net default: gateway 192.168.2.1
>>>> add host ::1: gateway lo0 fib 0: route already in table
>>>>
>>>>
>>>> *Something else to note about this setup output is that wlan0 did NOT
>>>> get the ssid or the security setup from /etc/hostapd.conf
>>>>
>>>> After boot I manually add the wlan0 to the bridge and then set the ssid
>>>>
>>>> root at bbb:~ # ifconfig bridge0 addm wlan0
>>>> root at bbb:~ # ifconfig wlan0 ssid freebsd
>>>>
>>>> I brought the interface down and back up again which made the AP is
>>>> available to the clients. I open the ipod and get the system to
>>>> associate with the ap and enter the following information
>>>>
>>>> static IP
>>>>
>>>> address: 192.168.2.102
>>>> subnet: 255.255.255.0
>>>> router: 192.168.2.1
>>>> dns : 192.168.1
>>>>
>>>> After numerous wrong attempts at configuring the client, I managed to
>>>> get exactly ONE request through. The freebsd.org page came up. I then
>>>> tried to search for the ookla page and my bbb kernel paniced! (yay!)
>>>> https://pastebin.com/zB9AnWTv
>>>>
>>>> The next time I booted the entire board hung right after the usb wifi
>>>> adapter loaded (chop of hung board output, full output here
>>>> https://pastebin.com/M09C5NEP):
>>>>
>>>> cpsw0: link state changed to UP
>>>> bridge0: link state changed to UP
>>>> Starting Network: lo0 cpsw0 bridge0.
>>>> lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
>>>> options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
>>>> inet6 ::1 prefixlen 128
>>>> inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
>>>> inet 127.0.0.1 netmask 0xff000000
>>>> groups: lo
>>>> nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
>>>> cpsw0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST>
>>>> metric 0 mtu 1500
>>>> options=8000b<RXCSUM,TXCSUM,VLAN_MTU,LINKSTATE>
>>>> ether a0:f6:fd:8a:c5:be
>>>> hwaddr a0:f6:fd:8a:c5:be
>>>> inet 192.168.2.101 netmask 0xffffff00 broadcast 192.168.2.255
>>>> media: Ethernet autoselect (100baseTX <full-duplex>)
>>>> status: active
>>>> nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
>>>> bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
>>>> ether 02:94:dd:d7:a3:00
>>>> id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
>>>> maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
>>>> root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
>>>> member: cpsw0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
>>>> ifmaxaddr 0 port 1 priority 128 path cost 55
>>>> groups: bridge
>>>> nd6 options=9<PERFORMNUD,IFDISABLED>
>>>> Starting devd.
>>>> run0 on uhub1
>>>> run0: <1.0> on usbus1
>>>>
>>>> U-Boot SPL 2015.10-00001-g143c9ee (Nov 06 2015 - 15:27:19)
>>>> bad magic
>>>>
>>>> Sometimes it boots, sometimes it hangs (I'd say 3 to 1). The lights on
>>>> the cpsw0 interface still blink but the serial console is dead. I'm
>>>> trying to *avoid* triggering that so I don't know the sequence that's
>>>> causing it. However, I can cause the kernel to panic on the BBB
>>>> relatively quickly from a handful of page requests on the ipod. No
>>>> more than three full page requests so far. It seems there is a bad
>>>> memory copy happening in bridge_broadcast() at bridge_broadcast+0x1c4?
>>>>
>>>> https://www.freebsd.org/cgi/man.cgi?apropos=0&sektion=9&query=m_dup
>>>>
>>>> Anyway, that's all the time I have for this weekend. I'm going to take
>>>> the chance that someone wants to see this and put it in bugzilla.
>>>>
>>>> Cheers,
>>>>
>>>> Russ
>>>
>>> https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=221845
>>>
>>> Cheers,
>>> Russ
>>
>> Hi,
>>
>> I've been digging into the code for if_bridge.c, which is found under
>> sys/net. bridge_broadcast only has one call to m_dup on line 2553.
>