atheros broadcast/multicast corruption with multiple hostap's

[Russell Yount]

On Sun, Jan 17, 2010 at 1:45 PM, Sam Leffler <sam at errno.com> wrote:

> Russell Yount wrote:
>
>
>>
>> On Sat, Jan 16, 2010 at 3:21 PM, Sam Leffler <sam at errno.com <mailto:
>> sam at errno.com>> wrote:
>>
>>    Russell Yount wrote:
>>
>>        It seems AP to client broadcasts/multicasts traffic is
>>        broken when using WPA2/802.11i with multiple hostapds in 8.0.
>>
>>        Only the SSID associated with the last hostapd to be started has
>>        AP to client broadcasts/multicasts being delivered correctly.
>>
>>        The AP and client are 8.0 freebsd systems althought I see same
>>        problems with windows XP as a client.
>>
>>        The AP has 4 hostapds configured to use TLS with client
>>        certificates for
>>        authentication. (hostapd recompiled with
>>        HOSTAPD_CFLAGS=-DEAP_SERVER)
>>        The AP and client radio are shown as ath0: AR5212 mac 5.9 RF5112
>>        phy 4.3
>>        in dmesg.
>>
>>        Client authenticate using client certificates associate correctly
>>        to all 4 SSIDs. Unicast traffic flows correctly between clients
>>        and AP
>>        for all for 4 SSIDs. Client to AP broadcast/multicast traffic works
>>        on of 4 SSIDs. AP to client broadcast/multicast traffic only works
>>        on 1 of the SSIDs. I have documented this using ARP broadcasts,
>>        but normal IP broadcasts also observed to corrupted.
>>
>>        When an ARP request is send through the AP to an associated client
>>        it seems to be trashed on any of the SSID except the one associated
>>        with the last hostapd to be started. Here is the output of
>>        client side
>>        tcpdump showing the problems.
>>
>>        In the first client side tcpdump with the hostapd associated
>>        with the SSID
>>        being associaed with the last hostapd started and the traffic
>>        flowing
>>        normally.
>>
>>        In the second client side tcpdump with the hostapd associated
>>        with the SSID
>>        being not the last hostapd started the ARP request is resent
>>        multiple times
>>        and appears corrupted.
>>
>>        I would really like to find a fix for this.
>>        Any help would be greatly appreciated.
>>
>>
>>    This sounds like the crypto encap of the frame is clobbering the
>>    mbuf contents.  You can verify this by setting up multiple vaps w/o
>>    WPA.  If this is the problem look for the mbuf copy logic for mcast
>>    frames and make sure a deep copy is done.
>>
>>           Sam
>>
>>    The four VAPs broadcast traffic works find without WPA if I do not
>> start hostapds on them
>>  I have been trying to discovery why broadcast traffic only works
>> correctly on the VAP associated with the last hostapd to be started. I have
>> move with VAP has the working broadcast traffic by restarting the hostapd
>> associated with it.
>>  It would seem something in the WPA/802.1x layer initialization remembers
>> which hostapd was started last and that affected the crypto encap.
>>  I keep looking but do not see any place in the code that could account
>> for this.
>>  It seems the corrupt crypto encap also happens on broadcast between
>> stations.
>> Please correct me if I am wrong:
>> but when using hostapd normally traffic is bridged withing the card.
>> So if a station sends to the VAP a broadcast it is actaully sending a non-
>> broadcast frame to the AP
>> and the AP sends the frame to all the other stations.
>>
>
> I told you waht the likely problem is.  Look in the net80211 layer in the
> kernel for the problem.
>
>        Sam
>



 I tried to find problems in mbuf corruption
in ieee80211_output.c by placing

  m = m_unshare(m,M_NOWAIT);
  if (m == NULL) {
      IEEE80211_DPRINTF(vap, IEEE80211_MSG_OUTPUT,
      "%s: cannot get writable mbuf\n", __func__);
      return NULL;
  }

at begining ieee80211_mbuf_adjust() and at
beginning of ieee80211_encap() with no change
in the broadcast traffic behaviour.

I tried then to in ieee80211_crypto.c substituting

  flags |= IEEE80211_KEY_SWCRYPT;

for the encryption capabilities test code

  if ((ic->ic_cryptocaps & (1<<cipher)) == 0) {
       IEEE80211_DPRINTF(vap, IEEE80211_MSG_CRYPTO,
       "%s: no h/w support for cipher %s, falling back to s/w\n",
           __func__, cip->ic_name);
       flags |= IEEE80211_KEY_SWCRYPT;
   }

to force all the encryption to be done in software.

This fixed the broadcast traffic problem but without
hardware support its very slow and really loads machine.
Enabling in the debug code to ath and net80211

and enabled ATH_DEBUG_KEYCACHE in if_ath.c and
IEEE80211_MSG_CRYPTO in net80211 code.

It seems that all the VAPS sets the broadcast key for
mac ff:ff:ff:ff:ff:ff in the ath device so I assume
they conflict and the last one setting the key is the
working one; that would explain why the last hostapd
started is the only one with working broadcast code
to clients.

In if_ath.c the code

 if ((k->wk_flags & IEEE80211_KEY_GROUP) && sc->sc_mcastkey) {
   /*
    * Group keys on hardware that supports multicast frame
    * key search use a mac that is the sender's address with
    * the high bit set instead of the app-specified address.
    */
     IEEE80211_ADDR_COPY(gmac, bss->ni_macaddr);
     gmac[0] |= 0x80;
     mac = gmac;
 } else
    mac = k->wk_macaddr;

seems to indicate that for multiple VAPs the ath chips
needs to be able to distinguish between broadcast keys
by using a permutation of VAPs bssid.

But in if_athvar.h the code does not seem complete
and sc->sc_mcastkey is also set false.

 #ifdef notyet
 #define ath_hal_hasmcastkeysearch(_ah) \
        (ath_hal_getcapability(_ah, HAL_CAP_MCAST_KEYSRCH, 0, NULL) ==
HAL_OK)
 #define ath_hal_getmcastkeysearch(_ah) \
        (ath_hal_getcapability(_ah, HAL_CAP_MCAST_KEYSRCH, 1, NULL) ==
HAL_OK)
 #else
 #define ath_hal_getmcastkeysearch(_ah)  0
 #endif

I am using cards with an AR5212 which does seem to have
multiple bssid support working so I hope they should also
support mcastkeysearch capability. Maybe only some
firmware revisions have this?

Do you know what the status of the HAL code is for
supporting this? Why it is commented out in if_athvar.h?

-Russ