atheros broadcast/multicast corruption with multiple hostap's

[Sam Leffler]

Russell Yount wrote:
> On Sun, Jan 17, 2010 at 1:45 PM, Sam Leffler <sam at errno.com
> <mailto:sam at errno.com>> wrote:
> 
>     Russell Yount wrote:
> 
> 
> 
>         On Sat, Jan 16, 2010 at 3:21 PM, Sam Leffler <sam at errno.com
>         <mailto:sam at errno.com> <mailto:sam at errno.com
>         <mailto:sam at errno.com>>> wrote:
> 
>            Russell Yount wrote:
> 
>                It seems AP to client broadcasts/multicasts traffic is
>                broken when using WPA2/802.11i with multiple hostapds in 8.0.
> 
>                Only the SSID associated with the last hostapd to be
>         started has
>                AP to client broadcasts/multicasts being delivered correctly.
> 
>                The AP and client are 8.0 freebsd systems althought I see
>         same
>                problems with windows XP as a client.
> 
>                The AP has 4 hostapds configured to use TLS with client
>                certificates for
>                authentication. (hostapd recompiled with
>                HOSTAPD_CFLAGS=-DEAP_SERVER)
>                The AP and client radio are shown as ath0: AR5212 mac 5.9
>         RF5112
>                phy 4.3
>                in dmesg.
> 
>                Client authenticate using client certificates associate
>         correctly
>                to all 4 SSIDs. Unicast traffic flows correctly between
>         clients
>                and AP
>                for all for 4 SSIDs. Client to AP broadcast/multicast
>         traffic works
>                on of 4 SSIDs. AP to client broadcast/multicast traffic
>         only works
>                on 1 of the SSIDs. I have documented this using ARP
>         broadcasts,
>                but normal IP broadcasts also observed to corrupted.
> 
>                When an ARP request is send through the AP to an
>         associated client
>                it seems to be trashed on any of the SSID except the one
>         associated
>                with the last hostapd to be started. Here is the output of
>                client side
>                tcpdump showing the problems.
> 
>                In the first client side tcpdump with the hostapd associated
>                with the SSID
>                being associaed with the last hostapd started and the traffic
>                flowing
>                normally.
> 
>                In the second client side tcpdump with the hostapd associated
>                with the SSID
>                being not the last hostapd started the ARP request is resent
>                multiple times
>                and appears corrupted.
> 
>                I would really like to find a fix for this.
>                Any help would be greatly appreciated.
> 
> 
>            This sounds like the crypto encap of the frame is clobbering the
>            mbuf contents.  You can verify this by setting up multiple
>         vaps w/o
>            WPA.  If this is the problem look for the mbuf copy logic for
>         mcast
>            frames and make sure a deep copy is done.
> 
>                   Sam
> 
>            The four VAPs broadcast traffic works find without WPA if I
>         do not start hostapds on them
>          I have been trying to discovery why broadcast traffic only
>         works correctly on the VAP associated with the last hostapd to
>         be started. I have move with VAP has the working broadcast
>         traffic by restarting the hostapd
>         associated with it.
>          It would seem something in the WPA/802.1x layer initialization
>         remembers which hostapd was started last and that affected the
>         crypto encap.
>          I keep looking but do not see any place in the code that could
>         account for this.
>          It seems the corrupt crypto encap also happens on broadcast
>         between stations.
>         Please correct me if I am wrong:
>         but when using hostapd normally traffic is bridged withing the card.
>         So if a station sends to the VAP a broadcast it is actaully
>         sending a non- broadcast frame to the AP
>         and the AP sends the frame to all the other stations.
> 
> 
>     I told you waht the likely problem is.  Look in the net80211 layer
>     in the kernel for the problem.
> 
>            Sam
> 
>  
>  
>  
>  I tried to find problems in mbuf corruption
> in ieee80211_output.c by placing
>  
>   m = m_unshare(m,M_NOWAIT);
>   if (m == NULL) {
>       IEEE80211_DPRINTF(vap, IEEE80211_MSG_OUTPUT,
>       "%s: cannot get writable mbuf\n", __func__);
>       return NULL;
>   }
>  
> at begining ieee80211_mbuf_adjust() and at
> beginning of ieee80211_encap() with no change
> in the broadcast traffic behaviour.
> 
> I tried then to in ieee80211_crypto.c substituting
>  
>   flags |= IEEE80211_KEY_SWCRYPT;
>  
> for the encryption capabilities test code
>  
>   if ((ic->ic_cryptocaps & (1<<cipher)) == 0) {
>        IEEE80211_DPRINTF(vap, IEEE80211_MSG_CRYPTO,
>        "%s: no h/w support for cipher %s, falling back to s/w\n",
>            __func__, cip->ic_name);
>        flags |= IEEE80211_KEY_SWCRYPT;
>    }
>  
> to force all the encryption to be done in software.
>  
> This fixed the broadcast traffic problem but without
> hardware support its very slow and really loads machine.
> Enabling in the debug code to ath and net80211
> 
> and enabled ATH_DEBUG_KEYCACHE in if_ath.c and
> IEEE80211_MSG_CRYPTO in net80211 code.
>  
> It seems that all the VAPS sets the broadcast key for
> mac ff:ff:ff:ff:ff:ff in the ath device so I assume
> they conflict and the last one setting the key is the
> working one; that would explain why the last hostapd
> started is the only one with working broadcast code
> to clients.
>  
> In if_ath.c the code
>  
>  if ((k->wk_flags & IEEE80211_KEY_GROUP) && sc->sc_mcastkey) {
>    /*
>     * Group keys on hardware that supports multicast frame
>     * key search use a mac that is the sender's address with
>     * the high bit set instead of the app-specified address.
>     */
>      IEEE80211_ADDR_COPY(gmac, bss->ni_macaddr);
>      gmac[0] |= 0x80;
>      mac = gmac;
>  } else
>     mac = k->wk_macaddr;
>  
> seems to indicate that for multiple VAPs the ath chips
> needs to be able to distinguish between broadcast keys
> by using a permutation of VAPs bssid.
>  
> But in if_athvar.h the code does not seem complete
> and sc->sc_mcastkey is also set false.
>  
>  #ifdef notyet
>  #define ath_hal_hasmcastkeysearch(_ah) \
>         (ath_hal_getcapability(_ah, HAL_CAP_MCAST_KEYSRCH, 0, NULL) ==
> HAL_OK)
>  #define ath_hal_getmcastkeysearch(_ah) \
>         (ath_hal_getcapability(_ah, HAL_CAP_MCAST_KEYSRCH, 1, NULL) ==
> HAL_OK)
>  #else
>  #define ath_hal_getmcastkeysearch(_ah)  0
>  #endif
>  
> I am using cards with an AR5212 which does seem to have
> multiple bssid support working so I hope they should also
> support mcastkeysearch capability. Maybe only some
> firmware revisions have this?
>  
> Do you know what the status of the HAL code is for
> supporting this? Why it is commented out in if_athvar.h?

Good work analyzing things; been a long time since I looked at this.  I
vaguely recall disabling mcastkey searching because of problems with
WEP.  I'm surprised WPA is broken as that was a standard test case.

You can try enabling the notyet code and see if the right thing happens.
 I don't see any indication of the mac rev for your part but I expect it
supports this as it was only very early parts that had issues.

If enabling the mcastkey search mechanism doesn't fix this I might've
broken things with changes to explicitly mark group keys when hostapd
plumbs their contents.  I recall doing this for mwl which doesn't have
an indexed key table like ath (it uses the mac address of the local bss
and/or associated station to find the data structure where crypto keys
are stored).

I haven't looked at this stuff in a long time and can't setup a system
to test but if you keep pushing on this I'll try to help w/ advise.

	Sam