svn commit: r341586 - head/sys/dev/mlx5/mlx5_en

[Slava Shwartsman]

On 16-Apr-19 17:39, Andrey V. Elsukov wrote:
> On 05.12.2018 17:25, Slava Shwartsman wrote:
>> Author: slavash
>> Date: Wed Dec  5 14:25:03 2018
>> New Revision: 341586
>> URL: https://svnweb.freebsd.org/changeset/base/341586
>>
>> Log:
>>    mlx5en: Implement backpressure indication.
>>    
>>    The backpressure indication is implemented using an unlimited rate type of
>>    mbuf send tag. When the upper layers typically the socket layer has obtained such
>>    a tag, it can then query the destination driver queue for the current
>>    amount of space available in the send queue.
>>    
>>    A single mbuf send tag may be referenced multiple times and a refcount has been added
>>    to the mlx5e_priv structure to track its usage. Because the send tag resides
>>    in the mlx5e_channel structure, there is no need to wait for refcounts to reach
>>    zero until the mlx4en(4) driver is detached. The channels structure is persistant
>>    during the lifetime of the mlx5en(4) driver it belongs to and can so be accessed
>>    without any need of synchronization.
>>    
>>    The mlx5e_snd_tag structure was extended to contain a type field, because there are now
>>    two different tag types which end up in the driver which need to be distinguished.
>>    
>>    Submitted by:   hselasky@
>>    Approved by:    hselasky (mentor)
>>    MFC after:      1 week
>>    Sponsored by:   Mellanox Technologies
>> @@ -587,27 +609,33 @@ mlx5e_xmit(struct ifnet *ifp, struct mbuf *mb)
>>   	struct mlx5e_sq *sq;
>>   	int ret;
>>   
>> -	sq = mlx5e_select_queue(ifp, mb);
>> -	if (unlikely(sq == NULL)) {
>> -#ifdef RATELIMIT
>> -		/* Check for route change */
>> -		if (mb->m_pkthdr.snd_tag != NULL &&
>> -		    mb->m_pkthdr.snd_tag->ifp != ifp) {
>> +	if (mb->m_pkthdr.snd_tag != NULL) {
>> +		sq = mlx5e_select_queue_by_send_tag(ifp, mb);
>> +		if (unlikely(sq == NULL)) {
>> +			/* Check for route change */
>> +			if (mb->m_pkthdr.snd_tag->ifp != ifp) {
>> +				/* Free mbuf */
>> +				m_freem(mb);
>> +
>> +				/*
>> +				 * Tell upper layers about route
>> +				 * change and to re-transmit this
>> +				 * packet:
>> +				 */
>> +				return (EAGAIN);
>> +			}
> 
> Hi,
> 
> I just discovered something strange and found that this commit is the
> cause.
> The test system has mlx5en 100G interface. It has two vlans: vlan500 and
> vlan100.
> Via vlan500 it receives some packets flows. Then it routes these packets
> into vlan100.
> But packets are dropped in mlx5e_xmit() with EAGAIN error code.
> 
> # dtrace -n 'fbt::ip6_output:return {printf("%d", arg1);}'
> dtrace: description 'fbt::ip6_output:return ' matched 1 probe
> CPU     ID                    FUNCTION:NAME
>   23  54338                ip6_output:return 35
>   16  54338                ip6_output:return 35
>   21  54338                ip6_output:return 35
>   22  54338                ip6_output:return 35
>   24  54338                ip6_output:return 35
>   23  54338                ip6_output:return 35
>   14  54338                ip6_output:return 35
> ^C
> 
> # dtrace -n 'fbt::mlx5e_xmit:return {printf("%d", arg1);}'
> dtrace: description 'fbt::mlx5e_xmit:return ' matched 1 probe
> CPU     ID                    FUNCTION:NAME
>   16  69030                mlx5e_xmit:return 35
>   23  69030                mlx5e_xmit:return 35
>   26  69030                mlx5e_xmit:return 35
>   25  69030                mlx5e_xmit:return 35
>   24  69030                mlx5e_xmit:return 35
>   21  69030                mlx5e_xmit:return 35
>   26  69030                mlx5e_xmit:return 35
> ^C
> 
> The kernel config is GENERIC.
> 13.0-CURRENT #9 r345758+82f3d57(svn_head)-dirty
> 

Hi Andrey,

Thanks for letting us know about this regression.
I would like to try to reproduce this issue in house.

Can you please share the exact steps to reproduce it?
- Can I reproduce the issue with B2B setup?
- What is the route command you used to make the route between the VLANs?
- What app are you using to generate the traffic?


Slava