Mellanox Case # 00320058 : RE: Slow speed with Mellanox IS5023 Switch and Connect-X3 cards was updated [ref:_00D507heg._50050q9Npg:ref ]
Mellanox Support Admin
supportadmin at mellanox.com
Mon Dec 12 21:41:48 UTC 2016
Dear Vinícius Ferrão,
Your Mellanox support case # 00320058/ has been updated By , with the following comment:
/-------------------------------------------------------------
Hi Vinicius,
As a side note, for your CentOS servers running Mellanox OFED Driver, if they are configured as UD and attempt to talk to nodes configured as RC, the default mode on both side will automatically be as UD.
Regards,
Sophie.
-------------------------------------------------------------/
Thank You,
Click on the link to access the case:
https://mymellanox.force.com/support/5005000000q9Npg
Don’t forget to check out our Online Academy, register now to learn about the products and technology, http://academy.mellanox.com/login/index.php
/------ Your Case Comments History ------
-------------------------------------------------
Created by: Sophie Naudin Pruitt 2016-12-12 21:41:39
Hi Vinicius,
As a side note, for your CentOS servers running Mellanox OFED Driver, if they are configured as UD and attempt to talk to nodes configured as RC, the default mode on both side will automatically be as UD.
Regards,
Sophie.
-------------------------------------------------
2016-12-09 22:04:18
Comment By Customer: ferrao at vmio.com.br
Sophie,
I think we can safely close this issue. The main problem was with the UD mode of IPoIB. Since we’ve already changed all the machines on the cluster for IPoIB Connected Mode and with the properly MTU settings everything worked as expected.
It’s an annoyance being unable to change from CM to UD, but we will stay with this since we cannot have a better solution, and moving the server from FreeBSD to Linux is unpractical for us.
But just for the sake of completeness everything was working correctly on the CentOS side using the Mellanox bundled OFED.
Thanks for helping us out.
V.
> On 9 Dec 2016, at 17:22, Mellanox Support Admin <supportadmin at mellanox.com>
-------------------------------------------------
Created by: Sophie Naudin Pruitt 2016-12-09 19:22:48
Thank you.
What is the driver in use for the RHEL environment? Mellanox OFED Driver or OFED?
What is the mode configured on the RHEL environment?
Is this fabric will communicate between servers with FreeBSD & RHEL?
Regards,
Sophie.
-------------------------------------------------
2016-12-09 13:26:56
Comment By Customer: denis at versatushpc.com.br
it's FreeBSD 10.3/Linux CentOS 7.2 Environment.
D.
2016-12-08 22:22 GMT-02:00 Mellanox Support Admin <supportadmin at mellanox.com
>:
> Dear Vinícius Ferrão,
>
> Mellanox support case # 00320058 has been updated By , with the following
> comment:
> /-------------------------------------------------------------
> Hi Vinicius,
>
> Our Mellanox OFED Driver currently only supports FreeBSD 10.1 and we don't
> have any indication of future release at this time supporting later version
> of this OS.
> Keep in mind that you are using the software stack released as OFED
> (OpenFabrics Enterprise Distribution) driver.
>
> Does this environment has servers running FreeBSD only or do you have
> servers with different Linux flavors?
>
> Regards,
> Sophie.
> -------------------------------------------------------------/
>
> Thank You,
>
>
>
> You can access all your closed/Open cases, just go to
> http://mellanox.com/content/pages.php?pg=customer_support
>
> Click on the link to access the case:
>
> https://mymellanox.force.com/support/5005000000q9Npg
>
>
> Don’t forget to check out our Online Academy, register now to learn about
> the products and technology, http://academy.mellanox.com/login/index.php
> /------ Your Case Comments History ------
> -------------------------------------------------
> Created by: Sophie Naudin Pruitt 2016-12-09 00:22:26
> Hi Vinicius,
>
> Our Mellanox OFED Driver currently only supports FreeBSD 10.1 and we don't
> have any indication of future release at this time supporting later version
> of this OS.
> Keep in mind that you are using the software stack released as OFED
> (OpenFabrics Enterprise Distribution) driver.
>
> Does this environment has servers running FreeBSD only or do you have
> servers with different Linux flavors?
>
> Regards,
> Sophie.
>
> -------------------------------------------------
> 2016-12-08 00:33:13
> Comment By Customer: ferrao at vmio.com.br
> Hello Sophie,
>
> Recompiling the software isn’t an viable option.
>
> I was able to mitigate the issue on this specific case raising the MTU to
> 65520 on all machines. It turns out that the old machines were configured
> on Connected Mode using only 2044 MTU and an MTU mismatch between the
> server and the clients was the real issue.
>
> But I’m worried about the viability of Infiniband on FreeBSD. Changing
> from Connected Mode to Unreliable Datagram should be easier than
> recompiling the kernel module.
>
> There’s a way to ask Mellanox to change the drivers to make the switch
> from UD to CM viable on FreeBSD? A feature request perhaps?
>
> Thanks in advance,
> V.
>
> > On 7 Dec 2016, at 19:05, Mellanox Support Admin <
> supportadmin at mellanox.com>
>
> -------------------------------------------------
> Created by: Sophie Naudin Pruitt 2016-12-07 21:05:15
> Hi Vinicius,
>
> Have you seen my last statement posted to this case and what is the
> current status?
> Please let me know.
>
> Thank you,
> Sophie.
>
> -------------------------------------------------
> Created by: Sophie Naudin Pruitt 2016-12-05 23:02:35
> Hi Vinicius,
>
> Searching some forums, I found the following information:
>
> Building
-------------------------------------------------
Created by: Sophie Naudin Pruitt 2016-12-09 00:22:26
Hi Vinicius,
Our Mellanox OFED Driver currently only supports FreeBSD 10.1 and we don't have any indication of future release at this time supporting later version of this OS.
Keep in mind that you are using the software stack released as OFED (OpenFabrics Enterprise Distribution) driver.
Does this environment has servers running FreeBSD only or do you have servers with different Linux flavors?
Regards,
Sophie.
-------------------------------------------------
2016-12-08 00:33:13
Comment By Customer: ferrao at vmio.com.br
Hello Sophie,
Recompiling the software isn’t an viable option.
I was able to mitigate the issue on this specific case raising the MTU to 65520 on all machines. It turns out that the old machines were configured on Connected Mode using only 2044 MTU and an MTU mismatch between the server and the clients was the real issue.
But I’m worried about the viability of Infiniband on FreeBSD. Changing from Connected Mode to Unreliable Datagram should be easier than recompiling the kernel module.
There’s a way to ask Mellanox to change the drivers to make the switch from UD to CM viable on FreeBSD? A feature request perhaps?
Thanks in advance,
V.
> On 7 Dec 2016, at 19:05, Mellanox Support Admin <supportadmin at mellanox.com>
-------------------------------------------------
Created by: Sophie Naudin Pruitt 2016-12-07 21:05:15
Hi Vinicius,
Have you seen my last statement posted to this case and what is the current status?
Please let me know.
Thank you,
Sophie.
-------------------------------------------------
Created by: Sophie Naudin Pruitt 2016-12-05 23:02:35
Hi Vinicius,
Searching some forums, I found the following information:
Building OFED in FreeBSD
Kernel
•Add the following options to your KERNCONF to compile the necessary protocols/protocol options into your kernel:
options OFED # Infiniband protocol stack and support
options SDP # Sockets Direct Protocol for infiniband
options IPOIB_CM # Use connect mode ipoib >>>recompile without that option and the mode should be datagram.
Regards,
Sophie.
-------------------------------------------------
Created by: Sophie Naudin Pruitt 2016-12-05 22:53:06
Hi Vinícius,
You are not using Mellanox OFED Driver if you are running OFED (OpenFabrics Enterprise Distribution), what is the version and did you consult the UM?
With our driver, you can change the mode with:
echo datagram > /sys/class/net/ib0/mode sets the mode of ib0 to UD
echo connected > /sys/class/net/ib0/mode sets the mode ib0 to CM
Though FreeBSD does not have a /sys.
Regards,
Sophie.
-------------------------------------------------
Created by: Anderson Dias Silva 2016-12-05 20:29:35
Hi Benny,
I am Anderson Dias Silva (LATAM SE). Please, see below an important consideration of our customer.
###########################################
After some testing we discovered that was a mismatch between the client and the server on operating modes of IPoIB. One side was using Connected Mode (the FreeBSD one) and the Linux client was on Datagram mode).
So I’m considering running FreeBSD in UD (Datagram) mode to see if I finally can get rid of the problems. But, how can I change between CM and UD in FreeBSD? We don't have a /sys folder on FreeBSD so I cant just set from 0 to 1 and change modes.
###########################################
So, how can we change between CM and UD in FreeBSD?
Ragards!
-------------------------------------------------------/
More information about the freebsd-infiniband
mailing list