Mellanox Case # 00320058 : RE: Slow speed with Mellanox IS5023 Switch and Connect-X3 cards was updated [ref:_00D507heg._50050q9Npg:ref ]
Mellanox Support Admin
supportadmin at mellanox.com
Fri Dec 9 13:32:06 UTC 2016
Dear Vinícius Ferrão,
Your Mellanox support case # 00320058/ has been updated By , with the following comment:
/-------------------------------------------------------------
Comment By Customer: denis at versatushpc.com.br
it's FreeBSD 10.3/Linux CentOS 7.2 Environment.
D.
2016-12-08 22:22 GMT-02:00 Mellanox Support Admin <supportadmin at mellanox.com
>:
> Dear Vinícius Ferrão,
>
> Mellanox support case # 00320058 has been updated By , with the following
> comment:
> /-------------------------------------------------------------
> Hi Vinicius,
>
> Our Mellanox OFED Driver currently only supports FreeBSD 10.1 and we don't
> have any indication of future release at this time supporting later version
> of this OS.
> Keep in mind that you are using the software stack released as OFED
> (OpenFabrics Enterprise Distribution) driver.
>
> Does this environment has servers running FreeBSD only or do you have
> servers with different Linux flavors?
>
> Regards,
> Sophie.
> -------------------------------------------------------------/
>
> Thank You,
>
>
>
> You can access all your closed/Open cases, just go to
> http://mellanox.com/content/pages.php?pg=customer_support
>
> Click on the link to access the case:
>
> https://mymellanox.force.com/support/5005000000q9Npg
>
>
> Don’t forget to check out our Online Academy, register now to learn about
> the products and technology, http://academy.mellanox.com/login/index.php
> /------ Your Case Comments History ------
> -------------------------------------------------
> Created by: Sophie Naudin Pruitt 2016-12-09 00:22:26
> Hi Vinicius,
>
> Our Mellanox OFED Driver currently only supports FreeBSD 10.1 and we don't
> have any indication of future release at this time supporting later version
> of this OS.
> Keep in mind that you are using the software stack released as OFED
> (OpenFabrics Enterprise Distribution) driver.
>
> Does this environment has servers running FreeBSD only or do you have
> servers with different Linux flavors?
>
> Regards,
> Sophie.
>
> -------------------------------------------------
> 2016-12-08 00:33:13
> Comment By Customer: ferrao at vmio.com.br
> Hello Sophie,
>
> Recompiling the software isn’t an viable option.
>
> I was able to mitigate the issue on this specific case raising the MTU to
> 65520 on all machines. It turns out that the old machines were configured
> on Connected Mode using only 2044 MTU and an MTU mismatch between the
> server and the clients was the real issue.
>
> But I’m worried about the viability of Infiniband on FreeBSD. Changing
> from Connected Mode to Unreliable Datagram should be easier than
> recompiling the kernel module.
>
> There’s a way to ask Mellanox to change the drivers to make the switch
> from UD to CM viable on FreeBSD? A feature request perhaps?
>
> Thanks in advance,
> V.
>
> > On 7 Dec 2016, at 19:05, Mellanox Support Admin <
> supportadmin at mellanox.com>
>
> -------------------------------------------------
> Created by: Sophie Naudin Pruitt 2016-12-07 21:05:15
> Hi Vinicius,
>
> Have you seen my last statement posted to this case and what is the
> current status?
> Please let me know.
>
> Thank you,
> Sophie.
>
> -------------------------------------------------
> Created by: Sophie Naudin Pruitt 2016-12-05 23:02:35
> Hi Vinicius,
>
> Searching some forums, I found the following information:
>
> Building
-------------------------------------------------------------/
Thank You,
Click on the link to access the case:
https://mymellanox.force.com/support/5005000000q9Npg
Don’t forget to check out our Online Academy, register now to learn about the products and technology, http://academy.mellanox.com/login/index.php
/------ Your Case Comments History ------
-------------------------------------------------
2016-12-09 13:26:56
Comment By Customer: denis at versatushpc.com.br
it's FreeBSD 10.3/Linux CentOS 7.2 Environment.
D.
2016-12-08 22:22 GMT-02:00 Mellanox Support Admin <supportadmin at mellanox.com
>:
> Dear Vinícius Ferrão,
>
> Mellanox support case # 00320058 has been updated By , with the following
> comment:
> /-------------------------------------------------------------
> Hi Vinicius,
>
> Our Mellanox OFED Driver currently only supports FreeBSD 10.1 and we don't
> have any indication of future release at this time supporting later version
> of this OS.
> Keep in mind that you are using the software stack released as OFED
> (OpenFabrics Enterprise Distribution) driver.
>
> Does this environment has servers running FreeBSD only or do you have
> servers with different Linux flavors?
>
> Regards,
> Sophie.
> -------------------------------------------------------------/
>
> Thank You,
>
>
>
> You can access all your closed/Open cases, just go to
> http://mellanox.com/content/pages.php?pg=customer_support
>
> Click on the link to access the case:
>
> https://mymellanox.force.com/support/5005000000q9Npg
>
>
> Don’t forget to check out our Online Academy, register now to learn about
> the products and technology, http://academy.mellanox.com/login/index.php
> /------ Your Case Comments History ------
> -------------------------------------------------
> Created by: Sophie Naudin Pruitt 2016-12-09 00:22:26
> Hi Vinicius,
>
> Our Mellanox OFED Driver currently only supports FreeBSD 10.1 and we don't
> have any indication of future release at this time supporting later version
> of this OS.
> Keep in mind that you are using the software stack released as OFED
> (OpenFabrics Enterprise Distribution) driver.
>
> Does this environment has servers running FreeBSD only or do you have
> servers with different Linux flavors?
>
> Regards,
> Sophie.
>
> -------------------------------------------------
> 2016-12-08 00:33:13
> Comment By Customer: ferrao at vmio.com.br
> Hello Sophie,
>
> Recompiling the software isn’t an viable option.
>
> I was able to mitigate the issue on this specific case raising the MTU to
> 65520 on all machines. It turns out that the old machines were configured
> on Connected Mode using only 2044 MTU and an MTU mismatch between the
> server and the clients was the real issue.
>
> But I’m worried about the viability of Infiniband on FreeBSD. Changing
> from Connected Mode to Unreliable Datagram should be easier than
> recompiling the kernel module.
>
> There’s a way to ask Mellanox to change the drivers to make the switch
> from UD to CM viable on FreeBSD? A feature request perhaps?
>
> Thanks in advance,
> V.
>
> > On 7 Dec 2016, at 19:05, Mellanox Support Admin <
> supportadmin at mellanox.com>
>
> -------------------------------------------------
> Created by: Sophie Naudin Pruitt 2016-12-07 21:05:15
> Hi Vinicius,
>
> Have you seen my last statement posted to this case and what is the
> current status?
> Please let me know.
>
> Thank you,
> Sophie.
>
> -------------------------------------------------
> Created by: Sophie Naudin Pruitt 2016-12-05 23:02:35
> Hi Vinicius,
>
> Searching some forums, I found the following information:
>
> Building
-------------------------------------------------
Created by: Sophie Naudin Pruitt 2016-12-09 00:22:26
Hi Vinicius,
Our Mellanox OFED Driver currently only supports FreeBSD 10.1 and we don't have any indication of future release at this time supporting later version of this OS.
Keep in mind that you are using the software stack released as OFED (OpenFabrics Enterprise Distribution) driver.
Does this environment has servers running FreeBSD only or do you have servers with different Linux flavors?
Regards,
Sophie.
-------------------------------------------------
2016-12-08 00:33:13
Comment By Customer: ferrao at vmio.com.br
Hello Sophie,
Recompiling the software isn’t an viable option.
I was able to mitigate the issue on this specific case raising the MTU to 65520 on all machines. It turns out that the old machines were configured on Connected Mode using only 2044 MTU and an MTU mismatch between the server and the clients was the real issue.
But I’m worried about the viability of Infiniband on FreeBSD. Changing from Connected Mode to Unreliable Datagram should be easier than recompiling the kernel module.
There’s a way to ask Mellanox to change the drivers to make the switch from UD to CM viable on FreeBSD? A feature request perhaps?
Thanks in advance,
V.
> On 7 Dec 2016, at 19:05, Mellanox Support Admin <supportadmin at mellanox.com>
-------------------------------------------------
Created by: Sophie Naudin Pruitt 2016-12-07 21:05:15
Hi Vinicius,
Have you seen my last statement posted to this case and what is the current status?
Please let me know.
Thank you,
Sophie.
-------------------------------------------------
Created by: Sophie Naudin Pruitt 2016-12-05 23:02:35
Hi Vinicius,
Searching some forums, I found the following information:
Building OFED in FreeBSD
Kernel
•Add the following options to your KERNCONF to compile the necessary protocols/protocol options into your kernel:
options OFED # Infiniband protocol stack and support
options SDP # Sockets Direct Protocol for infiniband
options IPOIB_CM # Use connect mode ipoib >>>recompile without that option and the mode should be datagram.
Regards,
Sophie.
-------------------------------------------------
Created by: Sophie Naudin Pruitt 2016-12-05 22:53:06
Hi Vinícius,
You are not using Mellanox OFED Driver if you are running OFED (OpenFabrics Enterprise Distribution), what is the version and did you consult the UM?
With our driver, you can change the mode with:
echo datagram > /sys/class/net/ib0/mode sets the mode of ib0 to UD
echo connected > /sys/class/net/ib0/mode sets the mode ib0 to CM
Though FreeBSD does not have a /sys.
Regards,
Sophie.
-------------------------------------------------
Created by: Anderson Dias Silva 2016-12-05 20:29:35
Hi Benny,
I am Anderson Dias Silva (LATAM SE). Please, see below an important consideration of our customer.
###########################################
After some testing we discovered that was a mismatch between the client and the server on operating modes of IPoIB. One side was using Connected Mode (the FreeBSD one) and the Linux client was on Datagram mode).
So I’m considering running FreeBSD in UD (Datagram) mode to see if I finally can get rid of the problems. But, how can I change between CM and UD in FreeBSD? We don't have a /sys folder on FreeBSD so I cant just set from 0 to 1 and change modes.
###########################################
So, how can we change between CM and UD in FreeBSD?
Ragards!
-------------------------------------------------
Created by: Benny Filin 2016-12-05 01:03:55
Hello Vinícius Ferrão,
Thank you for contacting Mellanox Technical Support.
1) DId you try changing switch ports when testing performance between two FreeBSD machines through switch?
2) What happens when you try a back-to-back connection between two FreeBSD machines? Do you get good results?
3) Did you test the performance between FreeBSD and a Linux machine through switch?
Please, follow the provided procedure in order to query your unmanaged switch:
1) Download and install MFT (Mellanox Firmware Tool):
http://www.mellanox.com/content/pages.php?pg=management_tools&menu_section=34
Refer to the User Manual for the installation instructions:
http://www.mellanox.com/pdf/MFT/MFT_user_manual.pdf
2) From one of the nodes in the fabric, invoke: ibswitches.
Match the relevant switch GUID from the Mellanox switch label to the LID.
3) From one of the nodes in the fabric , invoke: mst start -> mst ib add -> mst status
4) Identify the relevant unmanaged switch mst representation based on the switch LID (from previous step 2).
* Matching the remote switch mst device representation to the relevant switch procedure:
Locate the switch in the "ibswitches" output and retrieve the switch LID.
Convert the LID from Decimal to Hexadecimal base.
Locate the device with this LID (in hex base) from the "mst status" output.
5) Invoke the following command, and provide the output:
flint -d <remote_switch_mst_device> q
We're looking forward to hearing from you.
Best regards,
Mellanox Technical Support
-------------------------------------------------
Created by: Jubran Jubran 2016-12-01 16:41:35
Hello Vinícius Ferrão,
Thank you for contacting Mellanox Technical Support.
1) DId you try changing switch ports when testing performance between two FreeBSD machines through switch?
2) What happens when you try a back-to-back connection between two FreeBSD machines? Do you get good results?
3) Did you test the performance between FreeBSD and a Linux machine through switch?
Please, follow the provided procedure in order to query your unmanaged switch:
1) Download and install MFT (Mellanox Firmware Tool):
http://www.mellanox.com/content/pages.php?pg=management_tools&menu_section=34
Refer to the User Manual for the installation instructions:
http://www.mellanox.com/pdf/MFT/MFT_user_manual.pdf
2) From one of the nodes in the fabric, invoke: ibswitches.
Match the relevant switch GUID from the Mellanox switch label to the LID.
3) From one of the nodes in the fabric , invoke: mst start -> mst ib add -> mst status
4) Identify the relevant unmanaged switch mst representation based on the switch LID (from previous step 2).
* Matching the remote switch mst device representation to the relevant switch procedure:
Locate the switch in the "ibswitches" output and retrieve the switch LID.
Convert the LID from Decimal to Hexadecimal base.
Locate the device with this LID (in hex base) from the "mst status" output.
5) Invoke the following command, and provide the output:
flint -d <remote_switch_mst_device> q
We're looking forward to hearing from you.
Best regards,
Mellanox Technical Support
-------------------------------------------------------/
More information about the freebsd-infiniband
mailing list