InfiniBand supported hw

[John Nielsen]

> On May 26, 2016, at 12:33 PM, Grzegorz Junka <list1 at gjunka.com> wrote:
> 
> 
>> On 26/05/2016 17:44, John Nielsen wrote:
>>> On May 26, 2016, at 11:07 AM, Justin Clift <justin at postgresql.org> wrote:
>>> 
>>>> On 26 May 2016, at 14:00, Grzegorz Junka <list1 at gjunka.com> wrote:
>>>> Is 500EX-D supported on FreeBSD? I read that it's a Voltaire version and it is the same as Mellanox ConnectX, which supposed to be supported.
>>> Yep, they should work.  You may need to flash them with their Mellanox equivalent
>>> firmware, but I'm not sure.  (doing that flash isn't very hard btw)
>>> 
>>>> Also, most InfiniBand cards provide two ports. When connecting two computers together (IPoIB) is it possible to use both ports to double the speed?
>>> Not sure, as I've not needed to personally.  None of my storage (or other uses)
>>> even fills one DDR port.  Yet. :)
>>> 
>>> I *think* the answer to that is yes,
>> It depends on what you mean by "use both ports." You can of course use both ports at the same time, but if you're envisioning something like an IPoIB version of an Ethernet LAGG I don't think that is supported.
>> 
>> If you are using the IB for storage protocols, then active/active multipathing is one way to utilize the full bandwidth of multiple ports.
>> 
>>>> Lastly, is NFS RDMA properly supported on FreeBSD?
>>> Again not sure.  Other people here would likely know. :)
> 
> Yes, that's exactly what I was hoping for, a sort of LAGG to double the speed of the link between computer by utilizing both ports together. LAGG is independent from the network interface, if I could set up an IPoIB NIC for each port separately what could prevent me from creating a LAGG interface on top of them? And if that's not supported, what do people usually do with the other interface? Just leave it unconnected?

LAGG is designed to work with Ethernet. InfiniBand is a very different layer 2 technology with its own advantages and disadvantages. I don't want to get too far out of my depth so I'll just suggest you do your own research on the subject.

How you use the second port is entirely up to you. You can use it for separate applications, for redundancy or higher throughput. In my experience it usually makes sense to give it its own IP. If you want higher bandwidth than a single port can provide then you should at layer 3 or higher for ways to aggregate bandwidth. (You should also do some research on PCI-e throughput. A x8 PCI-e 3.0 slot maxes out well before 112Gbit/s.)

As I mentioned previously you can use active/active multipath to get more throughput for block storage applications. You could also run a routing daemon and do ECMP for any layer 3 traffic. Or your particular application may have other ways to intelligently use two distinct IPoIB paths.