Re: git: cdcd52d41e24 - main - irdma: Add RDMA driver for Intel(R) Ethernet Controller E810
Date: Fri, 09 Dec 2022 01:18:57 UTC
On 12/8/22 5:00 PM, Eric Joyner wrote:
> Yeah, I was able to repro it and get what I think is the error in a
> nicer-to-read set of lines:
>
> Building
> /root/build/head/obj/root/build/head/src/amd64.amd64/lib/ofed/libirdma/libirdma.a
>
> building static irdma library
>
> Building
> /root/build/head/obj/root/build/head/src/amd64.amd64/lib/ofed/libirdma/libirdma.so.1.full
> building shared library libirdma.so.1
>
> /usr/local/bin/x86_64-unknown-freebsd14.0-ld: duplicate expression `*' in
> version information
> collect2: error: ld returned 1 exit status
>
> *** [libirdma.so.1.full] Error code 1
>
>
> What does that even mean? I don't know anything about building libraries.
For libraries with versioned symbols, we use a separate file that lists the
versions assigned to exported symbols. You can find several examples in our
tree named Symbol.map. There are also several examples you can look at in
the ofed tree:
> find contrib/ofed/ -name '*.map'
contrib/ofed/libmlx5/libmlx5.map
contrib/ofed/libibumad/libibumad.map
contrib/ofed/libibnetdisc/libibnetdisc.map
contrib/ofed/opensm/libvendor/libosmvendor.map
contrib/ofed/opensm/complib/libosmcomp.map
contrib/ofed/opensm/opensm/libopensm.map
contrib/ofed/libibverbs/libibverbs.map
contrib/ofed/librdmacm/librdmacm.map
contrib/ofed/librdmacm/librspreload.map
contrib/ofed/libibcm/libibcm.map
contrib/ofed/libibmad/libibmad.map
contrib/ofed/libirdma/libirdma.map
The difference here is that the maps for other libraries explicitly list
the symbols being exported and what versions to export them under, but
the map for libirdma just uses the global '*' wildcard in multiple places,
which seems wrong.
> cat contrib/ofed/libirdma/libirdma.map
/* Export symbols should be added below according to
Documentation/versioning.md document. */
IRDMA_1.0 {
global: *;
local: *;
};
IRDMA_1.1 {
global: *;
} IRDMA_1.0;
Contrast this with the map for libmlx5:
> cat contrib/ofed/libmlx5/libmlx5.map
/* Export symbols should be added below according to
Documentation/versioning.md document. */
MLX5_1.0 {
global:
mlx5dv_query_device;
mlx5dv_init_obj;
local: *;
};
MLX5_1.1 {
global:
mlx5dv_create_cq;
} MLX5_1.0;
Here the wildcard is only used for the local rule (which I assume
is for private symbols not really exported?) Looking at the library
linked with lld, lld seems to be assigning all of the symbols to
the first namespace (IRDMA_1.0) and ignoring the duplicates. I suspect
that's probably an lld bug. A proper map should be explicitly listing
which symbols to export (and only exporting those).
I can't find a corresponding .map file for Linux anywhere (has this
library been upstreamed anywhere else?) The simplest fix might be
to just explicitly list the intended public symbols under version
1.0 and delete the clause for version 1.1 for now. I suspect you
weren't meaning to export '_init' and '_fini' as public symbols at
least. Here is the current list of exported symbols:
> readelf -s /usr/lib/libirdma.so | grep -v UND
Symbol table (.dynsym) contains 117 entries:
Num: Value Size Type Bind Vis Ndx Name
56: 0000000000008440 968 FUNC GLOBAL DEFAULT 14 irdma_uresize_cq@@IRDMA_1.0 (2)
57: 0000000000007c40 59 FUNC GLOBAL DEFAULT 14 irdma_uquery_qp@@IRDMA_1.0 (2)
58: 0000000000008430 10 FUNC GLOBAL DEFAULT 14 irdma_udetach_mcast@@IRDMA_1.0 (2)
59: 000000000000a6c0 35 FUNC GLOBAL DEFAULT 14 irdma_uk_cq_set_resized_cnt@@IRDMA_1.0 (2)
60: 000000000000b110 84 FUNC GLOBAL DEFAULT 14 irdma_get_sqdepth@@IRDMA_1.0 (2)
61: 000000000000b4b0 107 FUNC GLOBAL DEFAULT 14 irdma_uk_clean_cq@@IRDMA_1.0 (2)
62: 00000000000058f0 264 FUNC GLOBAL DEFAULT 14 irdma_ubind_mw@@IRDMA_1.0 (2)
63: 0000000000008a20 102 FUNC GLOBAL DEFAULT 14 irdma_uk_qp_post_wr@@IRDMA_1.0 (2)
64: 000000000000a690 33 FUNC GLOBAL DEFAULT 14 irdma_uk_cq_resize@@IRDMA_1.0 (2)
65: 0000000000008d90 123 FUNC GLOBAL DEFAULT 14 irdma_qp_get_next_recv_wqe@@IRDMA_1.0 (2)
66: 0000000000005680 154 FUNC GLOBAL DEFAULT 14 irdma_ualloc_pd@@IRDMA_1.0 (2)
67: 0000000000006030 80 FUNC GLOBAL DEFAULT 14 irdma_ucreate_cq@@IRDMA_1.0 (2)
68: 0000000000008310 213 FUNC GLOBAL DEFAULT 14 irdma_ucreate_ah@@IRDMA_1.0 (2)
69: 000000000000b0b0 89 FUNC GLOBAL DEFAULT 14 irdma_get_wqe_shift@@IRDMA_1.0 (2)
70: 000000000000b6b0 35 FUNC GLOBAL DEFAULT 14 irdma_fragcnt_to_wqesize_rq@@IRDMA_1.0 (2)
71: 0000000000005fd0 85 FUNC GLOBAL DEFAULT 14 irdma_udealloc_mw@@IRDMA_1.0 (2)
72: 0000000000007390 38 FUNC GLOBAL DEFAULT 14 irdma_munmap@@IRDMA_1.0 (2)
73: 000000000000b1c0 658 FUNC GLOBAL DEFAULT 14 irdma_uk_qp_init@@IRDMA_1.0 (2)
74: 0000000000008b80 513 FUNC GLOBAL DEFAULT 14 irdma_qp_get_next_send_wqe@@IRDMA_1.0 (2)
75: 0000000000005810 36 FUNC GLOBAL DEFAULT 14 irdma_udereg_mr@@IRDMA_1.0 (2)
76: 0000000000009e10 657 FUNC GLOBAL DEFAULT 14 irdma_uk_inline_send@@IRDMA_1.0 (2)
77: 000000000000b460 79 FUNC GLOBAL DEFAULT 14 irdma_uk_cq_init@@IRDMA_1.0 (2)
78: 0000000000007330 94 FUNC GLOBAL DEFAULT 14 irdma_mmap@@IRDMA_1.0 (2)
79: 000000000000a4c0 453 FUNC GLOBAL DEFAULT 14 irdma_uk_post_receive@@IRDMA_1.0 (2)
80: 00000000000089d0 73 FUNC GLOBAL DEFAULT 14 irdma_clr_wqes@@IRDMA_1.0 (2)
81: 0000000000008420 10 FUNC GLOBAL DEFAULT 14 irdma_uattach_mcast@@IRDMA_1.0 (2)
82: 0000000000009270 35 FUNC GLOBAL DEFAULT 14 irdma_fragcnt_to_quanta_sq@@IRDMA_1.0 (2)
83: 0000000000007230 153 FUNC GLOBAL DEFAULT 14 irdma_uarm_cq@@IRDMA_1.0 (2)
84: 00000000000065f0 770 FUNC GLOBAL DEFAULT 14 irdma_flush_thread@@IRDMA_1.0 (2)
85: 000000000000b170 78 FUNC GLOBAL DEFAULT 14 irdma_get_rqdepth@@IRDMA_1.0 (2)
86: 0000000000007f60 334 FUNC GLOBAL DEFAULT 14 irdma_udestroy_qp@@IRDMA_1.0 (2)
87: 00000000000054d0 227 FUNC GLOBAL DEFAULT 14 irdma_uquery_device_ex@@IRDMA_1.0 (2)
88: 000000000000a6f0 81 FUNC GLOBAL DEFAULT 14 irdma_uk_cq_request_notification@@IRDMA_1.0 (2)
89: 000000000000b95c 0 FUNC GLOBAL DEFAULT 16 _fini@@IRDMA_1.0 (2)
90: 0000000000006a70 78 FUNC GLOBAL DEFAULT 14 irdma_upoll_cq@@IRDMA_1.0 (2)
91: 0000000000007c80 723 FUNC GLOBAL DEFAULT 14 irdma_umodify_qp@@IRDMA_1.0 (2)
92: 0000000000009750 1099 FUNC GLOBAL DEFAULT 14 irdma_uk_send@@IRDMA_1.0 (2)
93: 000000000000e1e0 9 OBJECT GLOBAL DEFAULT 25 libirdma_version@@IRDMA_1.0 (2)
94: 0000000000008e10 1112 FUNC GLOBAL DEFAULT 14 irdma_uk_rdma_write@@IRDMA_1.0 (2)
95: 0000000000009ba0 620 FUNC GLOBAL DEFAULT 14 irdma_uk_inline_rdma_write@@IRDMA_1.0 (2)
96: 000000000000a2b0 524 FUNC GLOBAL DEFAULT 14 irdma_uk_mw_bind@@IRDMA_1.0 (2)
97: 000000000000a750 1808 FUNC GLOBAL DEFAULT 14 irdma_uk_cq_poll_cmpl@@IRDMA_1.0 (2)
98: 0000000000008a90 229 FUNC GLOBAL DEFAULT 14 irdma_qp_push_wqe@@IRDMA_1.0 (2)
99: 000000000000b94c 0 FUNC GLOBAL DEFAULT 15 _init@@IRDMA_1.0 (2)
100: 0000000000005720 36 FUNC GLOBAL DEFAULT 14 irdma_ufree_pd@@IRDMA_1.0 (2)
101: 00000000000055c0 119 FUNC GLOBAL DEFAULT 14 irdma_uquery_device@@IRDMA_1.0 (2)
102: 00000000000072d0 91 FUNC GLOBAL DEFAULT 14 irdma_cq_event@@IRDMA_1.0 (2)
103: 000000000000a0b0 512 FUNC GLOBAL DEFAULT 14 irdma_uk_stag_local_invalidate@@IRDMA_1.0 (2)
104: 0000000000006f10 209 FUNC GLOBAL DEFAULT 14 irdma_ibvcq_ex_fill_priv_funcs@@IRDMA_1.0 (2)
105: 000000000000e4a0 4 OBJECT GLOBAL DEFAULT 27 irdma_dbg@@IRDMA_1.0 (2)
106: 0000000000005640 59 FUNC GLOBAL DEFAULT 14 irdma_uquery_port@@IRDMA_1.0 (2)
107: 0000000000006900 367 FUNC GLOBAL DEFAULT 14 irdma_udestroy_cq@@IRDMA_1.0 (2)
108: 0000000000005a00 1477 FUNC GLOBAL DEFAULT 14 irdma_upost_send@@IRDMA_1.0 (2)
109: 00000000000083f0 36 FUNC GLOBAL DEFAULT 14 irdma_udestroy_ah@@IRDMA_1.0 (2)
110: 0000000000005840 161 FUNC GLOBAL DEFAULT 14 irdma_ualloc_mw@@IRDMA_1.0 (2)
111: 00000000000065c0 40 FUNC GLOBAL DEFAULT 14 irdma_ucreate_cq_ex@@IRDMA_1.0 (2)
112: 0000000000005750 179 FUNC GLOBAL DEFAULT 14 irdma_ureg_mr@@IRDMA_1.0 (2)
113: 00000000000073c0 1179 FUNC GLOBAL DEFAULT 14 irdma_ucreate_qp@@IRDMA_1.0 (2)
114: 00000000000092a0 1189 FUNC GLOBAL DEFAULT 14 irdma_uk_rdma_read@@IRDMA_1.0 (2)
115: 00000000000080b0 604 FUNC GLOBAL DEFAULT 14 irdma_upost_recv@@IRDMA_1.0 (2)
116: 000000000000b520 385 FUNC GLOBAL DEFAULT 14 irdma_nop@@IRDMA_1.0 (2)
>
> - Eric
>
> On Wed, Dec 7, 2022 at 1:05 PM John Baldwin <jhb@freebsd.org> wrote:
>
>> On 6/13/22 2:18 PM, Eric Joyner wrote:
>>> On Fri, Jun 10, 2022 at 3:29 PM John Baldwin <jhb@freebsd.org> wrote:
>>>
>>>> On 5/23/22 4:54 PM, Eric Joyner wrote:
>>>>> The branch main has been updated by erj:
>>>>>
>>>>> URL:
>>>>
>> https://cgit.FreeBSD.org/src/commit/?id=cdcd52d41e246ba1c0fcfad0769bd691487355ef
>>>>>
>>>>> commit cdcd52d41e246ba1c0fcfad0769bd691487355ef
>>>>> Author: Bartosz Sobczak <bartosz.sobczak@intel.com>
>>>>> AuthorDate: 2022-05-23 23:39:27 +0000
>>>>> Commit: Eric Joyner <erj@FreeBSD.org>
>>>>> CommitDate: 2022-05-23 23:52:49 +0000
>>>>>
>>>>> irdma: Add RDMA driver for Intel(R) Ethernet Controller E810
>>>>>
>>>>> This is an initial commit for RDMA FreeBSD driver for Intel(R)
>>>> Ethernet
>>>>> Controller E810, called irdma. Supporting both RoCEv2 and iWARP
>>>>> protocols in per-PF manner, RoCEv2 being the default.
>>>>>
>>>>> Testing has been done using krping tool, perftest, ucmatose,
>> rping,
>>>>> ud_pingpong, rc_pingpong and others.
>>>>>
>>>>> Signed-off-by: Eric Joyner <erj@FreeBSD.org>
>>>>>
>>>>> Reviewed by: #manpages (pauamma_gundo.com) [documentation]
>>>>> MFC after: 1 week
>>>>> Relnotes: yes
>>>>> Sponsored by: Intel Corporation
>>>>> Differential Revision: https://reviews.freebsd.org/D34690
>>>>
>>>> I'm trying to get the tree building again with GCC and ld.bfd does not
>> like
>>>> libirdma.map at all complaining with an error:
>>>>
>>>> duplicate expression '*' in version information
>>>>
>>>> I looked at the version map, and it seems rather bogus to me and not
>> like
>>>> version maps used anywhere else. It seems to be exporting all symbols
>> in
>>>> two different versions? That would seem to defeat the point of having
>>>> symbol versions?
>>>>
>>>> /* Export symbols should be added below according to
>>>> Documentation/versioning.md document. */
>>>> IRDMA_1.0 {
>>>> global: *;
>>>> local: *;
>>>> };
>>>>
>>>> IRDMA_1.1 {
>>>> global: *;
>>>> } IRDMA_1.0;
>>>>
>>>> (I'm also not sure why lld accepts such a version map.)
>>>>
>>>> --
>>>> John Baldwin
>>>>
>>>
>>> Hi,
>>>
>>> I don't know how to answer this; I'm going to forward this to the FreeBSD
>>> RDMA guy and another one (the first guy might still be out on vacation)
>> and
>>> see if they can answer this.
>>
>> FYI, this is now the current breakage for the new GCC 12 job in FreeBSD's
>> CI:
>>
>> https://ci.freebsd.org/job/FreeBSD-main-amd64-gcc12_build/47/console
>>
>> The error is buried in the log there, but should be obtainable by doing
>> 'pkg install amd64-gcc12' and 'make CROSS_TOOLCHAIN=amd64-gcc12 buildworld'
>> on main.
>>
>> --
>> John Baldwin
>>
>>
>
--
John Baldwin