From nobody Fri Apr 15 06:25:07 2022 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 622F41B4FC37 for ; Fri, 15 Apr 2022 06:25:21 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic306-19.consmr.mail.gq1.yahoo.com (sonic306-19.consmr.mail.gq1.yahoo.com [98.137.68.82]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange ECDHE (P-256) server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4KfmXc2XDJz3Pv2 for ; Fri, 15 Apr 2022 06:25:20 +0000 (UTC) (envelope-from marklmi@yahoo.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1650003913; bh=wyLEZMtOZHkxx34c7nz7QW0kbF3L5XhROZ98aLmOCkQ=; h=From:Subject:Date:To:References:From:Subject:Reply-To; b=JE7YCaRP7naFcN11NTuyx3Se1klqpunM5O/JlLPT9eMkTpxn1+5FaMgg9sNFE79/V5uQKEKzW4Eh6cdCwn3WTLVfVeptJ7mfIr1DwBhk1u37EfKvnHGLVtMGL7WvNNYN/sZ54lJNZm/6zVvfEI+nHI2F0XIOndoffy8qoplPkmCOVEqc3tzNs3CQ9fomC6hzgCUQvRE1zREqzW/3AKLJsDEVjE0jxX2pxa6WoL6mwx7+xB3shTLg3kFMq50WAEH6P/9EfD1bszHlUUaeheuF+qzt4lCnnQlqATqln8LXoeVGYxTzE0KWOi2rUll3xaHJ41kE58X0hgN5GubuGtbnIQ== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1650003913; bh=x/ZuAQ5rPOXslocncVVP+YHkJAhBCn56pgfX3gyI/JP=; h=X-Sonic-MF:From:Subject:Date:To:From:Subject; b=b8Iq0zPmNw+MtVnvnuMObTyt6p1h1giiGMyyN0eUi6Mv2tieEV0UYKPEo1f1oc4N9ofRqUW1DG6yrb8TrSFc1+ltwGAxgYJim5vZk9ysf/hz1AJl3v9Wrvc9i2dVIF8niw0MRyJNFaDCOI0qu6VQP0PbSgW0LLFgrZH8vJ0maiLhn+NP+MGvZwISlhLwNnRobY603PKy8dz/SUANcD8DQ5Lfap05O4Ye29rKg77RzXZ6CTZM0kxjv/v/lTc+1eNJ2NVKyEC9PGlbSl6xXbK4WBTNm6w1kflOu4Dopp03GpL+5JOOC6g83UPh5/TG/bkvhu6zUQ/V3VJell+g5wHsCw== X-YMail-OSG: 9TdufQoVM1lOsQ2R6HQZAtQqtgSx32PD7HKrREr9RnETl6Gl_20vWTjrhNWbMCM 4Yb2PlPUjYhC75m88oH4jLrdcbAAK5J_dbGRwZ1RbJwYcwbWu0EAcsALyZ7zyyOMhixDUscwCaqH hxo6i.ULgkTt8pRZwWuevvL6CcgluchHkzKD8LPTROLt2.cJQWskz3G7KX8pK2pYIGSGh.RtXYgK qul457DzPV8saGnYeoshWM7kBO0m8J2DkLBow.8STDRHpwxcD7mRu5I8z0FAspq4gYbTzwjhx32U g40h3c5g5QzhHinqsmERFF2pY1nwDQVHNEEqzb0mfYvya80bvOFIVdG.jFzuHyd.g39aP.yYoBaI 3p4n02vx7Xgs91z_bG0ruKw0BXNZNZGsIFwUM_xCwxGVPsWe4x3DLR39GJXnn4Sg5fzgSyJB8jCT zWl0p8PxrXPEnZNyiA2IDDLob7aKCSf1_AGMm3c4LDJ8e9QgFXVlsUr0uQbdvTtN9yp6JmswhNw8 kD43tZtIZjwCDpLIxDy8x2Rb1KkAH7SMQ2SP0Zqfcww5NBQKafSjXFnr93w4KMkC5TtOw0UKtywZ XppSnPQO07veHmt8H_ckfcl44J7C9J1q3LR_jA90wKfNcoLemxkVUCQGqDDKgnRc_Lr3skp4traS k2EPYzTvMGWsvYZ95R3D2B8rcRG9q8DMyry4O5D2lVggCTtKV.N2FK1MaFRRq1gr4eFy4mhxarLP J7aMU4fGi8TzRfBy0lBGWV7mUOO1mkgP_Z0HiSjU_uFfIjnXOA5rf.MOY1FCL0nJuIXAI4Sv.Klk OK_6qsPHQmsnDfakcqqKqSDWLpFToAZvxT._6jbz3cWqw.gG2VxDSPinlvoVu.ceXV4.iY6UzaTH Hihn2c8zBiRYEz21zNDzei17AksqfPtooZA9qunQRdWOxXZFC.ids9t71MI0otVD0gXxFnju.R2S kkGJBo9vJekmkVKQH9KoawoxiLCfmLlzlYKzJ8SeCODrmf062Z0wm6EyVKmXa8wfGnodj.XZ41QB iJ34xr84vyXyAILMGXLcFEp1_rVXNoRM6CIE.rmXtsuKvT9WwoS9YZu4RX0.01U00J8CKYg1Sk1I dqsQ4LPZpw__WiE78dPaF_xmJAFdK.n2ibN9aJ.4tMNmlskdSEjnWzmfeWb0LUslW17DW.mVBg50 CGuzkFQz1ZnZJrjVyixoJXNZf0om5b9hKKnj9sdAQN49G3.RQ.v6fUrsL59Ycu0Q10Dpl7BFCkv. 5cYlWnxFyOeUS3_1KbmSHO3GSTMxmENC5YqAQDi4DfUXpur.tVlRlVQ35oqCBOQc3fMd.X_OiCMC cDmiEglDoD67oFyF61BM5fek.aw42OXQ97z0cMmnPAlaWPk7qvI15ZKcru4TtABs6oWZUGqfAsi. orTy1D4qT0PrFLOWEPJmbc_QHGa3M8XNU8hU1t_XPoKjbJL4AHnnzskzAlEzjG_rNtss.jUI22v_ GOUiUYvHw3cVP5xhk9sDMfhRRBumNV6Vp8PDF66UBLxjBLTYr9eleg6Eos2skAtUjbDyL9lEdm4O 7y9KbAn2S2vsxkJ1gK9USD.5yfXVKLu1Ykt0b4rDfaHdXvbnLPBCuxHcbf7vFRNj1rQ6tgPwMY1W 2g2dEXxUz0F5K211iT0WRceG_QnMzwlzwMs0BsTOjlChaZcKRFE6.nVGEt0PpmBHzaB0._lO_XI8 wZhQbn4pC01KDFAIHR5jHXcE4TRTegJdLwoSCo6SQpw7TGBqokPJ5zsYLiXa8LFPQjQVtVs5NmZ2 3choYbYPgipVCmtsN1Pb6n7fcL2yTXznlRHOQazssVuIMVkF1b1Pdj.Fr5Q5PCNd3PVG6Req4W1. WSq.mDo7msdo9oRxhOw0nqc7k3X0Z.twAJ.Y1nKLHqE5oLXdsD1e8vz3HZEDVFZxCSSa5nTC4vE3 ubIRx7ZWdivbgWqLmbAd8vrpdLqpjWUbl8HnspQPlUWRxEfbLjbwmoHD2h2SauICdSTegMz0VmRU x1L_._xi3JNISratPMUgRrc_sBiVQijsDNme4rb6bZKHYbidPRJTTnFpzBQJEDqcBfwnAhgt8LAa VDPqS4nLEGGgf3w7Hi911X3Lmn2Ft9dg0K9nEdvQWUhFLluoGmqzce9ShUULobJWCi5NiImHW1Ma T7TROpHmrAriAii8aJUn.EvjI6v1K6a9W9GnE3lg0bOwHlHIoPu10Eem8ebS4hVrOvLNJbKERBUt LAb4sdZ1rYfvRVIArTVPpcNR2AlfCu3RDhd94wBSKzli6nvWL64PvMTQp5zlOyj8YIQQx8LxwX46 zpu7uhVepEwWgqg-- X-Sonic-MF: Received: from sonic.gate.mail.ne1.yahoo.com by sonic306.consmr.mail.gq1.yahoo.com with HTTP; Fri, 15 Apr 2022 06:25:13 +0000 Received: by hermes--canary-production-bf1-5f49dbcd6-plxjl (VZM Hermes SMTP Server) with ESMTPA ID 23988e620aa73933d54e1da5c9efc094; Fri, 15 Apr 2022 06:25:10 +0000 (UTC) From: Mark Millard Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 14.0 \(3654.120.0.1.13\)) Subject: Re: llvm & RTTI over shared libraries Message-Id: <9ADA04B1-2A0F-4B96-8510-88A5E4E1E2C0@yahoo.com> Date: Thu, 14 Apr 2022 23:25:07 -0700 To: jbo@insane.engineer, FreeBSD Hackers X-Mailer: Apple Mail (2.3654.120.0.1.13) References: <9ADA04B1-2A0F-4B96-8510-88A5E4E1E2C0.ref@yahoo.com> X-Rspamd-Queue-Id: 4KfmXc2XDJz3Pv2 X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=JE7YCaRP; dmarc=pass (policy=reject) header.from=yahoo.com; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.68.82 as permitted sender) smtp.mailfrom=marklmi@yahoo.com X-Spamd-Result: default: False [-3.45 / 15.00]; TO_DN_SOME(0.00)[]; FREEMAIL_FROM(0.00)[yahoo.com]; MV_CASE(0.50)[]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; DKIM_TRACE(0.00)[yahoo.com:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; NEURAL_HAM_SHORT(-0.99)[-0.988]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.96)[-0.958]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[98.137.68.82:from]; MLMMJ_DEST(0.00)[freebsd-hackers]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.68.82:from]; RCVD_COUNT_TWO(0.00)[2] X-ThisMailContainsUnwantedMimeParts: N From: wrote on Date: Thu, 14 Apr 2022 16:36:24 +0000 : (I've line-split the text.) > I'm in the middle of moving to FreeBSD as my primary development = platform (desktop wise). > As such, I am currently building various software tools I've written = over the years on > FreeBSD for the first time. Most of those were developed on either = Linux+GCC or on > Windows+Mingw (MinGW -> GCC). >=20 > Today I found myself debugging a piece of software which runs fine on = FreeBSD when > compiled with gcc11 but not so much when compiling with clang14. > I managed to track down the problem but I lack the deeper = understanding to resolve > this properly - so here we are. >=20 > The software in question is written in C++20 and consisting of: > - An interface library (just a bunch of header files). > - A main executable. > - A bunch of plugins which the executable loads via dlopen(). >=20 > The interface headers provide several types. Lets call them A, B, C = and D. where B, > C and D inherit from A. > The plugins use std::dynamic_pointer_cast() to cast an = std::shared_ptr (received > via the plugin interface) to the derived classes such as = std::shared_ptr. > This is where the trouble begins. >=20 > If everything (the main executable and the plugins) are compiled using = gcc11, everything > works "as I expect it". > However, when compiling everything with clang14, the main executable = is able to load the > plugins successfully but those std::dynamic_pointer_cast() calls = within the plugins > always return nullptr. >=20 > After some research I seem to understand that the way that RTTI is = handled over shared > library boundaries is different between GCC and LLVM. > This is where my understanding starts to get less solid. >=20 > I read the manual page of dlopen(3). It would seem like the flag = RTLD_GLOBAL would be > potentially interesting to me: "Symbols from this shared object [...] = of needed objects > will be available for re-solving undefined references from all other = shared objects." > The software (which "works as intended" when compiled with GCC) was so = far only calling > dlopen(..., RTLD_LAZY). > I'm not even sure whether this applies to my situation. My gut feeling = tells me that I'm > heading down the wrong direction here. After all, the main executable = is able to load > the plugins and to call the plugin's function which receives an = std::shared_ptr > asparameter just fine, also when compiled with LLVM. > Is the problem I'm experiencing related to the way that the plugin = (shared library) is > loaded or the way that the symbols are being exported? > In the current state, the plugins do not explicitly export any = symbols. >=20 > Here's a heavily simplified version of my scenario: The simplified example was not designed to compile and test. So I made guesses and made my own. The .cpp files have comments on the compile/link commands used and there are examples of c++ and g++11 compile/link/run sequences after the source code. The code is not well commented. Nor does it deal with error handling or the like. But it is fairly short overall. # more base_plugin.h #include // For its own libbase_plugin.so file, load time bound, no dlopen used = for it: struct base { virtual ~base(); }; struct base_plugin { virtual std::shared_ptr create_data_instance() = =3D 0; virtual void action(std::shared_ptr data) = =3D 0; virtual ~base_plugin(); }; extern "C" // for each derived plugin .so file: { using plugin_instance_creator=3D base_plugin* (*)(); const char plugin_instance_creator_name[] =3D = "create_plugin_instance"; // Lookup via dlsym. using plugin_instance_destroyer=3D void (*)(base_plugin*); const char plugin_instance_destroyer_name[] =3D = "destroy_plugin_instance"; // Lookup via dlsym. }; # more base_plugin.cpp // c++ -std=3Dc++20 -O0 -g -fPIC -lc++ -olibbase_plugin.so -shared = base_plugin.cpp // g++11 -std=3Dc++20 -O0 -g -fPIC -lstdc++ -olibbase_plugin.so -shared = base_plugin.cpp #include "base_plugin.h" base::~base() {} base_plugin::~base_plugin() {} # more main_using_plugin.cpp=20 // c++ -std=3Dc++20 -O0 -g -fPIC -lc++ -L. -lbase_plugin = -Wl,-rpath=3D. \ // -omain_using_plugin = main_using_plugin.cpp // g++11 -std=3Dc++20 -O0 -g -fPIC -lstdc++ -L. -lbase_plugin = -Wl,-rpath=3D. \ // -Wl,-rpath=3D/usr/local/lib/gcc11 -omain_using_plugin = main_using_plugin.cpp #include "base_plugin.h" #include int main() { auto dl=3D dlopen("./libsharedlib_plugin.so",RTLD_LAZY); // = hardcoded .so path for the example union { void* as_voidptr; plugin_instance_creator = as_plugin_instance_creator; } creator_plugin_func; creator_plugin_func.as_voidptr=3D = dlsym(dl,plugin_instance_creator_name); union { void* as_voidptr; plugin_instance_destroyer = as_plugin_instance_destroyer; } destroyer_plugin_func; destroyer_plugin_func.as_voidptr=3D = dlsym(dl,plugin_instance_destroyer_name); auto plugin=3D = (creator_plugin_func.as_plugin_instance_creator)(); { // Local scope for data std::shared_ptr = data{plugin->create_data_instance()}; plugin->action(data); } // Presume for the example that nothing requires the plugin = after here. (destroyer_plugin_func.as_plugin_instance_destroyer)(plugin); destroyer_plugin_func.as_voidptr=3D nullptr; dlclose(dl); } NOTE: So, other than the dlopen, the above has no direct tie to the specific dynamically loaded plugin. The base_plugin is in a .so but is load-time bound instead of using dlopen. That .so would be used by all the plugins found via dllopen. (I only made one example.) As for the .so used via dlopen/dlsym/dlclose . . . # more sharedlib_plugin.h #include "base_plugin.h" // For its own libsharedlib_plugin.so file, where dlopen is used to find = it: struct sharedlib : base { int v; }; struct sharedlib_plugin : base_plugin { std::shared_ptr create_data_instance() = override; void action(std::shared_ptr base) = override; }; # more sharedlib_plugin.cpp // c++ -std=3Dc++20 -O0 -g -fPIC -lc++ -olibsharedlib_plugin.so = -shared sharedlib_plugin.cpp // g++11 -std=3Dc++20 -O0 -g -fPIC -lstdc++ -olibsharedlib_plugin.so = -shared sharedlib_plugin.cpp #include "sharedlib_plugin.h" #include std::shared_ptr sharedlib_plugin::create_data_instance() { std::cout << "create_data_instance in use from dlopen'd .so\n"; return = std::static_pointer_cast(std::make_shared()); } void sharedlib_plugin::action(std::shared_ptr b) { std::cout << "action in use from dlopen'd .so class\n"; auto separate_share =3D std::dynamic_pointer_cast(b); if (separate_share->v || 1 < separate_share.use_count()) std::cout << "separate_share is not nullptr (would crash = otherwise)\n"; } extern "C" base_plugin* create_plugin_instance() { std::cout << "create_plugin_instance in use from dlopen'd = .so\n"; return new sharedlib_plugin(); } extern "C" void destroy_plugin_instance(const base_plugin* plugin) { std::cout << "destroy_plugin_instance in use from dlopen'd = .so\n"; delete plugin; } # c++ -std=3Dc++20 -O0 -g -fPIC -lc++ -olibbase_plugin.so -shared = base_plugin.cpp # c++ -std=3Dc++20 -O0 -g -fPIC -lc++ -L. -lbase_plugin -Wl,-rpath=3D. \ -omain_using_plugin main_using_plugin.cpp # c++ -std=3Dc++20 -O0 -g -fPIC -lc++ -olibsharedlib_plugin.so -shared = sharedlib_plugin.cpp # ./main_using_plugin create_plugin_instance in use from dlopen'd .so create_data_instance in use from dlopen'd .so action in use from dlopen'd .so class separate_share is not nullptr (would crash otherwise) destroy_plugin_instance in use from dlopen'd .so For reference: # ldd main_using_plugin main_using_plugin: libc++.so.1 =3D> /lib/libc++.so.1 (0x819d0000) libcxxrt.so.1 =3D> /lib/libcxxrt.so.1 (0x82735000) libbase_plugin.so =3D> ./libbase_plugin.so (0x8328d000) libm.so.5 =3D> /lib/libm.so.5 (0x83c47000) libgcc_s.so.1 =3D> /lib/libgcc_s.so.1 (0x85861000) libc.so.7 =3D> /lib/libc.so.7 (0x848f9000) # ldd ./libsharedlib_plugin.so ./libsharedlib_plugin.so: libc++.so.1 =3D> /lib/libc++.so.1 (0x3b69aeeb6000) libcxxrt.so.1 =3D> /lib/libcxxrt.so.1 (0x3b69af6f2000) libm.so.5 =3D> /lib/libm.so.5 (0x3b69afd1f000) libgcc_s.so.1 =3D> /lib/libgcc_s.so.1 (0x3b69b0303000) libc.so.7 =3D> /lib/libc.so.7 (0x3b69aafdb000) As for g++11 use . . . Testing with g++11 does involve additional/adjusted command line options: -Wl,-rpath=3D/usr/local/lib/gcc11/ ( for main_using_plugin.cpp ) -lstdc++ (for all 3 .cpp files) (FreeBSD's libgcc_s.so.1 does not cover everything needed for all architectures for g++11's code generation. I was working in a context where using /usr/local/lib/gcc11//libgcc_s.so.1 was important.) # g++11 -std=3Dc++20 -O0 -g -fPIC -lstdc++ -olibbase_plugin.so -shared = base_plugin.cpp # g++11 -std=3Dc++20 -O0 -g -fPIC -lstdc++ -L. -lbase_plugin = -Wl,-rpath=3D. \ -Wl,-rpath=3D/usr/local/lib/gcc11 -omain_using_plugin = main_using_plugin.cpp # g++11 -std=3Dc++20 -O0 -g -fPIC -lstdc++ -olibsharedlib_plugin.so = -shared sharedlib_plugin.cpp # ./main_using_plugin create_plugin_instance in use from dlopen'd .so create_data_instance in use from dlopen'd .so action in use from dlopen'd .so class separate_share is not nullptr (would crash otherwise) destroy_plugin_instance in use from dlopen'd .so For reference: # ldd main_using_plugin main_using_plugin: libstdc++.so.6 =3D> /usr/local/lib/gcc11//libstdc++.so.6 = (0x83a00000) libbase_plugin.so =3D> ./libbase_plugin.so (0x8213d000) libm.so.5 =3D> /lib/libm.so.5 (0x82207000) libgcc_s.so.1 =3D> /usr/local/lib/gcc11//libgcc_s.so.1 = (0x82c66000) libc.so.7 =3D> /lib/libc.so.7 (0x849c4000) # ldd ./libsharedlib_plugin.so ./libsharedlib_plugin.so: libstdc++.so.6 =3D> /usr/local/lib/gcc11/libstdc++.so.6 = (0x1c2a7b800000) libm.so.5 =3D> /lib/libm.so.5 (0x1c2a7bb1c000) libgcc_s.so.1 =3D> /lib/libgcc_s.so.1 (0x1c2a7c416000) libc.so.7 =3D> /lib/libc.so.7 (0x1c2a780e8000) Overall: Looks to me like both the system clang/llvm and g++11 contexts are working. (The platform context was aarch64 main [so: 14], in case it matters.) =3D=3D=3D Mark Millard marklmi at yahoo.com