[RFC] Adding a Rados block driver to bhyve

[Alan Somers]

On Tue, Mar 10, 2020 at 10:28 AM Willem Jan Withagen <wjw at digiware.nl>
wrote:

> On 10-3-2020 17:21, Alan Somers wrote:
>
> On Tue, Mar 10, 2020 at 9:41 AM Willem Jan Withagen <wjw at digiware.nl>
> wrote:
>
>> On 10-3-2020 16:15, Alan Somers wrote:
>>
>> On Tue, Mar 10, 2020 at 3:59 AM Willem Jan Withagen <wjw at digiware.nl>
>> wrote:
>>
>>> On 9-3-2020 14:46, Alan Somers wrote:
>>>
>>> On Mon, Mar 9, 2020 at 4:32 AM Willem Jan Withagen <wjw at digiware.nl>
>>> wrote:
>>>
>>>> Hi all,
>>>>
>>>> And sorry for crosspoing three groups, but the answer can/could be a mix
>>>> of things to do in these three areas.
>>>>
>>>> I have a prototype of bhyve running on Rados/Ceph working:
>>>>      https://github.com/freebsd/freebsd/pull/426
>>>>
>>>> ..........
>>>
>>> 4) Create a bhyve-blockrbd port.
>>>>      This is much like 3) but instead of building a bhyve-rbd
>>>> executable,
>>>>      it delivers a libblockrbd.so that is dynamically loadable by the
>>>>      standaard bhyve that comes with base.
>>>>
>>> ............
>>>
>>> > Great work!  I also agree that option 4 sounds like the best.  There's
>>> precedent for ports that
>>> > require the FreeBSD Sources.  For example, see devel/py-libzfs or
>>> emulators/virtualbox-ose.
>>> > You just need to define the SRC_BASE variable.
>>> Hi Alan,
>>>
>>> Thanx for the hint, and it made me check what is actually available
>>> within the poudriere jail
>>> And that does have full source, so the Makefile code is mainly for those
>>> that build in a different way.
>>>
>>> I've got a proto version working when compiling stuff with `make
>>> buildworld`, but run in the
>>> problem that libblock_rbd.so is stripped in such a way that the symbol I
>>> need is removed.
>>> Using the unstripped version does work.
>>>
>>> Is there an incantation for the SRC Makefiles that builds a dynamical
>>> loadable lib??
>>> And I'm still looking for a PORTS example of building a dynamical
>>> loadable lib.
>>> Or is there no generic code for that in the PORTS Mk files?
>>>
>>> --WjW
>>>
>>> BTW: Still haven't worked in your AIO code :(
>>>
>>
>> There are plenty of dynamic libraries built with the SRC makefiles.  For
>> example,
>> https://svnweb.freebsd.org/base/head/lib/libbsdstat/Makefile?view=markup
>> .
>>
>>
>> That looks dangerously close to what I have for libblock_rbd.
>> ===
>> > cat Makefile-librbd
>> #
>> # $FreeBSD$
>> #
>>
>> PACKAGE=lib${LIB}
>>
>> .include <src.opts.mk>
>>
>> LIB=            block_rbd
>> SHLIB_MAJOR=    1
>>
>> SRCS=   block_rbd.c
>>
>> CFLAGS+=-I${SRCTOP}/sys
>> CFLAGS+=-g -O0 -fPIC -rdynamic
>> LDFLAGS+=-Wl,-export-dynamic,-Bdynamic
>> CFLAGS+=-DWITHOUT_CAPSICUM
>>
>> LOCALBASE?=     /usr/local
>> CFLAGS+=        -I${LOCALBASE}/include
>> LDFLAGS+=       -L${LOCALBASE}/lib -lrados -lrbd
>>
>> WARNS?= 2
>>
>> ===
>>
>> This is the code that mk.lib.bsd runs:
>> objcopy --only-keep-debug libblock_rbd.so.1.full libblock_rbd.so.1.debug
>> objcopy --strip-debug --add-gnu-debuglink=libblock_rbd.so.1.debug
>> libblock_rbd.so.1.full libblock_rbd.so.1
>>
>> So still I get a stripped lib in /usr/lib. And then the one and only
>> symbol I need to load
>> is not found. Copying libblock_rbd.so.1.full actually works for me.
>>
>> So either I'm doing it the wrong way, like special options on the symbols
>> oid.
>> Or mk.lib.bsd cannot deliver dlopen/dlsym-able files?
>>
>> And there are plenty of ports that build shared libraries too, just look
>> at /usr/local/lib/*.so.  However, the ports framework doesn't have much
>> special code just to support building libraries.  Instead the hard work is
>> always done by the ports themselves.  Some use autotools, some cmake, etc
>> etc.  The simplest port I can find that uses both SRC_BASE and INSTALL_LIB
>> is this one: https://svnweb.freebsd.org/ports/head/devel/linux_libusb/ .
>>
>> Oke thanx, I'll have a look at it, and given that I can see most of the
>> compile build stuff
>> in the SRC_BASE version I'll get it to work.
>>
>> --WjW
>>
>
> Try setting "STRIP=    " in your makefile.  That should prevent the
> stripping.  However, I think there's something wrong with your library,
> too.  The library should be usable even if it's stripped.
>
> I checked with objdump, and the symbol that I need is definitly not
> present in the stripped version.
> And it does not really matter if I declare it static or not.
> But I'll give it a few more itterations to try it out.
> Including 'STRIP= '
>
> Thanx,
> --WjW
>

What does "nm --dynamic libblock_rbd.so.1" show?