[RFC] Adding a Rados block driver to bhyve

[Willem Jan Withagen]

On 10-3-2020 17:48, Alan Somers wrote:
> On Tue, Mar 10, 2020 at 10:28 AM Willem Jan Withagen <wjw at digiware.nl>
> wrote:
>
>> On 10-3-2020 17:21, Alan Somers wrote:
>>
>> On Tue, Mar 10, 2020 at 9:41 AM Willem Jan Withagen <wjw at digiware.nl>
>> wrote:
>>
>>> On 10-3-2020 16:15, Alan Somers wrote:
>>>
>>> On Tue, Mar 10, 2020 at 3:59 AM Willem Jan Withagen <wjw at digiware.nl>
>>> wrote:
>>>
>>>> On 9-3-2020 14:46, Alan Somers wrote:
>>>>
>>>> On Mon, Mar 9, 2020 at 4:32 AM Willem Jan Withagen <wjw at digiware.nl>
>>>> wrote:
>>>>
>>>>> Hi all,
>>>>>
>>>>> And sorry for crosspoing three groups, but the answer can/could be a mix
>>>>> of things to do in these three areas.
>>>>>
>>>>> I have a prototype of bhyve running on Rados/Ceph working:
>>>>>       https://github.com/freebsd/freebsd/pull/426
>>>>>
>>>>> ..........
>>>> 4) Create a bhyve-blockrbd port.
>>>>>       This is much like 3) but instead of building a bhyve-rbd
>>>>> executable,
>>>>>       it delivers a libblockrbd.so that is dynamically loadable by the
>>>>>       standaard bhyve that comes with base.
>>>>>
>>>> ............
>>>>
>>>>> Great work!  I also agree that option 4 sounds like the best.  There's
>>>> precedent for ports that
>>>>> require the FreeBSD Sources.  For example, see devel/py-libzfs or
>>>> emulators/virtualbox-ose.
>>>>> You just need to define the SRC_BASE variable.
>>>> Hi Alan,
>>>>
>>>> Thanx for the hint, and it made me check what is actually available
>>>> within the poudriere jail
>>>> And that does have full source, so the Makefile code is mainly for those
>>>> that build in a different way.
>>>>
>>>> I've got a proto version working when compiling stuff with `make
>>>> buildworld`, but run in the
>>>> problem that libblock_rbd.so is stripped in such a way that the symbol I
>>>> need is removed.
>>>> Using the unstripped version does work.
>>>>
>>>> Is there an incantation for the SRC Makefiles that builds a dynamical
>>>> loadable lib??
>>>> And I'm still looking for a PORTS example of building a dynamical
>>>> loadable lib.
>>>> Or is there no generic code for that in the PORTS Mk files?
>>>>
>>>> --WjW
>>>>
>>>> BTW: Still haven't worked in your AIO code :(
>>>>
>>> There are plenty of dynamic libraries built with the SRC makefiles.  For
>>> example,
>>> https://svnweb.freebsd.org/base/head/lib/libbsdstat/Makefile?view=markup
>>> .
>>>
>>>
>>> That looks dangerously close to what I have for libblock_rbd.
>>> ===
>>>> cat Makefile-librbd
>>> #
>>> # $FreeBSD$
>>> #
>>>
>>> PACKAGE=lib${LIB}
>>>
>>> .include <src.opts.mk>
>>>
>>> LIB=            block_rbd
>>> SHLIB_MAJOR=    1
>>>
>>> SRCS=   block_rbd.c
>>>
>>> CFLAGS+=-I${SRCTOP}/sys
>>> CFLAGS+=-g -O0 -fPIC -rdynamic
>>> LDFLAGS+=-Wl,-export-dynamic,-Bdynamic
>>> CFLAGS+=-DWITHOUT_CAPSICUM
>>>
>>> LOCALBASE?=     /usr/local
>>> CFLAGS+=        -I${LOCALBASE}/include
>>> LDFLAGS+=       -L${LOCALBASE}/lib -lrados -lrbd
>>>
>>> WARNS?= 2
>>>
>>> ===
>>>
>>> This is the code that mk.lib.bsd runs:
>>> objcopy --only-keep-debug libblock_rbd.so.1.full libblock_rbd.so.1.debug
>>> objcopy --strip-debug --add-gnu-debuglink=libblock_rbd.so.1.debug
>>> libblock_rbd.so.1.full libblock_rbd.so.1
>>>
>>> So still I get a stripped lib in /usr/lib. And then the one and only
>>> symbol I need to load
>>> is not found. Copying libblock_rbd.so.1.full actually works for me.
>>>
>>> So either I'm doing it the wrong way, like special options on the symbols
>>> oid.
>>> Or mk.lib.bsd cannot deliver dlopen/dlsym-able files?
>>>
>>> And there are plenty of ports that build shared libraries too, just look
>>> at /usr/local/lib/*.so.  However, the ports framework doesn't have much
>>> special code just to support building libraries.  Instead the hard work is
>>> always done by the ports themselves.  Some use autotools, some cmake, etc
>>> etc.  The simplest port I can find that uses both SRC_BASE and INSTALL_LIB
>>> is this one: https://svnweb.freebsd.org/ports/head/devel/linux_libusb/ .
>>>
>>> Oke thanx, I'll have a look at it, and given that I can see most of the
>>> compile build stuff
>>> in the SRC_BASE version I'll get it to work.
>>>
>>> --WjW
>>>
>> Try setting "STRIP=    " in your makefile.  That should prevent the
>> stripping.  However, I think there's something wrong with your library,
>> too.  The library should be usable even if it's stripped.
>>
>> I checked with objdump, and the symbol that I need is definitly not
>> present in the stripped version.
>> And it does not really matter if I declare it static or not.
>> But I'll give it a few more itterations to try it out.
>> Including 'STRIP= '
>>
>> Thanx,
>> --WjW
>>
> What does "nm --dynamic libblock_rbd.so.1" show?

Don't know what has changed, but I just rebuild everything.
And now it seems to work.... ;-)

One thing I'm wondering if th DATA_SET(block_backend_set, ....)
gets updated when dyn-loading the library it is in?

Or if not, how do I add an entry to this DATA_SET, once we are running
the program?

I'm dlsym() looking for block_backend_rbd:
                  w _Jv_RegisterClasses
                  U __assert
                  w __cxa_finalize
                  U __error
                  U __stack_chk_fail
                  U __stack_chk_guard
0000000000006098 D __start_set_block_backend_set
                  U __stderrp
00000000000060a0 D __stop_set_block_backend_set
000000000000475c T _fini
000000000000474c T _init
0000000000006008 D block_backend_rbd
0000000000006270 B blocklocal_backend
0000000000002170 T blockrbd_cleanup
0000000000006250 B blockrbd_debug
0000000000002080 T blockrbd_init

Which is the variable holding the block_if:jumptable, and gives me access to
all that I need to work with the driver.
==============
block_backend_t block_backend_rbd = {
     .bb_name      = "rbd",
     .bb_init      = blockrbd_init,
     .bb_cleanup   = blockrbd_cleanup,
     .bb_open      = blockrbd_open,
     .bb_size      = blockrbd_size,
     .bb_chs       = blockrbd_chs,
     .bb_sectsz    = blockrbd_sectsz,
     .bb_psectsz   = blockrbd_psectsz,
     .bb_queuesz   = blockrbd_queuesz,
     .bb_is_ro     = blockrbd_is_ro,
     .bb_candelete = blockrbd_candelete,
     .bb_read      = blockrbd_read,
     .bb_write     = blockrbd_write,
     .bb_flush     = blockrbd_flush,
     .bb_delete    = blockrbd_delete,
     .bb_cancel    = blockrbd_cancel,
     .bb_close     = blockrbd_close,
     .bb_opaque    = (void*)&block_backend_rbd,
};
DATA_SET(block_backend_set, block_backend_rbd);
==============

This is the Makefile:
#
# $FreeBSD$
#

PACKAGE=        lib${LIB}
LIB=    block_rbd

.include <src.opts.mk>

SHLIB_MAJOR=    1

SRCS=   block_rbd.c

CFLAGS+=        -I${SRCTOP}/sys
CFLAGS+=        -g -O0 -fPIC -rdynamic
LDFLAGS+=       -Wl,-export-dynamic,-Bdynamic
CFLAGS+=        -DWITHOUT_CAPSICUM

LOCALBASE?=     /usr/local
CFLAGS+=        -I${LOCALBASE}/include
LDFLAGS+=       -L${LOCALBASE}/lib -lrados -lrbd

WARNS?= 2

.include <bsd.lib.mk>

--WjW