[RFC] Adding a Rados block driver to bhyve

[Willem Jan Withagen]

On 10-3-2020 17:21, Alan Somers wrote:
> On Tue, Mar 10, 2020 at 9:41 AM Willem Jan Withagen <wjw at digiware.nl 
> <mailto:wjw at digiware.nl>> wrote:
>
>     On 10-3-2020 16:15, Alan Somers wrote:
>>     On Tue, Mar 10, 2020 at 3:59 AM Willem Jan Withagen
>>     <wjw at digiware.nl <mailto:wjw at digiware.nl>> wrote:
>>
>>         On 9-3-2020 14:46, Alan Somers wrote:
>>>         On Mon, Mar 9, 2020 at 4:32 AM Willem Jan Withagen
>>>         <wjw at digiware.nl <mailto:wjw at digiware.nl>> wrote:
>>>
>>>             Hi all,
>>>
>>>             And sorry for crosspoing three groups, but the answer
>>>             can/could be a mix
>>>             of things to do in these three areas.
>>>
>>>             I have a prototype of bhyve running on Rados/Ceph working:
>>>             https://github.com/freebsd/freebsd/pull/426
>>>
>>         ..........
>>>
>>>             4) Create a bhyve-blockrbd port.
>>>                  This is much like 3) but instead of building a
>>>             bhyve-rbd executable,
>>>                  it delivers a libblockrbd.so that is dynamically
>>>             loadable by the
>>>                  standaard bhyve that comes with base.
>>>
>>         ............
>>
>>         > Great work!  I also agree that option 4 sounds like the
>>         best.  There's precedent for ports that
>>         > require the FreeBSD Sources.  For example, see
>>         devel/py-libzfs or emulators/virtualbox-ose.
>>         > You just need to define the SRC_BASE variable.
>>         Hi Alan,
>>
>>         Thanx for the hint, and it made me check what is actually
>>         available within the poudriere jail
>>         And that does have full source, so the Makefile code is
>>         mainly for those that build in a different way.
>>
>>         I've got a proto version working when compiling stuff with
>>         `make buildworld`, but run in the
>>         problem that libblock_rbd.so is stripped in such a way that
>>         the symbol I need is removed.
>>         Using the unstripped version does work.
>>
>>         Is there an incantation for the SRC Makefiles that builds a
>>         dynamical loadable lib??
>>         And I'm still looking for a PORTS example of building a
>>         dynamical loadable lib.
>>         Or is there no generic code for that in the PORTS Mk files?
>>
>>         --WjW
>>
>>         BTW: Still haven't worked in your AIO code :(
>>
>>
>>     There are plenty of dynamic libraries built with the SRC
>>     makefiles.  For example,
>>     https://svnweb.freebsd.org/base/head/lib/libbsdstat/Makefile?view=markup
>>     .
>
>     That looks dangerously close to what I have for libblock_rbd.
>     ===
>     > cat Makefile-librbd
>     #
>     # $FreeBSD$
>     #
>
>     PACKAGE=lib${LIB}
>
>     .include <src.opts.mk <http://src.opts.mk>>
>
>     LIB=            block_rbd
>     SHLIB_MAJOR=    1
>
>     SRCS=   block_rbd.c
>
>     CFLAGS+=-I${SRCTOP}/sys
>     CFLAGS+=-g -O0 -fPIC -rdynamic
>     LDFLAGS+=-Wl,-export-dynamic,-Bdynamic
>     CFLAGS+=-DWITHOUT_CAPSICUM
>
>     LOCALBASE?=     /usr/local
>     CFLAGS+=        -I${LOCALBASE}/include
>     LDFLAGS+=       -L${LOCALBASE}/lib -lrados -lrbd
>
>     WARNS?= 2
>
>     ===
>
>     This is the code that mk.lib.bsd runs:
>     objcopy --only-keep-debug libblock_rbd.so.1.full
>     libblock_rbd.so.1.debug
>     objcopy --strip-debug --add-gnu-debuglink=libblock_rbd.so.1.debug
>     libblock_rbd.so.1.full libblock_rbd.so.1
>
>     So still I get a stripped lib in /usr/lib. And then the one and
>     only symbol I need to load
>     is not found. Copying libblock_rbd.so.1.full actually works for me.
>
>     So either I'm doing it the wrong way, like special options on the
>     symbols oid.
>     Or mk.lib.bsd cannot deliver dlopen/dlsym-able files?
>
>     And there are plenty of ports that build shared libraries too,
>     just look at /usr/local/lib/*.so. However, the ports framework
>     doesn't have much special code just to support building
>     libraries.  Instead the hard work is always done by the ports
>     themselves. Some use autotools, some cmake, etc etc.  The simplest
>     port I can find that uses both SRC_BASE and INSTALL_LIB is this
>     one: https://svnweb.freebsd.org/ports/head/devel/linux_libusb/ .
>
>     Oke thanx, I'll have a look at it, and given that I can see most
>     of the compile build stuff
>     in the SRC_BASE version I'll get it to work.
>
>     --WjW
>
>
> Try setting "STRIP=    " in your makefile.  That should prevent the 
> stripping.  However, I think there's something wrong with your 
> library, too.  The library should be usable even if it's stripped.
I checked with objdump, and the symbol that I need is definitly not 
present in the stripped version.
And it does not really matter if I declare it static or not.
But I'll give it a few more itterations to try it out.
Including 'STRIP= '

Thanx,
--WjW