CEPH + FreeBSD

[Rick Macklem]

Jordan Hubbard wrote:
> 
> > On Sep 3, 2015, at 4:17 AM, Rick Macklem <rmacklem at uoguelph.ca> wrote:
> > 
> > Slightly off topic but, btw, there is a port of GLusterFS and those folks
> > do seem
> > interested in seeing it brought "up to speed". I am not sure how mature it
> > is at
> > this point, but it has been known to build on amd64. (I don't have an amd64
> > machine,
> > so I haven't gotten around to building/testing it, but I do plan to try and
> > use
> > it as a basis for a pNFS server, if I can figure out how to get the FH info
> > out of it.
> > I'm working on that;-)
> 
> There are at least two distributed (multi-node) object stores for FreeBSD
> that I know of.
> 
> One is glusterfs, for which I’m not even really clear on the status of the
> ports for.  I don’t see any glusterfs port in the master branch of
> https://github.com/freebsd/freebsd-ports (or
> https://github.com/freebsd/freebsd-ports/tree/branches/2015Q3 for that
> matter).
> 
> Our FreeNAS ports tree (https://github.com/freenas/ports), in which we have a
> bit more latitude to add and curate our own ports, has both a net/glusterfs
> and sysutils/glusterfs, from separate sources (looks like we need to clean
> things up) - net/glusterfs lists craig001 at lerwick.hopto.org as the
> MAINTAINER and is at version 3.6.2.  The sysutils/glusterfs port lists
> bapt at FreeBSD.org as the MAINTAINER and is at version 20140811.
> 
> I’m not really sure about the provenance since we were simply evaluating
> glusterfs for awhile and may have pulled in interim versions from those
> sources, but obviously it would be best to have an official maintainer and
> someone in the FreeBSD project actually curating a glusterfs port so that
> all users of FreeBSD can use it.  It would also be fairly key to your own
> efforts, assuming you decide to pursue glusterfs as a foundation technology
> for pNFS.
> 
> The other object store, which is pretty mature and is currently leading the
> pack (of two :) ) for inclusion into FreeNAS is RiakCS from Basho.  There is
> a port currently in databases/riak but it’s pretty out of date at version
> 1.4.12 (the current version is 2.0.1, with 2.0 being a major upgrade of
> RiakCS).
> 
> We are very interested in investigating various ways of shimming RiakCS to
> NFS, using RiakCS a back-end store.   Is that something you’d be amenable to
> discussing?   I’d be happy to send you an amd64 architecture machine to
> develop on. :)
> 
Hmm. From a quick look at their web page (I looked once before as well), I don't
think RiakCS has what I need to do pNFS in a reasonable (for me) amount of effort.
Two things that glusterFS has that I am hoping to use (and I don't think RiakCS has
either of these) are:
- A Fuse file system interface which allows the kernel nfsd to access the store as
  a file system, so that it can provide the metadata services (NFS without the reads/writes).
- A userland NFSv3 server in each node which will allow the node to act as a data server.

If I am wrong and RiakCS does support a VFS file system interface (via Fuse or ???), then
please correct me. With that, it might be a reasonable alternative.
I'll admit I've spent a little time looking at the glusterFS sources and haven't yet
solved the problem of how to generate the file handles I need, but that sounds trivial
compared with an entire Fuse and/or VFS file system interface, I think?

In general, using a cloud object store to implement a pNFS server is a *mis*use of
the technology, imho. I think it may be possible with glusterFS, since that technology
seems to be based on a cluster file system, which is what a pNFS server can also use.

I think there would be a lot of work involved in mapping a POSIX file system onto the
Riak database and then exporting that via NFS, etc. It might also be more practical to
do this via a userland NFS service than the kernel based one currently in FreeBSD.
(glusterFS is starting to use the NFS-ganesha server, but I believe it is pretty Linux specific,
 so I doubt it would be useful for Riak running on FreeBSD?)

rick

> - Jordan
> 
>