DevHeads.net

building nfs-ganesha for ceph

Hi Niels and Kaleb,

I'm wondering how we could best build nfs-ganesha's ceph FSALs in CBS.

Right now, since we'll likely support multiple versions of nfs-ganesha
simultaneously for different versions of Ceph, I'm leaning towards
using a custom "dist tag" like thing in the Release field so we don't
collide with what Kaleb's doing at
<a href="http://cbs.centos.org/koji/packageinfo?packageID=98" title="http://cbs.centos.org/koji/packageinfo?packageID=98">http://cbs.centos.org/koji/packageinfo?packageID=98</a>

Eventually it would be nice to have a unified nfs-ganesha build shared
between Ceph and Gluster. If we built it in storage7-common-candidate,
we would have to tag gluster and ceph into that, which sounds kinda
messy, because you only get to choose one version of each. It'd be
more manageable to have a separate set of tags and targets just for
nfs-ganesha versions. Like:

storage7-nfs-ganesha-26-el7
storage7-nfs-ganesha-27-el7
... etc

But that still leaves the problem of shipping a nfs-ganesha-gluster
sub-package that will not have its runtime deps fulfilled in
<a href="http://mirror.centos.org/centos/7/storage/x86_64/ceph-nautilus/" title="http://mirror.centos.org/centos/7/storage/x86_64/ceph-nautilus/">http://mirror.centos.org/centos/7/storage/x86_64/ceph-nautilus/</a> . Vice
versa, we'd have a nfs-ganesha-ceph sub-package that lacks runtime
deps in <a href="http://mirror.centos.org/centos/7/storage/x86_64/gluster-5/" title="http://mirror.centos.org/centos/7/storage/x86_64/gluster-5/">http://mirror.centos.org/centos/7/storage/x86_64/gluster-5/</a>

What do you think? Just have separate nfs-ganesha builds for now?

- Ken

Comments

Re: building nfs-ganesha for ceph

By Kaleb S. KEITHLEY at 12/04/2018 - 18:08

On 12/4/18 5:38 PM, Ken Dreyer wrote:
I too would like to get to a place where we have a single ganesha package.

Gluster's GFAPI guarantees its ABI by using versioned symbols. A Gluster
FSAL built against glusterfs-4.1 (or earlier) will work with
glusterfs-5's libgfapi.so. And glusterfs-6's. And so on.

So we don't build separate ganesha-2.7 packages for both glusterfs-4.1
and glusterfs-5. We build a single ganesha-2.7 package with
glusterfs-4.1 and it is tagged into both the glusterfs-4.1 and
glusterfs-5 releases.

I think it should be achievable. I try and build ganesha with as many
FSALs enabled as possible. It's been a while though since I looked at
which version of Ceph was in CBS for the Storage SIG.

If Ceph doesn't guarantee its ABIs though that's going to make things
harder.

Re: building nfs-ganesha for ceph

By Ken Dreyer at 12/05/2018 - 11:11

On Tue, Dec 4, 2018 at 4:09 PM Kaleb S. KEITHLEY < ... at redhat dot com> wrote:
I confirmed with Matt Benjamin on ceph-devel today that this ok. We'll
be able to build nfs-ganesha-2.7 against Luminous and then cross-tag
that into our newer releases (eg mimic and nautilus). So we can follow
the same model that you're following with building nfs-ganesha in the
"oldest supported" build target.

In terms of sharing one single nfs-ganesha build across ceph and
gluster, I think that will require more work. We'd probably want:

1) A custom nfs-ganesha "-build" tag/target that contains the "oldest
supported" versions of both ceph and gluster,

2) cross-tag the nfs-ganesha builds from there into both the ceph and gluster
-testing tags,

3) Update the mash configuration to exclude the irrelevant sub-packages
when building ceph and gluster's "testing" and "release" repositories.

Unfortunately that last bit looks really tricky. I'm not sure there is
a way to tell mash itself to exclude a particular sub-package, and
https://git.centos.org/blob/sig-core!cbs-tools.git/master/scripts!mash_run.sh
uses a basic mash config template for all CBS tags.

- Ken

Re: building nfs-ganesha for ceph

By Niels de Vos at 12/05/2018 - 11:53

On Wed, Dec 05, 2018 at 09:11:58AM -0700, Ken Dreyer wrote:
This is something that I'm very much looking forward to see happening.
The 3rd point is indeed something that (from my understanding) is not
possible. I have thought of an alternative, but that is still a little
ugly. Not sure if we want to go that route.

- add a new repository with all nfs-ganesha-x.y packages (direct
dependencies and sub-packages), without ceph/gluster

- add a new repository definition in centos-release-gluster and
centos-release-ceph packages. This entry in the .repo file should use
the 'exclude=' directive to skip the Ceph sub-package for Gluster, and
the Gluster sub-package for Ceph.

For additional projects (like Samba) this would work in a similar way.

Opinions?
Niels

Re: building nfs-ganesha for ceph

By Ken Dreyer at 12/05/2018 - 12:47

On Wed, Dec 5, 2018 at 9:53 AM Niels de Vos < ... at redhat dot com> wrote:
The biggest reason I want to exclude the package server side is that
will allow repoclosure to pass for the repo. repoclosure provides me a
sanity-check that the entire "product" is installable and I'm not
missing any packages. With the way we're growing more and more
packages, I want to make this as automated as possible. Ideally
repoclosure would automatically gate a package going from -candidate
to -testing.

One problem I see with configuring this exclusion client side is that
I'm not sure I can rely on everyone using the exact same .repo file. I
eventually want to ship packaged mock configs for Ceph, so we'd have
to duplicate that there as well :(

I'd like to know if the CentOS admins plan to stay on mash long-term,
and if so, let's look into filtering at that level, like Pungi can do.
What do you think?

- Ken

Re: building nfs-ganesha for ceph

By Niels de Vos at 12/05/2018 - 13:14

On Wed, Dec 05, 2018 at 10:47:53AM -0700, Ken Dreyer wrote:
Yes, I understand that. This is an item on my wishlist for the Gluster
repositories for a looong time already.

Ah, yes. I have my mock-config files from the 'cbs' tool, but it is not
friendly for others to use. Packaging them would be really nice too.

This definitely would have my preference as well!

Thanks,
Niels