DevHeads.net

Modifying container storage for Fedora 26.

We would like to change the docker container storage to default to
Overlayfs2 in Fedora 26. But we have a problem on Atomic Host and
Fedora Server distributions.

Currently docker-storage-setup defaults to devicemapper and is hard
coded to setup a thinpool of 40% of remaining disk. Otherwise it sets
up loopback devices on the root file system. Devicemapper is nice
since it works with thinpools and can automatically expand the storage
if the disk space is getting used up.

Moving to Overlay, we can more easily use the root file system directly,
which would be fine for Fedora Workstation. We want to preserve the use
of the remaining storage for Overlay on AH and Fedora Server, since
this would give a user flexibility to switch back to using devicemapper
if they had problems with the Overlay driver. We can not as easily
support the expanding disk for Overlay since we will not use using thinpool.

We have looked at options to hard code OverlayFS with the defaults, or
we could just drop a /etc/sysconfig/docker-storage-setup that specified
Overlay and the percentage of remaining space to use for the
/var/lib/docker device. But what is the best way to set different
defaults for AH, Fedora Server and Fedora Workstation.

We would like to discuss with you guys about what you would think is
the best way to handle.

Comments

Re: Modifying container storage for Fedora 26.

By Vivek Goyal at 11/16/2016 - 15:40

On Wed, Nov 16, 2016 at 02:32:46PM -0500, Daniel J Walsh wrote:
And being able to do so basically involves following.

- docker-storage-setup creates a logical volume from free space
- Creates a filesystem on that logical volume
- Mounts that logical volume on the directory which docker is going to
use.

mount /dev/docker-vg/foo /var/lib/docker/

- Now when docker users overlay2 graph driver, all the images, containers
and associated metadata will be stored outside the root filesystem and
onto /dev/docker-vg/foo logical volume.

If we always mount /var/lib/docker on on /dev/vg/foo for overlay2 driver
this will be a regression w.r.t current behavior. So I would not
recommend changing current behavior. I think this should be an opt-in.
We are working on provide a config knob to elect this behavior and
atomic host and fedora server will have to opt-in somehow.

I think it will be easy for atomic host as they already drop something
in /etc/sysconfig/docker-storage-setup. Not sure how fedora server
variant will do it.

Thanks
Vivek

Re: Modifying container storage for Fedora 26.

By Josh Berkus at 11/22/2016 - 19:24

Vivek, Dan,

This is a change from current storage setup? Right now, containers go in the docker volume, but images do not. If we can do that, it would be worth having overlay right there.

My one concern for Atomic is ... how do existing users upgrade when we make this change? Does their devicemapper config still just work when they pull the new tree?

Re: Modifying container storage for Fedora 26.

By Vivek Goyal at 11/23/2016 - 17:14

On Tue, Nov 22, 2016 at 11:24:04PM -0000, Josh Berkus wrote:
[ Please don't remove emaild ids from oringal to, cc list. My emails
get filtered in a folder and I don't scan it often ]

Right now, with lvm thin pool, both images and containers get stored in
lvm thin pool. Some metadata associated with images and containers remains
on rootfs (in /var/lib/docker/ dir).

There will be no change to existing devicemapper functionality and upgrade
should continue to work fine.

This new functionality should kick in only if there is no existing setup.
(Signalled by the absense of /etc/sysconfig/docker-storage file).

Otherwise docker-storage-setup can detect that already there is a
configured driver and it will not setup a new one.

Vivek

Re: Modifying container storage for Fedora 26.

By Daniel J Walsh at 11/23/2016 - 09:36

On 11/22/2016 06:24 PM, Josh Berkus wrote:
Sadly I can not find it right now.

But you can do something like:

atomic storage export
atomic storage reset
atomic storage modify --driver overlay2
atomic storage import

Re: Modifying container storage for Fedora 26.

By Stephen Gallagher at 11/16/2016 - 15:49

On 11/16/2016 02:40 PM, Vivek Goyal wrote:

Today, Fedora Server relies on whatever is the default for docker-storage-setup.
We just tell Anaconda to reserve up to 15GiB by default for the / partition and
then it puts all remaining free space (on drives selected to be used by
Anaconda) into a single logical volume with no partitions.

It's a very easy thing for us to drop a different config file for
docker-storage-setup into place for Server. So if that's all we need to do, let
me know and I'll work it up.

Re: Modifying container storage for Fedora 26.

By Colin Walters at 11/16/2016 - 15:56

If anyone has time to work on this, I don't see a need for Atomic Host
and Server to have different partitioning defaults, so if we can just
merge them, that'd be nice.

Re: Modifying container storage for Fedora 26.

By Vivek Goyal at 11/23/2016 - 17:16

On Wed, Nov 16, 2016 at 02:56:35PM -0500, Colin Walters wrote:
Agreed.

Vivek

Re: Modifying container storage for Fedora 26.

By Vivek Goyal at 11/16/2016 - 15:56

On Wed, Nov 16, 2016 at 02:49:25PM -0500, Stephen Gallagher wrote:
Ok, that sounds good. We are working on providing a knob to opt-in new
behavior. I think all you have to drop in config file will be something
like.

/etc/sysconfig/docker-storage-setup

STORAGE_DRIVER=overlay2
YET_TO_BE_NAMED_OPTION=VAL

So upstream default will continue to be devicemapper. We will have to
modify fedora workstation, fedora server and atomic host infrastructure
to opt-in for overlay2.

Vivek

Re: Modifying container storage for Fedora 26.

By Stephen Gallagher at 11/16/2016 - 16:01

On 11/16/2016 02:56 PM, Vivek Goyal wrote:
Why exactly does this need to be opt-in? Why wouldn't we just change the default
on Fedora Server to use overlay2 instead of devicemapper?

I think I'm missing some key part of the problem here.

Re: Modifying container storage for Fedora 26.

By Vivek Goyal at 11/16/2016 - 16:09

On Wed, Nov 16, 2016 at 03:01:06PM -0500, Stephen Gallagher wrote:
I mean it will be devicemapper in upstream project. And distributions will
have to opt-in for overlay2.

And I think one reason being that rhel uses same git tree and we don't
want to switch to overlay2 by default for rhel yet.

overlay2 will be an experiment on fedora first as default and if it works
well, then change default upstream too.

Vivek

Re: Modifying container storage for Fedora 26.

By Stephen Gallagher at 11/16/2016 - 16:19

On 11/16/2016 03:09 PM, Vivek Goyal wrote:
OK, when you said "opt-in", I was assuming you meant that from the user's
perspective (as in having the user make an explicit choice). What you meant was
that Server would carry a different default from the upstream project. That's
easy enough to accomplish (we can do the same for Atomic as well).

The packaging guidelines for this are here:
<a href="https://fedoraproject.org/wiki/Packaging:Per-Product_Configuration" title="https://fedoraproject.org/wiki/Packaging:Per-Product_Configuration">https://fedoraproject.org/wiki/Packaging:Per-Product_Configuration</a>

Of course, that's only necessary if we want Server (and Atomic) to differ from
Workstation and/or non-productized Fedora. If they're all going to switch to
overlay2, then it's just a trivial matter.

Re: Modifying container storage for Fedora 26.

By Chris Murphy at 11/17/2016 - 00:30

Does it make sense to move all the products to XFS? An exception may
be possible for cloud images [1]

Server uses XFS. Cloud/Atomic have expressed a preference for it. I
found that ext4 allocates inodes at mkfs time, and there's no way to
get more later, and there's a real possibility with overlayfs that
there can be inode exhaustion. [2] Yes, you can just use non-default
mkfs options to create more inodes, but installer folks are typically
reluctant to use non-default mkfs options. XFS and Btrfs have
dynamically allocated inodes.

Is there any advantage of LVM for Cloud/Atomic? What about
Workstation? Dropping it for cloud/atomic may be an advantage, more
compact, less wasted space, less overhead, one less layer. For
Workstation it's probably a wash. Many users are confused by it, few
benefit from it. But the documentation refers to it.

[1]
Syslinux is preferred because it significantly reduces cloud image
size. The problem is this extlinux + ext4 bug. While syslinux 6.04
supports XFS, and builds are in koji, it's not finalized yet upstream.
So if this syslinux + ext4 bug gets fixed, then use use that
combination. Otherwise, GRUB + XFS.
Fedora 25 cloud images built with syslinux do not boot
<a href="https://bugzilla.redhat.com/show_bug.cgi?id=1369934" title="https://bugzilla.redhat.com/show_bug.cgi?id=1369934">https://bugzilla.redhat.com/show_bug.cgi?id=1369934</a>

[2]
<a href="https://lists.fedoraproject.org/archives/list/ ... at lists dot fedoraproject.org/message/47PC3YHBR7LTIDSDCZERROWRLEFOSVP7/" title="https://lists.fedoraproject.org/archives/list/ ... at lists dot fedoraproject.org/message/47PC3YHBR7LTIDSDCZERROWRLEFOSVP7/">https://lists.fedoraproject.org/archives/list/ ... at lists dot fedoraproject....</a>

Re: Modifying container storage for Fedora 26.

By Vivek Goyal at 11/16/2016 - 16:57

On Wed, Nov 16, 2016 at 03:19:06PM -0500, Stephen Gallagher wrote:
I think all are going to switch to overlay2 by default. But there will
be on difference. Fedora workstation will like to have its overaly2
setup on top of rootfs. While Fedora Server and Atomic Host will like
to carve out extra logical volume from free space and setup overlay2
on top of that.

So there will be total of two config options. One will be same in all
3 variants. While second one will be used only by server and atomic
host variants.

/me goes to read per product configuraiton link now.

Vivek

Re: Modifying container storage for Fedora 26.

By Subhendu Ghosh at 11/17/2016 - 20:54

Assuming cloud-init can also select the storage or is that too late in the
process?

Re: Modifying container storage for Fedora 26.

By Daniel J Walsh at 11/18/2016 - 08:46

No as long as the storage is modified before the first start of the
docker daemon and the first execution of docker-storage-setup, it should
be fine.

On 11/17/2016 07:54 PM, Subhendu Ghosh wrote:

Re: Modifying container storage for Fedora 26.

By Denise Dumas at 11/17/2016 - 21:42

So are we going to see you tomorrow at BoA? How is it going?
Denise