DevHeads.net

On not bumping the epoch in ceph-14, f30 and f31/rawhide

The epoch was inadvertently bumped (not by me) when ceph was rebased to
14.x in f30/rawhide.

I reset it to 1 in subsequent builds. Now adamwill is running builds with
it bumped to 2 again.

I would prefer that it not be bumped. Ceph has their own builds (for Fedora
even I think) where they have epoch=2. I see this as a feature that lets
someone install Ceph's epoch=2 packages on a system and not risk
inadvertently updating with the Fedora Ceph packages.

Is there really no other way to get rid of the one or two "bad builds"
where epoch=2 and keep shipping epoch=1 in Fedora? By untagging the builds
perhaps?

Thanks,

Comments

Re: On not bumping the epoch in ceph-14, f30 and f31/rawhide

By Dridi Boukelmoune at 03/14/2019 - 02:51

On Fri, Mar 8, 2019 at 9:08 PM Kaleb Keithley < ... at redhat dot com> wrote:
It just occurred to me that this was a normal update when epoch was
increased, right?

Maybe what we need is to have koji for example refusing epoch bumps
(and thus failing the build) if _not bumping_ epoch would _not break_
the upgrade path to ensure that epoch is only ever increased when we
need to force a downgrade or when upstream changes versioning schemes
in a way not considered an upgrade by RPM's NVR.

Dridi

Re: On not bumping the epoch in ceph-14, f30 and f31/rawhide

By =?UTF-8?B?TWlyb... at 03/14/2019 - 05:06

On 14. 03. 19 7:51, Dridi Boukelmoune wrote:
Or use Pull Requests in the workflow and have somebody else review the commit
before it gets to Fedora?

Re: On not bumping the epoch in ceph-14, f30 and f31/rawhide

By Dominik 'Rathan... at 03/14/2019 - 07:33

On Thursday, 14 March 2019 at 10:06, Miro Hrončok wrote:
While that's a good idea in general, how do you propose to implement
this for packages that have only a single maintainer?

Regards,
Dominik

Re: On not bumping the epoch in ceph-14, f30 and f31/rawhide

By =?UTF-8?B?TWlyb... at 03/14/2019 - 07:50

On 14. 03. 19 12:33, Dominik 'Rathann' Mierzejewski wrote:
Offer PR review swaps?

Re: On not bumping the epoch in ceph-14, f30 and f31/rawhide

By =?UTF-8?B?TWlyb... at 03/14/2019 - 07:55

On 14. 03. 19 12:50, Miro Hrončok wrote:
TBH I was mostly responding to the ceph package, where:

* there is more than 1 maintainer
* most of the commits are "dump hundreds of upstream changes here"

I'm concerned about this type of "maintenance". Giving the downstream spec some
real love could have prevented this issue (it would not eliminate human error
entirely).

Re: On not bumping the epoch in ceph-14, f30 and f31/rawhide

By Daniel P. Berrange at 03/11/2019 - 07:34

On Fri, Mar 08, 2019 at 03:07:09PM -0500, Kaleb Keithley wrote:
The ability to have multiple different builds of the same software which
users can choose between, sounds alot like the use case for modularity.
Abusing Epoch to try to address this kind of situation feels like a pretty
undesirable approach, as this problem with suddenly clashing Epochs will
illustrate.

If ceph in Fedora were a module, is it possible for Ceph upstream to
provide an alternate module stream of ceph too ? If so, then users
could use the normal modules features in DNF for deciding which stream
to have active on their systems.

Regards,
Daniel

Re: On not bumping the epoch in ceph-14, f30 and f31/rawhide

By Kaleb S. KEITHLEY at 03/11/2019 - 09:29

On Mon, Mar 11, 2019 at 7:35 AM Daniel P. Berrangé < ... at redhat dot com>
wrote:

If only there had been modularity before f29 that might have been
reasonable a reasonable claim, IMO.

But it wasn't. My issue is that there's no way to fix things when a
mistake is made.

Perhaps I misunderstand the purpose of rawhide. I appreciate that "we" try
to _not_ break things in rawhide, but when they do, there should be a way
to fix them.

Re: On not bumping the epoch in ceph-14, f30 and f31/rawhide

By Daniel P. Berrange at 03/11/2019 - 09:58

On Mon, Mar 11, 2019 at 09:29:52AM -0400, Kaleb Keithley wrote:
Sure, in the past it wasn't possible, but I think it is a more viable
long term approach to the general scenerio.

Well technically Fedora rawhide isn't actually broken by the epoch change.

All that's broken is a 3rd party's assumption that their Epoch setting
is greater than Fedora's. Assuming Ceph want to keep using Epoch in
this way, upstream can simply bump their Epoch again to be greater than
Fedora's new Epoch.

Regards,
Daniel

Re: On not bumping the epoch in ceph-14, f30 and f31/rawhide

By Dridi Boukelmoune at 03/11/2019 - 10:45

Or bump their Epoch to something much less likely to conflict in the
future, like 10 or 100.

Dridi

Re: On not bumping the epoch in ceph-14, f30 and f31/rawhide

By King InuYasha at 03/11/2019 - 08:20

On Mon, Mar 11, 2019 at 7:35 AM Daniel P. Berrangé < ... at redhat dot com> wrote:
If there was a material difference between upstream and downstream,
you might have a case for it. Today, there is not. Heck, the spec file
that is in Fedora is basically an openSUSE spec with Fedora
conditionals in it. It even has the (IMO dumb) license header thing
that openSUSE forces for all of its package spec files.

Not that it's necessarily a bad thing that the spec files are
identical, but I'd rather just say we should not care, since there's
no appreciable difference between the two.

Re: On not bumping the epoch in ceph-14, f30 and f31/rawhide

By Kaleb S. KEITHLEY at 03/11/2019 - 09:11

The ceph.spec file in Fedora is based on the upstream ceph.spec.in file;
not on anything in/from openSUSE.

The upstream ceph.spec.in file is full of Fedora and SUSE conditionals.

If openSUSE also used the upstream spec file then it shouldn't surprise
anyone that they are similar.

Re: On not bumping the epoch in ceph-14, f30 and f31/rawhide

By King InuYasha at 03/11/2019 - 09:26

On Mon, Mar 11, 2019 at 9:12 AM Kaleb Keithley < ... at redhat dot com> wrote:
I'm keenly aware of where that spec file comes from, since I saw how
it was developed. It did start out as something for Fedora, but SUSE
folks started actively contributing in 2012 and merging their
packaging into upstream, changing it from a Fedora-style package to a
SUSE-style one.

Today, Ceph packaging in OBS is fetched through a source service and
used pretty much verbatim from upstream. I imagine you do something
similar to bring it downstream, too.

I'm not begrudging them of it, mind you. But it's a lie to say that it
isn't an openSUSE-style spec file. It's nice to know that we're mostly
compatible these days...

Re: On not bumping the epoch in ceph-14, f30 and f31/rawhide

By Adam Williamson at 03/08/2019 - 17:20

On Fri, 2019-03-08 at 15:07 -0500, Kaleb Keithley wrote:
I did send you an email about this to explain.

The builds with epoch 2 made multiple Rawhide composes and the first
Fedora 30 compose. This means anyone who installed 30 or Rawhide with
ceph packages during those periods will never get ceph updated when
they do 'dnf update', because they will have a package installed with
epoch 2 and dnf will not see the package with epoch 1 as an update.

Untagging the builds does not help anyone who got them installed while
they were in Rawhide or Fedora 30 (in fact Fedora 30 still contains an
epoch 2 build right now, so anyone installing it at present is getting
epoch-2 ceph).

We can talk about potentially allowing epoch down bumps at a distro
upgrade boundary (though this rather hangs Rawhide users out to dry),
but even if we were to start allowing that, since an epoch 2 build made
it into a Fedora 30 compose, we can never bump the epoch down to 1 for
Fedora 30's lifetime.

Re: On not bumping the epoch in ceph-14, f30 and f31/rawhide

By King InuYasha at 03/08/2019 - 16:16

On Fri, Mar 8, 2019 at 3:08 PM Kaleb Keithley < ... at redhat dot com> wrote:
As of right now, no. Once it goes out in a compose, that's the way it goes...

I really wish we'd allow Epochs to be reset on distribution upgrades.
With dnf distro-sync (which is used by system-upgrade) Epochs don't
really matter and upgrades work as intended anyway...

And EVR games like this are really annoying. :(

Re: On not bumping the epoch in ceph-14, f30 and f31/rawhide

By =?UTF-8?B?TWlyb... at 03/08/2019 - 17:08

On 08. 03. 19 21:16, Neal Gompa wrote:
Let's do a Fedora change? Coordinate with FPC?

Allowing Epoch to be reset between releases (Was: On not bumping

By Jason L Tibbitts III at 03/08/2019 - 18:19

MH> On 08. 03. 19 21:16, Neal Gompa wrote:
MH> Let's do a Fedora change? Coordinate with FPC?

We (FPC) have talked about this before but I don't think it's really up
to FPC. The change process isn't really the right way to do it, either,
since this is really a policy revision. I just think FESCo needs to
decide whether or not it would like to relax the policy, and if so, how.

Here are the relevant points as I see them (unless I'm forgetting
something):

* dnf system-upgrade generally handles versions going backward without
issue. The specific case of epoch being reset is not an issue.

* dnf upgrade would not handle this, causing problems for those running
rawhide or using unsupported methods of upgrading between releases.

* Those running rawhide would have to run distro-sync in order to
upgrade (which would of course reset any locally built updated
packages and such). They would have to do this every time any
installed package drops epoch.

* Those using an unsupported method of upgrading would need to
incorporate distro-sync.

* Dropping epoch outside of rawhide would generally be bad.

* Koji and the compose process do handle things "going backwards", as
this has happened multiple times in the past without things dying.
What's important there is which version was most recently tagged.

* Bodhi shouldn't be involved here as this would be restricted to
rawhide.

Personally I'm in support of relaxing the restriction in some form, but
I prefer a single "drop Epoch: day" where epochs in rawhide are
_automatically_ removed and the packages rebuilt. This gives a single
point in time where rawhide users need to do a distro-sync in order to
properly track updates. Allowing epochs to be dropped without
coordination seems to me to be a burden on rawhide users, but I don't
think a single flag day would be problematic.

I would expect the first flag day to be busy. I see 756 Epoch: tags
currently, though 161 are set to 0 for whatever reason. Afterwards I
would expect no more than a small number of packages per cycle to
acquire Epoch: tags.

- J<

Re: Allowing Epoch to be reset between releases (Was: On not bum

By Randy Barlow at 03/11/2019 - 11:32

On Fri, 2019-03-08 at 16:19 -0600, Jason L Tibbitts III wrote:
Just a note that we do have plans to use Bodhi to manage Rawhide in the
future, and will hopefully have it doing this in 2019.

Bodhi is not currently Epoch-aware, but I wouldn't be opposed to
teaching it about Epochs if we wanted to make a change in this area.

Re: Allowing Epoch to be reset between releases (Was: On not bum

By =?ISO-8859-1?Q?... at 03/09/2019 - 08:00

Dne 08. 03. 19 v 23:19 Jason L Tibbitts III napsal(a):

If DNF were helpful and was able to report packages, which has higher
NVR then E(NVR), then I can imagine that reset of epoch could work.
Otherwise I am against resetting epochs.

Vít

Re: Allowing Epoch to be reset between releases (Was: On not bum

By =?ISO-8859-1?Q?... at 03/09/2019 - 08:10

Dne 09. 03. 19 v 13:00 Vít Ondruch napsal(a):

I meant (E)NVR actually.

V.

Re: Allowing Epoch to be reset between releases (Was: On not bum

By King InuYasha at 03/09/2019 - 10:37

On Sat, Mar 9, 2019 at 7:11 AM Vít Ondruch < ... at redhat dot com> wrote:
I'm confused, why would that matter? And DNF always reports NEVRA...

Re: Allowing Epoch to be reset between releases (Was: On not bum

By =?ISO-8859-1?Q?... at 03/11/2019 - 05:28

Dne 09. 03. 19 v 15:37 Neal Gompa napsal(a):
The epoch are used in cases like:

1. There is  foo version 1.0

2. foo is updated to version 2.0, because it seems it is safe.

3. It is discovered, that it breaks stuff, so the decision is go back to
1.0 and add epoch.

4. Eventually, we really want to have 2.0 in the release or even
something newer. Now, the epoch could be removed, but it is not
possible, because it has the highest priority.

In this case, if DNF said something like "you have installed foo-1:1.0,
but there is available foo-0:2.0" it would give me hint. From the start
it would be annoying, but once we would reach the point 4, I would, at
least, know that I should do distrosync or something.

V.

Re: Allowing Epoch to be reset between releases

By Jason L Tibbitts III at 03/11/2019 - 15:50

VO> In this case, if DNF said something like "you have installed
VO> foo-1:1.0, but there is available foo-0:2.0" it would give me
VO> hint. From the start it would be annoying, but once we would reach
VO> the point 4, I would, at least, know that I should do distrosync or
VO> something.

Under the proposal I put forward:

1. No releases except for rawhide would ever be affected by this,
assuming that users upgrade using supported methods.

2. Rawhide users would need to do this exactly once per cycle, on an
announced date.

So you would know that you should do distrosync because that would be
announced.

- J<

Re: Allowing Epoch to be reset between releases

By Zbigniew =?utf-... at 03/12/2019 - 05:52

On Mon, Mar 11, 2019 at 02:50:44PM -0500, Jason L Tibbitts III wrote:
This doesn't sound convincing at all. We *know* that people miss
announcements all the time. Dropping epochs would introduce yet
another case where a "magical" step is needed at a specific time.

We need to remember that dropping epochs also impacts any package
which uses Requires/BuildRequires/Recommends/Conflicts/Obsoletes
on the package dropping the epoch.

Elsewhere in the thread people raised:
- koji-shadow
- external build systems
- third party repos
- custom packages

All those will require periodic rebuilds. The problem is that
those things don't necessarily follow the cadence of Fedora releases.
The proposal to drop epochs sounds like a step that is problematic
and causes extra work now and ongoing for third-party packagers.
And the problem that it solves is niche. The cost of the solution
doesn't seem justified.

Zbyszek

Re: Allowing Epoch to be reset between releases

By Jason L Tibbitts III at 03/13/2019 - 18:43

ZJ> This doesn't sound convincing at all.

I was not attempting to be convincing.

ZJ> We *know* that people miss announcements all the time. Dropping
ZJ> epochs would introduce yet another case where a "magical" step is
ZJ> needed at a specific time.

Personally I don't see it as being excessive. Plus... if you're running
rawhide you do already have to deal with this, when it's done
accidentally. I believe a proposal to do it in a coordinated fashion
actually helps the situation.

ZJ> We need to remember that dropping epochs also impacts any package
ZJ> which uses Requires/BuildRequires/Recommends/Conflicts/Obsoletes on
ZJ> the package dropping the epoch.

Yes, that was covered in previous discussion.

ZJ> All those will require periodic rebuilds. The problem is that those
ZJ> things don't necessarily follow the cadence of Fedora releases.

Yes, all of these are confounded by epochs currently. I believe that
after the epochs have been removed, the situation is actually better
than it is now.

ZJ> The proposal to drop epochs sounds like a step that is problematic and
ZJ> causes extra work now and ongoing for third-party packagers.

Personally I just don't see the problem. I'm not saying that it has
nonzero cost, but I don't see it as being major.

ZJ> And the problem that it solves is niche. The cost of the solution
ZJ> doesn't seem justified.

I wonder how the existing RPM-based distros which allow epochs to go
away between releases handle this. Aren't we the last one that doesn't?

- J<

Re: Allowing Epoch to be reset between releases

By =?ISO-8859-1?Q?... at 03/12/2019 - 04:55

Dne 11. 03. 19 v 20:50 Jason L Tibbitts III napsal(a):

So maintainers would not be allowed to remove epoch, but there would be
some script/automation, which would remove epoch on demand, once per
release, in all packages? Interesting idea.

Anyway, I still believe DNF could report when there is package 0:2.0,
while there is also 1:1.0, because this change, if accepted, is going to
redefine the meaning of epoch anyway. Epoch would basically become some
temporary override no matter what is the precise process.

Vít

Re: Allowing Epoch to be reset between releases (Was: On not bum

By Stephen John Smoogen at 03/11/2019 - 06:47

On Mon, 11 Mar 2019 at 05:29, Vít Ondruch < ... at redhat dot com> wrote:
Whatever changes to EPOCH rules will need additional logic added to
all the various buildsystem logic where a human can't sit around and
choose 'oh yeah I want to go to that older epoch' because the build is
automated somewhere. This has been the major reason why various 'EPOCH
should go back' or 'I want an EPOCH to my EPOCH' conversations in the
past have floundered (I think we last looked at this in 2009 and I
know we did in 1999.. so maybe this is a 10 year cycle?). There is a
LOT of stuff written on the assumption that EPOCH's go forward and
never backwards. Changing the rules here have effects in many many
other build systems and install tools which sites are using and that
usually ends up being too big of a problem to solve. [Yes we can fix
our koji/bodhi/greenwave/waiverdb/pungi/mock/copr, and their koji and
maybe that other other koji... but how do we fix the plague server
building stuff at large industrial complex-1 or the clone of the OSBS
at large-government-research-place-2. ]

[If I remember the discussion from 2009 correctly, it was a similar
problem and the idea was to have an vip epoch which sat in front of
epoch and could override it so you go back to epoch.. this led to a
bunch of elephants standing on each other. The 1999 one was having
release logic in the installer which could over-ride epochs.]

Re: Allowing Epoch to be reset between releases (Was: On not bum

By King InuYasha at 03/11/2019 - 07:22

On Mon, Mar 11, 2019 at 6:48 AM Stephen John Smoogen < ... at gmail dot com> wrote:
I don't remember how Plague handles this anymore (forgive me, but I
haven't interacted with Plague since 2005!), but both Koji and OBS
don't care if the Epoch goes up or down. Koji uses NVR as a key
(without Epoch), and OBS freely allows EVRs to go up and down.

Any change requires a release bump anyway, so from that perspective,
things are usually fine.

So I think from that perspective, we should be okay.

Re: Allowing Epoch to be reset between releases (Was: On not bum

By Mikolaj Izdebski at 03/11/2019 - 07:40

On Mon, Mar 11, 2019 at 12:23 PM Neal Gompa < ... at gmail dot com> wrote:
Some parts of Koji do rely on epoch not doing down. Notably,
koji-shadow uses epoch and it will stop working correctly when epoch
is removed or decreased. koji-shadow is not currently used by Fedora
project, but it may be used in future to bootstrap new architectures.
And it is still used by third parties to rebuild Fedora packages.

Re: Allowing Epoch to be reset between releases (Was: On not bum

By =?UTF-8?B?TWlyb... at 03/08/2019 - 20:47

On 08. 03. 19 23:19, Jason L Tibbitts III wrote:
One thing to consider here is other packages that have Requires etc. on
something like "foo > 1:1.2", so if it is automated, this part needs to be
automated as well.

If we do this, we might have a "flag day" but not automated for all 756
packages, but opt in. Aka we create a window on rawhide, and packagers would
sign up for this, we announce the window, let them do it, and we close the
window... ?

This needs a lot of consideration for sure, but I think it can be done somehow.
However I'm not sure it's worth the effort.

Re: Allowing Epoch to be reset between releases

By Jason L Tibbitts III at 03/08/2019 - 21:06

MH> One thing to consider here is other packages that have Requires
MH> etc. on something like "foo > 1:1.2", so if it is automated, this
MH> part needs to be automated as well.

Indeed. And of course this breaks any such dependency outside of Fedora
as well. (Either in COPR repositories, RPMFusion, or local packages.)
I should have mentioned this as a potential downside, since it was part
of previous discussions.

MH> If we do this, we might have a "flag day" but not automated for all
MH> 756 packages, but opt in.

Sure, maybe at first. Stage it in over a couple of releases if
necessary, or having a couple of flag days. Though it's not quite as
many if you exclude the Epoch: 0 ones. (Though I recall that there is
some subtlety between Epoch: 0 and no epoch at all.)

But that's an implementation detail; the fundamental question is whether
there is any general support for dropping epochs, and more specifically
if FESCo would accept it on principle. And I can understand why they
may not, because while epochs are annoying, we've certainly been living
with them for quite some time.

MH> Aka we create a window on rawhide, and packagers would sign up for
MH> this, we announce the window, let them do it, and we close the
MH> window... ?

The issue is that if a compose happens while the window is open, we have
more than one rawhide update where distro-sync is needed. I think it's
worth spending a bit of effort to minimize the issues for those running
rawhide.

MH> However I'm not sure it's worth the effort.

Yes, that's basically the problem. History has shown that we can live
with epochs. But even if it's a limited effort, it would be nice to
give packagers who want to get rid of epochs a way to do so.

- J<