RFC: Multiple parallel side tags


Imagine situation that somebody is working on KDE rebase and me on
libgit2 rebase. Both involve rebuilding/updating some package, let's
say kf5-ktexteditor.

We both work in different side tags, in KDE rebase kf5-ktexteditor
gets updated to a new version. In libgit2 rebase, old version gets

Once any of us finished with side tag, we merge it. Let's say that was
KDE rebase. That means, new kf5-ktexteditor is merged into the rawhide
which is built against old libgit2. Then I finish with libgit2 things
and we merge it into the rawhide.. That means it just downgrades
kf5-ktexteditor version because koji looks at the build time and not
the NVR. And even if it did, that would mean I have to manually go and
rebuild all packages which are handled in multiple side tags.

Do you think that scales?


Re: RFC: Multiple parallel side tags

By =?ISO-8859-1?Q?... at 06/10/2019 - 04:34

I think there should be some feature in fedpkg, which would check for
side-tag builds and prevent any other builds if there is. The problem
is, that people does not know there are some builds in side tag and that
is understandable.


Dne 08. 06. 19 v 11:23 Igor Gnatenko napsal(a):

Re: RFC: Multiple parallel side tags

By Stephen John Smoogen at 06/08/2019 - 10:29

On Sat, Jun 8, 2019 at 6:12 AM Igor Gnatenko <

So I believe at one point we only allowed one side tag at a time, but
people complained that most of the time we were wasting packagers time
waiting for someone else to get stuff done. There was an idea about trying
to solve conflicts but that was seen as too much time to solve and too much
bureaucracy to live with. Instead we assumed that these would happen and
that packagers would work things out between themselves.

So no it doesn’t scale. The various solutions to make it scale are not
written and will not be written in any near time as they will need some
sort of project, some amount of resources added and a backlog of more
critical work done first. Until then packagers need to realize that there
are going to be build conflicts every now and then and factor in that they
will have to work out between themselves how to solve them.

Re: RFC: Multiple parallel side tags

By Zbigniew =?utf-... at 06/08/2019 - 13:20

On Sat, Jun 08, 2019 at 10:29:29AM -0400, Stephen John Smoogen wrote:
Is that true? If yes, that seems to be the wrong thing to do.
Instead, koji should say "not merging a-nn.rpm, because a-nn.rpm is
already in the tag", or "... a-nn+1.rpm is already in the tag".
Ideally, this would cause the whole merge to be rejected, and you
can bump and rebuild the offending packages, and try to merge again

Yes. But I think that'd be OK. This isn't *that* much work, and at
least there'd be a clear result and a precise list of packages that
need to be rebuilt. That's much better than a partial merge that
breaks the next compose.

Yeah, I don't think any technical solution is worth the trouble here.
But we can coordinate, e.g. by announcing on fedora-devel.


Re: RFC: Multiple parallel side tags

By =?UTF-8?B?TWlyb... at 06/10/2019 - 03:31

On 08. 06. 19 19:20, Zbigniew Jędrzejewski-Szmek wrote:
If this would block the merge, we would be stuck in an endless loop with big
side tags (read Python). While I would be rebuilding the 200 merge conflicts, I
would already get 100 more. Once libreoffice or clang conflicts, I would get
500. It's easier to just merge and than rebuild the remaining packages once more.

Last time (Python 3.7) I've asked maintainers not to rebuild Python packages in
rawhide unless really necessary. Some probably even read it, but mostly it was
just ignored. This time, I plan to CC everybody who's involved.

Re: RFC: Multiple parallel side tags

By Vitaly Zaitsev ... at 06/08/2019 - 06:00

Le samedi 08 juin 2019 à 11:23 +0200, Igor Gnatenko a écrit :

But, what is different here from the Fedora circles / Fedora modules /
etc endeavours? Isn’t the root problem synchronizing common code paths,
because free software means pervasive code reuse, apps ends up being
deployed together, and un-sharing generates collisions / API
incompatibilities / behaviour incompatibilities / config file
incompatibilities / un-adressed security issues?

What makes it possible in modules but not in side tags?


Re: RFC: Multiple parallel side tags

By Kevin Kofler at 06/08/2019 - 18:20

Nicolas Mailhot via devel wrote:
Modules are just as broken.

We really need to go back to just doing all the work directly in the one and
only Rawhide, no modules, side tags, or other such "features" that are just
conflicts waiting to happen. Yes, this means Rawhide will go back to being
broken for people trying to actually use it while a transition is ongoing,
but the point of Rawhide is to do development in it, NOT to be used. It
should be fully expected that Rawhide does NOT work as an actual distro,
ever. Its one and only purpose should be to coordinate the package
development so it all happens in one place.

Kevin Kofler

Re: RFC: Multiple parallel side tags

By Martin Kolman at 06/10/2019 - 07:28

Not to mention trying to debug the breakage when nothing really works and
more stuff is constantly landing. I thin there needs to be *some* sort
of balance.

Re: RFC: Multiple parallel side tags

By Kevin Kofler at 06/10/2019 - 18:00

Martin Kolman wrote:
In the past, the compose would not fail just because a release-blocking
deliverable failed to compose, the package compose would just ship without
the deliverables that failed to compose. That's what needs to happen again.
There is no technical reason why an error composing a live ISO prevents the
changed RPMs from going out. Rawhide is not a release, there should be time
until the actual release to fix the release-blocking deliverable.

And ideally, we would keep the last successful compose of each deliverable
on the mirrors, no matter whether it matches the current package compose or
not. (We did not do that in the past, the ISO was just missing when the
latest version failed to compose, which is suboptimal.)

We need to stop having unreasonable expectations about Rawhide's quality,
and ISOs always composing successfully is one such unreasonable expectation.

Kevin Kofler

Re: RFC: Multiple parallel side tags

By Kevin Fenzi at 06/17/2019 - 16:23

On 6/10/19 3:00 PM, Kevin Kofler wrote:
With the 'no more alphas' change, we strive to make rawhide always alpha
quality at least. This means all the release blocking deliverables are
there and working.

But the problem is there is not time. If you don't fix things as soon as
they happen, more pile up behind it making it harder and harder to fix.

I disagree. I think we need gating to block as much stuff that breaks
things from landing as we can and then we should find that keeping
composes going is much easier on all of us. Then things can be fixed
when gating catches them and it's on the person who broke things.


Re: RFC: Multiple parallel side tags

By Kevin Kofler at 06/17/2019 - 19:47

Kevin Fenzi wrote:
And that is going to make development completely cringe to a halt. It is the
nature of a distribution branch under development that things will sometimes
be completely broken for a couple weeks. There needs to be a place to do
development that can cause such temporary breakage.

The side tag approach already does not scale, as evidenced by this thread.
Forcing every small breakage into a side tag is going to make this issue
much worse. All the merge conflicts can be avoided if development is simply
done together in one place.

The purpose of Rawhide should be only and purely to develop future versions
of the distribution. If it accidentally happens to be usable at some point
in time, that is pure luck and no sane user should ever rely on that.

Kevin Kofler

Re: RFC: Multiple parallel side tags

By Kevin Fenzi at 06/17/2019 - 20:03

On 6/17/19 4:47 PM, Kevin Kofler wrote:
I again completely disagree. There is no reason for weeks of breakage.
Most of the issues that break composes are unannounced abi bumps where
just rebuilding dependent packages fixes it. Or broken deps (likewise).
Or mistakes made in kickstarts/comps. Or something that doesn't even
run. What good does having everyone broken for weeks do?

It does. You just need to communicate with others working in the same
area, IMHO. I don't think we need some technical thing for something
that happens rarely and can be solved by more communication.


Re: RFC: Multiple parallel side tags

By Kevin Kofler at 06/18/2019 - 18:48

Kevin Fenzi wrote:
There are several reasons why weeks of breakage are entirely reasonable for
a distribution under development:

1. A bump of a library to a new major version. It can take a few weeks for
dependent packages to be ported to the new version. But it is often worth
waiting rather than introducing a compatibility library with all the mess
that comes with it. (Of course, that does not hold for a library like Qt
where getting everything ported to the new version is not realistic even in
a couple weeks. Those are the cases where a compatibility library is needed
basically forever. But there are libraries where it makes sense to just port
everything.) The workflow would be to just bump the library to the new
version in Rawhide immediately and then let dependent packages get fixed
over time as their upstreams, Fedora packagers, or other distros' packagers
port them.

2. Vacation. The maintainer of the dependent package may be on vacation (or
even just busy with paid work, in case of volunteers) and therefore unable
to fix their package immediately. So you may have to wait a few weeks for
the broken dependency to get fixed if a simple rebuild is not enough.

This has all worked just fine in the past, before it was decided to make
basically every single broken dependency fail the entire Rawhide compose.
Soname bumps did not even have to be announced, they would get announced by
the broken dependency report within less than 24 hours anyway, and then
eventually fixed (without somebody unrealistically expecting maintainers to
fix the broken dependency in less than 24 hours to make the next compose
succeed, which is just plain impossible for the average volunteer packager,
especially if source code changes are needed).

Kevin Kofler

Re: RFC: Multiple parallel side tags

By Pierre-Yves at 06/19/2019 - 06:02

On Wed, Jun 19, 2019 at 12:48:49AM +0200, Kevin Kofler wrote:
I think both are valid reasons but I disagree that they should impact the entire
Fedora. So rather than breaking rawhide, use a side-tag, take all the time you
need and don't affect other people until everything is ready.
Whether you are the one doing all the rebuilds or working with other people,
the side tag model allows you to go at your own speed without affecting anyone

This is both true and false. Yes this is how it used to be, no it did not work
"just fine". It relied on other people doing heroic work to clean up someone's
mess. We've had volunteers that were spending hours just rebuilding broken
dependencies due to an unannounced soname bumps. I'm sure these people would
have preferred to spend this time with their family, playing video games or just
contributing to other parts of Fedora.

With the new model, you do not impose work on someone else (which may also be in
vacation as you point out here, and thus keeping things unfixable for a
potentially long period of time), you either fix everything yourself or work
with other to fix them and then you can affect the entire distribution.
And if you want to bypass the gating mechanism, it is possible, but then it's a
concious decision that you are making to make other people's life miserable and
you may be asked the reasons for which you made this decision.


Re: RFC: Multiple parallel side tags

By =?ISO-8859-1?Q?... at 06/18/2019 - 02:58

Dne 18. 06. 19 v 2:03 Kevin Fenzi napsal(a):

I think this has two sides. While you say there is not reason for weeks
of breakage, the broken compose means that my Rawhide can't be updated
and it might be broken for weeks. E.g. last Rawhide compose succeeded on
9th of June. If at least the Rawhide repository was updated, I would
probably notice the breakage.

Also, while we are informed about successful composes, there is
unfortunately no information about broken composes. I know, I was
pointed several times to some repositories with some tickets or what
not, but that is hardly useful.


Re: RFC: Multiple parallel side tags

By Igor Gnatenko at 06/17/2019 - 21:57

On Tue, Jun 18, 2019 at 3:01 AM Kevin Fenzi < ... at scrye dot com> wrote:
Actually with Rawhide Gating, this will happen a lot. Most of the
packages are independent, but there are way too many of those which
are tight with others. So people will have to use side tags for
updating 2-3 packages really a lot.

I don't think "more communication" is the solution. If people are
blocked by someone else, unless they have much free time they will
just walk away.

Re: RFC: Multiple parallel side tags

By King InuYasha at 06/17/2019 - 21:02

On Mon, Jun 17, 2019 at 8:53 PM Kevin Fenzi < ... at scrye dot com> wrote:
And that comes down to people shouldn't need to have to think about
these things when working in Rawhide. While I don't disagree that
Rawhide should be usable, I fundamentally disagree with making it
harder for people to put things in Rawhide. We should be developing
our tooling to make it _easier_ for stuff to go into Rawhide, and have
Rawhide fix itself when the issues are relatively trivial to fix (such
as reverse dependency rebuilding).

Side tags will happen a lot more often because the tooling is pushing
us to do it that way. Don't discount the potential for future
insanity. I'm still not sure side-tags are enough. Could we have a
concept of a "scratch side tag"? Something like a scratch build, but
contains a collection of builds and creates an overlay repo that can
be used to run checks on for auto-merging? If they're good, then it
would get auto-built properly into the main rawhide tag (or even
stable tag!).

Re: RFC: Multiple parallel side tags

By Aleksandra Bookwar at 06/18/2019 - 04:31

On Tue, Jun 18, 2019 at 3:05 AM Neal Gompa < ... at gmail dot com> wrote:
I think this is exactly what gating is supposed to do.

Let's compare it in this way:

currently to add a feature which may break Rawhide and some unknown
dependencies of the component you need to write a HEADS-UP e-mail, or,
better, submit a Change request with the analysis of the change.
People who read this e-mail would need to make a guess on whether or
not his change affects them, then they have to fetch it and test it
somehow, then they have to provide the feedback back to you. You need
to wait for feedback, then you get the reports, in best cases - bugs,
which you need to debug, requesting more info. Then you implement the
change, hit the unexpected bug, which was unnoticed, and block others
from building their packages and implementing their changes for
unknown amount of time.

With gating you can submit a code change, the tooling will take care
of building it, building its dependencies, informing you of possible
breakages, giving you the list of actual issues with all the debug
logs. And then based on this data you can proceed or stop and rework
the change a bit more in collaboration with exactly those people

I also think that we need a second point of view here: you were
talking about not driving away the developer, who makes the change.
But there are also other people who we shouldn't drive away. For
example developers who depend on the change. Or QA who need to react
on such changes. These people are have their part in the process.

But, to be honest, I think there is a bit of overreacting on the
entire Gating topic.

It doesn't do a hard block. It is included in the design that gating
can be bypassed. But it supposed to provide better analysis of the
change. Bypassing of the gate can happen, The key here is that it
won't be a surprise, rather informed decision.

Afaik, this is exactly the concept of a dynamic sidetag as Fedora
Infra is currently implementing. Sidetag in koji as a sort of
pull-request: you create sidetag, get repos and composes built out of
it, run tests, get results. Then "merge" this sidetag into the main

And while clashing sidetags is a problem, it is the same problem as it
happens with overlapping pull-requests when several people are working
on the source code.

There are several ways to address it:

1) create smaller pull-requests and merge them more often

This is one of the core points of CI workflows and it also seems to be
most often underestimated. There is an overhead in creating smaller
changes, which we are trying to address with tooling. But the overhead
is _linear_. While the complexity of one merge grows exponentially
with the size of the branch you are merging.

The smaller the changes - the easier the rebase.

2) reorganize the codebase

When working with source code, it usually means splitting files. If
there are many changes targeting the same file again and again,
causing conflicts, it might mean this file has too much logic bundled
in it and needs a split up.

It might also be merging files. For example if there are many changes
targeting two files and it causes problems when these two files get
out of sync.

When applied to RPM packages, it may be also a good question: if there
is circular dependency between packages, so that these are strictly
tied to each other by version and can only be processed together - why
these are separate packages and not subpackages?

3) Talk about changes in advance

This of course is the default, and it doesn't go anywhere with gating
or without it. Again I believe gating can make it easier. For example,
we can add a note on conflicting "pull-requests" and put a rebase
requirement as a gating check.

Re: RFC: Multiple parallel side tags

By King InuYasha at 06/18/2019 - 07:28

On Tue, Jun 18, 2019 at 5:21 AM Aleksandra Fedorova < ... at bookwar dot info> wrote:
I think this is the first time I've heard a coherent description of
the intent of this stuff. If it really is intended to work this way,
it'd be very helpful.

I think you kind of missed what I was asking: I want side tags that
are composed entirely of scratch builds. The result *must not* be used
to merge into the distribution. It *must* be thrown away and when the
pull request (or whatever) is merged, it should trigger the build
again, which would be a "real" side tag that would get auto-merged on

This is more or less the equivalent of a staging/integration overlay
that is used to test, where the artifacts are not preserved because
they aren't useful beyond that.

Re: RFC: Multiple parallel side tags

By Aleksandra Bookwar at 06/18/2019 - 10:54

On Tue, Jun 18, 2019 at 1:31 PM Neal Gompa < ... at gmail dot com> wrote:
You are right, I missed that part. And I agree this is needed. But I
am not sure if there is already a proposal for that.

There are several ways how one can implement it. I would personally
prefer pull requests to dist-git over scratch builds. I think it is
important to keep some visibility in the system, so that not just the
owner of a change, but also random lurkers can see what is being
prepared. (Also solves the )

And if we look into PR-based workflows, then Zuul team has a proof of
concept pipeline which builds packages on PR in Pagure.
Which is not your ordinary Jenkins PR-triggered pipeline.

The killer-feature of Zuul is that it can manage cross-project
dependencies. It understands links between pull-requests in different
git repositories, even in different Git forges (Pagure and GitHub for

If, let's say, you mark PR#123 in rpms/dnf as dependent on PR#765 in
rpms/rpm. Zuul fetches source code for both projects into the
workspace, and runs the pipeline in this workspace. Results of the
pipeline will be applied to both PR's.
Zuul could also do the merges for you, then it will also ensure that
the chain merge of dependent patches happen in the right order.

I like this concept, and I wonder if we can map this workflow cleanly
on the sidetag process, which happens after code is merged in

Re: RFC: Multiple parallel side tags

By =?ISO-8859-1?Q?... at 06/19/2019 - 04:27

Dne 18. 06. 19 v 16:54 Aleksandra Fedorova napsal(a):

Zuul team? Zuul feature? I am probably missing some context ...

It does not look like you are talking about
<a href="" title=""></a> which is the first reference to Zuul
returned by Google.


Re: RFC: Multiple parallel side tags

By Aleksandra Bookwar at 06/19/2019 - 04:38

On Wed, Jun 19, 2019 at 10:29 AM Vít Ondruch < ... at redhat dot com> wrote:
Yes, for me the OpenStack's Zuul is the default one :)

Project: <a href="" title=""></a>
It comes from OpenStack Infrastructure as an attempt to replace
Jenkins with smth scalable.

Fedora-related work <a href="" title=""></a>

Re: RFC: Multiple parallel side tags

By Aleksandra Bookwar at 06/18/2019 - 10:57

Sorry, missed part of the sentence.

Please read as: Also solves the issue when there are overlapping
sidetags prepared by different people.

One can subscribe to pull-requests in a project, and get early alerts
about planned changes.

Re: RFC: Multiple parallel side tags

By Fabio Valentini at 06/18/2019 - 06:24

On Tue, Jun 18, 2019 at 11:21 AM Aleksandra Fedorova < ... at bookwar dot info> wrote:
This was the most coherent argument on this topic I've read so far.
While I don't agree with all the points, the three points outlined
above are pretty on point, IMO.
Thank you.


Re: RFC: Multiple parallel side tags

By Igor Gnatenko at 06/08/2019 - 07:21

On Sat, Jun 8, 2019 at 12:54 PM Nicolas Mailhot via devel
< ... at lists dot> wrote:
I never said that it is better / possible in modules. I just wanted to
point out that expecting that people will do double, triple, … work in
side tags is not something what I would like to see (see thread about
libgit 0.28.x update). Also rawhide gating just makes this problem
even worse.

Re: RFC: Multiple parallel side tags

By Fabio Valentini at 06/08/2019 - 08:25

On Sat, Jun 8, 2019 at 2:10 PM Igor Gnatenko
< ... at fedoraproject dot org> wrote:
Would it be possible to block builds of packages for certain target
tags? Like a "mutex for packages"?

I'm thinking about specifying a list of packages at side-tag creation
time, and then builds of these packages targeting any other tag could
be blocked by koji, until the side "blocking" tag is merged back.


Re: RFC: Multiple parallel side tags

By King InuYasha at 06/08/2019 - 09:56

On Sat, Jun 8, 2019 at 9:15 AM Fabio Valentini < ... at gmail dot com> wrote:
As far as I know, the best we could do is block packages from being
built by other people (ACLs), but not block building in other tags.

Re: RFC: Multiple parallel side tags

By Igor Gnatenko at 06/08/2019 - 09:26

On Sat, Jun 8, 2019 at 3:21 PM Fabio Valentini < ... at gmail dot com> wrote:
That means we get huge bottleneck, why don't we just fix this problem
by automated rebuilding of packages like koschei does, but for all
side-tags in real mode (not scratch).