DevHeads.net

Fedora 31 System-Wide Change proposal: Gating Rawhide - Single package updates

<a href="https://fedoraproject.org/wiki/Changes/GatingRawhideSinglePackageUpdates" title="https://fedoraproject.org/wiki/Changes/GatingRawhideSinglePackageUpdates">https://fedoraproject.org/wiki/Changes/GatingRawhideSinglePackageUpdates</a>

== Summary ==
We want to gate packages on test results before they can land in
rawhide. This will reduce the amount of broken dependency,
uninstallable packages and broken composes leading to a more stable
rawhide as well as less work on the infrastructure and rel-eng teams
to keep composes working.

This project will be split in two phases, at first only single package
updates will be supported, in a second stage, we will add support for
multi-packages updates.

This proposal is about the phase 1 of this project.

== Owner ==
* Name: [[User:pingou|Pierre-Yves Chibon]]
* Email: pingou - pingoured.fr

People who are/will be involved in this:
* Coordinator: [[User:pingou|Pierre-Yves Chibon]]
* Bodhi: [[User:bowlofeggs|Randy Barlow]] and [[User:abompard|Aurélien Bompard]]
* Fedora CI: [[User:dperpeet|Dominik Perpeet]]

== Detailed Description ==

Querying the Bodhi database, it was revealed that 95% of all our
updates involve a single package.

To be exhaustive, here are the exact number found by Randy:

Of all time:

Single build updates: 123,657 (94.1%)
Multi build updates: 7,766 ( 5.9%)

Total: 131,423

Per release:

Fedora 29:

Single: 4,675 (93.7%)
Multi: 316 ( 6.3%)

Fedora 28:

Single: 9,153 (94.5%)
Multi: 536 ( 5.5%)

EPEL 7:

Single: 12,664 (94.0%)
Multi: 814 ( 6.0%)

When considering gating rawhide package updates on test results, we
need to consider two workflows: single package updates, multi-package
updates. This proposal only considers the single package updates
workflow. [[Changes/GatingRawhideMultiPackagesUpdates|Another
proposal]] will be submitted in the future to support the multi
package updates workflow.

At first we want to build the system allowing to gating rawhide,
packagers will '''opt-in into gating'''
[https://docs.pagure.org/greenwave/package-specific-policies.html
using greenwave's opt-in gating mechanism] as they want. We will also
offer a way to '''opt-out''' so packages depending on other packages
can still be built without issues.

We do not aim at making any tests mandatory as part of these
proposals. Once the two proposals have been deployed it will be up to
the community and likely FESCo to decide if they would like to enforce
some rules on all packages and which ones.

Note that this proposal completes previous initiative such as
[[Changes/NoMoreAlpha]]

=== Workflows ===

<gallery>
Single_package_GatingRawhide_bodhi.png|Single package update with gating
</gallery>

== Benefit to Fedora ==

* As more packagers opt-into gating and add tests to their packages,
rawhide becomes more stable
* More stable rawhide will lead to easier composes and a smoother
release process
* Ability to use chain-build in rawhide and stable releases

== Scope ==
The scope of this work is adding a mechanism to gate single package
updates before they enter the rawhide buildroot (and thus become
accessible to others) so broken packages are kept out and cannot
affect other packages or the compose until they are fixed by their
maintainers.

'''The CI system, the tests and the decision on which tests are used
to gate upon are out of scope for the present document.'''

* Proposal owners: pingou will be coordinating the work among the
different stack holder

* Other developers:
** Bodhi developers: Implement the needed changes
** infrastructure: deploy the new version of bodhi and the new koji plugin

* Release engineering: <a href="https://pagure.io/releng/issue/8183" title="https://pagure.io/releng/issue/8183">https://pagure.io/releng/issue/8183</a>

* Policies and guidelines: Once the entire system is in place, FESCo
may want to set a policy on distribution-wide gating (ie: tests that
would be enforced for all packages in the distributions). This is
however out of scope of this proposal.

* Trademark approval: N/A (not needed for this Change)
=== Application impacted ===
==== Bodhi ====

* Single package update
** We will need to write a message-bus listener that will
automatically create a bodhi update for each package built in the
rawhide-gated tag.
** We will need to write a message-bus listener to automatically push
bodhi updates created for rawhide that have passed CI (the decision
being announced by Greenwave).

* Bodhi will comment on the updates when greenwave's decision about an
update is known
** This will notify users about the test results and the corresponding
gating status.

==== Greenwave / WaiverDB ====
Nothing should change for these tools but we will have to confirm this
and test in staging that they behave as expected.

==== CI system ====
Nothing should change for the CI system but we will have to confirm
this and test in staging that they behave as expected.

== Upgrade/compatibility impact ==
N/A

== How To Test ==

# Add tests to a package you maintain in Fedora using the [[
CI/Quick_Start_Guide ]] documentation (see the full [[CI|CI
documentation ]] for more information).
# Update your package in rawhide
# Ensure that the bodhi update is automatically created
# Check that the tests have properly ran (see the Tests tab in the bodhi update)
# Ensure that the bodhi update went through to rawhide once the tests passed

== User Experience ==

=== Single package update ===

fedpkg clone rpms/foobar
cd foobar/
vim foobar.spec
git commit -am "Git commit message"
fedpkg build

If the CI tests pass, the package will land in rawhide, if they fail the user
will be able to go to bodhi to see what failed and why as well as waiving the
failed tests if needed.

=== Multi package update with single package update gating ===

fedpkg clone rpms/foobar
cd foobar/
vim foobar.spec
git commit -am "Update of foobar to update foo to 1.2"
fedpkg push
fedpkg build --target=rawhide-candidate
cd ..
fedpkg clone rpms/foobaz
cd foobaz
vim foobaz.spec
git commit -am "Update of foobaz to update foo to 1.2"
fedpkg push
fedpkg build --target=rawhide-candidate
cd ..
fedpkg clone rpms/foo
cd foo
vim foo.spec
git commit -am "Update foo to 1.2"
fedpkg push
fedpkg build --target=rawhide-candidate

Users are practically by-passing the gating process by specifying a
specific build target.

== Dependencies ==

* Bodhi
** bus listener to automatically create updates when a build happens in rawhide
** bus listener to automatically push updates who passed gating
according to greenwave
** comment on updates about gating status change/announce to notify
packagers about it

== Contingency Plan ==
We can keep doing rawhide as we are doing it today.

== Documentation ==
To be written/updated

List of documentation pages to update:

* <a href="https://fedoraproject.org/wiki/Updates_Policy" title="https://fedoraproject.org/wiki/Updates_Policy">https://fedoraproject.org/wiki/Updates_Policy</a>
* <a href="https://fedoraproject.org/wiki/Package_maintenance_guide" title="https://fedoraproject.org/wiki/Package_maintenance_guide">https://fedoraproject.org/wiki/Package_maintenance_guide</a>
* <a href="https://fedoraproject.org/wiki/Package_update_HOWTO" title="https://fedoraproject.org/wiki/Package_update_HOWTO">https://fedoraproject.org/wiki/Package_update_HOWTO</a>
* <a href="https://docs.fedoraproject.org/en-US/packaging-guidelines/" title="https://docs.fedoraproject.org/en-US/packaging-guidelines/">https://docs.fedoraproject.org/en-US/packaging-guidelines/</a>

Comments

Re: Fedora 31 System-Wide Change proposal: Gating Rawhide - Sing

By =?UTF-8?B?TWlyb... at 03/18/2019 - 02:44

On 01. 03. 19 22:19, Ben Cotton wrote:
I was thinking. Does this change retiring packages at all? E.g. do we "push"
removals trough bodhi, or not?

Re: Fedora 31 System-Wide Change proposal: Gating Rawhide - Sing

By =?UTF-8?B?TWlyb... at 03/03/2019 - 06:45

On 01. 03. 19 22:19, Ben Cotton wrote:
This is both good (specifying explicitly what is this change about and what it
is not about) and bad...

Since the CI system is not part of this change, we cannot say this change is not
complete if the CI system is not complete. So later we say we have rawhide
gating and we'll all be \o/ \o/ \o/ yet nobody will care that the CI is
unfortunately:

* not locally reproducible <a href="https://pagure.io/fedora-ci/general/issue/4" title="https://pagure.io/fedora-ci/general/issue/4">https://pagure.io/fedora-ci/general/issue/4</a>
* only working on on x86_64 <a href="https://pagure.io/fedora-ci/general/issue/16" title="https://pagure.io/fedora-ci/general/issue/16">https://pagure.io/fedora-ci/general/issue/16</a>
* unreadable <a href="https://pagure.io/fedora-ci/general/issue/2" title="https://pagure.io/fedora-ci/general/issue/2">https://pagure.io/fedora-ci/general/issue/2</a>
* unreliable (I file 1.75 issues per month on average)
* understaffed (my personal observation)
* tedious to start using (we focus on standards instead of UX)
* untested (the (sometimes fatal) issues I face go unnoticed until I'm hit)

My concern is that the CI experience still feels a bit rough and I'd rather see
us focusing on making it better. This can of course be done in parallel with
this change, yet I feel that we are building this on water.

Note that I don't blame the CI people, they are very helpful and great to work
with, they just don't have time/resources/people.

Re: Fedora 31 System-Wide Change proposal: Gating Rawhide - Sing

By Pierre-Yves at 03/04/2019 - 07:34

On Sun, Mar 03, 2019 at 11:45:31AM +0100, Miro Hrončok wrote:
I think you are raising very good points and that they should be looked into,
however, in the design of gating rawhide we tried to be test system agnostic,
so, as Adam already pointed out, you could gate on results any of our test
systems (we currently have three: the CI pipeline, taskotron, OpenQA) and
possibly other in the future.

I believe Dominik is working on a new CI initiative at the council level and
I'll let him reply if these (or some of these) concerns can be covered by this
new initiative.

Thanks again,
Pierre

Re: Fedora 31 System-Wide Change proposal: Gating Rawhide - Sing

By =?UTF-8?B?TWlyb... at 04/01/2019 - 16:31

On 04. 03. 19 12:34, Pierre-Yves Chibon wrote:
No word for almost a month. I've opened 2 more issues that I think must be
solved before the gating is considered:

CI errors are undecipherable:
<a href="https://pagure.io/fedora-ci/general/issue/43" title="https://pagure.io/fedora-ci/general/issue/43">https://pagure.io/fedora-ci/general/issue/43</a>

CI errors happen far to often:
<a href="https://pagure.io/fedora-ci/general/issue/44" title="https://pagure.io/fedora-ci/general/issue/44">https://pagure.io/fedora-ci/general/issue/44</a>

Re: Fedora 31 System-Wide Change proposal: Gating Rawhide - Sing

By Ben Cotton at 04/01/2019 - 17:30

For the record, the Change proposal I sent earlier without a title in
the subject is a follow-up to this proposal:
<a href="https://lists.fedoraproject.org/archives/list/ ... at lists dot fedoraproject.org/thread/PJVKI7DKPUL3MTT4GMKN5SMUGG5DGY6O/" title="https://lists.fedoraproject.org/archives/list/ ... at lists dot fedoraproject.org/thread/PJVKI7DKPUL3MTT4GMKN5SMUGG5DGY6O/">https://lists.fedoraproject.org/archives/list/ ... at lists dot fedoraproject....</a>

Re: Fedora 31 System-Wide Change proposal: Gating Rawhide - Sing

By =?UTF-8?B?TWlyb... at 03/11/2019 - 15:54

On 04. 03. 19 12:34, Pierre-Yves Chibon wrote:

Taskotron is planing to migrate to the CI pipeline.

<a href="https://pagure.io/fedora-ci/general/issue/36" title="https://pagure.io/fedora-ci/general/issue/36">https://pagure.io/fedora-ci/general/issue/36</a>

So that leaves 2 things:

* Fedora Ci with the above problems
* OpenQA

I honestly don't except many packagers writing their own OpenQA tests.

Re: Fedora 31 System-Wide Change proposal: Gating Rawhide - Sing

By Kevin Fenzi at 03/02/2019 - 17:36

On 3/1/19 1:19 PM, Ben Cotton wrote:
Might also test the 'opt out case' (ie, waiving or opting out
successfully lands the build in rawhide with failing tests?)

Do note:

We removed the rawhide target in koji (
<a href="https://pagure.io/releng/issue/7664" title="https://pagure.io/releng/issue/7664">https://pagure.io/releng/issue/7664</a> ). I have put in a symlink so it
still works (without the repo regenning, pointing to f31-build). So, we
might want to name the target here the number? Just thought I would
mention it, we can sort it out when we need to.

Other than those nit-picks, I think it looks great. ;)

Thanks for leading this and I hope we have it in place soon...

kevin

Re: Fedora 31 System-Wide Change proposal: Gating Rawhide - Sing

By Pierre-Yves at 03/04/2019 - 04:37

On Sat, Mar 02, 2019 at 01:36:10PM -0800, Kevin Fenzi wrote:
I have added a sentence about how to waive failing tests and a section about how
to test without opting in into gating.
I'll add a section about how to opt-out but before I do that I'd like to clear
out something, so it may take me a day or two before I can write this section.

I've kept "rawhide" and "rawhide-candidate" for now to be consistent in the
document (and because rawhide should remain a self-descriptive for a while), but
I'm fine changing this if needed.

Thanks :)

We should have a target date sometime this week :)

Pierre

Re: Fedora 31 System-Wide Change proposal: Gating Rawhide - Sing

By Ken Dreyer at 03/02/2019 - 17:05

What's the new Koji plugin?

- Ken

Re: Fedora 31 System-Wide Change proposal: Gating Rawhide - Sing

By Pierre-Yves at 03/04/2019 - 04:28

On Sat, Mar 02, 2019 at 03:05:56PM -0600, Ken Dreyer wrote:
Good catch, this is a left over from when I started this page and we thought we
may do single and multi packages updates in one go. The koji plugin is meant for
the multi-package workflow, I've edited this line.

Thanks!
Pierre

Re: Fedora 31 System-Wide Change proposal: Gating Rawhide - Sing

By Adam Williamson at 03/02/2019 - 03:09

On Fri, 2019-03-01 at 16:19 -0500, Ben Cotton wrote:
It might be a good idea to have a QA contact here, in case people
choose to block on tests maintained by the QA team (Taskotron or openQA
tests).

I'd suggest the implication that single-package updates are "the norm"
is slightly problematic, for two reasons:

1) Rawhide is very different from stable releases, and even from
Branched. Major changes like API breaks are not meant to be sent to
stable releases at all, by policy. So I don't think you can necessarily
rely very strongly on data regarding updates in Branched and stable
releases to draw conclusions about the likely nature of changes in
Rawhide.

2) This does not consider the not uncommon case that updates *should
have been* multi-package updates, but they were done wrong.

That Change envisaged *both* package build gating *and* compose gating,
so this alone does not really complete it, for the record.

Does this mean that the plan has changed once again and it will no
longer bypass Bodhi and use side tags and integration with fedpkg?

Re: Fedora 31 System-Wide Change proposal: Gating Rawhide - Sing

By =?ISO-8859-1?Q?... at 03/04/2019 - 04:40

Dne 02. 03. 19 v 8:09 Adam Williamson napsal(a):

These were my first thoughts as well.

Also, how are the mass rebuilds envisioned? I can't imagine that Ruby
rebuild will be held from entering rawhide due to some broken
dependency. Not sure how you want distinguish the package, which is
typically "single" from "multi" package.

Vít

Re: Fedora 31 System-Wide Change proposal: Gating Rawhide - Sing

By Pierre-Yves at 03/04/2019 - 04:47

On Mon, Mar 04, 2019 at 09:40:32AM +0100, Vít Ondruch wrote:
Just as anyone would: by opting-out of gating.
Maybe that term is overloaded. The way I see it there are two ways to opt-out:
- remove the gating.yml file in the package's git repo, not scalable especially
for mass-rebuild
- build in the koji candidate tag directly, thus by-passing the testing tag
where tests happen. This would be the approach releng would do for
mass-rebuild.

This is a decision left in the hands of the packagers, I'm pretty sure they know
more their packages than I do and thus I'm in no position to make this decision.

Does this help?
Pierre

Re: Fedora 31 System-Wide Change proposal: Gating Rawhide - Sing

By Kevin Fenzi at 03/04/2019 - 14:11

On 3/4/19 12:47 AM, Pierre-Yves Chibon wrote:

Well, to be clear, mass rebuilds already use a side tag and then are
merged in. So they would just be merged into the candidate tag or whatever.
I think perhaps it would be useful to easily show waivers?
like a weekly waiver report or something so we could tell what packages
and maintainers are waiving results?

kevin

Re: Fedora 31 System-Wide Change proposal: Gating Rawhide - Sing

By Pierre-Yves at 03/04/2019 - 15:57

On Mon, Mar 04, 2019 at 10:11:27AM -0800, Kevin Fenzi wrote:
We will definitely want to look into waivers at one point (both which tests as
well as which maintainers), I am not sure we need it from the get go though :)

Pierre

Re: Fedora 31 System-Wide Change proposal: Gating Rawhide - Sing

By Pierre-Yves at 03/02/2019 - 05:12

On Fri, Mar 01, 2019 at 11:09:48PM -0800, Adam Williamson wrote:
Yes and no, yes I'm all in favor of having a contact person from QA to help
volunteer to pick up the right tests for their package, but no as which tests
are used to gate is out of the scope of the proposal itself.

I'd be in favor of adding something like:
"""
If you want to volunteer to opt-in into this workflow and need help figuring out
which tests would be interesting to run in your package, contact foo at
<email/list> or open a ticket at <tracker>
"""

Would that make sense?
If so, where should we recommend packagers to get help?

This is fair, but plays on both side, we're also supposed to send more updates
to rawhide than to stable releases. So there will be more multi-packages updates
in rawhide but also more single-packages updates.
While we may not be at 95 to 5 in rawhide, it is still expected that single
package will be much higher.

That is true, I believe that by only onboarding packagers who want to (opt-in)
and providing a way to by-pass (opt-out) we should be able to cope with this.

[...]
We have considered a number of options, one of them was to no involve bodhi but
it had a number of disadvantage for Fedora for a similar level of work (amount
and complexity) which made us choose to keep bodhi involved.

Using side-tags and changes to fedpkg will be required for gating multi-packages
updates, they are not required for single package updates (there is a link in
this proposal to the follow-up proposal for multi packages updates, do note that
this is still a draft and thus may change when we get to this workflow).

Pierre

Re: Fedora 31 System-Wide Change proposal: Gating Rawhide - Sing

By Adam Williamson at 03/02/2019 - 11:21

On Sat, 2019-03-02 at 10:12 +0100, Pierre-Yves Chibon wrote:
It's more a case of: if people choose to gate on QA-maintained tests
and then there turns out to be some sort of problem involving tests not
running or flakes or whatever, it is going to be *perceived to be* a
part of this Change even if you're trying to keep it out of scope.

Re: Fedora 31 System-Wide Change proposal: Gating Rawhide - Sing

By Pierre-Yves at 03/02/2019 - 07:14

On Sat, Mar 02, 2019 at 10:12:00AM +0100, Pierre-Yves Chibon wrote:
I have a document with the different approaches considered with their pros and
cons, I'll put it on the wiki shortly and send the link here and add it to the
proposal as well. It should give some context about why this design was made
over other alternatives.

Pierre

Re: Fedora 31 System-Wide Change proposal: Gating Rawhide - Sing

By Tom Stellard at 03/01/2019 - 22:57

On 03/01/2019 01:19 PM, Ben Cotton wrote:
Does the gating prevent packages from being tagged at all so that they
won't even end up in the buildroot, or does it just gate packages from
going into a compose?

<one more comment below>

-Tom

Re: Fedora 31 System-Wide Change proposal: Gating Rawhide - Sing

By Pierre-Yves at 03/02/2019 - 05:02

On Fri, Mar 01, 2019 at 06:57:48PM -0800, Tom Stellard wrote:
It gates the package from entering the buildroot, which will impact the packages
going into a compose, but as a side effect.

[...]
Thanks for your feedback, I'll follow up on this ticket to see if we can get it
sorted out.

Pierre

Re: Fedora 31 System-Wide Change proposal: Gating Rawhide - Sing

By Zamir SUN at 03/07/2019 - 10:24

On 3/2/19 5:02 PM, Pierre-Yves Chibon wrote:
Given it blocks packages from entering buildroot, I wonder if it is a
good idea to start gating whole Rawhide lifetime, I mean, from the
starting of a ready-to-release release branched out of Rawhide?

My case is, we have a set of packages to update each release. They
cannot do in parallel - they depend on one another. Currently we only
update them in Rawhide, because each package can get into buildroot
shortly after we build it, and we don't need to file a
buildroot-override. Once even packages cannot get into Rawhide
automatically (for example, I need to click a "waive test result" or
something), this is more like a burden.

As for the Single build updates vs multi build updates ratio, I don't
quite understand what the number is from - does it comes out of Bodhi?
If it means the updates in Bodhi, I want to mention that, in my case, I
never want to update multi build updates in a stable (or post-freeze)
release. Thus I seldom file multi build updates in bodhi. Especially we
don't need to file Bodhi to get packages into Rawhide at this point,
this maybe misleading in deciding to enable gating for Rawhide.

As a summarize, I think it is a good idea to have some sort of gating.
But I think we need to think carefully if we do really need to gate Rawhide.

Re: Fedora 31 System-Wide Change proposal: Gating Rawhide - Sing

By Pierre-Yves at 03/07/2019 - 11:06

On Thu, Mar 07, 2019 at 10:24:00PM +0800, Zamir SUN wrote:
So what you are describing is basically the case of multi-packages update. You
want to ship as one packages that depend on each other and should be ship as one
unit.
The current approach specifically does not support this use case.
There are two ways to go about this:
- Do not opt-in into the gating (ie: do not add a gating.yaml in your dist-git
repo)
- Do opt-in but when you need something faster, opt-out, by simply building the
package in rawhide-candidate (or its equivalent) directly.

They do yes and indeed as you're mentioning they are informative but likely not
fully representative of rawhide.

The bodhi updates will be filled automatically and if tests pass (or if there
are no gating.yaml) the build will land in rawhide, automatically as well, just
as it does today.
The point of bodhi is to give us a way to see in which state is a package or why
a package that was built did not land in the rawhide buildroot in a place that
is accessible to everyone in the community.

I don't think there are so many ways to stabilize rawhide so that compose (for
example) pass more often than they fail (there hasn't been a successful rawhide
compose in a week). CI and gating is one of them and one that is pretty standard
in our industry.

Pierre

Re: Fedora 31 System-Wide Change proposal: Gating Rawhide - Sing

By King InuYasha at 03/07/2019 - 11:21

On Thu, Mar 7, 2019 at 10:07 AM Pierre-Yves Chibon < ... at pingoured dot fr> wrote:
I hate to be the downer here, but I'm generally opposed to this idea
as it stands.

My ability to develop against Rawhide is principally centered around
the fact I can push packages and have them depend on each other.
Today, there is no freely accessible way for people to atomically
merge changes into the distribution. There's no way for people to
freely create spaces to do a bunch of work on a bunch of packages at
once and then merge it.

We kind of fake this by having to ask releng to grant unto a packager
or packagers a side-tag, which work is done there, and then it's
merged back into the distribution. But that model has only worked so
far because we have nothing like this in place for Rawhide, so we can
push all the stuff there, make sure it works, and then merge it into
branches. This is how KDE updates are done, for example. And
occasionally, the GNOME desktop is updated in the same manner.

The fatal misunderstanding here is that people think there's a
distinction between "single" and "multi" updates in Rawhide, when in
fact that doesn't exist. This proposal is doomed to failure because it
assumes stable updates workflows translate to development workflows.
And they don't. The reason why single updates are more common is
because people are ever-so-slightly more cautious, and huge rebuilds
or revdep rebuilds don't happen in stable releases. It doesn't help
that the workflow in Fedora makes multi-package updates rather
painful, but I don't think anyone cares enough to fix that, either...

And I still believe Bodhi is a bad place to do this, because you've
still allowed the package to succeed building into the distribution,
but I think I won't win on this point.