DevHeads.net

Rolling out Phase I of rawhide package gating

Good Morning Everyone,

TL;DR: On July 24th we will turn on the first phase of Rawhide package gating,
for single build updates.
In a later phase, Rawhide updates that contain multiple builds will also be
enabled for gating. Our goal is to improve our ability to continuously turn out
a useful Fedora OS. So we hope and expect to get opt-in from as many Fedora
package maintainers as possible, including maintainers of the base OS. But this
phase of gating remains opt-in, and should not affect packagers who choose for
now not to opt in.

Last April FESCo approved a change proposal[1] allowing to gate Rawhide packages
based on test results. The proposal included gating updates with only a single
build as well as updates with multiple builds. It was designed to cause minimal
to no interference with the current workflow of packagers who do not opt-in.

The team has been working hard on this proposal, and decided to do a phased
roll-out of this change, so that we can gather feedback as early as possible
from the packagers interested in testing this workflow without impacting
everyone.

On July 24th, we plan to turn on the first phase of this change.

What does it mean for us as packagers?
When you run `fedpkg build` on Rawhide, your package will be built in a new koji
tag (which will be the default target for Rawhide). The package will be picked
up from this koji tag, signed and moved onto a second tag. Bodhi will be
notified by koji once this new build is signed and will automatically create an
update for it (you will be notified about this by email by bodhi directly) with
a “Testing” status. If the package maintainer has not opted in into the CI
workflow, the update will be pushed to “Stable” and the build will be pushed
into the regular Rawhide tag, making it available in the Rawhide buildroot, just
as it is today.

If the package maintainer has opted in into the CI workflow, the creation of the
update will trigger the CI pipeline which will send its results to resultsdb,
triggering greenwave to evaluate if all the required tests have passed or not,
thus allowing the update to go through the gate or not.

This is the first roll-out of this gating change, and so there may be additional
tuning and fixes until things are as smooth as we want them to be. With this
release we are looking for feedback on what can be improved. We have a dedicated
team working on this project and we will be taking your feedback into account to
improve the experience.

Small FAQ:
How do I enroll?
Great! We’re glad to see such enthusiasm! You can find the instruction in the
docs on how to enable gating: <a href="https://docs.fedoraproject.org/en-US/ci/gating/" title="https://docs.fedoraproject.org/en-US/ci/gating/">https://docs.fedoraproject.org/en-US/ci/gating/</a>

It does not work!
Bugs will be bugs. This is the first roll-out of this change, and more will
come. This rollout lets us gather feedback and iterate on the approach in an
open source fashion.
If you did not opt-in and you can’t do your packaging work as you used to,
please file a infrastructure ticket, since it’s likely a bug:
https://pagure.io/fedora-infrastructure/new_issue?title=[CI]
If you did opt-in and something in the gating of your update doesn’t work (for
example, CI ran but its results aren’t being considered, waiving didn’t work…),
file an infrastructure ticket:
https://pagure.io/fedora-infrastructure/new_issue?title=[CI]
If you opted-in and the tests don’t run the way you expect, file a fedora-ci
ticket: <a href="https://pagure.io/fedora-ci/general/new_issue" title="https://pagure.io/fedora-ci/general/new_issue">https://pagure.io/fedora-ci/general/new_issue</a>

I enrolled but now I want to step out for some reasons
:sad trombone:
We hope you reported all the issues you’ve found/faced and are helping us
resolving them.
In the meanwhile, you can simply remove the gating.yaml file you’ve added in
your git repo and that should be enough to make greenwave ignore your package.

Want to know more? Your question isn’t here? Check our documentation on
rawhide gating: <a href="https://docs.fedoraproject.org/en-US/rawhide-gating/" title="https://docs.fedoraproject.org/en-US/rawhide-gating/">https://docs.fedoraproject.org/en-US/rawhide-gating/</a> [2]
We’ll keep it up to date as we face or solve questions.

Pierre
For the Rawhide package gating team

[1] <a href="https://pagure.io/fesco/issue/2102" title="https://pagure.io/fesco/issue/2102">https://pagure.io/fesco/issue/2102</a>

[2] At the time of writing this, it looks like the website is having some
difficulties build the new version of the docs. This should get resolved in the
coming hours, sorry for the inconvenience.
(If only it had CI... :-))

Comments

Re: Rolling out Phase I of rawhide package gating

By Florian Weimer at 08/02/2019 - 14:13

* Pierre-Yves Chibon:

I see both “Status stable” and a “Push to Testing” button in the upper
right here:

<https://bodhi.fedoraproject.org/updates/FEDORA-2019-51c4168307>

Is this a UI issue?

Thanks,
Florian

Re: Rolling out Phase I of rawhide package gating

By Kevin Fenzi at 08/03/2019 - 14:20

On 8/2/19 11:13 AM, Florian Weimer wrote:
It's an issue of some kind definitely... either it should not show that
or not allow it.

kevin

Re: Rolling out Phase I of rawhide package gating

By Pierre-Yves at 08/04/2019 - 15:40

On Sat, Aug 03, 2019 at 11:20:35AM -0700, Kevin Fenzi wrote:
+1

Could you please make a ticket of this at:
<a href="https://github.com/fedora-infra/bodhi/issues/" title="https://github.com/fedora-infra/bodhi/issues/">https://github.com/fedora-infra/bodhi/issues/</a> ?

Thanks,
Pierre

Re: Rolling out Phase I of rawhide package gating

By Florian Weimer at 08/05/2019 - 02:19

* Pierre-Yves Chibon:

Done: <https://github.com/fedora-infra/bodhi/issues/3451>

Thanks,
Florian

Re: Rolling out Phase I of rawhide package gating

By Florian Weimer at 07/30/2019 - 10:15

* Pierre-Yves Chibon:

What are the actual tag names? I have a glibc build which appears to be
stuck in f31-updates-candidate:

<https://koji.fedoraproject.org/koji/buildinfo?buildID=1344222>

I wonder if this is caused by gating, or if it's something else.

Thanks,
Florian

Re: Rolling out Phase I of rawhide package gating

By Pierre-Yves at 07/30/2019 - 10:26

On Tue, Jul 30, 2019 at 04:15:31PM +0200, Florian Weimer wrote:
It's robosignatory acting up again :(

I think this may be related to the mass-rebuild being merged, cf Kevin's comment
in: <a href="https://pagure.io/fedora-infrastructure/issue/8041#comment-585310" title="https://pagure.io/fedora-infrastructure/issue/8041#comment-585310">https://pagure.io/fedora-infrastructure/issue/8041#comment-585310</a>

To answer your question:
- builds land in f31-updates-candidate
- robosignatory signs them and moves them to f31-updates-testing
- bodhi picks them up from there
- once CI passes, bodhi moves them to f31

Best,
Pierre

Re: Rolling out Phase I of rawhide package gating

By Richard W.M. Jones at 07/30/2019 - 13:00

On Tue, Jul 30, 2019 at 04:26:30PM +0200, Pierre-Yves Chibon wrote:
What is it about signing packages that takes so long? The crypto ops?

Rich.

Re: Rolling out Phase I of rawhide package gating

By Adam Williamson at 07/30/2019 - 13:04

On Tue, 2019-07-30 at 18:00 +0100, Richard W.M. Jones wrote:
The robot's pen keeps running out of ink...

Re: Rolling out Phase I of rawhide package gating

By Kevin Fenzi at 07/30/2019 - 14:11

On 7/30/19 10:04 AM, Adam Williamson wrote:
:)

In this case it's koji.

For every package in the mass rebuild (f31-pending tag) robosign asks
koji "hey, is foobar-1.0.1-1.fc31 signed' ? koji checks... "yes, it is".
robosign: "great, then I ask you to write out the signed rpms now"
koji: "ok, writing them out to disk again"

it's mostly this last step thats slow. I am not sure if koji is just
seeing if they were written out and returning, or actually re-writing
them out. It seems like it might be the latter, which makes me suspect
koji could optimize this somewhat.

kevin

Re: Rolling out Phase I of rawhide package gating

By Richard W.M. Jones at 07/31/2019 - 10:15

On Tue, Jul 30, 2019 at 11:11:34AM -0700, Kevin Fenzi wrote:
It's still taking a long time today to get builds through Koji and
into Rawhide. Is there a reason we need to sign builds in Rawhide?

Rich.

Re: Rolling out Phase I of rawhide package gating

By Pierre-Yves at 07/31/2019 - 11:10

On Wed, Jul 31, 2019 at 03:15:32PM +0100, Richard W.M. Jones wrote:
My canary took 14 minutes this morning, so that's within the usual time for it.

I'll run it again right to see if it is slower now.

Pierre

Re: Rolling out Phase I of rawhide package gating

By Tom Hughes at 07/31/2019 - 11:35

On 31/07/2019 16:10, Pierre-Yves Chibon wrote:
It seems to vary quite a bit. So far today I've seen about 45 minutes
then 15 and I'm now waiting on another one that's at 50 minutes and
counting.

Tom

Re: Rolling out Phase I of rawhide package gating

By Richard W.M. Jones at 07/31/2019 - 11:39

On Wed, Jul 31, 2019 at 04:35:11PM +0100, Tom Hughes wrote:
I've been waiting so far nearly 2 hours for this one to get into the
buildroot:

<a href="https://koji.fedoraproject.org/koji/buildinfo?buildID=1344542" title="https://koji.fedoraproject.org/koji/buildinfo?buildID=1344542">https://koji.fedoraproject.org/koji/buildinfo?buildID=1344542</a>

Rich.

Re: Rolling out Phase I of rawhide package gating

By Pierre-Yves at 07/31/2019 - 11:58

On Wed, Jul 31, 2019 at 04:39:09PM +0100, Richard W.M. Jones wrote:
My canary ran took 24 minutes, apparently the CI pipeline was slower than usual
but the rest of the workflow seemed fine.

$ koji buildinfo ocaml-result-1.2-12.fc31
returns:
Tags: f31 f31-updates-pending

So it should be in the buildroot. Is it not?

Pierre

Re: Rolling out Phase I of rawhide package gating

By Tom Hughes at 07/31/2019 - 12:06

Well wait-repo is not returning:

koji wait-repo --build=ocaml-result-1.2-12.fc31 f31-build

Tom

Re: Rolling out Phase I of rawhide package gating

By Richard W.M. Jones at 07/31/2019 - 12:29

On Wed, Jul 31, 2019 at 05:06:17PM +0100, Tom Hughes wrote:
It just went into the buildroot a few minutes ago. I think
total waiting time was about 2 hours 20 mins for this one.

Rich.

Re: Rolling out Phase I of rawhide package gating

By Kevin Fenzi at 07/31/2019 - 16:35

On 7/31/19 9:29 AM, Richard W.M. Jones wrote:
Pretty puzzling... it looks like from tagging it went through really
fast, but somehow didn't land in the buildroot. ;(

I'll look and see if there's some buildroot repo regen problem happening...

kevin

Re: Rolling out Phase I of rawhide package gating

By Fabio Valentini at 07/31/2019 - 16:45

FWIW, I just built two packages (snakeyaml and xbean), and for both of them
I received the "bodhi pushed this update to stable" email even before
"fedpkg build" exited. So it's pretty fast now, at least for small packages
like these.

Fabio

Re: Rolling out Phase I of rawhide package gating

By Kevin Fenzi at 07/31/2019 - 12:28

On 7/31/19 9:06 AM, Tom Hughes wrote:
It's there now.

Odd that it would take that long for a newrepo...

According to koji tag history it should have been done like 10 hours
ago, and signing took... 20 seconds.

Wed Jul 31 07:50:50 2019: ocaml-result-1.2-12.fc31 tagged into
f31-updates-candidate by rjones
Wed Jul 31 07:51:00 2019: ocaml-result-1.2-12.fc31 untagged from
f31-updates-candidate by autopen
Wed Jul 31 07:51:00 2019: ocaml-result-1.2-12.fc31 tagged into
f31-updates-testing-pending by autopen
Wed Jul 31 07:51:06 2019: ocaml-result-1.2-12.fc31 untagged from
f31-updates-testing-pending by bodhi
Wed Jul 31 07:51:09 2019: ocaml-result-1.2-12.fc31 tagged into f31 by
bodhi [still active]
Wed Jul 31 07:51:10 2019: ocaml-result-1.2-12.fc31 tagged into
f31-updates-pending by bodhi [still active]

kevin

Re: Rolling out Phase I of rawhide package gating

By Tomasz Torcz at 07/31/2019 - 10:35

On Wed, Jul 31, 2019 at 03:15:32PM +0100, Richard W.M. Jones wrote:
Because administrator of Fedora infrastructure run rawhide on laptops, and we
don't want them to be easily* hackable.

* or maybe not easily, but easier than users of regular releases

Re: Rolling out Phase I of rawhide package gating

By Kevin Fenzi at 07/31/2019 - 11:59

On 7/31/19 7:35 AM, Tomasz Torcz wrote:
Ha. No.

It's for a variety of reasons:

* Various groups that interact with the packages do not want to have to
code in exceptions or treat some things differently. (QA, CI, package
tools).

* Signing packages is a clear way to indicate where they are from. (look
at the 'keychecker' package. If you see a foo-1.0-1.fc29.x86_64.rpm
package you can check it's signature and see that it came from rawhide
or f29 or no where known, etc.

* If you use metalinks, rpm signatures are just gravy on top, in the end
you are still just trusing SSL CA's.

* Making sure everything is signed in rawhide allows us to test/develop
tooling that operates on composes instead of having to test those in
stable release branches.

There's likely other things too...

kevin

Re: Rolling out Phase I of rawhide package gating

By Jason L Tibbitts III at 07/31/2019 - 13:25

KF> * If you use metalinks, rpm signatures are just gravy on top, in the
KF> end you are still just trusing SSL CA's.

Only if you trust every mirror to always serve authentic content.

- J<

Re: Rolling out Phase I of rawhide package gating

By Vitaly Zaitsev ... at 07/31/2019 - 15:05

Le mercredi 31 juillet 2019 à 12:25 -0500, Jason L Tibbitts III a
écrit :
And, just to provide another data point, we tried this month to make
the network install iso talk to https dnf repos (a reposync of fedora
devel x86_64, without x86 packages, because we don't have the storage
budget to mirror 32 bit packages we don't have the use for them
anyway). The repos themselves worked fine from installed systems. But,
anaconda refused to use them, till they were re-exposed in plain un-
secured http.

TLS is a fine thing in theory, but relying on it requires a lot more
debugging capabilities, than the ones we built in our tools. TLS stacks
are heavily biaised towards refusing to connect as soon as something
does not matches their expectations (and they usually forget to tell
you what they didn't like).

Re: Rolling out Phase I of rawhide package gating

By Brian C. Lane at 07/31/2019 - 19:10

On Wed, Jul 31, 2019 at 09:05:21PM +0200, Nicolas Mailhot via devel wrote:
It's odd that they would work from an installed system and not anaconda.
Are you using a self-signed cert on them? If so you can pass
inst.noverifyssl to anaconda to tell it to ignore the error but still
use https.

Re: Rolling out Phase I of rawhide package gating

By Vitaly Zaitsev ... at 08/01/2019 - 02:41

Le mercredi 31 juillet 2019 à 16:10 -0700, Brian C. Lane a écrit :
No, a proper public cert, that even Firefox accepts without grumbling
(not an easy thing to manage those days).

Thanks for the suggestion, I had forgotten about it. Is it possible to
do that manually without a kickstart? Fot that installation workflow we
start from a minimal unmodified install, and customize it in a later
stage.

Regards,

Re: Rolling out Phase I of rawhide package gating

By Brian C. Lane at 08/01/2019 - 13:00

On Thu, Aug 01, 2019 at 08:41:32AM +0200, Nicolas Mailhot via devel wrote:
Very odd then, possibly a crypto support mismatch with the installer image?

dnf has *very* verbose logs in dnf.librepo.log if it is failing inside
dnf I'd expect something useful there.

You can also try curl from the cmdline while booted into the installer
image. I'd probably start with that.

inst.noverifyssl on the cmdline, or in your kickstart pass --noverifyssl
to the url and repo lines:

<a href="https://pykickstart.readthedocs.io/en/latest/kickstart-docs.html#url" title="https://pykickstart.readthedocs.io/en/latest/kickstart-docs.html#url">https://pykickstart.readthedocs.io/en/latest/kickstart-docs.html#url</a>

Re: Rolling out Phase I of rawhide package gating

By Samuel Sieb at 08/01/2019 - 03:27

On 7/31/19 11:41 PM, Nicolas Mailhot via devel wrote:
You can add that to the kernel command line when you boot.

Re: Rolling out Phase I of rawhide package gating

By Vitaly Zaitsev ... at 08/01/2019 - 03:41

Le jeudi 01 août 2019 à 00:27 -0700, Samuel Sieb a écrit :
Sor, from grub, before the keyboard layout is properly set up. Ok, not
too convenient, but a lot better than nothing. Thank you.

Re: Rolling out Phase I of rawhide package gating

By Kevin Fenzi at 07/31/2019 - 16:34

On 7/31/19 12:05 PM, Nicolas Mailhot via devel wrote:
Any errors? Bug filed? as long as the certs were valid/normal certs,
there should not be any reason that wouldn't work I wouldn't think.

kevin

Re: Rolling out Phase I of rawhide package gating

By Martin Kolman at 08/01/2019 - 13:29

On Wed, 2019-07-31 at 13:34 -0700, Kevin Fenzi wrote:

Re: Rolling out Phase I of rawhide package gating

By Vitaly Zaitsev ... at 08/01/2019 - 11:41

On 7/31/19 4:34 PM, Kevin Fenzi wrote:
My guess would be a protocol version or cipher suite negotiation
failure, presumably because the HTTPS end points use newer
configurations that exclude old versions and ciphers. Hopefully Nicholas
will find the real reason.

BTW, the new crypto systems like wireguard are eschewing crypto
negotiation: if the current protocols are determined  to be lacking, the
plan is to push a new version and force everyone to upgrade.

It's pretty harsh from the operational point of view, but they have a
point: if the crypto is vulnerable, it should not be possible to force a
downgrade on connections you care about, and you can still run the old
protocol specifically for endpoints which you cannot upgrade.

Re: Rolling out Phase I of rawhide package gating

By Vitaly Zaitsev ... at 08/01/2019 - 02:33

Le mercredi 31 juillet 2019 à 13:34 -0700, Kevin Fenzi a écrit :
A bug will be filed yes (we gave up on managing to make it work
yesterday). An error would have been mighty fine, IIRC there was
nothing useful returned by anaconda, just the same behaviour as if one
had made a mistake in the URL, that anaconda gave up on after a
timeout.

Regards,

Re: Rolling out Phase I of rawhide package gating

By Florian Weimer at 07/31/2019 - 13:49

* Jason L. Tibbitts, III:

At one point, there was a verified hash chain from the https:// metalink
service, to the repository metadata, down to individual packages. Any
tampering was detected then.

I don't know if all the pieces (including the installer) still use the
metalink service over https:// and verify the hashes.

Thanks,
Florian

Re: Rolling out Phase I of rawhide package gating

By Jason L Tibbitts III at 07/31/2019 - 13:58

FW> At one point, there was a verified hash chain from the https://
FW> metalink service, to the repository metadata, down to individual
FW> packages. Any tampering was detected then.

I understand that the metalink contains enough information to verify the
returnes repomd.xml files, but I guess I don't really know if there's
enough data to chase that down to the checksum of every file that's ever
expected to be on a mirror. If it is, then great, though signatures
still have value because there are other ways to get RPMs than letting
dnf hit the mirror network.

- J<

Re: Rolling out Phase I of rawhide package gating

By Florian Weimer at 07/31/2019 - 14:09

* Jason L. Tibbitts, III:

repomd.xml has hashes for primary.xml etc., and primary.xml contains
digests of the RPM files. In theory, it can all be checked.

At one point, RPM wrote unchecked file contents to disk, leading to
vulnerabilities such as CVE-2013-6435. At the time, it was not possible
to teach RPM to verify the data before writing it.

I think dnf only performs signature checking if the RPMs are downloaded
from repositories.

Thanks,
Florian

Re: Rolling out Phase I of rawhide package gating

By Kevin Fenzi at 07/31/2019 - 14:30

On 7/31/19 11:09 AM, Florian Weimer wrote:
Yes, it's all checked and if tampered with would fail.

You get the metalink via https from our mirrorlist containers running on
our proxies. This metalink has in it a list of mirrors that the
checksums for the repomd.xml file that is valid. You go to one of those
mirrors. If repomd.xml was tampered with, dnf will call it broken and go
on. If someone tampers with packages they would not match the other
checksums in the repomd.xml and be treated as corrupt.

If you are using metalink and not mirrorlist or pointing directly to a
mirror, you should be safe.

Yep. I am pretty sure that is the case.

kevin

Re: Rolling out Phase I of rawhide package gating

By King InuYasha at 07/31/2019 - 20:25

On Wed, Jul 31, 2019 at 2:45 PM Kevin Fenzi < ... at scrye dot com> wrote:
By default this is the case, but you can configure DNF to validate
signatures for all cases if you want.

You just set localpkg_gpgcheck=1 in /etc/dnf/dnf.conf

That said, you probably don't want to do that, since most downloaded
packages aren't signed...

Re: Rolling out Phase I of rawhide package gating

By Jason L Tibbitts III at 07/31/2019 - 21:35

NG> You just set localpkg_gpgcheck=1 in /etc/dnf/dnf.conf

NG> That said, you probably don't want to do that, since most downloaded
NG> packages aren't signed...

I think that the ideal behavior would be to always check, but
warn/prompt for unsigned packages or those with signature failures.
Certainly it's better to verify as much as possible as often as
possible.

- J<

Re: Rolling out Phase I of rawhide package gating

By John Florian at 08/01/2019 - 11:25

On 2019-07-31 21:35, Jason L Tibbitts III wrote:

Re: Rolling out Phase I of rawhide package gating

By Richard W.M. Jones at 07/31/2019 - 11:07

On Wed, Jul 31, 2019 at 10:22:36AM -0400, Stephen John Smoogen wrote:
Actually my question was wrong. Is there any reason we need to sign
builds while they are internal to Koji (ie. proving BuildRequires for
subsequent builds)? They could still be signed when they go out to
Rawhide.

Rich.

Re: Rolling out Phase I of rawhide package gating

By Kevin Fenzi at 07/31/2019 - 11:52

On 7/31/19 8:07 AM, Richard W.M. Jones wrote:
Can you define 'a long time'?

Do you have an example build for me to look at?

Packages are signed before CI runs on them. This is so the _exact_ thing
we are going to be using/shipping/building against is the thing that we
actually test. When you instead test something, then change it, you
leave yourself open to issues with whatever changes you are doing.

CI runs before they land in the buildroot as we want to not build
against anything that was gated for whatever reason.

kevin

Re: Rolling out Phase I of rawhide package gating

By Richard W.M. Jones at 07/31/2019 - 12:28

On Wed, Jul 31, 2019 at 08:52:52AM -0700, Kevin Fenzi wrote:
I waited 2 hours for ocaml-result-1.2-12.fc31. In fact it's just now
become available in the buildroot. I don't know if that helps.

The next build I will be waiting for (when it completes) is
ocaml-dune-1.10.0-4.fc31

Makes sense, thanks.

Rich.

Re: Rolling out Phase I of rawhide package gating

By Florian Weimer at 07/30/2019 - 10:30

* Pierre-Yves Chibon:

Oh. I guess the only thing to do for us waiting, then.

Thanks for these references. This will help to recognize the pattern if
it occurs again.

Florian

Re: Rolling out Phase I of rawhide package gating

By Fabio Valentini at 07/30/2019 - 10:20

On Tue, Jul 30, 2019 at 4:16 PM Florian Weimer < ... at redhat dot com> wrote:
I think that's probably caused by koji being *really* busy with
tagging and merging f31-rebuild into f31 for the past day or so.

Fabio

Re: Rolling out Phase I of rawhide package gating

By Richard W.M. Jones at 07/26/2019 - 09:41

I've been trying to build nbdkit which depends on libnbd >= 0.1.9
in Fedora Rawhide this morning.

I built libnbd a few hours ago, but it hasn't turned up in Rawhide.

I also discovered that you can now submit updates for Rawhide, so why
not: <a href="https://bodhi.fedoraproject.org/updates/FEDORA-2019-d3e8c3f7da" title="https://bodhi.fedoraproject.org/updates/FEDORA-2019-d3e8c3f7da">https://bodhi.fedoraproject.org/updates/FEDORA-2019-d3e8c3f7da</a>
However this didn't help either.

I guess this is something to do with this change, and if it is, what
am I supposed to do?

Rich.

Re: Rolling out Phase I of rawhide package gating

By Pierre-Yves at 07/26/2019 - 10:08

On Fri, Jul 26, 2019 at 02:41:44PM +0100, Richard W.M. Jones wrote:
Yes and no, robosignatory is swamped signing the builds from the mass-rebuild,
which means they aren't landing in the buildroot :(

My test of the pipeline this morning took about 5h to land in rawhide :(

I've opened an infra ticket to see how we can improve/avoid this in the future.

Also, there is no need to go via bodhi, the update you have created would have
been automatically created for you.
Basically in rawhide, we're using bodhi for its UI and its central place.
Everybody knows it and this makes it an obvious place to check what's going on
with a build/update.
Unless you've added tests to your packages or are interested in it, there is
basically no need to interact with bodhi.

Best,
Pierre

Re: Rolling out Phase I of rawhide package gating

By Richard W.M. Jones at 07/27/2019 - 11:26

On Fri, Jul 26, 2019 at 04:08:40PM +0200, Pierre-Yves Chibon wrote:
This is still taking a really long time. I built one package early
this morning which still hasn't gone into Rawhide probably 6+ hours
later.

Rich.

Re: Rolling out Phase I of rawhide package gating

By Kevin Fenzi at 07/28/2019 - 14:49

On 7/27/19 8:26 AM, Richard W.M. Jones wrote:
Yeah, the mass rebuild finished yesterday, but signing didn't catch all
the way up until last night. :(

In any case it should be all back to normal now.

There were 1760 failed builds from the mass rebuild. I am resubmitting
all those to try and catch at least some of the ones that just failed
due to transitory builder issues or the like.

kevin

Re: Rolling out Phase I of rawhide package gating

By Rex Dieter at 07/30/2019 - 13:33

I'm still seeing these in f31-updates-candidate, been there for over 24
hours now:

f5-attica-5.60.0-2.fc31
kf5-bluez-qt-5.60.0-2.fc31
kf5-karchive-5.60.0-1.fc31
kf5-kcodecs-5.60.0-2.fc31
kf5-kcoreaddons-5.60.0-2.fc31
kf5-kguiaddons-5.60.0-2.fc31
kf5-kidletime-5.60.0-2.fc31
kf5-kitemmodels-5.60.0-2.fc31
kf5-kwayland-5.60.0-2.fc31
kf5-kwindowsystem-5.60.0-2.fc31
kf5-modemmanager-qt-5.60.0-2.fc31
kf5-networkmanager-qt-5.60.0-2.fc31
kf5-prison-5.60.0-2.fc31
kf5-solid-5.60.0-2.fc31
kf5-sonnet-5.60.0-2.fc31
kf5-syntax-highlighting-5.60.0-2.fc31

-- Rex