DevHeads.net

signing status

Just wanted to update everyone on our current status.

As you know from the other thread:

* The mass rebuild happened and finished.
* The mass rebuild side tag was merged into the f30-pending tag
(to make sure everything was signed).
* In the middle of the night our autosign box stopped processing.
* The next morning this was noticed and a request for a replacement
motherboard was sent.
* Since that was going to take a day, we setup another machine to do
autosigning.
* That machine started processing the backlog, but also stopped
processing a few times (waiting on koji).
* Finally we set back up the normal listening process and I retagged all
the f30-pending builds to it would "see" they needed processing.

Currently it's processing along pretty fast, but it does make sure koji
has the signed rpms written out, which can take a few seconds on larger
packages.

I'm hopeful that it will catch back up today and we can go back to normal.

There will likely be a short outage next week to move back to the now
replaced hardware, but that should be only a few minutes.

kevin

Comments

Re: signing status

By =?UTF-8?Q?Tomas... at 02/11/2019 - 12:13

Is there anywhere some summary stats about mass update?
How many packages has been sent to rebuild vs those which rebuild failed?

kloczek

Re: signing status

By Kevin Fenzi at 02/11/2019 - 14:35

On 2/11/19 8:13 AM, Tomasz Kłoczko wrote:
There is:

<a href="https://kojipkgs.fedoraproject.org/mass-rebuild/f30-failures.html" title="https://kojipkgs.fedoraproject.org/mass-rebuild/f30-failures.html">https://kojipkgs.fedoraproject.org/mass-rebuild/f30-failures.html</a>

and

<a href="https://kojipkgs.fedoraproject.org/mass-rebuild/f30-need-rebuild.html" title="https://kojipkgs.fedoraproject.org/mass-rebuild/f30-need-rebuild.html">https://kojipkgs.fedoraproject.org/mass-rebuild/f30-need-rebuild.html</a>

The FTBFS bugs are being filed now I think...

kevin

Re: signing status

By Jan Pazdziora at 02/12/2019 - 04:46

On Mon, Feb 11, 2019 at 10:35:55AM -0800, Kevin Fenzi wrote:
My packages were caught in the crossfire:

<a href="https://bugzilla.redhat.com/show_bug.cgi?id=1675617" title="https://bugzilla.redhat.com/show_bug.cgi?id=1675617">https://bugzilla.redhat.com/show_bug.cgi?id=1675617</a>
<a href="https://bugzilla.redhat.com/show_bug.cgi?id=1675618" title="https://bugzilla.redhat.com/show_bug.cgi?id=1675618">https://bugzilla.redhat.com/show_bug.cgi?id=1675618</a>
<a href="https://bugzilla.redhat.com/show_bug.cgi?id=1675634" title="https://bugzilla.redhat.com/show_bug.cgi?id=1675634">https://bugzilla.redhat.com/show_bug.cgi?id=1675634</a>

All of them failed on armv7hl, with some package (for example
glibc-2.29-1.fc30.1.armv7hl.rpm) missing in the mock step.

How should maintainers proceed in such situation?

Re: signing status

By Peter Robinson at 02/12/2019 - 05:19

On Tue, Feb 12, 2019 at 8:49 AM Jan Pazdziora < ... at redhat dot com> wrote:
If the problem is now fixed you should just be able to do "fedpkg
build" to rebuild it and it'll proceed as usual.

Re: signing status

By Jan Pazdziora at 02/12/2019 - 07:14

On Tue, Feb 12, 2019 at 09:19:03AM +0000, Peter Robinson wrote:
Thanks, that passed now.

Re: signing status

By Hans de Goede at 02/11/2019 - 06:07

Hi,

On 08-02-19 17:41, Kevin Fenzi wrote:
Thank you and the entire infra team for quickly fixing this!

And more in general a big thank you to the entire infra team for all
the heroic work you are doing on an almost daily basis.

Regards,

Hans

Re: signing status

By =?UTF-8?Q?Bj=c3... at 02/09/2019 - 08:48

Am Freitag, den 08.02.2019, 08:41 -0800 schrieb Kevin Fenzi:

Thank you for the extra work, Kevin and all other being involved with
it!

Just asking as this came to my mind from a different thread:

Has that issue been adressed already, too?

Björn

Re: signing status

By Kevin Fenzi at 02/09/2019 - 14:34

On 2/9/19 4:48 AM, Björn 'besser82' Esser wrote:
Yes, I already fixed all those.

kevin

Re: signing status

By =?UTF-8?B?TWlyb... at 02/09/2019 - 06:00

On 08. 02. 19 17:41, Kevin Fenzi wrote:
Is this somehow relevant?

<a href="https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20190209.n.1/logs/global/pungi.global.log" title="https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20190209.n.1/logs/global/pungi.global.log">https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-201902...</a>

2019-02-09 08:44:00 [ERROR ] Compose run failed: RPM(s) not found for sigs:
['CFC659B9']. Check log for details. Unsigned packages:
dar-2.6.0-2.fc30
dar-debuginfo-2.6.0-2.fc30
dar-debugsource-2.6.0-2.fc30
electrum-3.3.2-3.fc30
libdar-2.6.0-2.fc30
libdar-debuginfo-2.6.0-2.fc30
libdar-devel-2.6.0-2.fc30
mysqltuner-1.7.19-2.git.59e5f40.fc30
openscap-1.3.0_alpha2-2.fc30
openscap-containers-1.3.0_alpha2-2.fc30
openscap-debuginfo-1.3.0_alpha2-2.fc30
openscap-debugsource-1.3.0_alpha2-2.fc30
openscap-devel-1.3.0_alpha2-2.fc30
openscap-engine-sce-1.3.0_alpha2-2.fc30
openscap-engine-sce-debuginfo-1.3.0_alpha2-2.fc30
openscap-engine-sce-devel-1.3.0_alpha2-2.fc30
openscap-perl-1.3.0_alpha2-2.fc30
openscap-perl-debuginfo-1.3.0_alpha2-2.fc30
openscap-python3-1.3.0_alpha2-2.fc30
openscap-scanner-1.3.0_alpha2-2.fc30
openscap-scanner-debuginfo-1.3.0_alpha2-2.fc30
openscap-utils-1.3.0_alpha2-2.fc30
openscap-utils-debuginfo-1.3.0_alpha2-2.fc30
python-oslo-messaging-8.0.0-1.fc29
python-oslo-messaging-doc-8.0.0-1.fc29
python2-oslo-messaging-8.0.0-1.fc29
python2-oslo-messaging-tests-8.0.0-1.fc29
python3-oslo-messaging-8.0.0-1.fc29
python3-oslo-messaging-tests-8.0.0-1.fc29

Re: signing status

By Kevin Fenzi at 02/09/2019 - 14:33

On 2/9/19 2:00 AM, Miro Hrončok wrote:
This was caused by my fixing upgrade path on f30.
ie, there was an older 'dar' tagged in, so I tagged in the 'newer' (by
NVR) one. It was signed, but apparently the written out rpms were
garbage collected already.

I'm fixing these up and will fire a new rawhide after.

kevin

Re: signing status

By =?ISO-8859-1?Q?... at 02/10/2019 - 12:01

On Sat, 2019-02-09 at 10:33 -0800, Kevin Fenzi wrote:

Hi, I don't if it helps but I don't know you notice that Fedora-
Rawhide- 20190207 [1] still running and pungi.log [2] last write was
2019-02-08 16:45:52 ( 2 days ago) ...

Best regards,

[1]
<a href="https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20190207.n.0/" title="https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20190207.n.0/">https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-201902...</a>

[2]
<a href="https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20190207.n.0/logs/global/pungi.global.log" title="https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-20190207.n.0/logs/global/pungi.global.log">https://kojipkgs.fedoraproject.org/compose/rawhide/Fedora-Rawhide-201902...</a>

Re: signing status

By Kevin Fenzi at 02/10/2019 - 13:02

On 2/10/19 8:01 AM, Sérgio Basto wrote:
It crashed/died and the status didn't update.

It's not still composing, it will just be stuck in started forever. ;(

Yesterdays failed in an anaconda traceback on the kde live, so I am
untagging the new anaconda(s) and restarting it.

Signing should be all caught up as of yesterday.

kevin

Re: signing status

By Martin Kolman at 02/11/2019 - 07:10

On Sun, 2019-02-10 at 09:02 -0800, Kevin Fenzi wrote:

Re: signing status

By Kevin Fenzi at 02/11/2019 - 14:27

On 2/11/19 3:10 AM, Martin Kolman wrote:

Yeah, sorry:

filed:

<a href="https://bugzilla.redhat.com/show_bug.cgi?id=1674605" title="https://bugzilla.redhat.com/show_bug.cgi?id=1674605">https://bugzilla.redhat.com/show_bug.cgi?id=1674605</a>

kevin

Re: signing status

By Alexander Bokovoy at 02/08/2019 - 15:11

On pe, 08 helmi 2019, Kevin Fenzi wrote:
And now I have a build without any tags.

Re: signing status

By Kevin Fenzi at 02/08/2019 - 17:32

...snip...

No, that all looks normal, the build was tagged into f30-pending as
expected. So once the autosigner gets to it, it will be signed and moved
to f30. So yeah, that is delayed due to the signing queue, but otherwise
it looks ok to me.

<a href="https://koji.fedoraproject.org/koji/buildinfo?buildID=1206959" title="https://koji.fedoraproject.org/koji/buildinfo?buildID=1206959">https://koji.fedoraproject.org/koji/buildinfo?buildID=1206959</a>

kevin