Fedora 30 Mass Rebuild

Hi all,

Per the Fedora 30 schedule[1] we started a mass rebuild for Fedora 30
on Jan 31st 2019. We did a mass rebuild for Fedora 30 for the changes
listed in:

<a href="" title=""></a>

Mass rebuild was done in a side tag (f30-rebuild) and are now getting moved
over to f30.

Failures can be seen
<a href="" title=""></a>

Things still needing rebuilt
<a href="" title=""></a>

19097 builds have been tagged into f30, there s currently 1787 failed
builds that need to be addressed by the package maintainers.

FTBFS bugs will be filed shortly.

Please be sure to let releng know if you see any bugs in the
reporting. You can contact releng in #fedora-releng on freenode, by
dropping an email to our list[2] or filing an issue in pagure[3]

Mohan Boddu.

[1] <a href="" title=""></a>
<a href="" title=""></a>
[3] <a href="" title=""></a>


Re: Fedora 30 Mass Rebuild

By =?UTF-8?B?TWlyb... at 02/14/2019 - 09:28

On 04. 02. 19 21:48, Mohan Boddu wrote:
Fore example: python2-django1.11

Re: Fedora 30 Mass Rebuild

By Fabio Valentini at 02/14/2019 - 09:41

On Thu, Feb 14, 2019 at 2:29 PM Miro Hrončok < ... at redhat dot com> wrote:
My 4 packages (agenda, elementary-photos, gitg, and
golang-github-oschwald-maxminddb-golang) vanished as well.


Re: Fedora 30 Mass Rebuild

By =?UTF-8?Q?Bj=c3... at 02/14/2019 - 11:08

Am Donnerstag, den 14.02.2019, 14:41 +0100 schrieb Fabio Valentini:

That might have happened, because I've run a dumb loop over all FTBFS
packages to have them build at least a second time. It actually fixed
about 180 packages by just submitting them a second time.

Sry, for br3aking stuff… :(

Anyways, this list still apears to be correct:

<a href="" title=""></a>


Re: Fedora 30 Mass Rebuild

By Chuck Anderson at 02/05/2019 - 08:06

On Mon, Feb 04, 2019 at 03:48:59PM -0500, Mohan Boddu wrote:
How many build failures were caused by infrastructure issues?

<a href="" title=""></a>

DEBUG zstd armv7hl 1.3.8-1.fc30 build 261 k
DEBUG Transaction Summary
DEBUG ================================================================================
DEBUG Install 145 Packages
DEBUG Total download size: 57 M
DEBUG Installed size: 218 M
DEBUG Downloading Packages:
DEBUG BUILDSTDERR: Error: Error downloading packages:
DEBUG BUILDSTDERR: Cannot download toplink/packages/util-linux/2.33.1/2.fc30/armv7hl/util-linux-2.33.1-2.fc30.armv7hl.rpm: All mirrors were tried
DEBUG Child return code was: 1

Re: Fedora 30 Mass Rebuild

By Rex Dieter at 02/12/2019 - 11:25

Small sample, but of the 20 I've looked at today, 17 were of this sort.

-- Rex

Re: Fedora 30 Mass Rebuild

By =?ISO-8859-1?Q?... at 02/05/2019 - 17:29

On Tue, 2019-02-05 at 12:06 +0000, Anderson, Charles R wrote:
An idea. Releng team could run a script to detect all builds that fails
in root.log and resubmit it. I also have a couple of fails in root.log
which aren't a real FTBFS .


Re: Fedora 30 Mass Rebuild

By =?ISO-8859-1?Q?... at 02/05/2019 - 06:33

Dne 04. 02. 19 v 21:48 Mohan Boddu napsal(a):

I might be wrong but think the number of failed packages is incorrect.
It seems that every package is counted as many times as many maintainers
it has.


Re: Fedora 30 Mass Rebuild

By Vascom at 02/05/2019 - 07:40

No. If count as many times as many maintainers it has it will be about
3300 fails.

вт, 5 февр. 2019 г. в 14:32, Vít Ondruch < ... at redhat dot com>:

Re: Fedora 30 Mass Rebuild

By Pete Walter at 02/05/2019 - 05:20

Re: Fedora 30 Mass Rebuild

By Peter Robinson at 02/05/2019 - 07:50

That'll be the signing queue

Re: Fedora 30 Mass Rebuild

By Kevin Fenzi at 02/05/2019 - 16:19

On 2/5/19 3:50 AM, Peter Robinson wrote:
Yeah, it wasn't processing these correctly I'm afraid.

I did add the mass rebuild side tag to autosign, but it was a day or two
after it started so likely these are the ones before that.

The autosign machine also had some issues the last few days (it lost 3
disks in a row) and we did some reboots to replace things, etc.

In any case, I have a loop running telling robosign to sign and move all
those f30-pending ones. It's slowly processing them down. Sadly, I don't
think it will be done tonight, but we will see.

Yes, thats a danger... we will have to retag those on a case by case
basis. The script releng uses to merge side tags won't merge a build if
there is a newer one, so those should be fine, it's just if the mass
rebuild build was added and then also someone did another rawhide build.

If you notice any cases of that, file a releng ticket and we can get it
sorted out.


Re: Fedora 30 Mass Rebuild

By =?ISO-8859-1?Q?... at 02/06/2019 - 11:23

Dne 05. 02. 19 v 21:19 Kevin Fenzi napsal(a):

It does not look better, does it? What is the status please?


Re: Fedora 30 Mass Rebuild

By Stephen John Smoogen at 02/06/2019 - 13:18

On Wed, 6 Feb 2019 at 11:15, Vít Ondruch < ... at redhat dot com> wrote:

The autosign hardware has developed a fault and is being worked on.

Re: Fedora 30 Mass Rebuild

By Kevin Fenzi at 02/06/2019 - 12:58

On 2/6/19 7:23 AM, Vít Ondruch wrote:
Our autosigning machine seems to have a failed tpm in it, which has
caused signing to fail to work. ;(

We are working on getting hardware fixed, or a replacement in place.

More updates as I know more. ;(

This is really a anoying time for hardware to die, but of course
sometimes it seems like it _knows_ when that is so it can fail.


Re: Fedora 30 Mass Rebuild

By Chris Murphy at 02/06/2019 - 15:49

On Wed, Feb 6, 2019 at 9:59 AM Kevin Fenzi < ... at scrye dot com> wrote:
Is it for sure dead, or is it possible it coincided with a kernel
update? I recently saw some messages of a TPM regression on the
linux-integrity@ list, but I'm not remembering the kernel versions

Re: Fedora 30 Mass Rebuild

By Kevin Fenzi at 02/06/2019 - 16:13

On 2/6/19 11:49 AM, Chris Murphy wrote:
The affected machine was running the rhel7 kernel, and we did boot into
the kernel it's been using for a long time without any change. ;(


Re: Fedora 30 Mass Rebuild

By Randy Barlow at 02/06/2019 - 15:13

On Wed, 2019-02-06 at 08:58 -0800, Kevin Fenzi wrote:
This is the beginning of the rise of the machines.

Re: Fedora 30 Mass Rebuild

By =?ISO-8859-1?Q?... at 02/06/2019 - 13:07

Dne 06. 02. 19 v 17:58 Kevin Fenzi napsal(a):

Thx for update.


Re: Fedora 30 Mass Rebuild

By Kevin Fenzi at 02/06/2019 - 15:37

On 2/6/19 9:07 AM, Vít Ondruch wrote:
ok. We have a new motherboard coming for the failed server, but it will
not be in until tomorrow.

So, we shuffled things around and got a machine to use for now, and it's
been configured and is signing the f30-pending backlog again. Many
thanks to Patrick for getting things up and running again.

As soon as the f30-pending backlog is done, I'll let it go back to
normal signing and also run the fix script to make sure we have the
latest stuff in f30 tag.

Sorry for the troubles here, hopefully we are back on track now.


Re: Fedora 30 Mass Rebuild

By Dominik 'Rathan... at 02/06/2019 - 17:33

On Wednesday, 06 February 2019 at 20:37, Kevin Fenzi wrote:
Thanks for all the updates, Kevin. They are much appreciated.
Kudos to you, Patrick and all the others involved in fixing this.


Re: Fedora 30 Mass Rebuild

By Kalev Lember at 02/05/2019 - 19:36

On 2/5/19 21:19, Kevin Fenzi wrote:
This sounds fairly bad: everything that got built today and yesterday is
going to get masked out by the older mass rebuild packages that are
being tagged "over" the newer builds as we speak.

Looks like that it was only the mass rebuild packages that got stuck in
the signing queue and other builds were processed normally (they moved
from f30-pending to f30), so they are going to be out of order when the
mass rebuild signing/tagging into f30 finishes.

Is there any chance releng could figure out the list of packages
affected once the signing/tagging is finished and fix all of them? I can
already tell that it's affecting at least 40-50 GNOME builds done in
that time frame.

Also, maybe next time around it would be better to tag mass rebuild
results directly into f31, skipping f31-pending (making sure everything
is signed before doing it)? It seems just painful to go through the
-pending signing queue which takes several days to process tens of
thousands of packages and can lead to out of order tagging.


Re: Fedora 30 Mass Rebuild

By Kevin Fenzi at 02/05/2019 - 21:27

On 2/5/19 3:36 PM, Kalev Lember wrote:
Yes, there's a releng script to fix this very case... can run after
everything is finished signing/tagging.

The reason we tag into pending is to make sure everything is signed, but
yes, we should just make sure to try and autosign the mass rebuild as it
builds. ;(


Re: Fedora 30 Mass Rebuild

By Kalev Lember at 02/06/2019 - 02:46

On 2/6/19 02:27, Kevin Fenzi wrote:
Great, thanks Kevin!