DevHeads.net

Updating Rawhide vs GPG keys

Hi,

Can somebody please enlighten me, how to update Rawhide after branching
and not using --nogpgcheck?

It seems that Rawhide keys were added in fedora-repos-30-0.4. So this is
the package which is still "rawhide" package and has "f31" keys. But
this package was not probably signed, because this directory is empty:

<a href="https://kojipkgs.fedoraproject.org/packages/fedora-repos/30/0.4/data/signed/" title="https://kojipkgs.fedoraproject.org/packages/fedora-repos/30/0.4/data/signed/">https://kojipkgs.fedoraproject.org/packages/fedora-repos/30/0.4/data/sig...</a>

Installing anything from Rawhide fails, because of missing GPG keys.

So is there a way to get the GPG keys via DNF? Would it be possible to
sign fedora-repos and fedora-release packages by older key to allow
smooth updates?

Vít

Comments

Re: Updating Rawhide vs GPG keys

By Kevin Fenzi at 03/11/2019 - 14:29

On 3/11/19 4:31 AM, Vít Ondruch wrote:
Can you expand on the case here?

What should happen is:

* branching
* f30 repos gets the f31 key
* you update your f30-repos
* you jump to rawhide and dnf just imports the key.

How did you get on rawhide?

Yeah, no longer shipped packages have their signed packages removed
after a while to save space. You just want any newer one there.
<a href="https://kojipkgs.fedoraproject.org/packages/fedora-repos/30/0.5/data/signed/cfc659b9/" title="https://kojipkgs.fedoraproject.org/packages/fedora-repos/30/0.5/data/signed/cfc659b9/">https://kojipkgs.fedoraproject.org/packages/fedora-repos/30/0.5/data/sig...</a>
for example.

The f30 repos should already include the f31 key.

<a href="https://src.fedoraproject.org/rpms/fedora-repos/c/81ae6bcd584818d890f7939658884c5c9ec2e85c?branch=f30" title="https://src.fedoraproject.org/rpms/fedora-repos/c/81ae6bcd584818d890f7939658884c5c9ec2e85c?branch=f30">https://src.fedoraproject.org/rpms/fedora-repos/c/81ae6bcd584818d890f793...</a>

You can also get it from there:
<a href="https://src.fedoraproject.org/rpms/fedora-repos/raw/f30/f/RPM-GPG-KEY-fedora-31-primary" title="https://src.fedoraproject.org/rpms/fedora-repos/raw/f30/f/RPM-GPG-KEY-fedora-31-primary">https://src.fedoraproject.org/rpms/fedora-repos/raw/f30/f/RPM-GPG-KEY-fe...</a>

kevin

Re: Updating Rawhide vs GPG keys

By =?ISO-8859-1?Q?... at 03/12/2019 - 07:14

Dne 11. 03. 19 v 19:29 Kevin Fenzi napsal(a):

They got the f31 key in -0.4, but the *signed* package with this key
without subsequent changes to the repositories done in -0.5 is not
available anywhere.

I did this try 30-0.5 of course, but this is wrong, since installing
this package makes F30 from my Rawhide, that is not what I want.

May the the whole problem is, that fedora-gpg-keys has to be updated
together with fedora-repos?

~~~

$ LC_ALL=C.UTF-8 sudo dnf update --disablerepo=rawhide
--enablerepo=updates-testing --release 30
<a href="https://kojipkgs.fedoraproject.org/packages/fedora-repos/30/0.5/data/signed/cfc659b9/noarch/fedora-gpg-keys-30-0.5.noarch.rpm" title="https://kojipkgs.fedoraproject.org/packages/fedora-repos/30/0.5/data/signed/cfc659b9/noarch/fedora-gpg-keys-30-0.5.noarch.rpm">https://kojipkgs.fedoraproject.org/packages/fedora-repos/30/0.5/data/sig...</a>
Last metadata expiration check: 0:01:12 ago on Tue Mar 12 12:12:22 2019.
Dependencies resolved.

 Problem: cannot install both fedora-gpg-keys-30-0.5.noarch and
fedora-gpg-keys-30-0.2.noarch
  - package fedora-repos-30-0.2.noarch requires fedora-gpg-keys =
30-0.2, but none of the providers can be installed
  - cannot install the best update candidate for package
fedora-gpg-keys-30-0.2.noarch
  - problem with installed package fedora-repos-30-0.2.noarch
==============================================================================================================================================================================================
 Package                                           
Architecture                             
Version                                  
Repository                                       Size
==============================================================================================================================================================================================
Skipping packages with conflicts:
(add '--best --allowerasing' to command line to force their upgrade):
 fedora-gpg-keys                                   
noarch                                   
30-0.5                                   
@commandline                                    102 k

Transaction Summary
==============================================================================================================================================================================================
Skip  1 Package

Nothing to do.
Complete!

~~~

Which is not what you expect?

Vít

Re: Updating Rawhide vs GPG keys

By Kevin Fenzi at 03/12/2019 - 14:53

On 3/12/19 4:14 AM, Vít Ondruch wrote:
Thats due to compose issues. A compose completed last night so this
should be there now?

The problem is that we are calling rawhide two things and have seperate
config for it. Once we call it 'rawhide' instead of a number and use the
same config as other branches I think most of these problems will go away.

Try again with --enablerepo=fedora ? You need the base repo not just
updates-testing?

kevin

Re: Updating Rawhide vs GPG keys

By =?ISO-8859-1?Q?... at 03/13/2019 - 06:49

Ok, so the process which worked for me was:

~~~

$ sudo dnf update
https://kojipkgs.fedoraproject.org/packages/fedora-repos/30/0.5/data/signed/cfc659b9/noarch/fedora-{gpg-keys,repos{,-rawhide}}-30-0.5.noarch.rpm

$ sudo dnf update --enablerepo=rawhide fedora-gpg-keys --release 31

~~~

But what I really hate about is that after the first step, my system
becomes F30. Yes, the other step changes it back to Rawhide, but in the
meantime, all the repos I had disabled (such as rawhide-modular) are
enabled again, because there is unnecessary fiddling with repositories.

I submitted
<a href="https://src.fedoraproject.org/rpms/fedora-repos/pull-request/29" title="https://src.fedoraproject.org/rpms/fedora-repos/pull-request/29">https://src.fedoraproject.org/rpms/fedora-repos/pull-request/29</a>, which
would enable the process to simplify and avoid the issues with the repo
files to either:

~~~

$ sudo dnf update
<a href="https://kojipkgs.fedoraproject.org/packages/fedora-repos/30/0.5/data/signed/cfc659b9/noarch/fedora-gpg-keys-30-0.5.noarch.rpm" title="https://kojipkgs.fedoraproject.org/packages/fedora-repos/30/0.5/data/signed/cfc659b9/noarch/fedora-gpg-keys-30-0.5.noarch.rpm">https://kojipkgs.fedoraproject.org/packages/fedora-repos/30/0.5/data/sig...</a>

~~~

or even to:

~~~

$ sudo dnf update --disablerepo=* --enablerepo=updates{,-testing}
fedora-gpg-keys

~~~

I have not tested the latter.

I wish the PR was merged, because the strict relation between repos and
gpg-keys serves no purpose IMO.

Vít

Dne 12. 03. 19 v 19:53 Kevin Fenzi napsal(a):

Re: Updating Rawhide vs GPG keys

By =?ISO-8859-1?Q?... at 03/12/2019 - 07:46

Dne 12. 03. 19 v 12:14 Vít Ondruch napsal(a):

<a href="https://src.fedoraproject.org/rpms/fedora-repos/pull-request/29" title="https://src.fedoraproject.org/rpms/fedora-repos/pull-request/29">https://src.fedoraproject.org/rpms/fedora-repos/pull-request/29</a>

V.

Re: Updating Rawhide vs GPG keys

By =?ISO-8859-1?Q?... at 03/12/2019 - 07:58

Dne 12. 03. 19 v 12:46 Vít Ondruch napsal(a):

I should have better explained this. Updating fedora-repos and
fedora-repos-rawhide together with fedora-gpg-keys to this specific
version makes F30 from Rawhide by disabling the "rawhide" repo (and
enabling the fedora repo, which should not be problem at all). Then all
the packages from F30 are taken including fedora-release, which then
make F30 from Rawhide.

V.

Re: Updating Rawhide vs GPG keys

By Fabio Valentini at 03/11/2019 - 14:45

I'm having a similar problem, but with Silverblue / rawhide.

I installed the system when rawhide was still f30, but now I can't run
"rpm-ostree upgrade" anymore, due to this error:

Enabled rpm-md repositories: rawhide
Updating metadata for 'rawhide'... done
error: Failed to download gpg key for repo 'rawhide': Curl error (37):
Couldn't read a file:// file for
<a href="///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-31-x86_64" title="///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-31-x86_64">file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-31-x86_64</a> [Couldn't open file
/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-31-x86_64]

So I'm *VERY* sure that I didn't mess with the system myself (since it's a
Silverblue installation), but still it's broken due to the missing GPG keys.

Fabio

Re: Updating Rawhide vs GPG keys

By Kevin Fenzi at 03/12/2019 - 14:49

Yeah, so the problem is that you were on f30 and then braching happend
and suddenly you were on f31 before you had a chance to pick up the
updated fedora-repos-30 with the 31 key in it.

We need to revamp this entirely, and as luck would have it, we have a plan:

<a href="https://lists.fedoraproject.org/archives/list/ ... at lists dot fedoraproject.org/thread/5UVGSBRLX352A4S2CBZ2CGBXPAGQTYKB/" title="https://lists.fedoraproject.org/archives/list/ ... at lists dot fedoraproject.org/thread/5UVGSBRLX352A4S2CBZ2CGBXPAGQTYKB/">https://lists.fedoraproject.org/archives/list/ ... at lists dot fedoraproject....</a>

How does this plan work with silverblue? Not sure... could use some
input from them.

kevin

Re: Updating Rawhide vs GPG keys

By Colin Walters at 03/13/2019 - 09:45

It's the same model for Fedora CoreOS btw; so terminology may be "ostree-based systems" or so.

An important thing to bear in mind here is there's a big difference between "dnf" and "libdnf" in this particular respect:
<a href="https://github.com/rpm-software-management/libdnf/issues/43" title="https://github.com/rpm-software-management/libdnf/issues/43">https://github.com/rpm-software-management/libdnf/issues/43</a>

rpm-ostree (and PackageKit) link libdnf. "dnf" today is still in progress of using libdnf in various places, and last I looked dnf still uses GPG code derived from yum.

So for anything using libdnf which auto-imports all keys in /etc/pki/rpm-gpg, AFAIK the only thing we need to do is ship the new version N+1 key as an update to version N systems.

Re: Updating Rawhide vs GPG keys

By =?ISO-8859-2?Q?... at 03/13/2019 - 06:33

Dne 12. 03. 19 v 19:49 Kevin Fenzi napsal(a):
I am afraid that this will not help in this situation, because even if $releasever will be equal to "rawhide" you still
will have in repo file:
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-$releasever-$basearch

which will have prior the branching *content* of F30 gpg key. Then after branching (let say 4 weeks later) you will run
'dnf upgrade'. It will try to download new fedodra-gpg-keys package, which will be signed by F31 gpg key.
IMO the only solution to this is:
* create new gpg keys several months before branching and add it to fedodra-gpg-keys package and
* gpgkey in repo file lists both gpg keys
or
* sign rpm packages in rawhide by both keys - and I'm afraid our infrastructure is not ready for this.

Miroslav

Re: Updating Rawhide vs GPG keys

By Kevin Fenzi at 03/14/2019 - 13:03

On 3/13/19 3:33 AM, Miroslav Suchý wrote:
Yeah.

Yep. We should do this, but note that this only partly solves it. What
if I have a rawhide machine from when rawhide was f29 say or older and
decide to try and update it? :) Of course you should always update your
rawhide machines frequently.

but it would help. We could even just generate them always at least a
release in advance. ie, make sure the f32 key goes out with f30.

So, you mean current rawhide should list the f31 key and the (not yet
made) f32 key? yeah, we could do that I think. I haven't tested, but man
dnf.conf implies you can specify multiple keys per repo.

I persued this. Our infrastructure is fine with it... but rpm isn't.
<a href="https://github.com/rpm-software-management/rpm/issues/189" title="https://github.com/rpm-software-management/rpm/issues/189">https://github.com/rpm-software-management/rpm/issues/189</a>

kevin

Re: Updating Rawhide vs GPG keys

By Colin Walters at 03/12/2019 - 08:54

On Mon, Mar 11, 2019, at 2:46 PM, Fabio Valentini wrote:
Hi, edit /etc/ostree/remotes.d/fedora-workstation.conf to look like this:

<a href="https://github.com/coreos/fedora-coreos-tracker/issues/143#issuecomment-461148282" title="https://github.com/coreos/fedora-coreos-tracker/issues/143#issuecomment-461148282">https://github.com/coreos/fedora-coreos-tracker/issues/143#issuecomment-...</a>

This will fix the GPG key issue and also give you much faster updates.
We have work in flight to try to ship this by default.

Re: Updating Rawhide vs GPG keys

By Fabio Valentini at 03/12/2019 - 10:57

Thanks for the response, but that didn't help with dnf failing the GPG
check.
I think ugrading from f30-rawhide to f31-rawhide is broken in general - I
had to disable gpgchecks for _both_ the ostree remote and the rawhide dnf
repo to make `rpm-ostree upgrade` succeed.

Fabio

Re: Updating Rawhide vs GPG keys

By Jan =?utf-8?Q?P... at 03/11/2019 - 10:01

On 11/03/19 12:31 +0100, Vít Ondruch wrote:
Good question; have observed this over several (if not all since
to-be-f26 based Rawhide) jumps like that, and always have solved this
inconvenience by hand and moved on.

GPG signatures related must-do consequence of branching for those
using mock with the branched + Rawhide targets is also to update
mock-core-configs, which is currently also not very straightforward
and could perhaps be handled more gracefully:
<a href="https://bugzilla.redhat.com/show_bug.cgi?id=1686869" title="https://bugzilla.redhat.com/show_bug.cgi?id=1686869">https://bugzilla.redhat.com/show_bug.cgi?id=1686869</a>

Re: Updating Rawhide vs GPG keys

By Jan =?utf-8?Q?P... at 03/11/2019 - 13:05

On 11/03/19 15:01 +0100, Jan Pokorný wrote:
Actually, fedora:rawhide container from Docker Hub (in CI scenario
here) is affected with this problem as well (cannot update with the
new-key-signed f30/f31 packages because of this, making the CI
procedure fail[*]).

Should not at least the "official container images" follow the
branching point automatically and promptly to prevent GPG signature
imposed disruptions ASAP?

[*] <a href="https://gitlab.com/poki/pacemaker/-/jobs/175445117" title="https://gitlab.com/poki/pacemaker/-/jobs/175445117">https://gitlab.com/poki/pacemaker/-/jobs/175445117</a>

Re: Updating Rawhide vs GPG keys

By Jan =?utf-8?Q?P... at 03/13/2019 - 05:41

On 11/03/19 18:05 +0100, Jan Pokorný wrote:
Not sure if this is thanks to the images indeed being
periodically updated (image's sha256 changed from
1c2065c021750bfa8b1530d2cce08dffda03f0fbe5fba4f21b29fc8e909954a9 to
eff97b05aaa6ef5b3444fc95cc8b340266ea63474dca3f557f07ccfd70f1fee1
between failed vs. successful run), but everything is back to normal
now, and this symptom of the failed "dnf update" with fedora:rawhide
went away, too:

All in all, Would be preferred if this was glitch-free all the time.

Re: Updating Rawhide vs GPG keys

By Steven A. Falco at 03/11/2019 - 09:05

On 3/11/19 7:31 AM, Vít Ondruch wrote:
I was able to update by first manually updating the keys via:

cd /var/cache/dnf/rawhide-2d95c80a1fa0a67d/packages
rpm -Uvh \
fedora-gpg-keys-31-0.1.noarch.rpm \
fedora-release-31-0.1.noarch.rpm \
fedora-release-common-31-0.1.noarch.rpm \
fedora-repos-31-0.1.noarch.rpm \
fedora-repos-rawhide-31-0.1.noarch.rpm

Then I was able to do a normal "dnf update --refresh".

Note that your package directory location may be slightly different. I don't know if the "rawhide-2d95c80a1fa0a67d" part is consistent or just where mine happened to be. But if you search for one of the "keys" packages inside the dnf cache area you should be able to find it.

Steve

Re: Updating Rawhide vs GPG keys

By Oron Peled at 03/12/2019 - 05:13

Hi,

I suspect it's a chicken and egg issue:
* Look at /etc/yum.repos.d/fedora-rawhide.repo
* You can see the line:
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-$releasever-$basearch
* But "$releasever" is determined by the version of "fedora-release" package.
* So dnf, tries to (re)import f30 gpg key.
* The import is OK, but doesn't help, because packages are signed with f31 key.

I cannot test it now, since I already did the "manual upgrade" workaround yesterday.
Anyone that want to check it:
* Temporarily edit "gpgkey" and modify "$releasever" to "31"
* Use dnf to upgrade and check if it imports the new GPG key and work correctly.

Bye,