DevHeads.net

always update the bootloader during major upgrades

Hi,

This is not a formal proposal, this is for discussion and identifying
liabilities. This email has an x86 GRUB bias only because that's the
bootloader regime I'm most familiar with. I think it should apply to
other archs as well, i.e. their bootloaders shouldn't be permitted to
become stale.

Short version: Fedora should take responsibility for the bootloader
being up to date, by updating it during major version upgrades. This
is already the case on UEFI with conventional installations. I'd like
to make sure it always happens on major version upgrades for BIOS
installations. What's the problem? This would step on any custom
bootloader configuration a user has for multiboot. There is no
reasonable mechanism on BIOS systems to determine if the bootloader is
a Fedora bootloader, and somehow only update a Fedora bootloader and
not touch any other bootloader.

Fedora should be responsible for keeping the bootloader it installs in
~80% of the use cases up-to-date; and ignore the fallout from the ~20%
who have a custom setup that would be stepped on by a forced
bootloader update. The former is a feature and security risk, by
allowing the bootloader to go stale over time. The latter is an
inconvenience.

Longer version:

terms:

bootloader - this is the pre-boot bootloader binaries; on BIOS it's
embedded in the MBR, and the MBR gap or BIOS Boot partition, and
modules found in /boot/grub2 on Fedora. On UEFI, this is the
/EFI/fedora/shimx64.efi and /EFI/fedora/grubx64.efi which in the
typical installation are built and signed by the Fedora build system.
Other bootloaders have different names but generally follow a similar
convention, the point here is to distinguish between the "bootloader"
which runs in the pre-boot environment, and the installed package
containing user space tools that are used to install (or contain) the
bootloader.

Fedora bootloader - this is a bootloader that is derived from Fedora
packages and effort; as contrasted to merely upstream GRUB, or
Ubuntu's GRUB, etc. as these derivatives can actually be substantially
different from each other.

Discussion:

Both gnome-software (pk-offline-update), and dnf system-upgrade, on
BIOS firmware x86 computers, do not update the bootloader. That is, we
do not run 'grub2-install' either before starting an upgrade, or as a
post install operation following an upgrade. Therefore the bootloader
will become stale if the user does not manually run 'grub2-install'
periodically.

On conventional Fedora installations on UEFI computers, the bootloader
is included in the shim and grub2 packages as a file, and is therefore
updated. This happens sporadically within a Fedora release, not only
at major upgrade time. This updating of the bootloader files doesn't
happen on Silverblue on UEFI, so it ends up being subject to the same
problem under discussion.

In the most recent release, Fedora 30, we saw quite a lot of people
run into a problem we knew about before release, directly related to
the bootloader becoming stale. And it only affected BIOS systems (and
Silverblue on both firmware types).
<a href="https://fedoraproject.org/wiki/Common_F30_bugs#blscfg-fail" title="https://fedoraproject.org/wiki/Common_F30_bugs#blscfg-fail">https://fedoraproject.org/wiki/Common_F30_bugs#blscfg-fail</a>
<a href="https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=1652806" title="https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=1652806">https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=1652806</a>

In the bug, you can find some dups as well but also quite a bit of
user aggravation. I'd like to avoid this going forward.

Windows and macOS never involved users in this stuff. They always
asserted complete and total domain over the bootloader, there wasn't
even an option to avoid updating it. And I think Fedora needs to do
the same. Just update it. That benefits most users. It's the
responsible thing to do. And those who need custom setups, can still
do that. It would only get stepped on at major upgrade time. And it's
decently likely we can warn them in advance.

There are some gotchas I'm already thinking of: MBR gap is too small,
there are multiple drives and it's ambiguous which one should get the
bootloader, and so on. I think it's sane to have a test for reasonable
certainty we can and should update the bootloader, and warn and not
update it in the cases that fail that test.

Thoughts?

Comments

Re: always update the bootloader during major upgrades

By Javier Martinez... at 07/04/2019 - 11:41

Hello Chris,

Thanks for bringing this topic and sorry for the late reply.

On Wed, Jun 26, 2019 at 10:20 PM Chris Murphy < ... at colorremedies dot com> wrote:
I agree with you on this. I've commented my thoughts in the
fedora-workstation issue you created:

<a href="https://pagure.io/fedora-workstation/issue/97#comment-580794" title="https://pagure.io/fedora-workstation/issue/97#comment-580794">https://pagure.io/fedora-workstation/issue/97#comment-580794</a>

Best regards,
Javier

Re: always update the bootloader during major upgrades

By Bruno Wolff III at 06/27/2019 - 10:02

On Wed, Jun 26, 2019 at 14:19:26 -0600,
Chris Murphy < ... at colorremedies dot com> wrote:
How do you envision this working for rawhide?
I had a problem where grub1 configs were no longer updated with kernel
updates, where I finally needed to upgrade to grub2.

Re: always update the bootloader during major upgrades

By Chris Murphy at 06/27/2019 - 13:26

On Thu, Jun 27, 2019 at 9:06 AM Bruno Wolff III < ... at wolff dot to> wrote:
Yep, small problem. And I'm not even sure how a 'grub2-install' on
BIOS systems would be initiated only at major upgrade time. But even
Fedora Rawhide does change versions when fcXX becomes fcXX+1, but is
that a sane trigger? I'm not sure what the alternatives are.

And could there be a policy setting in /etc/default/grub for this,
where the user can opt out? And should such an opt out exist? There is
no such opt out for UEFI. Anyway, the how to do this is a different
question than whether to do it, and whether the policy is the same
across all Fedora editions, spins, images, etc.

I think it's uncontroversial to say on IoT the bootloader must be
updated, I imagine no reasonable justification for making this user
domain on this class of device. That is probably also true for
cloud/virtual, and baremetal servers. It's really desktops that
there's an open question of what/who owns the bootloader. And there
I'd say the default really ought to be, maybe even must be, update the
bootloader with some regularity. And if that's true and agreed upon,
then fill in the gaps with some needed details: how often, how to do
it, should there be an opt out, and how to implement that, etc.

Re: always update the bootloader during major upgrades

By Vitaly Zaitsev ... at 06/28/2019 - 02:18

Le jeudi 27 juin 2019 à 12:26 -0600, Chris Murphy a écrit :
It's not, such magic “it's FCX time” break all the time for users that
only do dnf update, or use rawhide (which branches before the next
version logic is deployed), etc

That's, easy,

1. add a generic bootctl install command that knows the different
variants of bootloader used in Fedora, how to install them, how to
identify which variant is appropriate for a system (make grub and other
bootloaders packages install the corresponding info in a directory read
by this command)

2. make it write the id, generation, and deployment options used in a
lockfile every time it (re)installs the bootloader

3. add a config file with a variable to inhibit auto-redeployment

4. add a scriptlet to the bootctl package that calls bootctl install
and
– checks the variant in the lockfile is appropriate for the hardware
(in case of disk mode or copy)
– checks if the generation is current or unknown (unknown = future, do
not touch)
– and if any of those is false, reinstalls the bootloader, unless the
inhibit variable is set

5. add a --force switch to bootctl to allow the operator to force a
bootloader rewrite every time somethins else broke it

Re: always update the bootloader during major upgrades

By Chris Murphy at 06/28/2019 - 19:49

On Fri, Jun 28, 2019, 1:19 AM Nicolas Mailhot via devel
< ... at lists dot fedoraproject.org> wrote:

Twice a year for Rawhide. And upon initiation of major upgrade on
release versions. It wouldn't be all the time.

I think that's completely out of scope. It's inappropriate to wait 10
months let alone 10 years to resolve this problem. And it's an overly
complicated solution. It sounds like the bootloader install equivalent
of grubby that was just deprecated in Fedora 30, not least of which it
was so overly complicated it was difficult getting people to
contribute to or maintain it. Yet another one command to rule them
all? No thanks.

It might be sane for Anaconda to write out a file, or modify
/etc/default/grub, to indicate the user chose to prevent the Fedora
bootloader from being installed (as far as I'm aware this only
actually works on x86 BIOS computers even though it's presented in the
UI for all other firmwares and archs) when Fedora was installed. And
from that information, infer that they do not want the Fedora
bootloader (re)installed.

Re: always update the bootloader during major upgrades

By Vitaly Zaitsev ... at 06/29/2019 - 02:04

Le vendredi 28 juin 2019 à 18:49 -0600, Chris Murphy a écrit :
It's not an overcomplicated solution it's just putting a single stable
unchanging installation command in front of whatever anaconda does.
Which grubby was not since it was grub-specific with lots of options.

Not making the effort to put this single command in place, is the
reason why Fedora boot problems in 2019, are the same than RHL boot
problems in 2000:

1. no one knows exactly what to call to reinstall the bootloader except
bootloader people (it changes and is badly documented),

2. therefore no one can test the result on the scale needed for the
variety of hardware out there

3. when someone hits a bootloader problem, and tries to fix his system
with whatever lies on the internet (because the documentation is not
here, and the documentation is needed, because the actual commands to
type change), it's never exactly what bootloader people think he should
do, so the result is invalid (positive or negative), and not used to
improve the way Fedora does things

4. because there is no master command, and the actual commands change
over time, there is no incentive to do something in new sets of
commands, that salvages whatever was done in the previous set of
commands

5. it ends up in new full system installs just to get anaconda to the
point it does this part (and RHL supported in-place updates, while
Fedora will insist on blowing up the previous install)

6. and everyone involved insists it's "too complex" to set up an easy
to document master install command

7. something should be put in place "now" and not wait 10 months or 10
years

Well Fedora and RHL have spent 20 years avoiding cleaning up this mess
and it's still the same original mess so at one point 10 months seems
awfully short period to get it done.

Regards,

Re: always update the bootloader during major upgrades

By Chris Murphy at 06/30/2019 - 12:43

On Sat, Jun 29, 2019 at 1:05 AM Nicolas Mailhot via devel
< ... at lists dot fedoraproject.org> wrote:
grubby isn't GRUB specific. It supports LILO, ELILO, SILO, ZIPL,
extlinux, GRUB legacy, GRUB2, yaboot, and something else I'm
forgetting no doubt.

This also seems applicable
<a href="https://xkcd.com/927/" title="https://xkcd.com/927/">https://xkcd.com/927/</a>

a. Near as I can tell, the only system under discussion, x86 + BIOS +
multiboot, means exactly one bootloader: BIOS GRUB. That's the only
bootloader Fedora supports in that configuration.

b. Fedora only narrowly supports multiboot: Fedora + Windows, and
Fedora + macOS. As in, only the dual boot variety, and not any Linux
distros. Not even Fedora + Fedora is supported. By that I mean, there
is no release criteria that forms a basis for blocking the release if
something related to creating such a setup during installation is
found to be broken.

I just don't see how a single command to rule them all solves a single
problem. It's like playing musical chairs, except no seat has been
removed.

Fedora x86 BIOS, it's been 'grub2-install' since circa Fedora 14. And
on UEFI since early days it's reinstall shim and grub packages, also
hasn't changed.

But in effect you're proposing the inevitable day when the user still
invokes grub2-install, and the Fedora oracles still say "yeah that's
deprecated on Fedora, you ran the wrong command, that's why your
system is now hosed"

By the way, bootctl is already taken by systemd-boot.

Re: always update the bootloader during major upgrades

By Richard Shaw at 06/29/2019 - 06:23

I don't have the technical expertise in this area to comment on the details
of the implementation, but from a dumb user point of view, could there be a
simple or no-payload package that says "Fedora owns the bootloader" that's
installed by default and power users that wish to maintain it themselves
(or use other bootloaders) and remove the package?

Thanks,
Richard

Re: always update the bootloader during major upgrades

By Chris Murphy at 06/30/2019 - 12:52

On Sat, Jun 29, 2019 at 5:24 AM Richard Shaw < ... at gmail dot com> wrote:
I like this idea. Would bootloader projects be willing to use a single
canonical location and format for this indication? On the other hand,
if I don't want to use the Fedora GRUB bootloader, why do I have that
package still installed? If I were using the Ubuntu bootloader, or
systemd-boot, I would uninstall all the GRUB stuff to ensure it's not
being used, accidentally or otherwise.

Nevertheless, I'm OK with the idea that BIOS GRUB not be automatically
reinstalled everytime there's a package update, but rather postpone it
for major Fedora upgrades.

Re: always update the bootloader during major upgrades

By Adam Williamson at 06/28/2019 - 11:06

On Fri, 2019-06-28 at 09:18 +0200, Nicolas Mailhot via devel wrote:
I am not sure you quite understand the meaning of the word "easy". :P

Re: always update the bootloader during major upgrades

By Vitaly Zaitsev ... at 06/28/2019 - 11:27

Le vendredi 28 juin 2019 à 09:06 -0700, Adam Williamson a écrit :
It's the same core logic that already exists in anaconda, and in the
envisionned dist upgrade, except packaged in a sane reusable easy to
test unit.

Packaging it sanely does not make the core logic simpler sure, but it
removes the mass of problems caused by bootloader installations
happening in multiple badly defined in-between gray dimensions, without
any tracking of what was effectively done in the past.

And the core logic need to be maintained in any case.

So yes, I do believe that quick and dirty is more difficult in the long
run.