DevHeads.net

Fedora 31 Self-Contained Change proposal: DNF Make Best Mode the Default

<a href="https://fedoraproject.org/wiki/Changes/DNF_Default_Best" title="https://fedoraproject.org/wiki/Changes/DNF_Default_Best">https://fedoraproject.org/wiki/Changes/DNF_Default_Best</a>

== Summary ==
Currently, DNF prefers clean dependency resolution over package updates;
a package (almost) silently won't get updated to a newer version if the new
version has dependency problems. DNF will be changed to prefer updates and fail
if they have dependency resolution issues, while the failure has a
temporal or permanent workaround
hint for users who want to use the older behavior.

== Owner ==
* Name: [[User:jmracek| Jaroslav Mracek]]
* Email: <a href="mailto: ... at redhat dot com"> ... at redhat dot com</a>

== Detailed Description ==
Change the built-in default value of the `best` configuration option
from `0` (false) to `1` (true).

As a result, unless `best` is overridden in the `/etc/dnf/dnf.conf`
file or using `--setopt`, it will default to `1`. As a convenience, we
will also put the explicit `best=1` assignment in the shipped
`/etc/dnf/dnf.conf` file for better transparency, and introduce the
new `--nobest` command-line switch.

The purpose of the `--nobest` switch (as a shorthand for
`--setopt=best=0`) is to make it easy for the user to override the
default setting when needed, and it will also be
[https://github.com/rpm-software-management/dnf/pull/1311/commits/9a3e8fd0da49291d30fd1fef527cffb0bf3f047d#diff-6c823931c6d150295e5011fac6529ab9R144
suggested] in the DNF output when a dependency error occurs.

Relevant excerpt from the updated `dnf.conf(5)`:
<pre>
best boolean
When upgrading a package, always try to install its highest version
available, even only to find out some of its deps are not satisfiable.
Enable this if you want to experience broken dependencies in the
repositories firsthand. The default is True.
</pre>

Relevant excerpt from the updated `dnf(8)`:
<pre>
--nobest
Set best option as false, therefore transactions are not limited to
only best candidates.
</pre>

'''Change in DNF output - missing vim-enhanced-2:8.1.1561-1.fc30'''

Original output. DNF succeed with return code 0:
<pre>
sudo dnf upgrade
Last metadata expiration check: 2:16:40 ago on Mon 24 Jun 2019 04:27:16 PM CEST.
Dependencies resolved.

Problem: package vim-enhanced-2:8.1.1471-1.fc30.x86_64 requires
vim-common = 2:8.1.1471-1.fc30, but none of the providers can be
installed
- cannot install both vim-common-2:8.1.1561-1.fc30.x86_64 and
vim-common-2:8.1.1471-1.fc30.x86_64
- problem with installed package vim-enhanced-2:8.1.1471-1.fc30.x86_64
- cannot install the best update candidate for package
vim-common-2:8.1.1471-1.fc30.x86_64
- package vim-enhanced-2:8.1.1561-1.fc30.x86_64 is excluded
===================================================================================================================================
Package Architecture Version
Repository Size
===================================================================================================================================
Skipping packages with conflicts:
(add '--best --allowerasing' to command line to force their upgrade):
vim-common x86_64
2:8.1.1561-1.fc30 updates 6.7
M

Transaction Summary
===================================================================================================================================
Skip 1 Package

Nothing to do.
Complete!
</pre>

Output after the change. DNF fails with return code 1, but proposing
`--nobest` option as an option to resolve the issue:
<pre>
sudo dnf upgrade
Last metadata expiration check: 2:16:36 ago on Mon 24 Jun 2019 04:27:16 PM CEST.
Error:
Problem: package vim-enhanced-2:8.1.1471-1.fc30.x86_64 requires
vim-common = 2:8.1.1471-1.fc30, but none of the providers can be
installed
- cannot install both vim-common-2:8.1.1561-1.fc30.x86_64 and
vim-common-2:8.1.1471-1.fc30.x86_64
- problem with installed package vim-enhanced-2:8.1.1471-1.fc30.x86_64
- cannot install the best update candidate for package
vim-common-2:8.1.1471-1.fc30.x86_64
- package vim-enhanced-2:8.1.1561-1.fc30.x86_64 is excluded
(try to add '--allowerasing' to command line to replace conflicting
packages or '--skip-broken' to skip uninstallable packages or
'--nobest' to use not only best candidate packages)
</pre>

'''Q&A'''

Can be a default of the best configuration option overwritten easily
and permanently by user?
Yes, just add `best=false` to `/etc/dnf/dnf.conf`
<pre>
[main]
best=False
</pre>

Can be a default of the best configuration option overwritten easily
from commandline?
Yes, just add `--nobest` to command
<pre>
dnf upgrade --nobest
</pre>

What about PackageKit and Gnome Software?
<pre>
PackageKit and Gnome Software will be not affected by the change. In
case that the same behavior will be desired for PackageKit, It will
require changes in PackageKit code.
</pre>

What about Microdnf?
<pre>
Microdnf will be not affected by the change. There is a plan to unify
functional parity and behavior DNF with Microdnf but not before Fedora
33.
</pre>

== Benefit to Fedora ==

This change allows the users to be properly notified when a package
cannot be upgraded to the latest version, instead of silently ignoring
it as an upgrade candidate.

Right now, when DNF runs in `best=0` mode, if a package cannot be
upgraded due to dependency problems, it is skipped and a warning is
printed in the transaction summary table. However, this poses a risk
of important security fixes being overlooked by the user in case they
are broken for some reason, such as due to a repository
misconfiguration or inconsistency within the metadata itself.

Moreover, since DNF always exits with the return code `0` (success)
when in `best=0` mode, this mode is especially risky in automated
scripts invoking DNF in `assumeyes` mode in which case such
unsuccessful package upgrades could easily go unnoticed unless the
logs are manually examined after the fact.

The new behavior is also more in line with the generally accepted
software development practice of failing early and failing fast.

As a secondary benefit, broken upgrade paths in the Fedora
repositories will hopefully be noticed, reported and therefore fixed
sooner. Although, we would prefer if such problems would be detected
before we ship them to users.

'''Summary of benefits:'''

# No silently passed problems with updates
# Broken dependencies faster disappear from Fedora distribution
# Problems will be reported more often - opportunity to fix issues
# Increase in stability of Fedora distribution
# Less issues after branching
# Identical behavior of DNF in all distributions - Fedora/RHEL/Mageia/OpenSuse

== Scope ==
* Proposal owners:
The change is already part of the upstream (dnf-4.1.0) and reverted in
Fedora downstream. The change was composed by following pull requests:

<a href="https://github.com/rpm-software-management/libdnf/pull/678" title="https://github.com/rpm-software-management/libdnf/pull/678">https://github.com/rpm-software-management/libdnf/pull/678</a><br>
<a href="https://github.com/rpm-software-management/dnf/pull/1311" title="https://github.com/rpm-software-management/dnf/pull/1311">https://github.com/rpm-software-management/dnf/pull/1311</a><br>
<a href="https://github.com/rpm-software-management/dnf/pull/1316" title="https://github.com/rpm-software-management/dnf/pull/1316">https://github.com/rpm-software-management/dnf/pull/1316</a><br>
<a href="https://github.com/rpm-software-management/dnf/pull/1319" title="https://github.com/rpm-software-management/dnf/pull/1319">https://github.com/rpm-software-management/dnf/pull/1319</a>

We would like to stop the reverting the changes.

* Other developers: N/A (not a System Wide Change)
* Policies and guidelines: N/A (not a System Wide Change)
* Trademark approval: N/A (not needed for this Change)

== Upgrade/compatibility impact ==
N/A (not a System Wide Change)

== How To Test ==
N/A (not a System Wide Change)

== User Experience ==
Broken upgrades are recognized early, enabling the users to act upon
them by double-checking their repository configuration or filing bugs,
instead of assuming no upgrades are available.

== Dependencies ==
N/A (not a System Wide Change)

== Contingency Plan ==
If there is a massive negative feedback by the rawhide and pre-beta
users, we can revert the
change at the beta freeze. If there is a massive negative feedback by
the beta users, we can
revert the change at final freeze.

* Contingency mechanism: (What to do? Who will do it?) N/A (not a
System Wide Change)
* Contingency deadline: N/A (not a System Wide Change)
* Blocks release? N/A (not a System Wide Change), Yes/No

== Documentation ==
N/A (not a System Wide Change)

== Release Notes ==

Comments

Re: Fedora 31 Self-Contained Change proposal: DNF Make Best Mode

By Petr Pisar at 06/28/2019 - 03:34

On 2019-06-27, Ben Cotton < ... at redhat dot com> wrote:
Nevertheless I have a question whether the "best" strategy applies to
package NEVRAs only or if it also applies to Provides and Requires. E.g.
if a

package A provides FOO = 1

and a

package B provides FOO = 2,

will installing FOO insist on installing package B or will it keep
freedom to choose between A and B?

-- Petr

Re: Fedora 31 Self-Contained Change proposal: DNF Make Best Mode

By Jaroslav Mracek at 06/28/2019 - 03:40

If you request Package A you will always get package A. If you will request
package in lover version, you will get the package in requested version.

Jaroslav

Re: Fedora 31 Self-Contained Change proposal: DNF Make Best Mode

By Zbigniew =?utf-... at 06/27/2019 - 16:56

On Thu, Jun 27, 2019 at 10:37:28AM -0400, Ben Cotton wrote:
I wanted to update today, and I got the following report:
$ sudo dnf upgrade --best
Last metadata expiration check: 0:00:18 ago on Thu 27 Jun 2019 10:45:00 PM CEST.
Error:
Problem: package InsightToolkit-4.9.1-9.fc29.x86_64 requires libnetlib.so.1.14()(64bit), but none of the providers can be installed
- package InsightToolkit-4.9.1-9.fc29.x86_64 requires libv3p_netlib.so.1.14()(64bit), but none of the providers can be installed
- package InsightToolkit-4.9.1-9.fc29.x86_64 requires libvcl.so.1.14()(64bit), but none of the providers can be installed
- package InsightToolkit-4.9.1-9.fc29.x86_64 requires libvnl.so.1.14()(64bit), but none of the providers can be installed
- package InsightToolkit-4.9.1-9.fc29.x86_64 requires libvnl_algo.so.1.14()(64bit), but none of the providers can be installed
- cannot install both vxl-2.0.2-4.fc30.x86_64 and vxl-1.17.0-30.fc30.x86_64
- cannot install both vxl-1.17.0-30.fc30.x86_64 and vxl-2.0.2-4.fc30.x86_64
- cannot install the best update candidate for package vxl-1.17.0-30.fc30.x86_64
- cannot install the best update candidate for package InsightToolkit-4.9.1-9.fc29.x86_64
(try to add '--allowerasing' to command line to replace conflicting packages or '--skip-broken' to skip uninstallable packages)

The problem with this is that it's not possible to figure out what who is too blame.
Specifically:
1. How am I to know that vxl is related to libnetlib.so.1.14()(64bit) and
the other virtual provides?

2.
3.
So even in simple case with two packages, I'd need to do repoquery spelunking
to figure out what is going on. If dnf could tell me something like...

Problem: package InsightToolkit-4.9.1-9.fc29.x86_64 requires
libnetlib.so.1.14()(64bit), libv3p_netlib.so.1.14()(64bit),
libvcl.so.1.14()(64bit), libvnl.so.1.14()(64bit),
libvnl_algo.so.1.14()(64bit), currently provided by vxl-1.17.0.
- There is no upgrade candidate for InsightToolkit.
- Best upgrade candidate vxl-2.0.2-4.fc30.x86_64 does have those Provides.
→ cannot install both vxl-2.0.2-4.fc30.x86_64 and vxl-1.17.0-30.fc30.x86_64 because vxl is not an installonly package.
→ cannot install the best update candidate for package vxl

i.e. group relevant Provides that "connect" two package into one list instead
of repeating the whole set of messages every time,
don't talk about upgrade candidates that don't actually exist,
mention the connection between Provides and package names,
omit evra when not required,
provide more explanations in general,

then I'd see this change in a more positive light. I think the output
right now is just noise for most users.

The premise that bug reports from users will help us catch such cases
— I don't see this as true. We *already know* that InsightToolkit has a problem,
there's a FTBFS bug for it somewhere.

The idea of "fault tolerant systems" is that the system mostly
continues to work in face of small failures in components. The distro
is a big system with thousands of interacting components, and *some*
simply must fail occasionally. The proposal is to make the system
fault-intolerant to notice errors earlier. That just seems wrong.
Returning to the example, why can't dnf print in red letters
at the end of the transaction log:

Upgrade of vxl-1.17.0-30.fc30.x86_64 to vxl-2.0.2-4.fc30.x86_64 was held back because of dependency issues.

Users would see that too.

Zbyszek

Re: Fedora 31 Self-Contained Change proposal: DNF Make Best Mode

By =?ISO-8859-2?Q?... at 06/28/2019 - 04:11

Dne 27. 06. 19 v 23:56 Zbigniew Jędrzejewski-Szmek napsal(a):
This is message for people who are willing/are able to fix or report things. For regular user this should be
.. or run with "--nobest" to skip broken deps.

Re: Fedora 31 Self-Contained Change proposal: DNF Make Best Mode

By Kamil Paral at 06/28/2019 - 07:00

Can somebody clarify the difference between --skip-broken and --nobest?
Because even after reading the man page, I still don't get it. And I
believe most our users will not get it either.

Re: Fedora 31 Self-Contained Change proposal: DNF Make Best Mode

By Zbigniew =?utf-... at 06/28/2019 - 08:05

On Fri, Jun 28, 2019 at 02:00:22PM +0200, Kamil Paral wrote:
--nobest means: consider older versions of packages for installation, don't insist on upgrading everything.
--skip-broken means: skip packages which were selected but cannot be installed, instead of erroring out.

At least that's how I understand it.

Zbyszek

Re: Fedora 31 Self-Contained Change proposal: DNF Make Best Mode

By =?ISO-8859-1?Q?... at 06/28/2019 - 03:41

Dne 27. 06. 19 v 23:56 Zbigniew Jędrzejewski-Szmek napsal(a):

Yep, this is problem. I opened DNF tickets requesting improvement of the
messages, but apparently unsuccessfully :(

Vít

Re: Fedora 31 Self-Contained Change proposal: DNF Make Best Mode

By King InuYasha at 06/27/2019 - 13:07

On Thu, Jun 27, 2019 at 11:25 AM Ben Cotton < ... at redhat dot com> wrote:
So, while it *is* true that Mageia, OpenMandriva, and openSUSE are
shipping with this feature for DNF, the experience isn't what I
consider optimal. I would like to see user experience improvements as
part of making this change. Specifically, when such a dependency error
occurs, I would like to see it prompt the user on what to do, rather
than forcing them to re-run the whole thing over again with a switch
like `--no-best` (which is a terrible name...) or `--allowerasing`.

Zypper offers this capability, so I know libsolv provides a way to do
it. There's no reason DNF can't do that too.

Moreover, this change impacts all consumers of the DNF Python API,
including dnfdragora (through dnfdaemon). The behavior change needs to
be validated with that tool as well, and there's no way to run a
transaction with another flag from that program, so adjustments are
likely necessary there.

Re: Fedora 31 Self-Contained Change proposal: DNF Make Best Mode

By =?iso-8859-1?q?... at 06/27/2019 - 10:43

If I understand this change correctly, then:

· Before: If one package update is uninstallable, then that package
won't be updated, but other packages can still be updated.

· After: If one package update is uninstallable, then *nothing* will be
updated.

And you call that an improvement?

"Best" is an absolutely terrible name for this option. By what
definition is an unusable package "better" than a lower-numbered
package that can actually be installed?

If there is a significant risk that the warning will be overlooked,
then how about just making the warning more visible?

Would it not be possible to program DNF to update what can be updated
and then return a nonzero exit code?

Björn Persson

Re: Fedora 31 Self-Contained Change proposal: DNF Make Best Mode

By Florian Weimer at 06/27/2019 - 12:51

* Björn Persson:

It is. People have missed glibc updates because of the broken composes
without realizing it for quite some time, for example.

Thanks,
Florian