DevHeads.net

gross DNF bandwidth inefficiency if filesystem space limited

# dnf upgrade
(dnf nearly exhausts freespace downloading all packages before installing any
packages)
dnf then reports package xxx needs ##MB on / filesystem and exits without
doing any installing
dnf all deletes downloaded packages
# dnf upgrade (package subset, e.g. dnf* rpm* a* b* c* d* e* f* x* y* z*)
dnf downloads needed packages previously downloaded but later deleted, then
installs downloaded packages
# dnf upgrade
dnf downloads needed packages previously downloaded but later deleted, then
installs downloaded packages

Is the above deletion of freshly downloaded packages (wasted time, wasted
bandwidth) known or expected?

Comments

Re: gross DNF bandwidth inefficiency if filesystem space limited

By Radek Holy at 07/30/2015 - 03:48

Known, <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1220074" title="https://bugzilla.redhat.com/show_bug.cgi?id=1220074">https://bugzilla.redhat.com/show_bug.cgi?id=1220074</a>. Should be fixed in dnf-1.0.2.

Re: gross DNF bandwidth inefficiency if filesystem space limited

By Kevin Kofler at 07/30/2015 - 20:56

Radek Holy wrote:
I still don't understand why we don't just enable keepcache by default. Even
after a successful update/install, deleting the cached packages is a major
data loss because it prevents downgrading to them later, after a broken new
update comes out (which also removes the previous update from the mirrors).

Kevin Kofler

Re: gross DNF bandwidth inefficiency if filesystem space limited

By Radek Holy at 07/31/2015 - 12:27

One can say that the mirrors should keep the older versions for this purpose but I don't want to start a flame war.

AFAIK, the "local" plugin is what you are looking for.

Re: gross DNF bandwidth inefficiency if filesystem space limited

By Richard Hughes at 07/31/2015 - 15:14

On 31 July 2015 at 17:27, Radek Holy < ... at redhat dot com> wrote:
I would completely agree. As we can't rely that packages referenced in
metadata just one day old still being on the mirrors means that
PackageKit has to download hundreds of megabytes month more than it
has to.

Richard.

Re: gross DNF bandwidth inefficiency if filesystem space limited

By Nico Kadel-Garcia at 08/01/2015 - 07:33

On Fri, Jul 31, 2015 at 3:14 PM, Richard Hughes < ... at gmail dot com> wrote:
In the RHEL world, EPEL has bitten me really hard this way several
times, especially when packages are discarded and no longer present in
EPEL. So it's worth thinking about in general for RPM based systems.

Re: gross DNF bandwidth inefficiency if filesystem space limited

By Kevin Fenzi at 08/02/2015 - 12:15

On Sat, 1 Aug 2015 07:33:39 -0400

So, here's the things to consider:

* Keeping 2 versions of every package will double mirror space. This
may result in some mirrors dropping things or stopping bothering
mirroring Fedora at all.

* repodata will likewise be 2x (or at least increased a great deal).
Resulting in a bunch more downloading for everyone not just the folks
who might want to downgrade sometimes.

* There could be some nasty issues with keeping known vulnerable/broken
packages around. ie, foo-1.0 has a severe security bug, foo-1.1 fixes
it. You now just need to trick someone into downgrading or directly
installing foo-1.0 (which is in normal repos and signed and
completely valid looking).

But it's not clear exactly what you 3 are proposing (or even if it's
the same thing). :) So, perhaps you could clarify what exactly you want
to do?

kevin

Re: gross DNF bandwidth inefficiency if filesystem space limited

By Kevin Kofler at 08/02/2015 - 23:53

Kevin Fenzi wrote:
But there are plenty of even older packages in the GA repository, also
signed with the same key.

Kevin Kofler

Re: gross DNF bandwidth inefficiency if filesystem space limited

By Kevin Fenzi at 08/03/2015 - 10:35

On Mon, 03 Aug 2015 05:53:15 +0200

Sure, but this increases the exposure.

kevin

Re: gross DNF bandwidth inefficiency if filesystem space limited

By =?ISO-8859-1?Q?... at 08/03/2015 - 11:29

Dne 2.8.2015 v 18:15 Kevin Fenzi napsal(a):
This is actually not true. The repodata should contain just the latest
version, but if I have slightly older version of metadata already
downloaded, I would be probably fine with installation of slightly older
version of package.

Vít

Re: gross DNF bandwidth inefficiency if filesystem space limited

By Kevin Fenzi at 08/03/2015 - 11:45

On Mon, 3 Aug 2015 17:29:30 +0200

Well, as I noted in my reply, I wasn't actually sure what was being
proposed here.

So, you are proposing we do things exactly as we are now, but also keep
around all previous copies of the packages in the repos (but not in the
repodata)?

I'm not sure if that setup would work with dnf. I think it requires
whatever mirror(s) it uses to match the metadata. If you have a older
metadata and the mirror you hit has been updated, I think dnf will say
that the repodata doesn't match and try another.

kevin

Re: gross DNF bandwidth inefficiency if filesystem space limited

By =?ISO-8859-1?Q?... at 08/03/2015 - 12:06

Dne 3.8.2015 v 17:45 Kevin Fenzi napsal(a):
Keep the previous copies for some period of time, e.g. for one week.

I don't think so. At leas I interpret Richard's comment [1] differently.
I think that Gnome Software (DNF) fetches the metadata once a day, then
during that day, update is executed, based on the metadata got earlier.
Unfortunately, during period between fetching of metadata and the actual
update, the packages might not be available anymore, since the
repository content changed.

Vít

[1] <a href="https://lists.fedoraproject.org/pipermail/devel/2015-July/212998.html" title="https://lists.fedoraproject.org/pipermail/devel/2015-July/212998.html">https://lists.fedoraproject.org/pipermail/devel/2015-July/212998.html</a>

Re: gross DNF bandwidth inefficiency if filesystem space limited

By Bill Nottingham at 08/03/2015 - 11:52

Kevin Fenzi (<a href="mailto: ... at scrye dot com"> ... at scrye dot com</a>) said:
At some point, it might be worth doing cost/benefit analysis on continuing
down our existing mirroring strategy and designing for the limits of that
vs. the application of some sponsor funds towards the use of more standard
CDN service and methodology.

Bill

Re: gross DNF bandwidth inefficiency if filesystem space limited

By Kevin Fenzi at 08/03/2015 - 16:14

On Mon, 3 Aug 2015 11:52:01 -0400

Yeah.

We looked at one of the existing CDNs a while back, but the way it was
setup was not very friendly to the sort of content we have. They were
more expecting slowly changing static content, where we have lots of
changes all the time. Things may have changed or other vendors might be
different.

In any case even if we had a CDN we controlled, it wouldn't change the
fact that if you download repodata one day it may not be good the next.
We could fix that I suppose by simply only pushing updates once a week
or something, but I bet people wouldn't like that "solution". ;)

kevin

Re: gross DNF bandwidth inefficiency if filesystem space limited

By Zbigniew =?utf-... at 07/30/2015 - 23:47

On Fri, Jul 31, 2015 at 02:56:48AM +0200, Kevin Kofler wrote:
Most people don't downgrade or do it very rarely? It can be
said that downgrading is an advanced operation, and you can
set keepcache=1 if you need it.

Zbyszek

Re: gross DNF bandwidth inefficiency if filesystem space limited

By Reindl Harald at 07/31/2015 - 04:46

Am 31.07.2015 um 05:47 schrieb Zbigniew Jędrzejewski-Szmek:
and then there are they cases where deps are solved but files conflicts,
the transaction check fails like recently with two broken polkit updates

have fun typing "dnf --skip-broken upgrade" (yes i know NF lacks
--skip-broken ATM) and download the other packages again for *zero reason*

Re: gross DNF bandwidth inefficiency if filesystem space limited

By Zbigniew =?utf-... at 07/31/2015 - 13:25

On Fri, Jul 31, 2015 at 10:46:16AM +0200, Reindl Harald wrote:
Zbyszek

Re: gross DNF bandwidth inefficiency if filesystem space limited

By Reindl Harald at 07/31/2015 - 04:49

Am 31.07.2015 um 10:46 schrieb Reindl Harald:
BTW: all the akmods troubles on the rpmfusion list are coming from the
"dnf-makecache.timer" locking the rpmdb at random moments, wasting
traffic and i find it somehow pervert that packages are not cahced as
default while on the other side a *completly needless* cache job is
running for the same piece of software

Re: gross DNF bandwidth inefficiency if filesystem space limited

By Johnny Robeson at 07/30/2015 - 03:25

On Thu, 2015-07-30 at 03:21 -0400, Felix Miata wrote:
you should take your words in your email signature into account before
using words like "gross" and "wasted time"

Try to show some understanding to the DNF maintainers.

Re: gross DNF bandwidth inefficiency if filesystem space limited

By Felix Miata at 07/30/2015 - 03:39

Johnny Robeson composed on 2015-07-30 03:25 (UTC-0400):

What words would you like better?

s/gross/huge/
s/wasted time/lost time/

Different words, same meaning, exact same problem.

Does the time spent by people testing maintainers' work not count for
anything? Not everyone has unlimited bandwidth or time for testing.