DevHeads.net

Security updates and batched pushes

Hi all,

within the whole Meltdown and Spectre story I realized that Koji pushes
even security updates to batched only, not directly to stable. In
concrete case we have the firefox update [1], which already received 10
positive karma and many users complaining that it takes so long to get
it out as a stable update. Shouldn't we change that behaviour to get
such security updates out fast when maintainer relies on autopush when
karma threshold is reached?

Greetings,
Christian

[1] <a href="https://bodhi.fedoraproject.org/updates/FEDORA-2018-276558ff6f" title="https://bodhi.fedoraproject.org/updates/FEDORA-2018-276558ff6f">https://bodhi.fedoraproject.org/updates/FEDORA-2018-276558ff6f</a>

Comments

Re: Security updates and batched pushes

By Kevin Fenzi at 01/07/2018 - 19:36

On 01/07/2018 01:38 PM, Christian Dersch wrote:
The critera for bypassing batched is if the update is marked "urgent".

If they are urgent sure... perhaps this should have been so marked?
Or the maintainer/submittor can request stable anytime after it has
enough karma.

Anyhow, I have pushed it to request stable and will do another f27 push
after the current ones finish to get it out today.

kevin

Re: Security updates and batched pushes

By Martin Stransky at 01/10/2018 - 07:19

On 01/08/2018 01:36 AM, Kevin Fenzi wrote:
Seems to be my fault then, didn't know that the urgent/high state has
the meaning there.

ma.

Re: Security updates and batched pushes

By Peter Robinson at 01/08/2018 - 01:59

Or once you push it stable and it's queued for batched the maintainer
then has an option to push stable in the same spot as usual, just
needs the maintainer to push the button again.

Re: Security updates and batched pushes

By Kevin Kofler at 01/07/2018 - 23:01

Kevin Fenzi wrote:
The problem is, this appears to be insufficient.

I really don't understand why we do this "batched" thing to begin with.
Users who want to batch updates have always been able to do it, GNOME
Software will even do it for them. Now, those users who want to batch their
updates are forced to follow Fedora rel-eng's schedule for the batches
rather than being able to pick their own weekday, so how does the server-
side batching help them? And those users (like me) who want to get their
updates, including security fixes (!) as we see here, as soon as they passed
testing are now screwed.

If people insist on that "batched" misfeature, can we please at least get a
fast track repository that contains all the batched updates (but no updates
that are still in testing and have not been batched yet!)?

Kevin Kofler

Re: Security updates and batched pushes

By Kevin Fenzi at 01/08/2018 - 12:39

On 01/07/2018 08:01 PM, Kevin Kofler wrote:
To reduce the constant flow of updates that are very minor or affect
very few mixed in with the major updates that affect lots of people and
are urgent. To save users downloads of repodata.

rel-eng's schedule is a cron job at 03:00 on tuesday (so the batch
appears on wed's pushes).

There was some discussion about changing the gnome batching based on the
Fedora batching, but I don't know whats happened there.

I haven't seen a bunch of urgent updates get blocked by this process.
Do you have more data for updates that hit this?

I would be very much against additional repos like this.

kevin

Re: Security updates and batched pushes

By Peter Robinson at 01/08/2018 - 21:17

On Mon, Jan 8, 2018 at 5:39 PM, Kevin Fenzi < ... at scrye dot com> wrote:
I thought for some reason that all updates marked as security were
automatically urgent, maybe I'm misremembering, but if not it might be
good to do that as a RFE that way all security updates go out non
batched.

Re: Security updates and batched pushes

By Matthew Miller at 01/09/2018 - 08:43

On Tue, Jan 09, 2018 at 02:17:43AM +0000, Peter Robinson wrote:
There was some discussion. There are plenty of updates which have some
security implication but which aren't really urgent. I'd rather them
stay described as non-urgent security updates than people start not
calling them security updates. When they _are_ important to get to
users quickly, they should be marked that way. But that's always a
tradeoff -- more time in testing gives more of a chance to find
regressions or other probems.

Also, I think everyone in this discussion on _this_ list who would like
updates faster should probably be using updates-testing. Or at least
_looking_ at updates-testing. You can always pull individual updates
from there on a per-package basis, and doing this helps everyone else.

Re: Security updates and batched pushes

By Kevin Kofler at 01/09/2018 - 16:06

Matthew Miller wrote:
I want tested updates faster. I don't want untested updates.

I also don't think it would be a productive use of my time to go through all
testing updates every day to see which ones are batched for stable. This is
something that could easily be automated by software (i.e., by having Bodhi
push the updates to a fast track repository), why should I be doing this by
hand?

What I do normally do is read the update notes of every update that I am
about to apply. This is also a reason why I hate the batching, because the
batches are huge and painful to check through. I prefer more frequent
smaller pushes that I can easily read through. But what I can definitely
tell you is that most updates do not contain enough information in the
update notes to really know their impact, so using that as a basis for
applying or not applying some update from updates-testing is not going to
scale either.

By the way, the reason I did not voice these complaints in the thread
initially announcing this change is that I was both too angry and too busy
at that time to come up with a polite reply, and besides, the change was
already implemented when announced anyway.

Kevin Kofler

Re: Security updates and batched pushes

By Michael Catanzaro at 01/08/2018 - 23:00

On Mon, Jan 8, 2018 at 8:17 PM, Peter Robinson < ... at gmail dot com>
wrote:
Most security updates are simply not urgent, and we really don't want
to pester users with these on a daily basis. So it would be nice if
security updates went through batched by default.

Still, this update was marked as a high-severity security update. It
probably makes sense that those should skip batched, in addition to
updates marked as urgent.

Michael

Re: Security updates and batched pushes

By Kevin Kofler at 01/08/2018 - 13:53

Kevin Fenzi wrote:
Urgency is always in the eye of the beholder. I as a user consider all
security updates "urgent", and in addition, I want ALL updates as soon as
they passed testing no matter whether they actually are urgent.

But the users were already able to opt to update only weekly. So why force a
fixed schedule on them?

This does not work in practice because there is almost always at least one
urgent update that requires downloading the whole repodata. (And also
because maintainers are free to skip batched without giving a reason. I
always do this because I consider batching a disservice to my users.)

But what if this is a company that wants to do weekly updates on Monday
morning in order to be free from disruptions for the working week? Then they
will get the updates up to almost 2 weeks late rather than up to 1 week as
both you and they intend.

Batching really needs to be left to the client side!

I have empirically seen security updates end up in the batches, but I have
not checked each of them to see whether it actually went through "batched"
or just happened to go out on that day.

But again, I think it is essential for users to be able to opt to getting
updates without arbitrary delays.

Why? It would allow you to keep the server-side batching while still
allowing those users like me to opt out of it. And the repodata download
size for fast track would be minimal if updates that went out to stable get
removed from fast track.

Even RHEL has a fast track repository.

Kevin Kofler

Re: Security updates and batched pushes

By Kevin Fenzi at 01/09/2018 - 12:59

On 01/08/2018 10:53 AM, Kevin Kofler wrote:
You also don't want updates-testing to even exist right?

To save all the Fedora users in the world from having to update metadata
for minor changes. Since there's a hourly dnf makecache every user in
the world pulls down new metadata ever time we update a repo. If we
update a repo for some minor enhancements it means everyone in the world
has to pay for that. If we just push all those out every tuesday and
don't update those unless there's something urgent we save everyone a
lot of bandwith and us computing time/resources.

There are definitely more days when there are no updates for a
particular repo now. Of course there would be even more if you (or those
who do likewise) wouldn't skip batched, but probibly we need to explain
why more clearly.

...snip...
because it would be a ton more infrastructure and resources.

kevin

Re: Security updates and batched pushes

By Andrew Lutomirski at 01/10/2018 - 14:23

Could Fedora, perhaps, come up with a way to make incremental metadata
updates fast? This shouldn't be particularly hard -- a tool like
casync or even svn should work pretty well. Or it could be a simple
ad-hoc thing. Have the mirrors serve up both the whole metadata blob
and a list of metadata changes. The client could grab the list of
changes from last time, compute a bitwise-identical blob, and verify
the signature on that, deltarpm style. (But on the decompressed data,
of course -- no need to repeat the mistakes of the past.)

It seems that all this batched stuff is a rather weak hack to work
around the extreme inefficiency of Fedora's metadata distribution
model.

Re: Security updates and batched pushes

By King InuYasha at 01/11/2018 - 08:58

On Wed, Jan 10, 2018 at 2:23 PM, Andrew Lutomirski < ... at mit dot edu> wrote:
This was actually prototyped two years ago with zsync:
<a href="https://github.com/rh-lab-q/deltametadata-prototype" title="https://github.com/rh-lab-q/deltametadata-prototype">https://github.com/rh-lab-q/deltametadata-prototype</a>

It worked, but it was slow since it wasn't implemented in librepo and
of course we never got it implemented in the infrastructure.

Details: <a href="https://github.com/rh-lab-q/deltametadata-prototype/wiki/Deltametadata-of-repo-md-files" title="https://github.com/rh-lab-q/deltametadata-prototype/wiki/Deltametadata-of-repo-md-files">https://github.com/rh-lab-q/deltametadata-prototype/wiki/Deltametadata-o...</a>

The Plan of 2017 that never got done:
<a href="https://github.com/rh-lab-q/deltametadata-prototype/wiki/Plan-2017" title="https://github.com/rh-lab-q/deltametadata-prototype/wiki/Plan-2017">https://github.com/rh-lab-q/deltametadata-prototype/wiki/Plan-2017</a>

Re: Security updates and batched pushes

By Stephen John Smoogen at 01/10/2018 - 14:38

On 10 January 2018 at 14:23, Andrew Lutomirski < ... at mit dot edu> wrote:
This sounds a lot like the Atomic project and how it does things...

All things are a weak hack to work around some inefficiency and an
extremely important process to deal with needs. Trying to label
something without taking into consideration of the other is where most
arguments go pear shaped.

Re: Security updates and batched pushes

By Colin Walters at 01/11/2018 - 08:41

On Wed, Jan 10, 2018, at 2:38 PM, Stephen John Smoogen wrote:
Atomic could mean either (rpm)-ostree or Docker/OCI. In the
Docker/OCI world search isn't standardized yet AIUI but there's
progress on that upstream. I am not aware of the state of things
overall there - it'd be really interesting to have someone who is
clued in comment.

I do ostree and rpm, which is used for Atomic Host (and
the Atomic Workstation). Now we're in
the midst of a deeply fundamental rework:
<a href="https://github.com/projectatomic/rpm-ostree/issues/1081" title="https://github.com/projectatomic/rpm-ostree/issues/1081">https://github.com/projectatomic/rpm-ostree/issues/1081</a>

And very specifically on this topic:
<a href="https://github.com/projectatomic/rpm-ostree/issues/1127" title="https://github.com/projectatomic/rpm-ostree/issues/1127">https://github.com/projectatomic/rpm-ostree/issues/1127</a>

(That issue includes my thoughts on e.g. ostree vs git for metadata)

That said, if someone is actually planning to *work* on this,
the best place to discuss is <a href="http://lists.rpm.org/pipermail/rpm-ecosystem/" title="http://lists.rpm.org/pipermail/rpm-ecosystem/">http://lists.rpm.org/pipermail/rpm-ecosystem/</a>
as you're more likely to have the relevant people involved.
I do plan to take some of my ideas from the above rpm-ostree
issue to that list, but it's going to come after implementing jigdo.

Re: Security updates and batched pushes

By Colin Walters at 01/11/2018 - 08:58

One followup that should help people understand things:

When someone pushes an update to a package that isn't
in Atomic Host (or Workstation), *and* one is using rpm-ostree
in "pure ostree" mode (i.e. you never ran `rpm-ostree install`),
then checking for updates just uses libostree, which like any
sane image system, makes this a very cheap operation.

In the case of libostree, checking for "no changes" is just a few
HTTP requests for tiny files.

Say for example you've got a bunch of CentOS containers
on your FAH system (IMO this is not just a valid use case but
one I'd encourage) - you don't see an available update whenever
a new Firefox or whatever appears.

We were also doing ~two week batching for FAH updates before
Bodhi started doing batching...the interaction between those
really could be better...that's a whole other topic.

Anyways though to finish the explanation:
the second you do `rpm-ostree install libvirt` for example
(as I have done on my home server), you're doing *both* systems;
now libdnf comes into play, and it's the same RPM metadata from Fedora
with all the same performance penalty. Smoothing out this dramatic cliff of
a transition is part of the reason I'm working on rpm-ostree jigdo.

Re: Security updates and batched pushes

By Andrew Lutomirski at 01/10/2018 - 14:46

On Wed, Jan 10, 2018 at 11:38 AM, Stephen John Smoogen < ... at gmail dot com>
wrote:

--Andy

Re: Security updates and batched pushes

By Stephen John Smoogen at 01/10/2018 - 15:01

On 10 January 2018 at 14:46, Andrew Lutomirski < ... at mit dot edu> wrote:
OK clearly I should not try to help with a single sentence email as it
didn't help anyone. I should have asked more questions on what you
meant by metadata and what you meant by distribution and how svn would
have been used instead. I should have also asked if you knew enough
about how atomic did things to give an informed opinion on how that
was similar to the request. Without that we ended up talking past each
other.

Re: Security updates and batched pushes

By Andrew Lutomirski at 01/10/2018 - 15:19

On Wed, Jan 10, 2018 at 12:01 PM, Stephen John Smoogen < ... at gmail dot com> wrote:

SVN is probably a poor model, but imagine that the various metadata
files (repodata, I think) were check in, uncompressed, to an SVN repo.
Then the client could do 'svn up' and take advantage of delta
compression. I suspect the server load would be too high, though.

Basically, what I'm getting at is that Fedora seems to distribute
comps, prestodelta, primary, filelists, and updateinfo, and they add
up to 20MB compressed for the updates repo. AFAICT it gets
re-downlaoded from scratch every time, even though the actual list of
changes is probably minimal from day to day.

I don't know too much about the details. I suspect that, if the
.xml.gz files were, instead, one file per package, then they could be
served up from an ostree repo.

Re: Security updates and batched pushes

By Tomasz Torcz at 01/10/2018 - 07:01

So, if there ARE updates to be pushed (marked urgent?) and we update metadata
ANYWAY, this is a good point to flush everything from batched in this push.

Re: Security updates and batched pushes

By Kevin Fenzi at 01/10/2018 - 11:12

On 01/10/2018 04:01 AM, Tomasz Torcz wrote:
We could yeah. We could also only ever push when there are urgent updates...

kevin

Re: Security updates and batched pushes

By Tim Landscheidt at 01/09/2018 - 17:09

BTW, are there technical reasons why the metadata is updated
en bloc and not incrementally like for example delta RPMs,
or is it just that nobody bothered to implement something
like that yet for metadata?

Tim

Re: Security updates and batched pushes

By Samuel Sieb at 01/10/2018 - 15:37

On 01/09/2018 02:09 PM, Tim Landscheidt wrote:
This has been discussed for a very long time:
<a href="https://bugzilla.redhat.com/show_bug.cgi?id=850896" title="https://bugzilla.redhat.com/show_bug.cgi?id=850896">https://bugzilla.redhat.com/show_bug.cgi?id=850896</a>

Re: Security updates and batched pushes

By Kevin Fenzi at 01/10/2018 - 11:06

On 01/09/2018 02:09 PM, Tim Landscheidt wrote:
There is no implementation that I know of. It's been talked about for
many years, but not been implemented.

kevin

Re: Security updates and batched pushes

By Jonathan Dieter at 01/11/2018 - 10:54

On Wed, 2018-01-10 at 08:06 -0800, Kevin Fenzi wrote:
We talked a bit about using casync for delta metadata at Flock last
summer, and I even did a small proof of concept to see what kind of
savings we'd see, but there were a couple of issues that came up:

* casync creates a large number of very small files (roughly 4k files
for yesterday's metadata) that mirrors would need to sync
* Users would need to download a significant portion of those files to
delta against (anywhere from 0 to all of them, depending on how
similar their metadata is)

Downloading any significant number of tiny files over http/1.x is
really, really slow as the downloads are serial.

On reflection, one idea that might fix both problems would be if casync
could concatenate the cacnk files into one large block (cablk,
perhaps), and have the caidx file contain the location of each cacnk in
the block file. Using http ranges, you would then be able to download
just the parts of the file you wanted (though I have not actually
tested whether specifying a large list of http ranges will download
faster than serially downloading each individual file).

FWIW, casyncing Jan 4th's metadata over Dec 15th's downloads 9.6MB (in
just over 1k cacnk files), while downloading the same metadata directly
would total 16MB. More frequent updates would be smaller, but I'm not
convinced the difference from an average updates push to another would
be less than 2MB.

Jonathan

Re: Security updates and batched pushes

By Kevin Kofler at 01/09/2018 - 15:57

Kevin Fenzi wrote:
That is not true. I want to leave the decision whether and for how long an
update needs to be tested to the package maintainer instead of enforcing
minimum testing requirements in the software, because the software can never
understand the exact context. Removing updates-testing entirely is not what
I want! But this is unrelated to the current issue of artificially delaying
updates that satisfy all the criteria for being pushed to stable.

So to save people the download, you make a change that totally defeats the
point of dnf checking for updates every hour to begin with?

This does not work in practice because there are always updates that are not
batched.

Are there really? The last couple days, there were basically daily pushes
with around 2 updates each.

The batching only makes the daily pushes smaller and not empty, which does
not help at all for reducing repodata download size, because there are still
no repodata deltas implemented.

Really? Bodhi composes (or triggers the compose of, let's please not discuss
the technical details down to that level) 2 repositories per release at each
push, of which one (updates-testing) already works pretty much the way the
fast track repository would work (updates transit there, but leave again
when they reach the next level). How would adding a third level require a
ton more resources? It would just need some small changes to the Bodhi
implementation ("pushes to batched" would have to be actually pushed, to the
fast track repository) and could otherwise use all the existing mirroring
infrastructure. And users would be able to opt in to getting stable updates
without batching. And even if those users keep the regular repodata
downloads enabled, the downloads for fast track would still be much smaller
than the repodata downloads for all of stable. And having fast track
available would also reduce the proportion of updates that skip batched. (I
know I would think twice about skipping batched for my updates if I knew
that the users have a way to opt out of the batching. Right now, I don't
even consider using batched.)

I see only advantages of such an approach, for minimal infrastructure costs.
Almost everything you need is already there!

Kevin Kofler

Re: Security updates and batched pushes

By Kevin Fenzi at 01/10/2018 - 11:05

On 01/09/2018 12:57 PM, Kevin Kofler wrote:
Agreed. Thanks for clarifying.

It doesn't do that though. dnf wants the latest metadata so it can let
users use that cache for things like searching for packages or listing
them or the like.
I... have seen updates pushes that do not take place when I have been
pushing updates, so I assert you are incorrect. True, it doesn't happen
as often as I was hoping, but it does and has happened.
At least one of those cases was me pushing firefox, right after a
f27-updates push just finished, so yeah, it only had 2 updates in it.
but the technical details are what matters here. ;) Making another repo,
building drpms, and doing all the compose will take time, disk space and
cpu cycles and slow all other updates pushes down.

Anyhow, I don't want this to be a back and forth between just us, I'd
like to hear some other opinions and proposals and have FESCo decide on
some adjustment ehre.

I agree the current setup is not ideal and personally I'm open to
adjusting/changing it, but I don't know that I think we should just drop
batched entirely. We should come up with something that balances all the
various needs (you want updates as soon as they are tested, I want not
to update metadata on people all the time, etc).

kevin

Re: Security updates and batched pushes

By Kevin Kofler at 01/17/2018 - 08:30

Kevin Fenzi wrote:
This week, there have been almost daily nonempty update pushes (listing only
the SRPMs here, and only the updates that affected me):
* Jan 10 (previous batch)
* Jan 11 (not batched, gtk3 and microcode-ctl)
* Jan 12 (not batched, kernel, dhcp, dnfdragora, hplip, webkitgtk4)
* none on Sat Jan 13 (some updates from RPM Fusion, but those have separate
repodata, so they don't count)
* Jan 14/15 (night, not batched, bluez, python-nbconvert, python-qtconsole)
* Jan 15/16 (night, not batched, llvm with all its revdeps, hplip again)
and now today (Jan 17)'s batch includes (for me) 135 (!) binary packages
(less if you count the SRPMs, but in the end, the binary packages are what
really matter).

I don't see how this is helpful.

Kevin Kofler

Re: Security updates and batched pushes

By Kevin Kofler at 01/18/2018 - 10:09

I wrote:
And today (Jan 18), the day after the batch, there's yet another nonempty
push: redhat-rpm-config/nim-srpm-macros, cmake, jnr-*, python-sphinx,
twolame moved from RPM Fusion to Fedora.

Kevin Kofler

Re: Security updates and batched pushes

By Michael Catanzaro at 01/17/2018 - 08:47

Thanks, Kevin. Knowing when the updates are actually going out adds
important context to this discussion.

On Wed, Jan 17, 2018 at 7:30 AM, Kevin Kofler <kevin. ... at chello dot at>
wrote:
Kevin has a point here. Clearly what we have now is, in practice, not
working as intended.

Michael

Re: Security updates and batched pushes

By Randy Barlow at 01/17/2018 - 09:37

On 01/17/2018 08:47 AM, <a href="mailto: ... at gnome dot org"> ... at gnome dot org</a> wrote:
The original intent as I understood it from the thread long ago[0] was
to reduce the number of updates that go out on non-Tuesdays, and make
most updates happen on Tuesdays. The data that Kevin cited seems to be
accomplishing that purpose.

[0]
<a href="https://lists.fedoraproject.org/archives/list/ ... at lists dot fedoraproject.org/thread/HFTB6Z5DAXZV6O4OGFK4I2GQIE42XMHR/#M5EH2C3CDRZ477PO5ADKAVRGO444P34M" title="https://lists.fedoraproject.org/archives/list/ ... at lists dot fedoraproject.org/thread/HFTB6Z5DAXZV6O4OGFK4I2GQIE42XMHR/#M5EH2C3CDRZ477PO5ADKAVRGO444P34M">https://lists.fedoraproject.org/archives/list/ ... at lists dot fedoraproject....</a>

Re: Security updates and batched pushes

By Kevin Kofler at 01/18/2018 - 10:13

Randy Barlow wrote:
But whom does this help? There are still updates going out daily, the
repodata download cost is still there, the notifications too if you aren't
doing client-side batching (and if you are, you don't need server-side
batching to begin with).

The only difference I see is that some fixes take longer to reach the users
and that the huge Tuesday (Wednesday morning for most users) pushes are a
huge pain to go through if you want to actually check what you are updating.
(Am I the only one who bothers reading update notes?)

Kevin Kofler

Re: Security updates and batched pushes

By Randy Barlow at 01/23/2018 - 15:39

On 01/18/2018 10:13 AM, Kevin Kofler wrote:
It would help people with more minimal package sets more often. On days
with just a handful of updates, it's possible that some users don't use
any of them and thus don't have to get notified.

There is some discussion around possibly forming a formal policy around
batching updates[0]. If we did batch updates more then users would more
often get days when they don't need to download metadata, which would be
nice.

[0] <a href="https://pagure.io/fesco/issue/1820" title="https://pagure.io/fesco/issue/1820">https://pagure.io/fesco/issue/1820</a>

Re: Security updates and batched pushes

By Kevin Kofler at 02/02/2018 - 07:24

Randy Barlow wrote:
Even if they don't get notified, they still have to download the whole
metadata in the background.

Kevin Kofler

Re: Security updates and batched pushes

By Andrew Clayton at 01/08/2018 - 20:46

On Mon, 08 Jan 2018 19:53:01 +0100

Right.

I did wonder what was going on with the updates...

dnf-cron, gnome-software etc could all *default* to weekly.

But if I'm doing a dnf update, I want the updates *now*.