MBI (Playground 2.0)

MayBe I …(can do something useful)?


We've been discussing some (hopefully) nice idea with Mikolaj, Neal and
Jakub how we could improve packager (and user) experience and we have some
proposal which will be described below.

We would like to ask you to read it, understand it and ask us any questions
you have. From the Fedora Infrastructure we would like to ask for some
machines to implement this idea (you can find some more information in
"Implementation details" part).

I would like to apologize for HTML email, but it is much easier to have it
that way to have better visibility and reading easiness.

Feel free to reply to this email or comment in google doc (there is a link
on the bottom).
Proposal Owners

Mikolaj Izdebski (mizdebsk) < ... at redhat dot com> - Java SIG, Fedora
Igor Gnatenko (ignatenkobrain) < ... at fedoraproject dot org> - Rust
SIG, Golang SIG, Neuro SIG, RPM and libsolv contributor
Neal Gompa (ngompa) < ... at gmail dot com> - Rust SIG, Golang SIG, RPM
Jakub Čajka (jcajka) < ... at fedoraproject dot org> - Golang SIG, Container

This proposal aligns with the objective of improving the packager experience
<> by
developing a platform whereby the proposal owners and others can experiment
with improvements that can be funneled back into the official production
ProblemsProblem №1: Build-only packages

Some ecosystems have many build-only packages (packages which are used to
build other packages, without having them installed on end-user systems).
Those ecosystems include Java, Rust and Go.

It is extremely hard to keep up maintaining them due to lack of
time/people. Upstreams are usually changing quickly (sometimes when you
update just one such package, you’d need to update tens of the
dependencies). Few facts about such packages:

They are almost always outdated in released versions of distribution;
They are often FTBFS (e.g. because there was compiler update but not
package update, broken deps, broken APIs due to deps rebases, …).

In Rust ecosystem, we abuse Rawhide to build and store such packages there
and then generate end-user application (e.g. ripgrep) which uses some of
those packages and produces binary for all supported releases. Those
applications have high quality and are supported.

Rawhide gating makes this much more complicated because builds appear in
buildroot slower, updating group of packages would need side tags and it’s
just painful to work with.

<a href="" title=""></a>

And, after all, those packages shouldn’t be shipped to users.
Problem №2: Testing of new rpm/koji/mock features/configuration

When developing new features in RPM (e.g. rich dependencies) or testing
different Koji configuration (e.g. removing gcc/gcc-c++ from the
buildroot), it is required to make custom configuration and try building
Problem №3: Developing modules

Modules are built in MBS as a single unit. It is hard to develop big
modules by progressively improving individual packages or small package
groups. Scratch builds for modules are not implemented. Local builds work
differently from official module builds, they don’t scale and don’t allow
groups of people to work collaboratively. All these problems can be solved
by first developing a flat repository of packages in a shared environment
and then generating modulemd from such package set.
Problem №4: Testing things before push

Primary Fedora Koji and dist-git are not suited for packaging
experimentation. Packagers are expected to experiment on their own systems
and push changes of successful experiments only. But this approach doesn’t
scale with number of maintained packages. Often you can find commits like
“d’oh, forgot to upload sources” or “let’s try with this settings”. People
need to build things somewhere quickly, do development and testing. And
only after that, they should push commit(s) to Fedora.

Separate Koji + Koschei deployed in Fedora infrastructure cloud;
Builders are optimized for the best performance (see below
People can have their own targets where they can break things as they
wish without affecting others;
Package changes are eventually contributed to Fedora proper by merging
changes to Fedora dist-git and rebuilding packages in official Fedora Koji;
All improvements can eventually be contributed back to official Fedora


All ideas which you’ll find below are just an ideas and do not have to be
Idea №1: Automated legal review

openSUSE released their review app called Cavil
<> which we could try running in automated
Idea №2: Automated package review

Currently review process is burden and we could run automated legal review,
create a repo, run set of checks and notify person. Somewhat similar to
fresque <>. In future might be
integrated with approval process and auto-imported into Fedora.
Idea №3: Building packages for multiple distributions

In Rust ecosystem, we managed to get have same packaging guidelines in
openSUSE, Mageia and Fedora. We could automatically build some packages on
multiple distributions and be kinda upstream.
Idea №4: Building custom images with user content

Deploy (or build) a tool(s) to enable user-built install, appliance and
container images with their content (modulo restrictions similar to COPR)
on top of Fedora.
Implementation details

Koji and Koschei deployed in
A few builders constantly running, with a possibility to add more
builders as needed and as available cloud resources allow
Deployed and maintained by proposal owners - not supported by Fedora
Other contributors can have access granted by proposal owners (for the
time being only users in “packagers” group will be eligible to get access)
Possibility to have some builders running outsides of Fedora
infrastructure - contributed by proposal owners
Koji builds can be done from SCM (src.fp.o,, github, …) or
from SRPM upload

FAQWhy not COPR?

COPR has been starved of resources for years, which has impaired its
ability to provide reliable service at scale. Fedora Infrastructure refuses
to consider supporting it officially and Fedora Release Engineering seems
to have some undefined issues with COPR.
It is not official build system of Fedora which is not helping with Problem
№2: Testing of new rpm/koji/mock features/configuration
COPR is not extensible enough, more specifically:
No query API (e.g. it is not possible to find out from which SCM
commit the package has been built)
Builds always have access to all previous builds in a repository
(i.e. not possible to control when/how repo is created)
GC doesn’t exist
No scratch builds

Why not staging Koji?

Staging Koji is meant for testing Koji itself and not for package
Staging Koji is often in broken state where it is not possible to build
No one can touch staging Koji, so it’s pointless for offering a useful

How can you make Koji builds faster?

By tuning Koji for performance, not correctness and reliability.
By building only on a single, fast architecture.
By extensive caching. Files like RPM packages, repodata and files in
lookaside cache don’t change after they are initially stored, so builders
can cache them aggressively.
By keeping build repositories small. Some package sets don’t need to be
built against full Fedora, but can use a minimal subset of Fedora. Such
repositories can be generated by Koji in seconds, not minutes.

Why not the Open Build Service (OBS)?

OBS is not yet packaged for Fedora officially. The upstream code lacks
adaptations necessary to deploy and run on Fedora and in Fedora
OBS is not official build system of Fedora, which does not help with Problem
№2: Testing of new rpm/koji/mock features/configuration

What does MBI stand for?

M for middlestream, module, mizdebsk, …
B for build, bootstrap, …
I for infrastructure, initiative, ignatenkobrain, …
MBI might be pronounced as “maybe I …(can make something cool)?”
MBI is also initials of mizdebsk (Mikolaj Bozydar Izdebski)
MBI is also IBM spelled backwards :)

Is it somehow related to Fedora Playground

Yes, it is. Although it is more developer-focused. Users could enrol for
some specific things like experimental Java/Rust packages.

MBI (Playground 2.0)


Re: MBI (Playground 2.0)

By Dan =?utf-8?B?x... at 02/14/2019 - 07:51

This idea sounds pretty good to me.

As a less active contributor: How can I help out? Do you have a task
list? (Sorry if this got already answered in one of the follow up



Igor Gnatenko < ... at fedoraproject dot org> writes:

Re: MBI (Playground 2.0)

By =?ISO-8859-2?Q?... at 02/01/2019 - 10:58

Dne 31. 01. 19 v 13:24 Igor Gnatenko napsal(a):
That is because Copr has been always more popular than we had had resources. We are adding more resources over time, but
we are a bit behind Copr's popularity.
If you want to put your service into Fedora Cloud than you are talking about the same (shared) resource as Copr has.

1) First we are working on making it officially supported.
2) When I released new version of Mock, then Copr is usually the first build system to use. And we always use the latest
version. Compare that to Koji, which until recently used very old version of Mock. This changed few weeks ago.
But I generally try to dogfood Mock in Copr first so I hit potential issues before it can hit Koji, which is more important.
3) I do not think that allowing completely custom Mock configs in any build system is wise solution.

Every time, we got a reasonable request backed by good user-story we implemented it in reasonable time frame.
Feel free to submit an issue to our Pagure.

There is a garbage collection. Albeit it remove just the build artifacts, not the DB entries, which I seen as advantage
in past. Most people does not share the same opinion, so we are working right now on option to remove the DB entries as
well so it does not poulute WebUI.
At the same time we are finishing with GC of outdated chroots, which should allow us to reclaim huge amount of storage.
Expect an announce about it pretty soon.

Yes. Because everything in Copr is basicaly a scratch build. You can delete the build. Or you can crate new project
which will use the first project. And after a build you can delete the new project. It is easy and very cheap.


Re: MBI (Playground 2.0)

By Stephen John Smoogen at 01/31/2019 - 19:34

On Thu, 31 Jan 2019 at 07:28, Igor Gnatenko <

My main problem with this is the above. Yes Joe Desktop User isn't going to
see/need that package.. but we have a LOT of packagers who take our stuff
and rebuild it for themselves for various reasons. I find it hard to put
together the proposal which is supposed to make developing/packaging easier
with making developing/packaging harder. Whether you want it to or not,
this comes across as "If you want to use Go or Rust, you will need our
special set of tools which we keep hidden."

Re: MBI (Playground 2.0)

By King InuYasha at 01/31/2019 - 20:52

On Thu, Jan 31, 2019 at 6:58 PM Stephen John Smoogen < ... at gmail dot com> wrote:
This is actually something I really don't like either. But the Fedora
leadership has pushed very hard on the concept of having packages that
aren't available to "normies", and require special tooling to be able
to leverage (that for various reasons, I can't even use as a third
party packager!). That's a big part of the core to how Modularity is
being used. The CRB in RHEL 8 is another example of this. But fixing
this is very hard when few others see it as a problem. There was even
talk of not publishing SRPMs of packages that make up modules a while
ago. That idea died very quickly thankfully, but I don't know how
people think of these things anymore.

I'm tired of fighting... Our tools haven't improved enough to handle
our new challenges in the Fedora way, because it's hard for people in
the community to experiment and explore in this manner. My hope is
that with something like MBI, we can finally put the power to
experiment and develop tooling that actually makes sense for Fedora
(rather than clearly being designed for RHEL and shoehorned into
Fedora) without all the pain and agony of people imposing RHELish
requirements. There's no way we can evolve and meet the needs of our
communities without some way for people to "play".

COPR was supposed to be that outlet, but no one gives a damn about it.
Everyone complains that the service is "bad" and that the design is
"bad" but no one wants to actually constructively improve it. The
quality of service on COPR has fallen due to lack of care and
unwillingness to invest, so what are we supposed to do? The horrible
thing is that COPR is successful *despite* that. COPR is an
interesting and cool service, and I would have loved to see an
eventual merger of the Koji and COPR systems (because there are
awesome attributes to both and we should have a large and strong team
actually supporting the literal core of the distribution), but it
won't happen because everyone thinks the COPR system is terrible even
though it's not really.

I've been arguing for literally years about our shortcomings. I've
built tools for working around pitfalls in the Fedora workflow for my
personal use. I've personally explored how other people and other
distributions do things. I got my hands dirty to learn about other
ways of doing things. I've used that knowledge and expertise to avoid
those mistakes when deploying package and image construction
infrastructure for $DAYJOB. But it deeply saddens me that I have not
been able to make any meaningful progress in convincing anyone that it
was something that we should invest in.

I still believe that Modularity, as designed, is a mistake. But if I
don't have a way to prove a better way to do things, there's no way
anything will improve. The folks running the show like what they have
now, so it'll take a lot to trigger change.

Re: MBI (Playground 2.0)

By Josh Boyer at 02/01/2019 - 08:17

On Thu, Jan 31, 2019 at 8:19 PM Neal Gompa < ... at gmail dot com> wrote:
I'd like to understand why you see RHEL 8 Code Ready Linux Builder as
an example of what you're talking about, but I think that's for a
different email.

I don't understand what is preventing anyone from building such tools
or communities. Why can't you do what you want?

You've used the word "invest" twice in your email, so perhaps the
answer to my above question is "because I/we don't have enough funds
and/or time to make it happen"? If that's the case, and I'm thinking
it likely is, then I have no good answer for you. People choose to
invest their money and time in the things they are interested in or
see long-term value out of. Corporations (which are not people)
choose to invest resources (machine, monetary, human) in things they
see as providing value to their long-term business interests.

It can be frustrating to not get that critical mass needed to move an
idea forward. I'm in a similar boat with many (minor) things in
Fedora right now. I can empathize with you on that front.

Here is my issue with many of the ideas/proposals I've seen lately in
Fedora. Someone describes a problem (good!), they come up with a plan
(ok!), that plan immediately requires a significant investment in
resources and time of *other people*. There is no evolution from
existing tools, there is no small prototype they've done on the side
to prove things out or tackle some of the issues. There's no evidence
the idea is even going to pay off at all. It's an immediate "here's
my idea!" with a hand out to get started.

Fedora is a project, not a venture capital firm. Things get done
because people do them, prove they work, get buy-in, and evolve over
time. If that means ideas or projects start small and outside
existing Fedora infrastructure, that's OK!

For the record, I think Modularity started much the same way.
"Everything will be a module!" without proving any of it out in a
small scale hurt the adoption and messaging around why it's useful.
When it was reset and rescoped, it started making more progress and
continues to gain steam. I am not blind to the fact that Modularity
did not suffer from lack of investment, but it should serve to
illustrate that even if you have a corporate backer your plan or idea
still might not succeed at first and will need to evolve over time.


Re: MBI (Playground 2.0)

By King InuYasha at 02/02/2019 - 12:21

On Sat, Feb 2, 2019 at 10:37 AM Josh Boyer < ... at fedoraproject dot org> wrote:
The problem is that it's a symptom of a greater issue, which is that
RHEL is no longer self-consistent as a distribution. Wiring up RHEL 8
into my build process is functionally impossible without
rebootstrapping and building all the module components as "normal"
packages. But there are a class of packages that simply don't exist in
any form: packages that are used exclusively build-time. We started
seeing this problem come about with RHEL 7 over time as CentOS 7
rebuilt new point releases. RHEL 8 takes that to a whole new level.
And the CRB wasn't even published as part of the RHEL 8 beta public
repos, which made it impossible to build anything using it.

Fedora is starting to do the same thing. One of the major points of
contention I've had with Igor in the Rust SIG was the choice to stop
publishing Rust crate packages that were used to build applications
themselves when we switched to modules. It's essentially vendoring all
in but name. But as we've talked about it, I realized that we
basically have no choice, because the people in control of the tooling
have not been receptive to caring about our problems and working with
us to improve it.

But it means that downstream people cannot easily take our stuff,
modify it, rebuild it, etc.

That's a huge loss from my point of view.

Right now, it doesn't matter what I want to do. I don't have the
resources to host infrastructure and I don't have the buy-in from
people to consider new ideas that *I want to implement*.

While money is helpful, the real problem is that people don't see it
as a priority to work with the community to resolve these issues. I'm
happy to help with doing the work, but I can't do it alone. This is
one of those things I *literally* cannot do alone because I don't have
the power, resources, or time to do alone.

But as an example of something for COPR that I'd been working on is an
autorebuilder service that is dependency-aware[1]. Basically, when the
repositories configured for your COPR project update, it would trigger
rebuilds of your packages in the correct order. This is very similar
to the build scheduler present in the Open Build Service. It wouldn't
require anything more than you to configure the rebuilder service on
which projects and packages to watch, and an API key for COPR.

Ideally, such a thing would be integrated into COPR itself and done
with a bit more finesse, but I wanted to build this out to support my
own use-case: rebuilding kmod packages automatically when new kernels
became available. But the idea would be trivial to extend to anything.

I hadn't gotten the initial code working yet, so there's only a
skeleton pushed, but it's something I have been hacking on

[1]: <a href="" title=""></a>

That has *never* been the way I roll with things. It doesn't take much
to see that I'm fairly involved in a wide variety of projects.
However, an important litmus for me is that the people in control of
the project have to be receptive and willing to work with me. I do not
have the mental bandwidth or time to sometimes carry out an idea to
its full conclusion. I can build small-scale things to prove out the
concept, and maybe help with scaling the idea up.

But this is not my job, this is my passion I work on in nights and
weekends. If I do it alone, it'll take a long time to get there. I
*know* I need help, and I'm happy to accept it. Perversely, it also
means if there's a fundamental disagreement that can't be worked past,
I probably am more likely to quit rather than fork the project and
keep going with it. I just simply don't have the ability to drive
something alone.

The problem with Modularity today is that it's still working from
several biases that will hurt the overall Fedora ecosystem. It assumes
people don't want to do local builds of packages reliably. It assumes
that everyone has a Koji with an MBS scheduler. It assumes that Mock
is used build-side and DNF is used for runtime. It assumes you never
use PackageKit. It does not even account for ecosystem assumptions
about repository metadata, which is extremely painful for me

Many of the improvements in MBS should not be module-exclusive. They
should be part of Koji and usable by regular packages too. But there's
no opportunity because the Koji team is incredibly unresponsive. To
the point that even basic, trivial patches go without response for
months. And doing PR reviews for them doesn't seem to help either.

I'm incredibly frustrated and dismayed because it's Fedora's (and
maybe even Red Hat's) own fault that our tooling is in such poor
shape. To me, it seems like we don't actually want to be better.

I haven't given up yet because I care very deeply about Fedora as a
whole. The Project, the distribution, and the people. I want us to be
the best we can be.

I just don't know what to do anymore...

Re: MBI (Playground 2.0)

By Nicolas Mailhot at 02/01/2019 - 09:04

Le 2019-02-01 13:17, Josh Boyer a écrit :

I think there is a misunderstanding of what the "Friends" Fedora value
actually means. A lot of @RH contributors are developers at core and
only see Friends in a dev context: sharing source code with others, and
to hell with distro packages, and Fedora as a binary package
distribution in general.

But, there are already lots of instances where you can collaborate on
code (the Apache Foundation, etc). Fedora isn't one of those no matter
how devs push for it to become one (sometimes our desktop friends seem
to see Fedora as some form of GNOME Foundation sugar daddy nothing

What "Friends" mean in a Fedora context, is being to collaborate on
binary packages (rpms), and a best of breed integrated system, no matter
what your team size is, no matter if you do it as a hobbyist or as part
of some job, no matter what level of the software stack you're
interested in.

Whenever Fedora chooses to deploy infra solutions that are unmanageable
except @RH, it's hurting its core Friends value. Whenever Fedora
separates its offerings in special Editions (basically declaring "we
don't care about those other use cases") it is hurting its Friends
value. Whenever Fedora invests in deployment solutions, which can only
be applied on the desktop, only applied on openshift, or whatever, it is
hurting its Friends value. Whenever the Fedora main sponsor, chooses to
segment EL in unmanageable optional channels and modules that require
some friendly commercial help to be navigated, it's hurting its
"Friends" value (and it *is* hurting Fedora when EPEL rules are chosen
to accommodate unfriendly @RH policy).

And, why am I writing all this? Because the old proprietary behemoths
like Oracle, that justified investing in the Fedora/Centos/RHEL
ecosystem, are slowly fading away.

That is going to make Debian a direct Fedora competitor to attract
contributions and getting things done, when before anyone saddled with
Oracle or SAP or whatever wouldn't even look at Debian. And, because
Debian is a competent distribution, and because Debian has the same
software reach (or more) than Fedora, a huge part of the choice is going
to be made on the Friends axis.

So yes Fedora is not a venture capital firm. That does not mean it is
not competing. It's just competing for contributions, not money.

Thus stating clearly how initiatives like MBI, contribute to "Friends"
excellence, is not some form of weird request, it's a survival
requirement for Fedora as a project. That is, as long as Friends is
supposed to be a core value and a core differentiator of the Fedora

Re: MBI (Playground 2.0)

By Josh Boyer at 02/01/2019 - 10:31

On Fri, Feb 1, 2019 at 9:00 AM Nicolas Mailhot
<nicolas. ... at laposte dot net> wrote:
I agree. Fedora is not a typical developer project. It is a
distribution project.

Maybe so. I'm not convinced that's really what "Friends" means, but
the sentiment around collaboration isn't really wrong.

I disagree that is what Editions is doing.

I can't comment on EPEL. I'm only a user, not a contributor. It
works well for my uses though.

Really? Can you illustrate that with data? I don't mean to be
argumentative, but there is a very old, large, and blue behemoth that
just decided to plunk down a $34B investment on all of those things.

"going to"? If you believe that Fedora and Debian are not *already*
in competition for contributions with other distros or projects then
I'm somewhat confused.

OK? I'm missing your point on this one.

I didn't say it was a weird request. I didn't even say it was a bad
idea! I said it's unproven and requires *further* investment on the
part of the project as a whole, which seems premature.

I can appreciate your perspective and focus on Friends to argue your
position, but it can easily be turned around as well. I would rather
see things mature in Fedora on technical merit and not because of
pitch around a social construct that frankly doesn't exist. The
Fedora project is friendly and welcoming and respectful. That doesn't
mean we're all friends nor does it mean that everyone's ideas get
equal investment.


Re: MBI (Playground 2.0)

By Nicolas Mailhot at 02/01/2019 - 03:45

Le jeudi 31 janvier 2019 à 19:52 -0500, Neal Gompa a écrit :
If MBI is about hiding build packages I don't see the point and I'm not

What I *am* interested in is being able to import in one go the
hundred(s) of srpms involved in building the complex highly modularized
stuff package projects output nowadays, without the two months of review
getting each spec audited separately would entail.

Our main infra and main administrative processes just don't scale.

Re: MBI (Playground 2.0)

By Stephen John Smoogen at 02/01/2019 - 11:34

On Fri, 1 Feb 2019 at 03:26, Nicolas Mailhot <nicolas. ... at laposte dot net>

Re: MBI (Playground 2.0)

By King InuYasha at 02/01/2019 - 08:12

On Fri, Feb 1, 2019 at 6:59 AM Nicolas Mailhot
<nicolas. ... at laposte dot net> wrote:
MBI isn't about that. That's what modularity is (partly) about.

Indeed. Those were ideas that we put into the proposal of things we
want to improve.

Re: MBI (Playground 2.0)

By Mikolaj Izdebski at 02/01/2019 - 08:10

On Fri, Feb 1, 2019 at 8:46 AM Nicolas Mailhot
<nicolas. ... at laposte dot net> wrote:
It's the opposite - it's about opening build process to others and
allowing for collaboration. Instead of doing development by yourself
in private, you use a public, shared environment. But if you are not
interested then no one is forcing you to use it, you can keep
developing packages your current way.

Re: MBI (Playground 2.0)

By Kevin Fenzi at 01/31/2019 - 21:27


I've heard you and some others say this, but can you perhaps expand on
it? How has quality of service of copr fallen?

There are times when a bunch of builds are dumped on it that it takes a
bit to catch up, but it's usually like an hour not days or anything.
Is it the build time you refer to? Or something(s) else?


Re: MBI (Playground 2.0)

By King InuYasha at 02/04/2019 - 20:03

On Mon, Feb 4, 2019 at 5:14 AM Kevin Fenzi < ... at scrye dot com> wrote:
Aside from the times when it falls over for various reasons, I've had
entire days where I wait for a build to even start, because people who
use it for doing things like building KDE, chromium, or the Linux
kernel occupy literally all the available builder slots for a long
time. There aren't that many slots and it's easy to fill that up.
There's usually a large queue of packages to build, but not enough
builders to allow them to get done.

That indicates two things:
1. The builders are weak and so builds take a long time (which means
slots are held up longer)
2. The demand and popularity of the service isn't being handled
appropriately (i.e. it should get more builders provisioned).

I don't do things like build kernels often, but when I do, it usually
doesn't take all day. But stuff like Chromium is hard to build
locally, so I appreciate that we have somewhere to build and publish.

But, as of right now, there are 16 tasks running, with 85 tasks
waiting for a builder.

I wish we had visualizations like the ones OBS has[1][2], so that we
have an idea of how stuff is occupied and know at a glance that we're
over capacity.

All I know right now is that it's easy to see that COPR gets into a
state where I just wind up waiting for builds to even start.

[1]: <a href="" title=""></a>
[2]: <a href="" title=""></a>

Re: MBI (Playground 2.0)

By Kevin Fenzi at 02/05/2019 - 00:06

Well, thats not supposed to be the case. Each user is given a limited
amount of slots, so if say I dumped 100 kernels on it, it would only do
a few of those at a time. So, sounds like a bug in allowing people more
slots than they should have?
Could be indeed. Could look at increasing the size...

Well, there's another issue here that seems to be a copr bug (see below)

Yeah, but copr is allocated the following:

150 instances
200 vcps
488 GB ram

So, either copr is loosing track of it's builders or openstack is doing
something odd and not really providing that. I know openstack has
accounting issues, but I do see about 65 or so builder instances, so I
think this is on the copr side. So, fixing this would go a long way to
helping out.

Additionally, sometimes jobs finish, but copr shows them as still
running. For example, the oldest job now:

<a href="" title=""></a>

which copr shows running for 11 hours, the logs show it's done (I don't
know how long ago as there's no timestamps).

Well, we do have:
<a href="" title=""></a>
but some of it doesn't match up with osbs, since copr builders are dynamic.


Re: MBI (Playground 2.0)

By Fabio Valentini at 02/07/2019 - 08:22

FWIW, it looks like COPR's connectivity / download issues when
installing the buildroot have gotten even worse over the past two
The last 13 builds for my elementary-nightly repository **all** failed
due to some download issue in root.log, and the situation was similar
yesterday morning.

At this point, COPR is becoming completely useless to catch upstream
issues, since builds fail for COPR related issues 99% of the time, and
only 1% are real issues.
I usually try to re-submit builds until they either succeed or fail
for real, but at this rate (and with the slow web interface), this
isn't really feasible anymore.


On Tue, Feb 5, 2019 at 6:07 AM Kevin Fenzi < ... at scrye dot com> wrote:

Re: MBI (Playground 2.0)

By Stephen John Smoogen at 02/07/2019 - 09:53

On Thu, 7 Feb 2019 at 07:34, Fabio Valentini < ... at gmail dot com> wrote:
I need links and data to go with for any troubleshooting. I don't know
where these reports you have are and without that I can't know which
download servers are causing a problem.

Re: MBI (Playground 2.0)

By Fabio Valentini at 02/07/2019 - 10:15

On Thu, Feb 7, 2019 at 3:01 PM Stephen John Smoogen < ... at gmail dot com> wrote:
Sorry, forgot to add an explicit link:
<a href="" title=""></a>

As you can see, *all* builds since about 14 hours ago failed with
either "Error: Failed to synchronize cache for repo 'fedora'" (or the
same message for the "updates" repo).


Re: MBI (Playground 2.0)

By Stephen John Smoogen at 02/07/2019 - 10:40

On Thu, 7 Feb 2019 at 09:21, Fabio Valentini < ... at gmail dot com> wrote:
The page was dynamic and gave me all good ones when I went there. I
think this failed one shows the problem:

<a href="" title=""></a>

2019-02-07T00:01:52Z DEBUG fedora: using metadata from Wed 24 Oct 2018
10:20:15 PM UTC.
2019-02-07T00:01:52Z DDEBUG repo: downloading from remote: updates,
_Handle: metalnk:
<a href=";arch=x86_64" title=";arch=x86_64">;arc...</a>,
mlist: None, urls [].
2019-02-07T00:02:29Z DEBUG Cannot download
Cannot prepare internal mirrorlist: Curl error (28): Timeout was
reached for <a href=";arch=x86_64" title=";arch=x86_64">;arc...</a>
[Operation too slow. Less than 1000 bytes/sec transferred the last 30
2019-02-07T00:02:29Z DDEBUG Cleaning up.
2019-02-07T00:02:29Z SUBDEBUG
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/dnf/", line 230, in _perform
return super(_Handle, self).perform(result)
File "/usr/lib64/python3.6/site-packages/librepo/", line
1522, in perform
_librepo.Handle.perform(self, result)
librepo.LibrepoException: (8, 'Cannot prepare internal mirrorlist:
Curl error (28): Timeout was reached for
<a href=";arch=x86_64" title=";arch=x86_64">;arc...</a>
[Operation too slow. Less than 1000 bytes/sec transferred the last 30
seconds]', 'An Curl handle error')

That says the proxy it was trying to get to was too slow.. I will see
if I can find which one it was at that timezone and see if I can

Re: MBI (Playground 2.0)

By Stephen John Smoogen at 02/07/2019 - 13:21

On Thu, 7 Feb 2019 at 09:40, Stephen John Smoogen < ... at gmail dot com> wrote:
OK the problem does not look like it is mirrorlist related but
something to do with the copr host/network at that time - - [07/Feb/2019:00:01:59 +0000] "GET
/metalink?repo=updates-released-f29&arch=x86_64 HTTP/1.1" 200 18343
"-" "dnf/2.7.5"

says it sent the data but it looks like something that the client was
looking for timed out. I will work with the copr team to diagnose

Re: MBI (Playground 2.0)

By Adam Williamson at 02/08/2019 - 15:30

On Thu, 2019-02-07 at 12:21 -0500, Stephen John Smoogen wrote:
FWIW, openQA tests have been failing relatively frequently with
timeouts while refreshing repos or 503s from mirrormanager for the last
few weeks. I'm pretty sure this didn't used to be anything like as

Re: MBI (Playground 2.0)

By Dridi Boukelmoune at 02/09/2019 - 04:28

Whether it's dnf upgrade of my system or mock builds, both problems
have been increasingly frustrating me by orders of magnitude.
Especially for mock build, I had to make sure to --offline a given
build after a successful repo refresh & rpm download because it became
ridiculously hard to get mock --buildsrpm and mock --rebuild to
succeed in a row!

And my internet access has been flaky for "the last few weeks" too so
I thought it was causing the failures, but then even on a good
connection it kept happening.


Re: MBI (Playground 2.0)

By Fabio Valentini at 02/10/2019 - 07:09

On Sat, Feb 9, 2019 at 9:29 AM Dridi Boukelmoune
<dridi. ... at gmail dot com> wrote:
That's exactly my experience, as well.

First I've put it off as "my internet connection being unreliable".
But, since almost everything works fine, but mock builds continue to fail
(mostly because of "Failed to synchronize cache for repo 'foo'", where
foo is either fedora, updates, or rawhide),
I now think this really is some infra problem.

Also, I think some of the problems that happen in COPR are caused by
its very burst-y behavior:

<a href="" title=""></a>

For example, today, some of my builds were pending for over 7 hours,
during which time almost no running build completed, but pending tasks
blew up from the average 0-10 to over 140, since no pending tasks
moved to running at all, during that time, from what I can tell.
Then, a lot of builds (about 35) were moved to the "running" state at
once, and some of them failed with connectivity issues, again - which
doesn't surprise me, when there are possibly hundreds of mock builds
triggered by COPR at about the same time, obviously overwhelming the
system serving the repositories.


Re: MBI (Playground 2.0)

By Kevin Fenzi at 02/11/2019 - 14:23

On 2/10/19 3:09 AM, Fabio Valentini wrote:
Well, that was caused by copr backend becoming unresponsive.

So, no jobs were processed until a copr maintainer was able to reboot it.

The repo issues seem to be our mirrorlist containers sometimes throwing
a 503 when they shouldn't. ;( This is made worse by dnf never retrying
them, so if it happens once thats it. We are investigating and trying to
fix whatever is causing this.


Re: MBI (Playground 2.0)

By Adam Williamson at 02/11/2019 - 14:26

On Mon, 2019-02-11 at 10:23 -0800, Kevin Fenzi wrote:
Note for the record this is causing spurious openQA test failures too,
so it'd be really nice if it could be fixed!

Re: MBI (Playground 2.0)

By Kevin Fenzi at 02/11/2019 - 14:41

On 2/11/19 10:26 AM, Adam Williamson wrote:

Re: MBI (Playground 2.0)

By Dridi Boukelmoune at 02/11/2019 - 14:48

FWIW: <a href="" title=""></a>

Re: MBI (Playground 2.0)

By Stephen John Smoogen at 02/08/2019 - 17:00

On Fri, 8 Feb 2019 at 14:31, Adam Williamson < ... at fedoraproject dot org> wrote:
I am checking if we are seeing an increase and when. There are
generally some every hour when mirrormanager changes out pkls but
sometimes it grows.

Re: MBI (Playground 2.0)

By Stephen John Smoogen at 02/10/2019 - 12:17

On Fri, 8 Feb 2019 at 16:00, Stephen John Smoogen < ... at gmail dot com> wrote:
OK looking at the general stats for the last 2 months, we were seeing
an average of 0.2% of all connections generating a 503 (Out of
22114356 connections on 2018/12/03 only 41263 were 503 and the rest
are 200, similar to all days). There doesn't seem to be any large
uptick in them that stands out but I am not ruling out specific
mirrors, times of requests or clients yet that could make these more

Re: MBI (Playground 2.0)

By Fabio Valentini at 02/11/2019 - 08:51

On Sun, Feb 10, 2019 at 5:18 PM Stephen John Smoogen < ... at gmail dot com> wrote:
Either I am unreasonably unlucky, or your statistics are wrong:
Looking at [0], 35 of the last 44 build tasks failed due to network
issues (not counting all sub-tasks), which is about 80%.

[0]: <a href="" title=""></a>


Re: MBI (Playground 2.0)

By Nicolas Mailhot at 02/11/2019 - 09:46

Le 2019-02-11 13:51, Fabio Valentini a écrit :

You're not unreasonably unlucky, and the stats are not wrong. You're
just not talking about the same things.

A repo sync that works will results in hundreds or more of requests just
to get the various packages and files. So it will easily account for
hundreds or more of successful requests in stats. OTOH a failing sync
will account for at most 4-5 retries before dnf gives up. So the raw
stat number is heavily skewed against counting actual copr build

Re: MBI (Playground 2.0)

By Kevin Fenzi at 02/11/2019 - 14:34

On 2/11/19 5:46 AM, Nicolas Mailhot wrote:
This is made worse by:

<a href="" title=""></a>

basically dnf gets a mirror, downloads packages from it, but if it gets
a 503, it just retries the same mirror and gives up, it doesn't go to
the next mirror. ;(


Re: MBI (Playground 2.0)

By Fabio Valentini at 02/07/2019 - 16:20

Thank you!


Re: MBI (Playground 2.0)

By Mikolaj Izdebski at 02/01/2019 - 08:21

On Fri, Feb 1, 2019 at 2:28 AM Kevin Fenzi < ... at scrye dot com> wrote:
There are some of current/recent major problems with COPR that did not
exist initially:

- builds failing due to failure to download packages from official
Fedora mirror
- builds failing due to problems with keygen (was not a problem before
keygen was introduced)
- builds failing to import to COPR distgit (was not a problem before
dist-git was introduced)
- outages caused by read-only filesystem after reboot - copr is
unusable until someone remounts it rw
- multi-day long outages caused by out of disk space
- multi-hour or even day- long build queues
- outdated SOP preventing non-COPR people like me from fixing COPR outages

Re: MBI (Playground 2.0)

By Stephen John Smoogen at 02/01/2019 - 11:40

This one is on us in infrastructure. The problem is mainly because dl was
never scoped or built to handle the load that COPR brings. It is meant to
mostly deal with yum updates of last resort and mirror traffic. When a
bunch of COPR builds happen it gets overloaded and will drop those. It does
need to be fixed for this... but the MBI or similar tools will run into
this also.

The others I can't really speak for (beyond the disk space where
Infrastructure provides an equallogics and it would need a budget increase
to get more space.)

Re: MBI (Playground 2.0)

By =?ISO-8859-2?Q?... at 02/01/2019 - 10:37

Dne 01. 02. 19 v 13:21 Mikolaj Izdebski napsal(a):
This is not first time I hear this. So I will open discussion to (again) alter builder's mock config to point to PHX
location, which is just rack away.

This is actually new to me. Can you point me to some report?

This was actually recently fixed.

That is because I actually care about the resources. It is very easy to submit a proposal and state that "it will run in
Fedora Cloud", but the cloud is not infinite. I can easily increase the soft limits in Cloud dashboard. But we are
either hitting hard limits (storage) or we would drastically limit other users using the same cloud (vCPU in case more
Copr is not designed for spikes, it is designed for the average traffic. Which is handling pretty well. You can check it
<a href="" title=""></a>
If you submit 300+ build today (like happened today) you will simply have to wait some time. At the same time, it should
not affect (too much) other users. This is IMO big difference from Koji which is IIRC strictly FIFO.

Feel free to raise any issue. Last update in this document was in October 2018. Checking it right now I see outdated
part about ppc builders, but otherwise it should be ok.
Thou, I will review that document after weekend.


Re: MBI (Playground 2.0)

By King InuYasha at 02/03/2019 - 16:45

On Sun, Feb 3, 2019 at 6:19 AM Miroslav Suchý < ... at redhat dot com> wrote:
Wouldn't it make sense to maintain a local mirror of the distributions
enabled in COPR and just have all the mock configs point to that? That
eliminates all the network pressure and the issues we keep seeing
there when trying to use it.

I can't say I've experienced this issue. However, I do wish we could
interact with COPR Dist-Git the same way we do with the official
Fedora Dist-Git, with a Pagure instance and everything...

Would it be possible to extend COPR to support multiple locations for
builders? For example, in addition to an OpenStack system, builders
could be hosted on an oVirt system, or AWS, or GCP, and so on? That
way it can support on-premise deployments and cloudy deployments too.

Perhaps supporting even some limited form of auto-scaling for when it
needs to "burst" to support more build traffic using clouds or
hypervisors? For example, you could use AWS spot instances to do it on
the cheap for the time a builder needs to run, and then have it go
away afterward. I've done builds this way with buildbot and you can do
thousands of builds for really cheap (on the order of hundreds of
dollars each year).

I don't know anything about this...

Re: MBI (Playground 2.0)

By Pavel Raiskup at 02/06/2019 - 03:55

On Sunday, February 3, 2019 9:45:06 PM CET Neal Gompa wrote:
I have implemented something along those lines in the Red Hat internal
Copr deployment. There's something like middleware between Copr and VM
providers which allocates the VMs a flexible way on multiple places.
This feature is tracked in <a href="" title=""></a> .

This would be really trivial to deploy for Fedora Copr - if there was use
for it (are there any resources which could rise throughput in Fedora
Copr?). That said, this is not much of technical problem to me, but
rather policy/funding problem nowadays.

This is really interesting idea...


Re: MBI (Playground 2.0)

By Stephen John Smoogen at 02/04/2019 - 06:33

On Sun, 3 Feb 2019 at 16:43, Neal Gompa < ... at gmail dot com> wrote:
So problem 1 is that is 1 rack away. There is
some sort of problem with the setup of the dl servers which is no
longer using memory efficiently. I think it needs a local varnish or
similar caching server but i haven't had time to analyze.

COPR doesn't have the disk space to mirror all that at speed. [Yes you
can cheaply stick them on slow large drives but trying to run the
several dozen builders would be worse than going across the rack to
the other network server. So you then need a lot of slow large drives
and then you are no longer cheap and it seems like you have a lot of
empty disk space that isn't being used so you might as well go with
smaller expensive SAS/SSD. ]

I expect it is possible but it is going to come down to a
credential/funding problem. Going out to the cloud like GCP/AWS
requires funding and setup to make sure that the systems aren't
compromised. [We have had several cloud offers from smaller vendors
but they then wanted our root password and other credentials.. just to
remind you that this isn't your hardware and they can look in it any
time you want.]

There would also be the speed issue of getting the data from the
on-premise to the cloud. That takes time and sometimes seems to take
longer than waiting in queue locally. Then there is the mirroring part
of the data. [That seems to be slower than molasses at times on the
order of a day. I expect it is partially that we are doing something
naive and would be better dealt with elsewhere.]

That would be useful.

Re: MBI (Playground 2.0)

By Mikolaj Izdebski at 02/01/2019 - 12:55

On Fri, Feb 1, 2019 at 3:37 PM Miroslav Suchý < ... at redhat dot com> wrote:
I will try to find a reference.

Already reported in RFR ticket:
<a href="" title=""></a>

Re: MBI (Playground 2.0)

By Nicolas Mailhot at 02/01/2019 - 09:08

Le 2019-02-01 13:21, Mikolaj Izdebski a écrit :
And builds failing is just an annoyance when you are building a single
leaf package. When your build execution plan covers hundreds of
interdependent packages, with bootstrapping phases, having something
failing in the middle is not just an annoyance, it often means
restarting the full plan from scratch, because trying to salvage the
result manually is just too much work


Re: MBI (Playground 2.0)

By Kevin Fenzi at 01/31/2019 - 18:14

Note that the rawhide gating plans include self service side tags, so
you can make one anytime you want to update a collection of packages.

This of course doesn't solve the above problem, but it's worth
mentioning I think.

This of course means however that users who want to build their own
verions of the applications or whatever can't start from our repos, they
have to use whatever setup we use for developing them. I don't know any
answers here but it seems to me we keep making it harder for people to
use Fedora to develop applications.


Turns out we are going to be retiring our cloud, so no, this is not a
place to put it. :)

Would that be entirely self service? Or ?

Where is the source for the changes from? Would this need it's own
pagure/src/dist-git? Or would it pull from PR's in prod dist git? Or ?

Sounds great!

I guess this would be up to legal to look at and tell us if it's worth

Sounds great if someone has time to build the app(s)

So, this setup will need to distribute things... would it need mirrors
as well? or ?

Yeah, this has been attempted about 3 times, but if someone has cycles
to work on a app/tool again, great!

Nope. There are options of course, but the cloud won't be one. ;)
We could do a remote cloud provider possibly, or we are considering
another openshift cluster with kubevirt or perhaps just some simple
virthost/ansible controlled thing. In any case I think if it's important
we can get resources.
koji doesn't have much way to dynamically deal with builders...

I'd be carefull of this, as it can cause a lot of issues, but of course
up to you as you are doing the work. :)

Ah, so it would not have it's own git/pagure?

Thats kind of glossing over a lot. The reason infrastructure doesn't
want to support it is that its running on a ancient version of openstack
thats very hard to fix when broken. The only issues I know otherwise is
that copr is just designed differently from koji. koji is set to record
everything and keep anything needed to rebuild things, but cope is much
more free form.

In any case with our cloud retirement there will be changes to copr.
Not determined yet, but hopefully it will have more resources and a
solid base to run on.

For the case of this proposal, I agree copr might not be a good fit, as
you are trying to fold all the changes back to koji, so best to just run
a koji to make it as close as possible.

Note that if you also are testing new things you could break packagers
that are trying to do other work...


This may well bite you when you merge back to koji and build for all


Overall sounds very nice... we will have to work out details if everyone
is on board with it. Do note that it's going to be a lot of work for you
all... :)


Re: MBI (Playground 2.0)

By Mikolaj Izdebski at 02/01/2019 - 08:06

On Thu, Jan 31, 2019 at 11:16 PM Kevin Fenzi < ... at scrye dot com> wrote:
All packages built in MBI are meant to be eventually contributed back
to Fedora. Just like developing a package on your own system and
building in local mock. Eventually you push the code to dist-git and
rebuild packages in Fedora Koji. With MBI the difference is that
people can collaboratively devlop changes and they have scalable build

Cloud doesn't necessarily mean OpenStack. There are other options. For
example, a baremetal, out-of-warranty virthost in an isolated network
would work - better that OpenStack.

MBI was ran this way (baremetal server with KVM guests) for some time.
But it was a private setup, usably only by a few people. Among other
things it was used for GCC removal from buildroot change - all
packages were mass-rebuilt several times and then build logs were
scanned to see which packages failed due to missing gcc/g++ and would
need to have BuildRequires added. The change was eventually pushed to

No, I would be setting up these tags/targets myself, manually.

Changes would be kept somewhere, depending on packager preferences.
Ideally in a publicly-available git repository where people can
collaborate. Ideally in a fork on to make sure
that contributors have FPCA signed. Giving myself as example, I have
forked all my packages and created "mbi" branches for them, for
example: <a href="" title=""></a>

Nothing is distributed to end users. This system is for use by Fedora
contributors - packagers/QA/etc. Of course anyone interested is able
to download these packages, but they are not meant for end users. Just
like Koji scratch builds are not meant for end users, but anyone can
download them and use in production, if they like.

Koji builders can be added and removed dynamically very easily. Adding
a builder in public cloud is as easy as spawning some VM, installing
koji-builder, putting config file in place and starting kojid - of
course automated using Ansible. Removing a builder only requires
terminating the VM, nothing more. Adding a builder that is installed
on your laptop is as easy as booting up the VM containing in, that's

The idea was to have just a few builders (eg. 3) running constantly.
If someone wants to do massive amounts of builds (and have them
complete quickly) then they can plug in their own hardware or
instances in public cloud that they would pay for, which would be
added to a separate Koji channel. Then their builds would use their
builders. Builders have very low requirements - in particular they
don't need public IP, can be behind firewall/NAT and only need to talk
to hub over https. Even someone's personal laptop could be used as a

No. Builds would be allowed from any configured SCM - it could be from
forks on, from repositories on,
repositories on other git hosting services. It's up to people using
the system to choose the workflow that suits them best. They could
even build from SRPM uploads.

Testing new features would be done in separate tags. So no, others
would not be affected.

I don't see this as a problem.Most of the packages that we are
planning to build in MBI (Java/Rust/Go) are noarch anyway.

Actually there's not that much work. Most of the work is already done.

I've developed and I'm running MBI setup myself in AWS, using my
personal account. Others liked this workflow and asked whether they
could use it. I kindly refused as I can't pay for all that myself.
Then we come up with a proposal to have this hosted by Fedora.

Re: MBI (Playground 2.0)

By Kevin Fenzi at 02/01/2019 - 11:47


Well, I was responsing to "Fedora Infrastructure cloud" which completely
does mean openstack. ;) But yes, as I mentioned we can come up with
other options if it's desired.


Sure, except they exist in the database by that name forever.

Sure, note there would be some setup here for you... adding channel,
keytabs or certs for builders, etc.

I'm not sure how you could test a new koji hub or rpm on the hub with
tags, but sure, you could test some things by having different builders.

ok. Unless some other folks start using it.


Sure, makes sense.

Thanks for the answers!


Re: MBI (Playground 2.0)

By Mikolaj Izdebski at 02/01/2019 - 13:06

On Fri, Feb 1, 2019 at 4:48 PM Kevin Fenzi < ... at scrye dot com> wrote:
Testing new rpm features, global macros etc. can be done by tagging a
build (of rpm, redhat-rpm-config etc.) into one tag without affecting
other tags.
Testing new kojid/mock configuration can be done on builders in
non-default channels.

For testing new Koji releases we have staging setup.

Then we can reconsider more architectures later. Or they can
contribute their builders of any architectures, even not supported by
Fedora Koji.

Re: MBI (Playground 2.0)

By Stephen John Smoogen at 01/31/2019 - 14:44

On Thu, 31 Jan 2019 at 07:28, Igor Gnatenko <

Re: MBI (Playground 2.0)

By Ben Cotton at 01/31/2019 - 13:38


This is great. It seems like it would fit in really well with the Packager
Experience objective proposal[1] that Ben Rosser was working on. I know you
weighed in on that thread at one point. Is this a part of that proposed
objective or is it separate?

<a href=" ... at lists dot" title=" ... at lists dot"> ... at lists dot fedoraproject....</a>

Re: MBI (Playground 2.0)

By Josh Boyer at 01/31/2019 - 08:35

On Thu, Jan 31, 2019 at 7:28 AM Igor Gnatenko
< ... at fedoraproject dot org> wrote:
Why does this need to be deployed in the fedora infrastructure cloud?
Seems like you could stand it up in AWS or somewhere else.


Re: MBI (Playground 2.0)

By Mikolaj Izdebski at 01/31/2019 - 08:38

On Thu, Jan 31, 2019 at 1:36 PM Josh Boyer < ... at fedoraproject dot org> wrote:
Because we (Fedora contributors) don't have budget to pay AWS bills.
If someone is willing to sponsor this then AWS would work well.

Re: MBI (Playground 2.0)

By Brian (bex) Exe... at 01/31/2019 - 10:08

On Thu, Jan 31, 2019 at 1:40 PM Mikolaj Izdebski < ... at redhat dot com> wrote:
I encourage you to specify what you need and then the Project can
figure out if and how it wants to provide it. So if you need a
specific kind of access, note that. If you just need machines, note
that. If you need sysadmin work, note that, etc.