DevHeads.net

modular repositories in mock configs: please don't

Hi everybody,

Recently, modular repositories were enabled in the mock configs for fedora 29+.

Now, I can't build at least one of my packages (elementary-music) in
fedora 29 chroots, due to dependency issues within modules. dnf just
gives up with this rather unhelpful message:

Problem: cannot install the best candidate for the job
- package libpeas-devel-1.22.0-9.module_2123+73a9ef6f.x86_64 is excluded

I don't want or need modules installed for this package to build.

See: <a href="https://bodhi.fedoraproject.org/updates/FEDORA-2019-cca3e242eb#comment-901551" title="https://bodhi.fedoraproject.org/updates/FEDORA-2019-cca3e242eb#comment-901551">https://bodhi.fedoraproject.org/updates/FEDORA-2019-cca3e242eb#comment-9...</a>

IMO it was a mistake to enable modular repositories in mock configs by
default. Now dnf only downloads even more metadata for no benefit (or,
it even breaks dependency resolution, as in this case).

Do I really have to manually edit mock's config files to disable
modular repos, to get builds equivalent to koji (where modules aren't
available / usable either)? I want to test builds locally, before I
push them to koji builders ...

Any insights why this was done?

Can it be fixed please?

Or am I the only one having problems?

Fabio

Comments

Re: modular repositories in mock configs: please don't

By Petr Sabata at 03/04/2019 - 05:01

Replying in general.

While it's never been entirely the same, I'd also like our mock
configs to be as close to koji environment as possible. In the
current situation that would mean no modules being available.

Once/if we proceed with one of the
modules-in-the-non-modular-buildroot proposals, I would like them
to include the same module set that is available in the
non-modular buildroot in koji.

If you're building content that depends on modular packages at
this point, you should be building a module anyway. In that case
your local MBS manages the build and pulls the relevant packages
for you.

But I also kinda like the idea of having two configs -- one that
aims to somewhat replicate the koji experience and one that's
close to installations. Not sure if there's really any value in
it, though.

P

Re: modular repositories in mock configs: please don't

By Pavel Raiskup at 03/04/2019 - 05:22

On Monday, March 4, 2019 10:01:14 AM CET Petr Šabata wrote:
I don't understand your point of view. If it is safe to enable modular
repositories on user boxes by default (that's what we have now), why we
can not enable them in koji and why we can not enable them in mock?

Can you elaborate? How the 'non-modular buildroot in koji' differs from
modules-in-the-non-modular-buildroot?

Please elaborate on "why" on this, too.

Ok, but consider that

$ mock -r fedora-rawhide-x86_64 \
--config-opts module_enable=postgresql:10
--rebuild my.srpm

is much more convenient and economical than approaching the whole MBS
"thingy".

Pavel

Re: modular repositories in mock configs: please don't

By =?UTF-8?B?TWlyb... at 02/28/2019 - 19:22

On 01. 03. 19 0:05, Fabio Valentini wrote:
No you are not. Rawhide mock is broken for the very same reason.

Mock should IMHO bring the exact same (or at least the most similar) results as
building in koji. I don't want to get different packages in mock and Koji just
because the configurations are different.

Let's make the defaults the sme as Koji (currently, that means no modular repos).

Re: modular repositories in mock configs: please don't

By =?ISO-8859-2?Q?... at 03/01/2019 - 03:59

Dne 01. 03. 19 v 0:22 Miro Hrončok napsal(a):
It may be true if your specific case. But generally this is not true. AFAIK Some packages are not available in normal
repo any more and are available only in modules. E.g., stratis* packages.
The general expectaion is that more and more packages will move to modules.

Mock is in fact just easy tool to run 'rpmbuild' in minimal chroot of Fedora/CentOS. So running

mock -r fedora-29-x86_64 foo.src

should give you the same results as running `rpmbuild --rebuild foo.src` on normal installation of fedora-29 with only
minimal installation.
And /etc/yum.repos.d/fedora-modular.repo *is* part of Fedora 29+. Therefore it is in mock configs. If you do not need it
appeal either:
* maintainer of fedora-repos package
* modularity team
* FESCO representatives

Nope, it is the other way round. Koji use Mock and therefore Koji builds should be the same as your local builds with
local mock.
Only Koji admins do not jump on every released version and should (and I hope they do) test every new released version
if it does not break Koji builds.

Miroslav

Re: modular repositories in mock configs: please don't

By =?UTF-8?B?TWlyb... at 03/01/2019 - 05:24

On 01. 03. 19 8:59, Miroslav Suchý wrote:
Done <a href="https://pagure.io/fesco/issue/2003#comment-557358" title="https://pagure.io/fesco/issue/2003#comment-557358">https://pagure.io/fesco/issue/2003#comment-557358</a>

Re: modular repositories in mock configs: please don't

By Mikolaj Izdebski at 03/01/2019 - 04:56

On Fri, Mar 1, 2019 at 9:00 AM Miroslav Suchý < ... at redhat dot com> wrote:
Koji doesn't use upstream mock configs. For each task it generates a
new config dynamically. These generated configs don't include any
modular repos.

Re: modular repositories in mock configs: please don't

By Pavel Raiskup at 03/01/2019 - 07:02

On Friday, March 1, 2019 9:56:48 AM CET Mikolaj Izdebski wrote:
Looks like a serious bug in Koji -> if modular repos are enabled by
default in Fedora, Koji, Copr (and mock) should it do as well, no?

Or the other way around - if enabling modules causes _any_ problems
anywhere, it should be disabled by default (and let people opt-in, in
mock, copr, koji, and plain fedora).

The status quo - inconsistency - is really bad for Fedora.

Pavel

Re: modular repositories in mock configs: please don't

By Richard Shaw at 03/01/2019 - 10:25

Not replying to anyone in particular but just a nit for me...

I've been a Fedora packager for probably 10 years now (need to check!) but
I *STILL* don't really understand modules other than at a high level (it
lets you use dependencies that aren't available in the main repos). I've
read through some of the documentation but it's still not clear. I know I
would have to crate some kind of module file which I think tools could be
developed to mostly automate (spec->module template converter?).

While modularity is more of a Fedora thing I think it's sad that for more
standard tools I reference Arch documentation more often than Fedora
documentation. Of course there's the, "I could help with that" part and I
might, but frankly I can barely keep up with my packages between $DAYJOB,
$FAMILY, Fedora, and RPM Fusion.

Thanks,
Richard

Re: modular repositories in mock configs: please don't

By Michael Cronenworth at 03/04/2019 - 17:23

On 3/1/19 8:25 AM, Richard Shaw wrote:
+1. I stumbled upon this email among the hundreds I get hourly. My sentiment mirrors
yours bit for bit.

Re: modular repositories in mock configs: please don't

By Ken Dreyer at 03/04/2019 - 12:34

I'm there with you Richard. I don't really get how I can get started
building a module outside of the Fedora infrastructure's system (Koji or
Copr).

I'm hoping this gets a lot clearer after modules come in RHEL 8 and we get
concrete examples.

- Ken

Re: modular repositories in mock configs: please don't

By =?ISO-8859-2?Q?... at 03/05/2019 - 14:52

Dne 04. 03. 19 v 17:34 Ken Dreyer napsal(a):
In fact, Copr support building modules for ages - even before the modularity has been finalized.

You just create project in Copr, build regular packages there. Then you click on "Modules" tab, then "Create new
module", select which packages should be part of module and submit the form. Few seconds later your module should be ready.

Copr does not support all features of modularity, because the format of modules changed every few week and it was hard
for us to keep pace. But you are simply build simple module in Copr.

Miroslav

Re: modular repositories in mock configs: please don't

By Ken Dreyer at 03/06/2019 - 08:52

What I meant was that I don't know how to do this outside of Koji or Copr.
My use cases:

* I want to experiment on my laptop,
* I want to build modules I cannot distribute through Copr,
* I have a large project with many changes every day, and if I sent all of
those into copr.fedoraproject.org via Jenkins, it could melt down

With Fedora regular RPMs, fedpkg has "mockbuild" to do local builds. How
can I do something similar on my laptop for a module?

Re: modular repositories in mock configs: please don't

By Stephen John Smoogen at 03/06/2019 - 10:11

On Wed, 6 Mar 2019 at 07:53, Ken Dreyer < ... at ktdreyer dot com> wrote:
I think you are looking for the same thing a lot of people are
wanting: A detailed "How the henry do I do this locally without using
your buildsystem?" howto. Followed by "how to I use this for my own
build system." and then "How do I do integrate this with some other
system?" I am saying this because I see a lot of the frustration and
venting seems to be that this information isn't easily found if it
exists. I think that until such stuff is written, then we can get to
informed frustrations of 'why did you do it this way?'

Re: modular repositories in mock configs: please don't

By Mikolaj Izdebski at 03/06/2019 - 09:00

On Wed, Mar 6, 2019 at 1:54 PM Ken Dreyer < ... at ktdreyer dot com> wrote:
Modules are just sets of RPMs with extra metadata. To build a module
it is enoug to:
- create a proper modulemd document
- build some (zero or more) RPM packages using rpmbuild
- create YUM repodata from built packages using createrepo_c
- attach modulemd to repodata using modifyrepo_c

Re: modular repositories in mock configs: please don't

By =?ISO-8859-2?Q?... at 03/07/2019 - 19:53

Dne 06. 03. 19 v 14:00 Mikolaj Izdebski napsal(a):
Yes. But the first and last steps needs to be automated. No ones wants to do that manually.

Miroslav

Re: modular repositories in mock configs: please don't

By Stephen Gallagher at 03/04/2019 - 17:07

On Mon, Mar 4, 2019 at 11:36 AM Ken Dreyer < ... at ktdreyer dot com> wrote:

This is a known gap. We're having a Modularity hackfest in Boston
right now and looking into how we're going to make this simpler and
easier. Stay tuned.

Re: modular repositories in mock configs: please don't

By Petr Sabata at 03/04/2019 - 05:04

On Fri, Mar 01, 2019 at 08:25:32AM -0600, Richard Shaw wrote:
You can view them as virtual repositories with dependencies. I
think that might be the simplest way to put it.

You can try playing with fedmod to generate your modulemd file or
you can just write it from scratch; it's not all that complicated.

P

Re: modular repositories in mock configs: please don't

By Michael Cronenworth at 03/04/2019 - 17:25

On 3/4/19 3:04 AM, Petr Šabata wrote:
Why should I care? Please, win me over. (I'm being serious.)

I view the current RPM system as reliable, standard, and well-documented. It's
served me and the people I know that use it well for decades. I view Modularity as
an answer to a question no one asked. It presents new problems that never existed
and has been somewhat silently taking over RPM-land with very little community
involvement.

Re: modular repositories in mock configs: please don't

By Adam Samalik at 03/05/2019 - 04:08

I think you should care if:

a) You need to maintain multiple versions of the same app/runtime/set of
tools tools in Fedora (or an alternative to something that already exists),
or
b) You only want to maintain one source branch per package for all Fedora
releases, or
c) You could benefit from having a build recipe in case you maintain
app/runtime/set of tools tools represented by multiple packages that need
to be built in the right order

But if any of the above is something that you need/want to do, then
Modularity won't help you nor hurt you in any way.

Re: modular repositories in mock configs: please don't

By Ralf Corsepius at 03/06/2019 - 00:29

On 3/5/19 9:08 AM, Adam Samalik wrote:
All of what list above can be handled at the rpm-level by means of
"proper" system integration.

That said, to me, modules are an approach to circumvent system integration.

Ralf

Re: modular repositories in mock configs: please don't

By stan at 03/05/2019 - 12:01

On Tue, 5 Mar 2019 09:08:19 +0100

I'm not a packager, I only follow devel to see what is coming down
the pipe as a user. I'm concerned about modularity. Like Michael, I
find rpm works fine for my use case, an individual workstation. I have
a couple of comments, but since I am not the target of modularity, take
them as curiosity.

It seems that modularity will create one more layer of indirection, so
security holes will be more prone to exist and be harder to find and
fix for the user, particularly on mixed rpm / modularity systems.

Modularity seems to be a workaround to avoid making the system fully
multi version. Admittedly, that is a lot of work, gcc has to be
modified to create a dictionary of libraries and api calls needed in
executables, libraries have to be modified to publish their api calls,
the kernel has to sort through multiple versions of a binary checking
priorities (somehow). rpm has to be modified to allow multiple version
installation, and to do library checking to be sure that an existing
library will cover needed calls. There has to be garbage collection for
libraries no longer needed by executables on the system. etc. The up
side is there is no more shared library so numbers for libraries, so
updates will *always* succeed, and the garbage collector will take
care of no longer needed packages. Is the following a good summarization
of the purpose of modularity? Modularity is a workaround using the
pareto principle to get 80% of the benefits of multi version with only
20% of the work.

Would it be practical to have a new hierarchy under /usr for
modularity, say /usr/mod, or move the current /usr to /usr/rpm and
let /usr be for modularity? This would make it easier to specify which
version of a binary to run, and which should be default in the path.
So, if I install a module with old, possibly insecure components,
because I need it for legacy purposes, I can specify it in a script, and
let the newer version be the default for general use.

Re: modular repositories in mock configs: please don't

By King InuYasha at 03/05/2019 - 12:18

On Tue, Mar 5, 2019 at 11:01 AM stan < ... at zoho dot com> wrote:
Modules are merely collections of rpms built in a special way.
Sometimes that's with macro definitions to force overrides for
specific configurations, and sometimes that's with filtering packages,
or customizing build and runtime content.

RPM already allows multiversion installation with the caveat that
there are no file collisions. The other aspects are things that can be
handled by DNF. Modules are only about some of those use cases. The
other major cases involve build-time customization and partitioning
built packages into supportable or non-supportable groups.

No. It should be possible for modular content to become non-modular
and vice versa.

Re: modular repositories in mock configs: please don't

By stan at 03/05/2019 - 15:19

On Tue, 5 Mar 2019 11:18:04 -0500

Thanks for the response and the information.

Re: modular repositories in mock configs: please don't

By =?ISO-8859-1?Q?... at 03/05/2019 - 04:36

Dne 05. 03. 19 v 9:08 Adam Samalik napsal(a):

I am afraid you are missing one point. You need to care if somebody else
believes the points you list below are relevant. This is the issue.

Vít

Re: modular repositories in mock configs: please don't

By Dan =?ISO-8859-... at 03/01/2019 - 04:48

On Fri, 1 Mar 2019 08:59:35 +0100

koji provides own mock configs for the builds pointing to the internal
repos, it doesn't use the configs distributed with mock at all

Dan

Re: modular repositories in mock configs: please don't

By Adam Samalik at 03/01/2019 - 04:58

I'm glad Modularity is getting popular, however, we should coordinate such
big changes so we keep consistency among various build environments.

The ability to enable modules in a Koji buildroot is being discussed in a
FESCo ticket [1] — although that discussion is a bit longer than initially
anticipated. But when that's done, we'll be finally able to move content
from traditional packages to default modules when desired. And that'll be
the best time to enable the modular repo in Mock so it behaves consistently
with Koji.

[1] <a href="https://pagure.io/fesco/issue/2003" title="https://pagure.io/fesco/issue/2003">https://pagure.io/fesco/issue/2003</a>

Re: modular repositories in mock configs: please don't

By Fabio Valentini at 03/01/2019 - 05:28

On Fri, Mar 1, 2019 at 9:58 AM Adam Samalik < ... at redhat dot com> wrote:
I'd call it "notorious" or "infamous" instead of "popular", but that's
just a difference perspective ;)

I agree, this change to mock-core-configs seems to have been rushed
and uncoordinated, since it's not possible to use modules in koji at
all.

Can we find some middle ground here? Maybe two different configs (e.g.
fedora-29-x86_64 and fedora-29-modular-x86_64), where the first one
preserves the expected behaviour, and the new one enables modular
repositories for the people who want to test this out?

Either way, I'm just strongly opposed to enable modular repos in mock
by default (yet).
(Maybe I should have added a (yet) to the subject to this e-mail, as well.)

Fabio

Re: modular repositories in mock configs: please don't

By =?ISO-8859-2?Q?... at 03/01/2019 - 06:38

Dne 01. 03. 19 v 10:28 Fabio Valentini napsal(a):
*nod*
With the all issues it brought, it is pragmatic move to remove the modular repo. In fact, I will just set enabled=0.

Done:
<a href="https://github.com/rpm-software-management/mock/commit/65d87447c44e0a4ae628306b701b9d805fed0b24" title="https://github.com/rpm-software-management/mock/commit/65d87447c44e0a4ae628306b701b9d805fed0b24">https://github.com/rpm-software-management/mock/commit/65d87447c44e0a4ae...</a>
Expect Bodhi update in a few hours.

However, everyone should be aware that Mock can deliver different results than pure `rpmbuild` on the same platform.

And I strongly encourage the Modularity team to resolve this ASAP.

Miroslav

Re: modular repositories in mock configs: please don't

By Fabio Valentini at 03/01/2019 - 06:43

On Fri, Mar 1, 2019 at 11:39 AM Miroslav Suchý < ... at redhat dot com> wrote:
Thank you, Miroslav! I think this is a good solution for now.
With these changes, people who *explicitly want* modular repos enabled
can then just invoke mock with "--enablerepo=fedora-modular" as
needed, right?

Fabio

Re: modular repositories in mock configs: please don't

By Mikolaj Izdebski at 03/01/2019 - 06:27

On Fri, Mar 1, 2019 at 10:29 AM Fabio Valentini < ... at gmail dot com> wrote:
It is possible to use modules in Koji. But Fedora Koji is not
configured to use modules.

Re: modular repositories in mock configs: please don't

By Fabio Valentini at 03/01/2019 - 06:30

On Fri, Mar 1, 2019 at 11:28 AM Mikolaj Izdebski < ... at redhat dot com> wrote:
Thanks, that's what I meant to say. Of course koji can produce modules.
It's just that "normal" builds can't consume modules (yet?).
Guess I pressed "Send" too quickly.

Fabio

Re: modular repositories in mock configs: please don't

By Mikolaj Izdebski at 03/01/2019 - 06:42

On Fri, Mar 1, 2019 at 11:31 AM Fabio Valentini < ... at gmail dot com> wrote:
"Normal" builds in Koji *can* already depend on modules. Modules can
be enabled in Koji buildroots using "module_enable" mock config
option. But Fedora Koji is not configured that way.

"Normal" builds in Koji can also depend on modular packages from
non-modular repositories. This can be configured by adding YUM
repositories without modlar metadata to Koji tags. Ursa-Major uses
this approach, but it can be configured without Ursa-Major too.

Re: modular repositories in mock configs: please don't

By =?UTF-8?B?TWlyb... at 03/01/2019 - 05:27

On 01. 03. 19 9:58, Adam Samalik wrote:
Agreed.

I'm quite concerned with this approach. It feels like modularity is breaking
stuff and asking for solutions once they are broken. Why don't we do it the
other way around?

Re: modular repositories in mock configs: please don't

By M A Young at 02/28/2019 - 20:08

As is RHEL8 public beta 1. I think this is actually the bug
<a href="https://bugzilla.redhat.com/show_bug.cgi?id=1673851" title="https://bugzilla.redhat.com/show_bug.cgi?id=1673851">https://bugzilla.redhat.com/show_bug.cgi?id=1673851</a>
with 2 related rhel8 bugs
<a href="https://bugzilla.redhat.com/show_bug.cgi?id=1677583" title="https://bugzilla.redhat.com/show_bug.cgi?id=1677583">https://bugzilla.redhat.com/show_bug.cgi?id=1677583</a>
<a href="https://bugzilla.redhat.com/show_bug.cgi?id=1678911" title="https://bugzilla.redhat.com/show_bug.cgi?id=1678911">https://bugzilla.redhat.com/show_bug.cgi?id=1678911</a>

in which case you can probably work around it by setting best=0 in the
relevant file in /etc/mock (or you could take a copy of the file and
edit that, then select the edited copy with the -r option of mock).

Michael Young