DevHeads.net

Module build proposal

Hi all,

as promised on the last Modularity WG meeting [0], here's
a module build proposal we've been pondering for a while.

As always, this is nowhere near final and there are still many
open questions. The purpose of this mail is to get some early
feedback. Feel free to comment, especially if you see anything
obviously wrong or have ideas how to make the whole thing better.

Note the last module metadata proposal is a little outdated and
will have to be adapted to make this possible.

Key points:
- building a module means building the components it contains
and composing a module deliverable (a repository, an image, other)
- the build must be reproducible
- the module's buildroot is defined by "buildrequiring" other,
already existing modules
- modules should be tracked in a VCS, such as dist-git [1]

Build proposal, step by step:
- we have a module, let's name it foo
- the module definition says it's composed of RPM packages
bar and baz
- the module definition also says where to find the sources
for these packages, e.g. a link to a git repository, the
commit hash and a link to the lookaside cache [2]
- the build service clones bar's and baz's repositories and
checks out the specified commits [3]
- the build service prepares the buildroot using the modules
listed as its build dependencies
- the build service builds bar and baz in the prepared buildroot;
some modules might require buildroot overrides and build their
packages in a particular order
- for whatever reason, we might not always want all the built
packages in our module; we can filter some out at this point
- the service optionally includes (well, bundles) dependencies
present in the buildroot but not present in the module's runtime
environment in the currently built module
- the service gathers any extra metadata it needs and records
them in the module -- this could include automatically generated
lists of component licenses, builder IDs, vendor tags, timestamps,
module life cycle data and more
- the build service composes the deliverable(s)

Challenges:
- we need to ensure the module sources will be available for the
entire lifetime of the module (e.g. SPEC files, tarballs)
- plenty more, point them out

Most of this can be done with the tools we already have or
with minimal changes to them. This proposal doesn't address
parallel installations of conflicting modules but that's a
somewhat different topic and may be solved on a different level.

Again, share your comments and feel free to ask any questions.

Cheers,
P

[0] <a href="https://meetbot.fedoraproject.org/teams/modularity-wg/modularity-wg.2016-04-28-15.01.html" title="https://meetbot.fedoraproject.org/teams/modularity-wg/modularity-wg.2016-04-28-15.01.html">https://meetbot.fedoraproject.org/teams/modularity-wg/modularity-wg.2016...</a>
[1] Working on that; our staging dist-git and pkgdb already
include support for this.
[2] Obviously some of these couldn't be set by the module author
when building official Fedora modules; our build service should
always use our dist-git and our cache. It is still useful for
development, testing and third-party module vendors.
[3] We've considered using git tags but it's more trouble than it's
worth.