New "tests" namespace to share test code


While working on adding CI tests [0] using the Standard Test
Interface a need arose to have a shared git repository where tests
could be stored:

* A large number of test files makes a dist-git repository more
difficult to maintain

* Tests might follow a different branching pattern than the
dist-git repo, leading to code duplication

* Shared maintenance for tests sometimes benefits from different
access levels than the release dist-git repository

The plan is to create a new “tests” namespace in Fedora git/pagure
dedicated to storing the shared test code. To enable execution of
these tests by the CI pipeline, tests.yml file in dist-git will be
used to link the tests in the standard way as defined by the
Standard Test Interface [1].

This approach should help to efficiently maintain tests & minimize
test code duplication. Using a dedicated git repo for test code
also means to keep dist-git more as a place for storing metadata
only: Build metadata (spec file = how to build the package) and
test metadata (tests.yml = how to test the package) rather than
mixing spec files with test code itself.

Please note that this does not mean that all tests should now go
into this new namespace. You can still link tests directly from
upstream (like GitHub) or any other source. Also, for unit tests
it makes more sense to be kept directly with the project source
and executed there. Shared tests namespace in Fedora will be
suitable especially for functional and integration testing which
should help to continuously ensure the OS works as a whole.

For more detailed motivation and real-life examples see the Share
Test Code proposal on the Fedora CI list [2]. If you have any
questions or comments feel free to share them here or in the
pagure issue requesting the new namespace:

<a href="" title=""></a>



[0] <a href="" title=""></a>
[1] <a href="" title=""></a>
[2] <a href=" ... at lists dot" title=" ... at lists dot"> ... at lists dot</a>


Re: New "tests" namespace to share test code

By Petr Splichal at 02/14/2018 - 12:28


During the last days there have been concerns raised regarding
what is an appropriate content for the tests namespace. [1] My
original idea was to enable sharing tests even across branches of
the same component, not only for tests to be used by completely
different packages. The initial examples might have been a bit
misleading in this respect. One of the main points still holds:

From the feedback from developers I feel they always keep on mind
and care a lot about the maintenance costs. So it perfectly makes
sense to me if they want to keep and maintain tests in a separate
repo instead of merging/cherry-picking between dist-git branches.

When, for a particular package, it is the most efficient way to
maintain tests in a separate repo why should we discourage from
this approach? There are packages where it makes more sense to
store test code directly in dist-git. And it is still an option.
But why should we enforce this for all?

Please share your thoughts and real-life examples. For those who
are not familiar with the topic there is a new wiki page with a
summary of the Share Test Code approach [2].


[1] <a href="" title=""></a>
[2] <a href="" title=""></a>

On 7 December 2017 at 10:38, Petr Splichal < ... at redhat dot com> wrote:

Re: New "tests" namespace to share test code

By Pierre-Yves at 02/22/2018 - 10:50

On Wed, Feb 14, 2018 at 05:28:20PM +0100, Petr Šplíchal wrote:
My worries are basically that this mechanism is hiding the tests from the
package maintainers. It splits the concerns between people maintaining the
artifacts and people maintaining the tests, which is exactly what the original
plan to bring CI was *not* about.
The idea has always been to bring the tests where the code lives, so that both
can be worked on as one, to make tests be a concern of the package maintainer
very much like updating the artifact (rpms, image..) is, while getting support
from QE for them (the tests).

In addition, this is what I fear most:
The tests for the package are stored elsewhere, we're not sure where (the tests
namespace, a random git repo on the internet...), the packager update package
and the tests fail.
What do you think will happen?
a) the packager will look for $random_place_of_the_internet where tests are and
spend time trying to fix them.
b) the packager will turn off the tests because they get in the way

If the packager wants to turn off the tests, they will have to go through
dist-git to do it, they may very well end up turning the tests off anyway, but
if the tests are right there, they may as well have a quick look at them to see
if they can fix them quickly before deciding.

In addition, if the packager turn the tests off and the people maintaining the
tests do not realize that, they will end up spending time maintaining
$somewhere_else tests that aren't being used.
Having them interact directly with the dist-git repo will increase the chances
that they see/realize when something is turned off.

If that means we have less tests in dist-git because the maintainers do not want
them, I'd say so be it.
In the long term this is their loss, they are the ones who will get the bug
reports and will have to deal with them, they are the ones who will have to dive
into old code rather than going back into something that is still fresh in their
As long as, it is clear that they have been approached and that it is their
choice to not merge pull-requests adding tests, I think the people offering to
help should not be the ones blamed.


Re: New "tests" namespace to share test code

By Zbigniew =?utf-... at 02/25/2018 - 08:12

On Thu, Feb 22, 2018 at 03:50:38PM +0100, Pierre-Yves Chibon wrote:

pingou, I share your thinking in general, but I think your concern is
overstated. Petr's original e-mail suggests that the new separate
namespace should be used *in* *preference* to the in-dist-git, and your
reply concentrates on that. But I think that we want to have both
possibilities, because both will work best for different cases, and it's
just the emphasis that needs to change.

I agree that having tests in a separate namespace is not any simpler
or more manageable for most packages. Thus, I think that *by* *default*
we should put tests in dist-git. IMHO this will apply to 90% of all
tests and packages.

OTOH, there will be cases where it'll be useful to split the tests out
into a separate area. The documentation for tests/ has the example of
shared tests for bash/zhs/dash. I'm sure that there will be other
cases, like testing of stacks consisting of multiple packages
apache+mysql+php or glibc+gcc+binutils, etc, where it'll be beneficial
to keep the test separate. The management of permissions in this case
will be _much_ more complicated, because to meaningfully update the
tests sometimes it'll be necessary to change the test repo and _each_
of the packages. But having them separate is better than randomly
sticking the shared tests in dist-git for one of the involved packages.
Nevertheless, because of the overhead of more repos and additional
permissions, IMHO this should never be advertised as the default
approach, but as an "advanced" possibility.


Re: New "tests" namespace to share test code

By Pierre-Yves at 02/25/2018 - 15:59

On Sun, Feb 25, 2018 at 12:12:33PM +0000, Zbigniew Jędrzejewski-Szmek wrote:
I violently agree with this :) I am very much advocating for using the test
namespace to share tests across multiple packages.
What trigger this discussion has been the request to create repositories in the
test namespace to store the tests of a single package (ie: instead of just
adding them to the corresponding dist-git repo).


Re: New "tests" namespace to share test code

By Honza Horak at 03/01/2018 - 13:12

On 02/25/2018 08:59 PM, Pierre-Yves Chibon wrote:
Even a single package is then maintained in multiple Fedora branches
(and don't forget about modules in the picture) and while RPM spec files
and sources might be easily different release to release, the tests have
much bigger chance to be same across the releases, so having one copy of
them is what I expect in most cases. That's what I like about the shared
place variant.


Re: New "tests" namespace to share test code

By Pavel Raiskup at 02/23/2018 - 05:33

On Thursday, February 22, 2018 3:50:38 PM CET Pierre-Yves Chibon wrote:
Since we'll have to "hook" the remote repo into dist-git code, I must be
missing something. Where do you see the hidden magic?

If the tests live in separate repo, they still can "work as one".

Yes, please don't force us to use random_place_of_the_internet, but rather
allow maintainers to use standardized place.

Then e.g. 'rpmtests/<PKGNAME>' repo can automatically grant commit access
to package maintainers of 'rpms/<PKGNAME>' to not overcomplicate things,
but still, we can grant non-packagers access to 'rpmtests/<PKGNAME>'.

What about proventesters, could we grant them access to all tests, or could we
create new role?

Polemics (even below) about turning-off the tests is FUD. Since tests are
opt-in feature, why would anyone thought this way?

There's no loss, no. Everything is as transparent as with in-dist-git
testing, except it gives us more flexibility
- to allow more people to contribute.
- keep the package's git smaller and concentrated on packaging

The help here is very much welcome, I wouldn't blame anyone though (I for
one only wait for a bit more flexible form of CI).

Btw., the far more important problem is that we can not setup CI for
non-atomic host packages, and that we can not use the CI for Rawhide


Re: New "tests" namespace to share test code

By Pierre-Yves at 02/23/2018 - 06:09

On Fri, Feb 23, 2018 at 10:33:57AM +0100, Pavel Raiskup wrote:
How do they work as one if they are both working in different places?

We're working on figuring a way to give non-packagers commit to dist-git, so
giving more access would work on the main repo as well.

I may very well be miss-seeing this, but what I see is people refusing to add
tests to their repository because it doesn't suit their workflow.
So I am thinking that if they finally think "whatever let's just merge" because
they don't want to cause troubles, whenever the tests are going to impact that
workflow even more because they fail, chances are high that said person will
just turn them off.
I'd be happy to be wrong about that though.

Pull-requests and improvement to our dist-git set up seems to me better ways to
achieve this.

That one is fair.

Definitely agreed on this one, I've been pushing for it for the better part of a
year, I'd be happy to see more people pushing for it.


Re: New "tests" namespace to share test code

By Petr Splichal at 02/23/2018 - 05:19

On 22 February 2018 at 15:50, Pierre-Yves Chibon < ... at pingoured dot fr> wrote:
Thanks for sharing your thoughts, Pierre.

I do not see a split here but rather potential for enhanced or
extended collaboration. Repositories in the tests namespace can be
set up with more open access than rpms git. In this way devel can
invite qe for direct collaboration on tests. That's one of the
reasons why we should try to fix the non-packager access as soon
as possible.

<a href="" title=""></a>

I see the benefits of having code with tests at the very same
place, I find this very useful especially for unit tests in the
project upstream. However, the situation here is a bit different
in two respects:

1) The rpms dist git does not include package code itself but
rather build metadata only (mainly spec files describing how to
build the package) so it makes very good sense to host at this
place testing metadata only (how to test the package). This is
quite nicely consistent approach.

2) As far as I can see, within the Always Ready Operating System
effort we are not looking to cover every and each feature in all
packages but rather ensuring the OS works as a whole. Thus basic
functionality and integration tests which guard the stability of
the packages are more appropriate for testing. These do not need
to be kept so close to the package code.

I believe we don't want to duplicate upstream unit testing here.
We should probably clarify more which test types are appropriate
or expected in the Fedora CI. As part of the docs consolidation I
added a brief section about test types to the main CI portal:

<a href="" title=""></a>

Does it decribe well what we are looking for? Shall we extend it?

I see no hiding of tests from maintainers: Tests are clearly
referenced and discoverable from the rpms dist git as defined by
the Standard Test Interface and devels have access, see:

<a href="" title=""></a>

Devel decides which tests are good for testing, it does not matter
where the test code lives. However there needs to be trust to the
referenced repo. But this is the same as with any other open
source project.

It's not random_place_of_the_internet, it's well selected and
trusted source where devel can (usually) directly contribute. I
meet devels around who see benefit in having tests enabled. So
I don't share your concern about turning the test off so easily.

Working on something that is not being used supposes there is no
communication/collaboration between devel and qe. And I doubt that
enforcing inefficient workflow with test code duplication and less
ACL flexibility will help to increase it. I believe we should set
up the workflow as smooth as possible to encourage better

The question is not about devels who do not want tests enabled in
dist-git but about those who do not want to inefficiently maintain
the test code there. So diving into old code does not apply here.


Re: New "tests" namespace to share test code

By Pierre-Yves at 02/23/2018 - 05:59

On Fri, Feb 23, 2018 at 10:19:35AM +0100, Petr Šplíchal wrote:
I don't see how that's not possible with tests being in dist-git.

Agreed and being worked on.

I still believe it creates a separation of concerns that is in opposition with
the original idea.

Referenced maybe, discoverable, much less than if they are right there.

If the tests are defined in dist-git and adding new tests means committing these
changes to dist-git's yaml file, how does that solve the concern about "noise"
in the git commits?

It seems to me that whether the commit touches just the yaml file or the yaml
and tweak the tests themselves, the outcome is the same.

I don't see how this duplication works, if you need to touch the yaml file
anyway, there will be just as much "duplication".

How are ACLs less flexible than with any of the open source project out there?
You start by submitting pull-request and at one point the people trust you
enough to grant you direct commit access.

I don't see what's inefficient here sorry.


Re: New "tests" namespace to share test code

By Petr Splichal at 03/01/2018 - 09:44

What I've heard from devels is that they often don't feel like
giving direct commit access to the rpms dist git to QE as spec
files and packaging in general is a big responsiblity. But they
are quite OK with directly collaborating with QE on test coverage.

Not necessarily. You could use advanced metadata to select desired
tests only once. The very first proof of concept of using Flexible
Metadata Format for testing selinux quite nicely illustrates this:

<a href="" title=""></a>
<a href="" title=""></a>

Instead of listing individual tests you could say: Run all tests
for this selinux component + all Tier1 selinux integration tests:

fmf --key test --filter 'component:libselinux | tags:Tier1'

For tests covering multiple components the benefit of the tests
namespace approach is unquestionable. However, I see benefit of
the approach for single components as well (and developers seem
to often feel the same): That is not having to cherry-pick/merge
test coverage between supported branches. So even for these cases
there is the benefit of sharing the test code.

I believe that extending the integration testing coverage is one
the most essential goals of Fedora CI. This is the crucial piece
of the puzzle how to make an Always Ready Operating System: It is
to ensure that packages play nicely together. So I guess there
will be quite many tests which will benefit from this approach.

Do we really want/need to assess every new request for a tests
repo if it is good-enough for the test namespace? Will there be
a committee to decide whether a particular set of tests has a
potential to be used by other components in the future or not? :)

To sum up what I've heard so far from the developer side:

* I would like to enable tests for my component (yes, I want)
* I will take care of them (really, I see the benefit in CI)
* I want to easily collaborate on tests with qe (direct commits)
* I don't want to give qe commit access to the rpms dist git
* I want to efficiently maintain tests long-term
* I want to have just a single place for my tests (no duplication)
* I want to keep rpms dist git clean (just metadata & patches)
* I want to easily enable testing for all supported branches
* I don't want to backport new tests to old branches (enable once)
* I want to run all available relevant tests (not to list them)
* I want to execute set of tests based on a tag (e.g. Tier1)
* I want an easy way to clone tests (fedpkg clone tests/pkg)

I believe the tests namespace resolves them all.


Re: New "tests" namespace to share test code

By Zbigniew =?utf-... at 03/01/2018 - 10:15

On Thu, Mar 01, 2018 at 02:44:37PM +0100, Petr Šplíchal wrote:
Personally, I'm perfectly happy to give write access to any repo
to people who have shown that they know what they are doing.
We have pull requests in dist-git pagure now, and I think this is
a better approach:
1. ask QE contributors to submit PRs
2. if enough cooperation happens and trust is established, give
privileges to write to the repo directly, possibly with an agreement
that this specific person should only touch tests, and not the

I think it's perfectly fine to never get to point 2.: many many
upstream projects require a review from a second person, or sometimes
even two reviews before a PR is merged, which means that one _never_
merges their own PR, and another contributor is always involved. We
usually don't do this in dist-git, but I'm quite sure that requiring
reviews for dist-git changes would raise quality and catch many silly
mistakes. Either way, it's nowadays possible to cooperate using a
single repo without fully trusting the other person, frictionlessly.

While I agree that keeping tests separate avoids a bit of effort
required to update multiple branches, this effort isn't very big. If
the tests indeed apply cleanly to all branches, then it's just a
matter of doing 'git merge' or 'git cherry-pick' once per branch.


Re: New "tests" namespace to share test code

By Petr Splichal at 03/01/2018 - 11:00

Good point. However, if there is a qe/devel team which prefers to
collaborate on tests in a separate repo because this is the most
efficient way they found so far, why should we stop them and try
to enforce a less efficient workflow? (Which they can workaround
by a different git repo.)

But "fedpkg clone tests" is easier than cloning from a "random"
git repository where I was forced to save my tests because I was
not allowed to save them in Fedora tests namespace.

Why default? Test namespace should be an option. Not the default.
Storing tests directly in dist git is and will be possible.
Anybody who finds this as a better way can do so. But why
enforcing this approach to all?


Re: New "tests" namespace to share test code

By Honza Horak at 03/01/2018 - 13:06

On 03/01/2018 04:00 PM, Petr Šplíchal wrote:
+1 Exactly! I don't really see any reason why to force people to follow
one way when there are obvious disadvantages. All concerns against
tests/ namespace in this thread are based on expectations that it won't
work, but how can anybody know at this point? The only proof we have is
that the tests/ namespace works internally.

I believe that what we really need here is flexibility -- let people
find the way that works best for them.


Re: New "tests" namespace to share test code

By Petr Splichal at 03/13/2018 - 19:16


We've had some more discussions regarding this topic within the CI
team. Here's a brief summary of the recommendation we agreed upon:

Tests may be written in different ways, but are enabled in a
standard way directly in the package git repository as defined
by the Standard Test Interface. [1]

Test code can be stored directly in the package git repository
(recommended as default) or fetched from another location hosted
in the Fedora infrastructure such as the Tests Namespace. [2] The
decision where the test code is stored is up to the package maintainer.

[1] <a href="" title=""></a>
[2] <a href="" title=""></a>


On 1 March 2018 at 18:06, Honza Horak < ... at redhat dot com> wrote:

Re: New "tests" namespace to share test code

By Adam Williamson at 02/14/2018 - 13:18

On Wed, 2018-02-14 at 17:28 +0100, Petr Šplíchal wrote:
I don't have any problem with the concept *in itself*; in fact I know
of another reason to have something like it. That is 'generic' tests,
tests we want to run on all packages, or at least a very large set of
packages - like the tests currently running in Taskotron, which are
generally run on all packages (rpmgrill, rpmdeplint...) or a large
subset (the Python tests).

What I did think was maybe a concern is that we should figure out in
advance the squishy human consequences of different technical

Basically this boiled down to "who is responsible for these 'shared'
tests, and who gets to decide which 'shared' tests can/should block

Looking at the proposal, I think it actually has at least workable
initial explicit/implicit answers to this, if I'm reading it correctly.
Anyone can create a shared test repository (and is therefore
"responsible" for maintaining it), but the definition of which tests
are run on which packages remains in the package repositories. Thus the
package maintainer(s) retain the ultimate choice over which tests are
run against their packages (and thus can pick which shared tests are
run, and disable shared tests if they are no longer properly testing
their package).

The question 'who decides which tests block which packages' is left a
bit up in the air, but in fact no more so than it already was for
package-specific tests...

Re: New "tests" namespace to share test code

By Petr Splichal at 02/15/2018 - 08:11

On 14 February 2018 at 18:18, Adam Williamson
< ... at fedoraproject dot org> wrote:
Yes, sharing tests useful for multiple packages makes sense in
general. Test code for generic tests can be stored basically
anywhere but not in specific components. However I see some
additional benefits of having a dedicated tests namespace:

* Source of inspiration and real-life examples (at one place)
* Potential for workflow automation (tests CI, fedpkg --tests)
* Easy cherry-picking/backporting upstream/downstream tests

The difference I see when comparing these shared tests to generic
tests mentioned above like rpmdeplint is that you don't want to
reference these from the individual package repos (e.g. using the
Standard Test Interface), but just run them on all packages.

If the expectation is to run tests only on a subset of rpms then a
question comes: How to store the list of relevant components or
criteria to decide which packages are to be tested or not. Here
the Flexible Metadata Format might come handy in the future:

<a href="" title=""></a>

Yes. This is exactly true. The maintainer of the package repo has
always the last word on what is to be executed for qualification.
It can be done by referencing only suitable tests from the shared
repo or selecting an appropriate branch or even a specific commit
to ensure that future test development does not break stable
gating (and only manually update reference after review of shared
test updates) or even completely disable tests by removing the

Right. Do we want to change this? Specify this more strictly?
Like exactly defining requirements which an Always Ready Operating
System has to meet? And then mapping these requirements to the
test coverage? (Here again FMF mentioned above might come handy.)


Re: New "tests" namespace to share test code

By Adam Williamson at 02/15/2018 - 15:03

On Thu, 2018-02-15 at 13:11 +0100, Petr Šplíchal wrote:
Maybe we should just *formalize* it a bit, so folks know where they
stand. So far we've basically done three things that implicitly
describe the policy on this:

1) Bodhi has a mechanism by which the person submitting an update can
specify some tests that block the update

2) FESCo accepted, voted on and approved a ticket which involved a
decision that certain tests blocked some/all updates

3) We built and deployed WaiverDB and Greenwave such that Greenwave
currently accepts waivers submitted to WaiverDB by...(anybody??? this
point I'm not sure on)

So, effectively, we have communicated that FESCo can set mandatory
policies here, and packagers can also make voluntary choices. Point 3)
is rather a crucial one, and I don't know that it's been formally
discussed anywhere; I think we have been rather heavily focused on
building the tools as opposed to setting the policy so far. But we've
at least clearly implied so far that FESCo claims to some extent the
power to set automated test gating requirements for updates, and that
packagers have some ability to extend those requirements voluntarily,
and also submit waivers. I guess the natural thing to do from here, if
we want to be a bit more explicit about it, would be to extend the
Updates Policy:

<a href="" title=""></a>

either in-line or with a subsidiary document, to define this stuff a
bit more clearly/formally.

Re: New "tests" namespace to share test code

By Honza Horak at 02/14/2018 - 12:46

On 02/14/2018 05:28 PM, Petr Šplíchal wrote:

From my PoV, it makes perfect sense to share tests between branches, or
between different versions of packages available in modules streams (soon).