DevHeads.net

Interest in doing Fedora CI with test subpackages

Hey all-
I was starting to setup CI for one of my packages in Fedora (cscope),
which requires that I have access to the sources to run my test (cscope uses its
own source tree to search for various symbols to confirm that its working
properly). Getting the sources in the CI environment is a bit of a pain, so I
started working on trying to do this by creating a test subpackage (specifically
named -citest) to package up the sources solely for the purpose of getting them
installed and available during CI runs. It occured to me that this offers
several advantages, among them:
1) the ability to codify dependencies within the ame spec file, rather than
having to copy them to the test.yml file, and keep them in sync

2) The ability to use a file format (rpm spec files) that I'm more familiar with

3) Easy access to tests that are embedded in the source tree

4) minimizing the test harness setup in test.yml

For anyone interested, I've got a pull request started here:
<a href="https://src.fedoraproject.org/rpms/cscope/pull-request/2" title="https://src.fedoraproject.org/rpms/cscope/pull-request/2">https://src.fedoraproject.org/rpms/cscope/pull-request/2</a>

If anyone wants to take a look at the changes I had to make to do this (fair
warning, its still very rough).

That all said, I was wondering if perhaps there was general interest in making
this kind of test model somewhat more formal (i.e. creating an rpm macro library
to make test package generation a bit easier, creating a standard entry point to
run tests, etc).

Thoughts welcome

Comments

Re: Interest in doing Fedora CI with test subpackages

By Dan =?utf-8?B?x... at 09/11/2019 - 18:06

Hi Neil,

Neil Horman < ... at redhat dot com> writes:

This is imho a pretty big advantage, as it ensures that the tests and
the source don't diverge.

I am not sure whether a generalization makes sense, as there are so many
languages with such a wide range of test suites. What would make sense
to standardize would be the generation of a -citest subpackage though,
so that it is setup correctly and consistently.

I like this idea and you're actually not the first one ;-). Something
comparable is being done in openSUSE's Ruby RPM packages: if the gem
ships a testsuite, then a -test subpackage is created with the tests
inside it. (In practice these packages are unfortunately never used, as
they often lack the necessary dependencies to be installable and even
if, the testsuite usually doesn't run outside of bundler, but that's a
different story).

I think this approach makes especially sense for packages which ship an
extensive test suite that is not feasible to run during %check, but can
be run in the gating CI.

Re: Interest in doing Fedora CI with test subpackages

By =?ISO-8859-1?Q?... at 07/08/2019 - 08:17

I am skeptical about this proposal. While this might work for your
package, I am afraid it won't work generally and trying to do something
like this is wasted energy. Let me explain.

When RubyGems were designed in 2004, TDD, BDD and testing in general was
becoming good practice. Therefore they decided that it is good idea to
ship code together with its test suite and also execute the tests during
installation. So you were supposed to be able to run something like "gem
install -t foo". But it proven to be problematic. Generally this "-t"
option never worked and could be used just for the most naive test
suites. So the option was removed [1]. Nowadays, more and more gem
packages does not ship their tests suites and even the support for
shipping the test suites is deprecated [2].

If you want to understand how complex it might be to run the test suite
of some packages, I welcome you to explore the %check sections of
rubygem- packages. Some are simple, but most are not that simple,
starting with small differences as load path, ending with test suites
which needs to run various servers or check against cloud services. Not
mentioning test matrices.

Also, for all these tests we usually try to simulate the environment as
similar as possible to upstream repository, because this is how these
test suites are designed. It would be even harder to execute the test
suites against installed packages.

Vít

[1]
<a href="https://github.com/rubygems/rubygems/commit/429f883210f8b2b38ea310f7fc6636cd0e456d5c" title="https://github.com/rubygems/rubygems/commit/429f883210f8b2b38ea310f7fc6636cd0e456d5c">https://github.com/rubygems/rubygems/commit/429f883210f8b2b38ea310f7fc66...</a>

[2] <a href="https://github.com/rubygems/rubygems/issues/735" title="https://github.com/rubygems/rubygems/issues/735">https://github.com/rubygems/rubygems/issues/735</a>

Dne 05. 07. 19 v 19:49 Neil Horman napsal(a):

Re: Interest in doing Fedora CI with test subpackages

By Neil Horman at 07/08/2019 - 10:40

On Mon, Jul 08, 2019 at 02:17:44PM +0200, Vít Ondruch wrote:
and a few others, all of which have very mature test suites embedded into their
soruce code. For these pacakages, being able to run their test code in the CI
environment would be very helpful to me (and I think, by extension), others who
have simmilarly constructed packages.

Neil