DevHeads.net

Fedora 31 Self-Contained Change proposal: Move test.support module to python3-test subpackage

<a href="https://fedoraproject.org/wiki/Changes/Move_test.support_module_to_python3-test_subpackage" title="https://fedoraproject.org/wiki/Changes/Move_test.support_module_to_python3-test_subpackage">https://fedoraproject.org/wiki/Changes/Move_test.support_module_to_pytho...</a>

== Summary ==

Move <code>test.support</code> from <code>python3-libs</code> to
<code>python3-test</code> subpackage.

== Owner ==
* Name: [[User:lbalhar| Lumír Balhar]]
* Email: [mailto: ... at redhat dot com| ... at redhat dot com]

== Detailed Description ==

Python test modules should be used only for testing Python itself.
However, some packages have buildtime or runtime dependency on parts
of Python test modules. The aim of this change is to move the most
popular Python test module <code>test.support</code> from
<code>python3-libs</code> to <code>python3-test</code> subpackage
which will help us discover packages which depend on it and also
identify parts of the module which might be useful to move to standard
library.

Also, [https://docs.python.org/3/library/test.html#module-test.support
the Python documentation] says test.support is not a public module.
Other things should not depend on it, and if something is useful
outside the CPython test suite, we should ideally take it out of
test.support.

== Benefit to Fedora ==

The main benefit here is that moving it all to -test is less fragile
and more consistent.

It also helps us understand which parts of Python test suite are
useful and which is worth to move to the standard library.

There were also a few bugs caused by parts of the test module packaged
in different subpackages and by differences between Python 2 and 3 -
[https://bugzilla.redhat.com/show_bug.cgi?id=1651245 RHBZ#1651245],
[https://bugzilla.redhat.com/show_bug.cgi?id=1528899 RHBZ#1528899],
[https://bugzilla.redhat.com/show_bug.cgi?id=596258 RHBZ#596258]

== Scope ==
* Proposal owners:
** Move <code>test.support</code> from <code>python3-libs</code> to
<code>python3-test</code> subpackage.
** Identify as many as possible affected packages and try to fix their
buildtime/runtime issues caused by the change.

* Other developers:
** If a package depends on <code>test.support</code> module which we
move to -test subpackage, change the build/runtime dependencies
definition accordingly.

* Release engineering: N/A (not a System Wide Change)

* Policies and guidelines: N/A (not a System Wide Change)

* Trademark approval: N/A (not needed for this Change)

== Upgrade/compatibility impact ==
Test modules should not be used as a runtime dependency so there is no
impact on an upgrade.

== How To Test ==

N/A (not a System Wide Change)

== User Experience ==
The user experience should not be affected.

== Dependencies ==
N/A (not a System Wide Change)

== Contingency Plan ==

* Contingency mechanism: Changes will be mostly done in Python
specfile and they can be easily reverted in case of some unexpected
issues. The second possibility is to set Provides to temporarily
deactivate the impact of the change.
* Contingency deadline: N/A (not a System Wide Change)
* Blocks release? N/A (not a System Wide Change)
* Blocks product? No.

== Documentation ==
N/A (not a System Wide Change)

== Release Notes ==
Since this affects only a few packages, no release notes are necessary.

Comments

Re: Fedora 31 Self-Contained Change proposal: Move test.support

By Yaakov Selkowitz at 06/24/2019 - 15:06

On Mon, 2019-06-24 at 12:09 -0400, Ben Cotton wrote:
The main reason for this change is to *experiment* and see what breaks
when it is moved? Why can't that be done in copr instead?

Re: Fedora 31 Self-Contained Change proposal: Move test.support

By =?UTF-8?B?TWlyb... at 06/24/2019 - 15:10

On 24. 06. 19 22:06, Yaakov Selkowitz wrote:
The main reason for this change is to make packaging of Python easier (and more
explicit) and less error prone in the future. Discovering packages that need
test.support is a secondary goal that help upstream but does not help Fedora much.

Should the change be reworded so this is more clear?

Re: Fedora 31 Self-Contained Change proposal: Move test.support

By Yaakov Selkowitz at 06/24/2019 - 15:25

On Mon, 2019-06-24 at 22:10 +0200, Miro Hrončok wrote:
I would say the scoping should happen (e.g. in copr) first, then we'll
know what the scope (and hence feasibility) of this change truly is.

Re: Fedora 31 Self-Contained Change proposal: Move test.support

By =?UTF-8?B?TWlyb... at 06/24/2019 - 16:09

On 24. 06. 19 22:25, Yaakov Selkowitz wrote:
What will the scoping actually tell us? If it tells us that X hundred packages
need to add BR for python3-test, we'll just do that and report the list to
upstream. We can do that during the mass rebuild.

Re: Fedora 31 Self-Contained Change proposal: Move test.support

By Yaakov Selkowitz at 06/24/2019 - 17:18

On Mon, 2019-06-24 at 23:09 +0200, Miro Hrončok wrote:
If there would indeed be X hundred packages affected, would this move
still make sense? The python3-test subpackage is even larger than most
of the rest of python3 combined, and the vast majority of it is
irrelevant to other packages. Such usage would indicate to me that
some work needs to be done within the ecosystem to respect its official
status as internal-only. (Perhaps, in that case, a move to python3-
devel could be considered instead.)

Re: Fedora 31 Self-Contained Change proposal: Move test.support

By =?UTF-8?B?TWlyb... at 06/24/2019 - 17:26

On 25. 06. 19 0:18, Yaakov Selkowitz wrote:
Good questions. What number of affected packages do you think would be crucial?
I say X hundred, because I don't anticipate it will go to thousands. However
with 3000 Python packages, couple hundreds is IMHO not a big deal. Note that
this is build dependency, not a runtime one.

Re: Fedora 31 Self-Contained Change proposal: Move test.support

By Yaakov Selkowitz at 06/24/2019 - 18:37

On Tue, 2019-06-25 at 00:26 +0200, Miro Hrončok wrote:
Nevertheless, there has been a great deal of effort recently to shrink
buildroots and speed up build times. Having to add BR: python3-test to
packages would mean adding a download of ~9MiB and an installation of
~3K files to every single affected package's build time. IMO this
needs to be fully scoped before consideration.

Re: Fedora 31 Self-Contained Change proposal: Move test.support

By Pierre-Yves at 06/25/2019 - 02:50

On Mon, Jun 24, 2019 at 07:37:19PM -0400, Yaakov Selkowitz wrote:
Wait a minute, adding a BR python3-test would mean an additional 9MiB ok, but
currently these files are part of python3-libs which *every packages* get.
In other words, for the packages that require python3-test the amount of data
downloaded doesn't change while for all the packages that do *not* require
python3-test, the amount of data is reduced.
This sounds like a win to me :)

Pierre

Re: Fedora 31 Self-Contained Change proposal: Move test.support

By =?UTF-8?B?TWlyb... at 06/25/2019 - 03:48

On 25. 06. 19 9:50, Pierre-Yves Chibon wrote:
Unfortunately, this is not correct.

python3-test already is large.

We just move a small bit of python3-libs (test.support) and we move it to
python3-test for consistency.

test.support is 396K installed (for Python 3.8).

Re: Fedora 31 Self-Contained Change proposal: Move test.support

By Pierre-Yves at 06/25/2019 - 05:07

On Tue, Jun 25, 2019 at 10:48:42AM +0200, Miro Hrončok wrote:
Arf, then it's a lesser win than I thought.
Have you considered doing a python3-test-support subpackage? This would allow
the separate package and thus give you the information you're looking for as
well as avoid downloading the entire python3-test for these few files.

Pierre

Re: Fedora 31 Self-Contained Change proposal: Move test.support

By =?UTF-8?B?TWlyb... at 06/25/2019 - 05:25

On 25. 06. 19 12:07, Pierre-Yves Chibon wrote:
The purpose of this change is to make packaging of Python easier and avoid
problems, when the test.support module cannot be properly used without the rest
of the test module. What you are proposing will make packaging more complicated
and would not eliminate the problems at all.

I think we are talking about couple packages here. Would it make sense to have a
number and say that if more than X packages suddenly need to add BR for
python3-test, we revert the change and consider a different approach?

BTW python3-test is now buildrequired by 7 packages:
python-bz2file
python-honcho
python-iniparse
python-kitchen
python-typing-extensions
python-urwid
python-zodbpickle

Re: Fedora 31 Self-Contained Change proposal: Move test.support

By Lumir Balhar at 06/28/2019 - 02:25

On 6/25/19 12:25 PM, Miro Hrončok wrote:
I've prepared a special COPR [0] where test.support module is not
available at all and then I rebuilt all Python packages. From 2952
packages 2769 built successfully and only 182 failed. Most failures were
caused by missing dependencies (python2-), incompatible pytest or sphinx
version, etc. and only 7 packages fail to build because of the missing
test.support module. From those 7 packages, 4 already require
python3-test so there will be need to change only 3 packages after we'll
move test.support module from python3-libs to python3-test.

[0] <a href="https://copr.fedorainfracloud.org/coprs/lbalhar/test.support_change/" title="https://copr.fedorainfracloud.org/coprs/lbalhar/test.support_change/">https://copr.fedorainfracloud.org/coprs/lbalhar/test.support_change/</a>

Re: Fedora 31 Self-Contained Change proposal: Move test.support

By Yaakov Selkowitz at 06/28/2019 - 12:45

On Fri, 2019-06-28 at 09:25 +0200, Lumir Balhar wrote:
Thank you for doing that. This information should be included in the
Change proposal and the "help us discover" wording removed.