DevHeads.net

Fedora 31 System-Wide Change proposal: Python means Python3

<a href="https://fedoraproject.org/wiki/Changes/Python_means_Python3" title="https://fedoraproject.org/wiki/Changes/Python_means_Python3">https://fedoraproject.org/wiki/Changes/Python_means_Python3</a>

== Summary ==
In package and command names, "Python" will mean "Python 3".

Users installing and running Python or Python packages without
specifying a version will get Python 3.

Running <code>python</code> will run <code>python3</code>. Running
<code>pytest</code> will run the Python 3 version of pytest, and
similarly for <code>pydoc</code>, <code>pylint</code>, etc.

<code>dnf install python</code> will install {{package|python3}}, and
similarly for other <code>python-*</code> provides, e.g. <code>dnf
install python-requests</code> will install
{{package|python3-requests}}.

This is the final preparation for
[https://fedoraproject.org/wiki/Changes/RetirePython2 the retirement
of Python 2 in Fedora 32] done in line with the
[https://github.com/python/peps/pull/989 soon to be finalized]
upstream recommendations.

== Owner ==
* Name: [[User:Churchyard| Miro HronĨok]]
* Name: [[User:Pviktori| Petr Viktorin]]
* Email: <python- ... at redhat dot com>

== Detailed Description ==

=== Motivation ===

The final upstream release of Python 2 is planned for January 2020. No
further fixes will be made upstream. Most of Fedora 31's lifetime is
after that date. Python 2 will be maintained only by its Fedora
maintainers.

In preparation for removing Python 2 from Fedora entirely, we will
make the unqualified name "Python" refer to the fully supported
version.

This is in line with upstream changes to
[https://www.python.org/dev/peps/pep-0394/ PEP 394] recommendations.
(At the time of this writing, these changes are still
[https://github.com/python/peps/pull/989 being finalized], but
recommendations for Linux distributions are accepted.)

This completes the work started with Fedora 23's
[https://fedoraproject.org/wiki/Changes/Python_3_as_Default "Python 3
as Default"] change, with only
[https://fedoraproject.org/wiki/Changes/RetirePython2 removing Python
2] left to do.

=== What is being changed ===

The following changes will be made both in the
[https://docs.fedoraproject.org/en-US/packaging-guidelines/Python/
Python packaging guidelines] and the actual packages.

==== Package provides ====

'''Before:''' Packages with Python 2 modules used to provide the
unversioned <code>python-</code> name. Users could do <code>dnf
install python-requests</code> and the {{package|python2-requests}}
package would be installed.

'''After:''' Packages with Python 3 modules will provide the
unversioned <code>python-</code> name. Users can do <code>dnf install
python-requests</code> and the {{package|python3-requests}} package
will be installed.

This applies to the {{package|python3}}/{{package|python2}} package as
well as packages with Python modules. <code>dnf install python</code>
will install {{package|python3}}.

The vast majority of packages will be updated by changing
[https://docs.fedoraproject.org/en-US/packaging-guidelines/Python/#_the_python_provide_macro
the %python-provide macro], with no changes in the individual spec
files. The change owners will change the macro before the mass
rebuild.

==== Python command ====

'''Before:''' The <code>/usr/bin/python</code> command was a symbolic
link to <code>/usr/bin/python2</code> living in the
{{package|python-unversioned-command}} subpackage of
{{package|python2}}. {{package|python2}} recommended
{{package|python-unversioned-command}}, so most users got the command
by default when {{package|python2}} was installed.

'''After:''' The <code>/usr/bin/python</code> command will be a
symbolic link to <code>/usr/bin/python3</code> living in the
{{package|python-unversioned-command}} subpackage of
{{package|python3}}. {{package|python3}} will recommend
{{package|python-unversioned-command}}, so most users will get the
command by default when {{package|python3}} is installed (and it is
installed by default).

==== Other commands ====

Similarly, the Python-specific commands will switch from Python 2 to
Python 3. This includes at least the following commands:

* pip
* python-config
* wheel
* idle
* pydoc
* pytest
* nosetest
* pycodestyle
* pylint
* epylint
* pyreverse
* symilar
* unit2
* msgfmt.py
* pygettext.py
* flask
* ipython
* f2py
* ipdb
* easy_install
* ...

This list is incomplete, automation is yet to be created to discover
all such commands.

Note that applications like <code>pygmentize</code> or
<code>cython</code>, whose behavior doesn't depend on the Python
version, are not affected. (By current guidelines, they should be
already using Python 3 if possible.)

=== Changes needed by Python package maintainers ===

This section of the change covers action needed from Python package
maintainers. Most of the packages need no change, but there are
several exceptions.

==== Packages with ambiguous names ====

Tracked at: <a href="https://fedora.portingdb.xyz/namingpolicy/" title="https://fedora.portingdb.xyz/namingpolicy/">https://fedora.portingdb.xyz/namingpolicy/</a>

If your package with a Python 2 module or plugin is named with
unversioned <code>python</code> (such as for example
<code>claws-mail-plugins-python</code> or <code>PyQt4</code>), it
needs to be removed or renamed to have <code>python2</code> in the
name (such as, for example, <code>claws-mail-plugins-python2</code> or
<code>python2-PyQt4</code>).

If your package is an application that happens to be written in Python
2 (such as {{package|calibre}}), no renaming is needed (applications
don't need the <code>python-</code>, <code>python2-</code> or
<code>python3-</code> prefix).

==== Packages with ambiguous requires ====

Tracked at: <a href="https://fedora.portingdb.xyz/namingpolicy/" title="https://fedora.portingdb.xyz/namingpolicy/">https://fedora.portingdb.xyz/namingpolicy/</a>

If your package depends on (Requires, BuildRequires, Recommends, etc.)
a Python package with unversioned Python name (such as for example:
systemd-python, python-setuptools, PyQt5, pycairo), it will now get
resolved to the Python 3 version. Such dependencies need to be updated
to specific Python 2 or Python 3 names (such as for example:
pythonX-systemd, pythonX-setuptools, pythonX-qt5, pythonX-cairo where
X is either 2 or 3).

==== Packages with ambiguous provides ====

Most of the backwards compatibility <code>python-*</code> provides are
handled by the %python_provide macro and the change owners will change
the macro behavior, so you don't have to worry about those.
However, sometimes manual provides were added. If your package has
some manual backwards compatibility provides for Python 2 packages,
those need to be moved to the Python 3 packages or removed.

For example, before this change, the {{package|python2-m2crypto}}
package provided and obsoleted <code>m2crypto</code>. Instead, the
{{package|python3-m2crypto}} package should do that after this change.

==== Packages with missing %python_provide ====

The <code>%python_provide</code> macro used to do nothing for Python 3
packages. As such, it was often forgotten and only used with the
Python 2 packages.
Packagers of Python 3 packages should make sure to use the
<code>%python_provide</code> macro according to the guidelines:

%package -n python3-%{srcname}
...
%{?python_provide:%python_provide python3-%{srcname}}

==== Packages with Python versioned commands and tools ====

Some packages have two versions of the provided commands. For example,
the {{package|python2-pytest}} package has
<code>/usr/bin/pytest-2</code> and <code>/usr/bin/pytest-2.7</code>,
the {{package|python3-pytest}} package has
<code>/usr/bin/pytest-3</code> and <code>/usr/bin/pytest-3.X</code>.
The unversioned <code>/usr/bin/pytest</code> command used to be a
symbolic link to <code>/usr/bin/pytest-2</code>. Now it needs to be
changed to <code>/usr/bin/pytest-3</code> and moved to the
{{package|python3-pytest}} package.

For some packages, such duplication makes no sense, because the user
sees no difference. For example, there should be just one
<code>/usr/bin/pygmentize</code> -- the user doesn't care if it runs
on Python 3, Python 2 or if it is written in Rust. This is not a new
rule, but if your package is not following it, now is a good time to
make sure the tool uses Python 3.

==== Packages that need unversioned Python to be Python 2 ====

Since Fedora 29, a
[https://fedoraproject.org/wiki/Changes/Move_usr_bin_python_into_separate_package
specific workaround is needed] to use the <code>python</code> command
as Python 2.
This workaround will now bring Python 3.
If that does not work for your package, you'll need to fix it (patch
it) or retire it from Fedora.

If you only need the <code>python</code> command to mean Python 2
during package build, you can do something like this:

mkdir tmp_path
ln -s %{__python2} tmp_path/python
PATH=$(pwd)/tmp_path:$PATH make ...

However, if your package uses the <code>python</code> command during
runtime, this ugly workaround won't work.

== Benefit to Fedora ==
The name "Python" will not refer to software that will be unmaintained
upstream for most of Fedora 31's lifetime and
[https://fedoraproject.org/wiki/Changes/RetirePython2 retired from
Fedora 32].

== Scope ==

* Proposal owners:
** Changes in {{package|python3}}, {{package|python2}} packages:
*** make <code>/usr/bin/python</code> link to
<code>/usr/bin/python3</code> instead of <code>/usr/bin/python2</code>
(+ the same for other executables there)
*** make {{package|python-unversioned-command}} a subpackage of
{{package|python3}} instead of {{package|python2}}
*** make {{package|python3}} (instead of {{package|python2}})
recommend {{package|python-unversioned-command}}
** Changes in other commands:
*** switch pytest, nosetests, ipython, pip... to python3 (see Detailed
Description for the actual list)
** Changes in {{package|python-rpm-macros}}:
*** make <code>%python_provide</code> provide <code>python-foo</code>
for <code>python3-foo</code> instead of for <code>python2-foo</code>
** Let the mass rebuild change the provides based on the
<code>%python_provide</code> change
** Examine the failures and provide help to package maintainers
** File bugs for remaining packages that provide <code>python-x</code>
or <code>x-python</code> for Python 2

* Other developers (Python package maintainers, see Detailed Description):
** Everybody: Make sure that you require and use python2 or python3
packages explicitly
** If your package provides <code>python-*</code> or
<code>*-python</code> for a Python 2 package by other means than the
<code>%python_provide</code> macro, move that to the Python 3 package
(or remove entirely if not applicable)
** Fix or retire your package if it requires the unversioned
<code>python</code> command to be Python 2

* Release engineering: [https://pagure.io/releng/issue/8482 #8482]

* Policies and guidelines:
** Slightly adapt
[https://docs.fedoraproject.org/en-US/packaging-guidelines/Python/#_the_python_provide_macro
the %pyton_provide macro] section of the Python packaging guidelines
and the [https://docs.fedoraproject.org/en-US/packaging-guidelines/Python_Appendix/
Python Appendix].
* Trademark approval: not needed

== Upgrade/compatibility impact ==
Custom scripts with <code>python</code> shebangs will invoke Python 3
by default, whereas previosuly they invoked Python 2 by default. See
the User Experience section for details.

== How To Test ==
For your favorite Python 3 package, check that it can be installed
with the ambiguous Python name. For example, check that <code>dnf
install python-pip</code> installs {{package|pyhon3-pip}} instead of
{{package|pyhon2-pip}}.

For your favorite Python command, check that it invokes the Python 3
version. For example, check that <code>python</code> runs Python 3.

== User Experience ==
Users who run <code>python</code> directly will get Python 3.
Users who run <code>pip</code> directly will get pip for Python 3.
Users who run <code>pytest</code> directly will get Python 3 pytest
for Python 3.
...

Scripts with ambiguous Python shebangs (<code>#!/usr/bin/python</code>
or <code>#!/usr/bin/env python</code>) will be executed by Python 3 by
default.

If users need the <code>python</code> command or the
<code>#!/usr/bin/env python</code> shebang to run Python 2, they can
easily do that by:

ln -s /usr/bin/python2 ~/.local/bin/python

Similarly, sysadmins can do that system-wide:

ln -s /usr/bin/python2 /usr/local/bin/python

If users don't want the python command at all, they can <code>dnf
remove python-unversioned-command</code>.

== Dependencies ==
Most packages that provide Python version-specific functionality will
be affected. However, the Change owners include proven packagers and
they maintain python-rpm-macros, so most will not need packager
attention.

We depend on the Fedora mass rebuild to adjust macro-generated package provides.

(Also, the PEP 394 update could be delayed, or even rejected/changed.
Even if that happens, this Change will not be affected: Fedora would
depart from following the upstream recommendations.)

There are currently (2019-06-25)
[https://fedora.portingdb.xyz/namingpolicy/ 90 packages left with
ambiguous Python requires]. Those need to be adapted. Lot of them
unfortunately Fail to Build From Source and will be retired before the
Fedora 31 branching.

== Contingency Plan ==
* Contingency mechanism: revert the changes, mass rebuild packages
with the original <code>%python_provide</code> macro
* Contingency deadline: beta freeze
* Blocks release? No
* Blocks product? No

== Documentation ==
* Updated Packaging guidelines
* This Change page

== Release Notes ==
TBD. Check User Experience section.

Comments

Re: Fedora 31 System-Wide Change proposal: Python means Python3

By Adam Williamson at 06/26/2019 - 14:07

On Wed, 2019-06-26 at 13:57 -0400, Ben Cotton wrote:
Oh, man. I thought we'd decided against this in the past? I'm worried
about the cost/benefit ratio on such a change.

Re: Fedora 31 System-Wide Change proposal: Python means Python3

By =?utf-8?q?Jos=C... at 06/26/2019 - 16:13

On Wednesday, 26 June 2019 19.07.27 WEST Adam Williamson wrote:
If this were only related with python I would agree with you.

Yet this also applies to lots of other tools like ipython and probably it
could be excluded from this but programs (a development IDE in this case) like
spyder also have the ***** suffix (I am not used to curse but I think that I
could use an exception for this).

It is really, really, annoying to have to type/remember an extra suffix when
there are no other options available.

So even although you have a point all those cases, to me, look a lot like the
paper cuts they are very but they really hurt. :-)

Regards,