DevHeads.net

F28 Self Contained Change: Avoid /usr/bin/python in RPM build

= Proposed Self Contained Change: Avoid /usr/bin/python in RPM build =
<a href="https://fedoraproject.org/wiki/Changes/Avoid_usr_bin_python_in_RPM_Build" title="https://fedoraproject.org/wiki/Changes/Avoid_usr_bin_python_in_RPM_Build">https://fedoraproject.org/wiki/Changes/Avoid_usr_bin_python_in_RPM_Build</a>

Change owner(s):
* Petr Viktorin <python-devel at lists.fedoraproject.org>
* Miro Hrončok <python-devel at lists.fedoraproject.org>

Deprecate, and later disable, running /usr/bin/python (as opposed to
/usr/bin/python3 or /usr/bin/python2) during RPM build.
Changes will be driven by Python SIG, but a few packages may fail to
build (with the failure message providing an easy workaround).

== Detailed Description ==
=== Motivation ===

Currently in Fedora (package names, executable names, etc.), python
means Python 2.
We would like to change it to mean Python 3, but to do that, we need
to free it of the current meaning.
This means explicitly using either "python2" or "python3" throughout Fedora.
This is a multi-release effort tracked in Python SIG's "Finalizing
Fedora Switch to Python3" document.
This page describes a very focused subset of it.

Renaming packages (and associated changes to, for example, "Requires:"
directives) is relatively straightforward, but making all
Fedora-provided code avoid /usr/bin/python (as opposed to
/usr/bin/python2) is both harder to achieve and harder to keep track
of.

We would like to start deprecating /usr/bin/python (as opposed to
/usr/bin/python2) at RPM build time.
RPM build is a controlled environment: changes to it will not be felt
by end users, breakages are almost immediately visible, and output of
Koji builds can be nicely tracked and analyzed.

=== Specification ===

Python 2, when called as python or /usr/bin/python at RPM build time
(as identified by the RPM_BUILD_ROOT environment variable), will print
a deprecation warning to stderr. (Any program invoked during build
that invokes /usr/bin/python will cause this warning as well.)

A new Taskotron check will be implemented to look for this warning and
fail if it's found.
We will look at the Taskotron results and work with packagers to
switch to update the affected packages. (We'll look at the results to
determine if we'll use automated pull requests, mass bug filing, or
something else.)

The warning itself may cause some packages to fail to build, for
example if a test relies on exact stderr contents.
For these cases, it will be possible to turn the warning off using an
environment variable, but we ask packagers that use of this workaround
is tracked in Bugzilla. See Opt-Out below.

After all packages that BuildRequire python2 are re-built with this
check passing, python will be switched to fail after printing the
message.

The warning will not be effective if stderr output is hidden. So,
after switching /usr/bin/python to fail, some packages may start
failing to build. We will work with the packagers to fix these. (We'll
look at results from the "warnings phase" to see how proactive we'll
need to be here.)

The warning text will be:

DEPRECATION WARNING: python2 invoked with /usr/bin/python.
Use /usr/bin/python3 or /usr/bin/python2
/usr/bin/python will be removed or switched to Python 3 in the future.
If you cannot make the switch now, please follow instructions at
<a href="https://fedoraproject.org/wiki/Changes/Avoid_usr_bin_python_in_RPM_Build#Quick_Opt-Out" title="https://fedoraproject.org/wiki/Changes/Avoid_usr_bin_python_in_RPM_Build#Quick_Opt-Out">https://fedoraproject.org/wiki/Changes/Avoid_usr_bin_python_in_RPM_Build...</a>

(Using the Wiki link allows us to easily revise the instructions.)

=== Quick Opt-Out ===

In case switching to /usr/bin/python2 or /usr/bin/python3 is
non-trivial, do these two things:

* Set the environment variable
DISABLE_AMBIGUOUS_PYTHON_VERSION_WARNING=1 when calling
/usr/bin/python
* File a Bugzilla, and make it block our tracking bug (XXX - link)

All such bugs will need to be fixed before we make /usr/bin/python fail hard.

== Scope ==
* Proposal owners:
Patch python2, write the Taskotron check, deal with warnings and failures.

* Other developers:
Switch to using /usr/bin/python3 or /usr/bin/python2 explicitly at RPM
build time (with help from Proposal owners if needed). Also tools that
are invoked during build time of other packages that themselves use
/usr/bin/python will need to be fixed.

* Release engineering:
#7257: <a href="https://pagure.io/releng/issue/7257" title="https://pagure.io/releng/issue/7257">https://pagure.io/releng/issue/7257</a>
Note: Depending on the timing, separate targeted rebuilds may be
needed, but we can do these as Proven Packagers, no side-tags are
required

* List of deliverables:
N/A

* Policies and guidelines:
Already existing "packages in Fedora ... MUST call the proper
executable for the needed python major version directly, either
/usr/bin/python2 or /usr/bin/python3 as appropriate" from
Packaging:Python#Multiple_Python_Runtimes

* Trademark approval:
N/A (not needed for this Change)

Comments

Re: F28 Self Contained Change: Avoid /usr/bin/python in RPM buil

By Jason L Tibbitts III at 01/09/2018 - 21:30

JK> Python 2, when called as python or /usr/bin/python at RPM build time
JK> (as identified by the RPM_BUILD_ROOT environment variable), will
JK> print a deprecation warning to stderr. (Any program invoked during
JK> build that invokes /usr/bin/python will cause this warning as well.)

I applaud the effort and think the idea is a good one. However, I'm
vaguely uneasy about hanging this off of the seemingly unrelated
RPM_BUILD_ROOT.

My initial idea was to simply decide on another variable to use such as
AMBIGUOUS_PYTHON_VERSION_WARNING, and then set that in our rpm
configuration. (Most likely in %__build_pre, which is defined by the
macros in the rpm package, not redhat-rpm-config.) But it occurs to me
that in the end this has the same result; you get complaints even if
you're building non-Fedora packages (where our guidelines don't apply).
It's just that the involved variable is slightly more obvious.

I guess if we did that Python would only need to check one environment
variable, but that's not really much of a concern. It would also mean
that "unset RPM_BUILD_ROOT" would never occur to someone as a potential
solution.

In the end I just can't shake the notion that it's bad to have some
random non-python-related environment variable basically breaking
python.

- J<

Re: F28 Self Contained Change: Avoid /usr/bin/python in RPM buil

By Nick Coghlan at 01/09/2018 - 22:16

On 10 January 2018 at 11:30, Jason L Tibbitts III < ... at math dot uh.edu> wrote:
Aye, I think you've hit on the main problem: if this is keyed off
RPM_BUILD_ROOT, then it will impact all RPM builds that use a Fedora
provided Python, even if those builds aren't otherwise required to
abide by Fedora's policy settings.

With a dedicated environment variable instead, that could look something like:

PYTHON_DISALLOW_AMBIGUOUS_VERSION=0 # Status quo
PYTHON_DISALLOW_AMBIGUOUS_VERSION=1 # Hard failure
PYTHON_DISALLOW_AMBIGUOUS_VERSION=warn # Deprecation warning

Then either Koji can take care of setting that (and including it in
the mock configs it generates), or else it can be included in some of
the Fedora specific RPM setup.

Cheers,
Nick.

P.S. Using a dedicated environment variable would have the advantage
of allowing anyone else that *also* wanted to look for and remove
unqualified references to Python 2 to set it.

Re: F28 Self Contained Change: Avoid /usr/bin/python in RPM buil

By =?UTF-8?B?TWlyb... at 01/10/2018 - 04:33

On 10.1.2018 03:16, Nick Coghlan wrote:
This sounds good to me. It didn't occur to me that we can actually set a
dedicated env vars for our builds (which is even better).

This would potentially even make it possible to push this patch to upstream.

Re: F28 Self Contained Change: Avoid /usr/bin/python in RPM buil

By Jason L Tibbitts III at 01/10/2018 - 12:34

MH> This sounds good to me. It didn't occur to me that we can actually
MH> set a dedicated env vars for our builds (which is even better).

Well, doing so requires that we change an rpm macro to export this new
env var in the same place that RPM_BUILD_ROOT is exported. I believe
the thing we need to override is in the rpm package itself, not
redhat-rpm-config, so it does mean asking the maintainers of the rpm
package nicely to add that. It's not technically difficult at all.

I could of course be wrong; it may be possible to do this entirely
within redhat-rpm-config, which is a bit easier to have changed.

- J<

Re: F28 Self Contained Change: Avoid /usr/bin/python in RPM buil

By Zbigniew =?utf-... at 01/10/2018 - 06:25

On Wed, Jan 10, 2018 at 09:33:44AM +0100, Miro Hrončok wrote:
Sounds like a good idea, but it's not entirely clear what "our builds"
means. If it's just koji, then I think this is not enough. We really want
'fedpkg local' and 'fedpkg mockbuild' to emit those warnings too, so that
people can find and fix them easily. And it should not just be possible to
get the warning locally (e.g. by setting a variable from the command line),
but also the default should be the same as in koji, so that the results
of local builds and remote builds match. If this can't be done, the
implementation of this change is going to be harder for packagers.

Zbyszek

Re: F28 Self Contained Change: Avoid /usr/bin/python in RPM buil

By =?UTF-8?B?TWlyb... at 01/10/2018 - 07:06

On 10.1.2018 11:25, Zbigniew Jędrzejewski-Szmek wrote:
Noted. Will think about that, added a note to the change page.

Re: F28 Self Contained Change: Avoid /usr/bin/python in RPM buil

By Petr Viktorin at 01/10/2018 - 05:28

On 01/10/2018 09:33 AM, Miro Hrončok wrote:
+1, let's do that!

Re: F28 Self Contained Change: Avoid /usr/bin/python in RPM buil

By =?UTF-8?B?TWlyb... at 01/10/2018 - 06:43

On 10.1.2018 10:28, Petr Viktorin wrote:
Changed
<a href="https://fedoraproject.org/wiki/Changes/Avoid_usr_bin_python_in_RPM_Build" title="https://fedoraproject.org/wiki/Changes/Avoid_usr_bin_python_in_RPM_Build">https://fedoraproject.org/wiki/Changes/Avoid_usr_bin_python_in_RPM_Build</a>
accordingly.

Re: F28 Self Contained Change: Avoid /usr/bin/python in RPM buil

By Kevin Kofler at 01/09/2018 - 19:56

Jan Kurik wrote:
Are you sure that this is worth the pain? Unsuffixed qmake is still the Qt 3
QMake (not even Qt 4), unsuffixed kde-config is still the KDE 3 kde-config
(not even KDE 4), etc., for backwards compatibility. I think changing the
meaning of "python" is unhelpful and counterproductive. Sure, without this
change, more and more users will eventually end up with not having an
unsuffixed "python" installed at all, but IMHO that's better than silently
changing its meaning.

Kevin Kofler

Re: F28 Self Contained Change: Avoid /usr/bin/python in RPM buil

By =?UTF-8?B?TWlyb... at 01/10/2018 - 04:39

On 10.1.2018 00:56, Kevin Kofler wrote:
Whether we change it or get rid of it is not yet set in stone. However,
Python 2 will be gone one day. And that day, everything will break if we
don't start breaking it now by little chunks.