DevHeads.net

About making noarch package arch specific, when contents differ.

I had this build failure:

Package: sagemath-6.5-7.fc22
Status: failed
Built by: pcpa
ID: 672175
Started: Sat, 25 Jul 2015 22:52:10 UTC
Finished: Sun, 26 Jul 2015 07:57:28 UTC

Closed tasks:
mismatch when analyzing sagemath-doc-en-6.5-7.fc22.noarch.rpm, rpmdiff
output was:
error: cannot open Packages index using db5 - Permission denied (13)
error: cannot open Packages database in /var/lib/rpm
error: cannot open Packages database in /var/lib/rpm
added /usr/share/doc/sagemath/output/html/en/reference/hecke
[...]

The interesting part, is that the above build generated
packages, but ended up with failed state.
After that, I checked contents of the armv7hl and x86_64
trees, and noticed that they are, indeed not identical.
The way documentation is generated with sphinx, should
be part of the cause (when it pickle/unpicke python states, etc),
and sometimes it even adds the location of a file to the docs,
e.g. telling where source is located, causes diffs, from
/usr/lib/python2.7/... vs /usr/lib64/python2.7/...

Should I make the doc packages arch specific?

Thanks,
Paulo

Comments

Re: About making noarch package arch specific, when contents dif

By Florian Weimer at 07/28/2015 - 03:58

No, this is not a reason to make them arch-specific. A lot of packages
give different results when built twice in a row, on the *same*
architecture.

There is an effort under way to change that, called “reproducible
builds”. The hard part is any reproducibility at all, identical noarch
builds across architectures are likely just some additional work on top
of it.

Re: About making noarch package arch specific, when contents dif

By Ralf Corsepius at 07/28/2015 - 11:59

On 07/28/2015 10:58 AM, Florian Weimer wrote:
Actually, reproducable builds wrt. docs have been subject to Fedora
Packaging since Fedora day #1 and repeatedly have been subject to
discussions of details (e.g. doxygen repeatedly had introduced docs
breakages)

Packages which do not comply to this rule are broken.

Ralf

Re: About making noarch package arch specific, when contents dif

By Florian Weimer at 08/01/2015 - 16:05

On 07/28/2015 06:59 PM, Ralf Corsepius wrote:
Can you provide a citation in the guidelines? As far as I can tell,
javadoc hasn't been patched not to put the build date into the resulting
HTML documentation, so a lot of Java packages are not compliant with the
above.

Re: About making noarch package arch specific, when contents dif

By Reindl Harald at 08/01/2015 - 16:09

Am 01.08.2015 um 23:05 schrieb Florian Weimer:
it's simply not true - there are no reproducable builds in Fedora and
frankly currently very few distributions have them at all

Re: About making noarch package arch specific, when contents dif

By =?ISO-8859-1?Q?... at 07/28/2015 - 09:52

2015-07-28 5:58 GMT-03:00 Florian Weimer < ... at redhat dot com>:
I believe that if there is a check for bit by bit identical noarch
packages, it would also be mandatory some way to tell that
any minor difference is ok and expected, and use the noarch
built on that arch...

Thanks,
Paulo

Re: About making noarch package arch specific, when contents dif

By Nico Kadel-Garcia at 08/01/2015 - 06:59

On Tue, Jul 28, 2015 at 10:52 AM, Paulo César Pereira de Andrade
<paulo.cesar.pereira.de. ... at gmail dot com> wrote:
Please note that *changing* a package from arch to noarch requires
extra attention. The switch of architecture is not normally regarded
as an upgrade path, by default, so you may have to insert an
"Obsoletes" statement to ensure that older versions of the alternative
architecture get upgraded.

Re: About making noarch package arch specific, when contents dif

By Dan Callaghan at 07/27/2015 - 20:34

Excerpts from paulo.cesar.pereira.de.andrade's message of 2015-07-27 00:05 +10:00:
Rather than trying to make Sphinx spit out bitwise-identical output on
every arch (which sounds like fighting a losing battle), could you just
build the doc subpackage on only one arch?

%ifarch x86_64
%package doc
BuildArch: noarch
...
%endif

I think Koji still counts this a regular noarch subpackage and it should
therefore be included in the Fedora trees for all arches.

Re: About making noarch package arch specific, when contents dif

By Peter Robinson at 07/28/2015 - 03:01

On Tue, Jul 28, 2015 at 2:34 AM, Dan Callaghan < ... at redhat dot com> wrote:
Dan,

This is completely NOT appropriate, it breaks on secondary arches
where they then end up with no documentation due to the lack of any
x86_64. Please DO NOT do this and please revert the change on any
packages you might have made this change.

Re: About making noarch package arch specific, when contents dif

By Dan Callaghan at 07/28/2015 - 17:35

Excerpts from Peter Robinson's message of 2015-07-28 18:01 +10:00:
I haven't used that hack on any of my packages but I learnt about it
from the ipxe package which uses it to produce noarch subpackages
containing boot images. I guess ipxe is broken on secondary arches as
well, but it's acceptable for that package because the only alternative
would be to make it x86-only?

Re: About making noarch package arch specific, when contents dif

By Ralf Corsepius at 07/27/2015 - 23:51

On 07/28/2015 03:34 AM, Dan Callaghan wrote:
This is a blatant hack.

Packages builds must be reproducable on any supported architecture.

Ralf

Re: About making noarch package arch specific, when contents dif

By =?ISO-8859-1?Q?... at 07/27/2015 - 22:37

2015-07-27 22:34 GMT-03:00 Dan Callaghan < ... at redhat dot com>:
This looks like a very wise way of handling it. Actually, while debugging
it, I found that the translated documentation was not being properly
generated, and after fixing it, it would take like 3 to 4 times longer
to generate docs, and doc generation was already almost 80% of
the package build time...

In the worst case, it would generate -doc packages only for x86_64,
where most users interested on reading it would be using.

Thanks! I will try this way :)
Paulo

Re: About making noarch package arch specific, when contents dif

By Peter Robinson at 07/28/2015 - 03:03

On Tue, Jul 28, 2015 at 4:37 AM, Paulo César Pereira de Andrade
<paulo.cesar.pereira.de. ... at gmail dot com> wrote:
It's not, it breaks all secondary architectures.

That tells me the process of generating docs is broken, or they're
very good docs and worth the wait!

And won't generate docs for any of the secondary arches which don't
have any x86_64 build capacity, please don't do this.

Re: About making noarch package arch specific, when contents dif

By =?ISO-8859-1?Q?... at 07/28/2015 - 09:57

2015-07-28 5:03 GMT-03:00 Peter Robinson < ... at gmail dot com>:

It is the later. It has its problems of course, but the documentation
is really very good and complete, documenting every single interface.
It is "live", once running the sagemath "notebook", one can modify
the examples, run with different input, etc.

Thanks,
Paulo

Re: About making noarch package arch specific, when contents dif

By =?ISO-8859-1?Q?... at 07/26/2015 - 10:20

On Dom, 2015-07-26 at 11:05 -0300, Paulo César Pereira de Andrade wrote:
I suspect package is using %{_libdir} and libdir can't be used in noarch
packages. %{_libdir} expands to /usr/lib on arm and /usr/lib64/ on
x86_64. This is a general problem when we try translate Debian packages
to Fedora, I don't have time now to explain better but we had some
topics about related subjects in Packaging mailing list ...

Python also have different scriptlets for arch and noarch packages. On
x86_64 systems, we can find some noarch packages in /usr/lib/python2.7/
and python arched in /usr/lib64/python2.7/ .

Re: About making noarch package arch specific, when contents dif

By =?ISO-8859-1?Q?... at 07/26/2015 - 10:50

2015-07-26 12:20 GMT-03:00 Sérgio Basto < ... at serjux dot com>:
A good share of the diffs is documentation is printing addresses of
objects, at the time the documentation was generated, for example:

---8<---
diff -rNu armv7hl/usr/share/doc/sagemath/output/html/en/reference/calculus/sage/gsl/dft.html
x86_64/usr/share/doc/sagemath/output/html/en/reference/calculus/sage/gsl/dft.html
--- armv7hl/usr/share/doc/sagemath/output/html/en/reference/calculus/sage/gsl/dft.html
2015-07-26 12:25:01.039419016 -0300
+++ x86_64/usr/share/doc/sagemath/output/html/en/reference/calculus/sage/gsl/dft.html
2015-07-26 12:25:06.410419222 -0300
@@ -226,7 +226,7 @@

<dl class="method">
<dt id="sage.gsl.dft.IndexedSequence.dft">
-<tt class="descname">dft</tt><big>(</big><em>chi=&lt;function
&lt;lambda&gt; at 0xac336d70&gt;</em><big>)</big><a class="headerlink"
href="#sage.gsl.dft.IndexedSequence.dft" title="Permalink to this
definition">¶</a></dt>
+<tt class="descname">dft</tt><big>(</big><em>chi=&lt;function
&lt;lambda&gt; at 0x7f27a03f4140&gt;</em><big>)</big><a
class="headerlink" href="#sage.gsl.dft.IndexedSequence.dft"
title="Permalink to this definition">¶</a></dt>
<dd><p>A discrete Fourier transform &#8220;over <img class="math"
src="../../_images/math/7bab5d1f8fefbcece570be83647d253afa56e0df.png"
alt="\QQ"/>&#8221; using exact
<img class="math"
src="../../_images/math/9373b89a1028693ea48d96bef05ec06f42c4ba41.png"
alt="N"/>-th roots of unity.</p>
<p>EXAMPLES:</p>
---8<---
Example of the above, "live":
<a href="http://doc.sagemath.org/html/en/reference/calculus/sage/gsl/dft.html#sage.gsl.dft.IndexedSequence.dft" title="http://doc.sagemath.org/html/en/reference/calculus/sage/gsl/dft.html#sage.gsl.dft.IndexedSequence.dft">http://doc.sagemath.org/html/en/reference/calculus/sage/gsl/dft.html#sag...</a>

About %{_libdir}, there are some in this pattern:
---8<---
diff -rNu armv7hl/usr/share/doc/sagemath/output/html/en/reference/todolist.html
x86_64/usr/share/doc/sagemath/output/html/en/reference/todolist.html
--- armv7hl/usr/share/doc/sagemath/output/html/en/reference/todolist.html
2015-07-26 12:25:03.271419101 -0300
+++ x86_64/usr/share/doc/sagemath/output/html/en/reference/todolist.html
2015-07-26 12:25:08.671419308 -0300
@@ -83,36 +83,36 @@
<p class="last">Deprecate this in favor of a method called <img
class="math" src="_images/math/107b545f3b4d727d47ab3a406ddd5b3a92cd2fda.png"
alt="center()"/> once
subalgebras are properly implemented in Sage.</p>
</div>
-<p class="todo-source">(The <a class="reference internal"
href="algebras/sage/algebras/clifford_algebra.html#index-0"><em>original
entry</em></a> is located in
/usr/lib/python2.7/site-packages/sage/algebras/clifford_algebra.py:docstring
of sage.algebras.clifford_algebra.CliffordAlgebra.center_basis, line
12.)</p>
+<p class="todo-source">(The <a class="reference internal"
href="algebras/sage/algebras/clifford_algebra.html#index-0"><em>original
entry</em></a> is located in
/usr/lib64/python2.7/site-packages/sage/algebras/clifford_algebra.py:docstring
of sage.algebras.clifford_algebra.CliffordAlgebra.center_basis, line
12.)</p>
---8<---

Others are in some generated js, after searching the first
difference, it looks like this,
diff at char 255:
armv7hl:
Search.setIndex({envversion:42,terms:{represent:[1,2],all:[5,1,3,6,8,12,13],x3y:13,partial:0,sch:3,joyner:14,illustr:[0,2,4,6,7,8,15],skip:9,lfsr_sequenc:9,hamburg:14,roe:7,month:3,'_k1':0,plot_list:15,short_weierstrass_model:10,'_k2':0,ellipt:13,legal:3,edu:3,follow:
x86_64:
Search.setIndex({envversion:42,terms:{represent:[1,2],all:[5,1,3,6,8,12,13],x3y:13,partial:0,sch:3,joyner:14,illustr:[0,2,4,6,7,8,15],skip:9,lfsr_sequenc:9,hamburg:14,roe:7,month:3,'_k1':0,plot_list:15,short_weierstrass_model:10,'_k2':0,ellipt:13,legal:3,classic:12,edu:3,follow

Thanks,
Paulo