DevHeads.net

s390x: hanging koji build

Hi,

I'm trying to create a new build borgbackup for rawhide/F30 (borgbackup is
currently broken on F30).

However it seems like the s390x build just hangs:
<a href="https://koji.fedoraproject.org/koji/taskinfo?taskID=34771060" title="https://koji.fedoraproject.org/koji/taskinfo?taskID=34771060">https://koji.fedoraproject.org/koji/taskinfo?taskID=34771060</a>

The problem seems to be reproducible as a previous Koji scratch build for F30
has the same problem: <a href="https://koji.fedoraproject.org/koji/taskinfo?taskID=34754384" title="https://koji.fedoraproject.org/koji/taskinfo?taskID=34754384">https://koji.fedoraproject.org/koji/taskinfo?taskID=34754384</a>

Any idea how to fix this?

Felix

Comments

Re: s390x: hanging koji build

By Felix Schwarz at 05/16/2019 - 14:36

I'd like to thank everyone who contributed time + provided guidance on this
issue. I just build a working version of borgbackup and which is on its way to
the F30 testing repository.

Re: s390x: hanging koji build

By Jerry James at 05/16/2019 - 21:45

On Thu, May 16, 2019 at 12:37 PM Felix Schwarz
<felix. ... at oss dot schwarz.eu> wrote:
Great news! I really like the spirit of helpfulness and collaboration
that the Fedora community has. It has kept me hanging around lo these
many years. :-)

Regards,

Re: s390x: hanging koji build

By Jerry James at 05/11/2019 - 18:25

On Sat, May 11, 2019 at 2:42 AM Felix Schwarz
<felix. ... at oss dot schwarz.eu> wrote:
Execution is getting stuck in this test:

build/lib.linux-s390x-3.7/borg/testsuite/archiver.py::RemoteArchiverTestCase::test_extract_hardlinks

The outer while loop in call_many(), src/borg/remote.py, lines 769
through 894, never exits. On the first iteration through the loop, we
have:

self = <RemoteRepository ssh://__testsuite__/tmp/tmp45ii70au/repository>
self.dictFormat = False
self.msgid = 0
self.preload_ids = []
self.shutdown_time = None
self.to_send = b''
cmd = 'negotiate'
calls = [{'client_data': {b'client_version': (1, 1, 9, -1)}}]
wait = True
is_preloaded = False
async_wait = True

Here is a backtrace from inside the loop:

Python backtrace:
File "/builddir/build/BUILD/borgbackup-1.1.9/build/lib.linux-s390x-3.7/borg/remote.py",
line 876, in call_many
if not self.to_send and self.preload_ids:
File "/builddir/build/BUILD/borgbackup-1.1.9/build/lib.linux-s390x-3.7/borg/remote.py",
line 938, in get_many
for resp in self.call_many('get', [{'id': id} for id in ids],
is_preloaded=is_preloaded):
File "/builddir/build/BUILD/borgbackup-1.1.9/build/lib.linux-s390x-3.7/borg/archive.py",
line 222, in fetch_many
for id_, data in zip(ids, self.repository.get_many(ids,
is_preloaded=is_preloaded)):
File "/builddir/build/BUILD/borgbackup-1.1.9/build/lib.linux-s390x-3.7/borg/archive.py",
line 608, in extract_item
for data in self.pipeline.fetch_many(ids, is_preloaded=True):
File "/builddir/build/BUILD/borgbackup-1.1.9/build/lib.linux-s390x-3.7/borg/archiver.py",
line 765, in do_extract
stripped_components=strip_components, original_path=orig_path, pi=pi)
File "/builddir/build/BUILD/borgbackup-1.1.9/build/lib.linux-s390x-3.7/borg/archiver.py",
line 167, in wrapper
return method(self, args, repository=repository,
manifest=manifest, key=key, archive=archive, **kwargs)
File "/builddir/build/BUILD/borgbackup-1.1.9/build/lib.linux-s390x-3.7/borg/archiver.py",
line 154, in wrapper
return method(self, args, repository=repository, **kwargs)
File "/builddir/build/BUILD/borgbackup-1.1.9/build/lib.linux-s390x-3.7/borg/archiver.py",
line 4387, in run
return set_ec(func(args))
File "/builddir/build/BUILD/borgbackup-1.1.9/build/lib.linux-s390x-3.7/borg/testsuite/archiver.py",
line 109, in exec_cmd
ret = archiver.run(args)
File "/builddir/build/BUILD/borgbackup-1.1.9/build/lib.linux-s390x-3.7/borg/testsuite/archiver.py",
line 289, in cmd
ret, output = exec_cmd(*args, fork=fork, exe=self.EXE,
archiver=self.archiver, **kw)
File "/builddir/build/BUILD/borgbackup-1.1.9/build/lib.linux-s390x-3.7/borg/testsuite/archiver.py",
line 829, in test_extract_hardlinks
self.cmd('extract', self.repository_location + '::test',
'--strip-components', '2')
File "/usr/lib64/python3.7/unittest/case.py", line 615, in run
testMethod()
File "/usr/lib64/python3.7/unittest/case.py", line 663, in __call__
return self.run(*args, **kwds)
File "/usr/lib/python3.7/site-packages/_pytest/unittest.py", line
221, in runtest
self._testcase(result=self)
File "/usr/lib/python3.7/site-packages/_pytest/runner.py", line 123,
in pytest_runtest_call
item.runtest()
File "/usr/lib/python3.7/site-packages/pluggy/callers.py", line 187,
in _multicall
res = hook_impl.function(*args)
File "/usr/lib/python3.7/site-packages/pluggy/manager.py", line 62,
in <lambda>
firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
File "/usr/lib/python3.7/site-packages/pluggy/manager.py", line 68,
in _hookexec
return self._inner_hookexec(hook, methods, kwargs)
File "/usr/lib/python3.7/site-packages/pluggy/hooks.py", line 289, in __call__
return self._hookexec(self, self.get_hookimpls(), kwargs)
File "/usr/lib/python3.7/site-packages/_pytest/runner.py", line 198,
in <lambda>
lambda: ihook(item=item, **kwds), when=when, reraise=reraise
File "/usr/lib/python3.7/site-packages/_pytest/runner.py", line 226,
in from_call
result = func()
File "/usr/lib/python3.7/site-packages/_pytest/runner.py", line 198,
in call_runtest_hook
lambda: ihook(item=item, **kwds), when=when, reraise=reraise
File "/usr/lib/python3.7/site-packages/_pytest/runner.py", line 173,
in call_and_report
call = call_runtest_hook(item, when, **kwds)
File "/usr/lib/python3.7/site-packages/_pytest/runner.py", line 93,
in runtestprotocol
reports.append(call_and_report(item, "call", log))
File "/usr/lib/python3.7/site-packages/_pytest/runner.py", line 78,
in pytest_runtest_protocol
runtestprotocol(item, nextitem=nextitem)
--Type <RET> for more, q to quit, c to continue without paging--c
File "/usr/lib/python3.7/site-packages/pluggy/callers.py", line 187,
in _multicall
res = hook_impl.function(*args)
File "/usr/lib/python3.7/site-packages/pluggy/manager.py", line 62,
in <lambda>
firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
File "/usr/lib/python3.7/site-packages/pluggy/manager.py", line 68,
in _hookexec
return self._inner_hookexec(hook, methods, kwargs)
File "/usr/lib/python3.7/site-packages/pluggy/hooks.py", line 289, in __call__
return self._hookexec(self, self.get_hookimpls(), kwargs)
File "/usr/lib/python3.7/site-packages/_pytest/main.py", line 270,
in pytest_runtestloop
item.config.hook.pytest_runtest_protocol(item=item, nextitem=nextitem)
File "/usr/lib/python3.7/site-packages/pluggy/callers.py", line 187,
in _multicall
res = hook_impl.function(*args)
File "/usr/lib/python3.7/site-packages/pluggy/manager.py", line 62,
in <lambda>
firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
File "/usr/lib/python3.7/site-packages/pluggy/manager.py", line 68,
in _hookexec
return self._inner_hookexec(hook, methods, kwargs)
File "/usr/lib/python3.7/site-packages/pluggy/hooks.py", line 289, in __call__
return self._hookexec(self, self.get_hookimpls(), kwargs)
File "/usr/lib/python3.7/site-packages/_pytest/main.py", line 249, in _main
config.hook.pytest_runtestloop(session=session)
File "/usr/lib/python3.7/site-packages/_pytest/main.py", line 209,
in wrap_session
session.exitstatus = doit(config, session) or 0
File "/usr/lib/python3.7/site-packages/_pytest/main.py", line 242,
in pytest_cmdline_main
return wrap_session(config, _main)
File "/usr/lib/python3.7/site-packages/pluggy/callers.py", line 187,
in _multicall
res = hook_impl.function(*args)
File "/usr/lib/python3.7/site-packages/pluggy/manager.py", line 62,
in <lambda>
firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
File "/usr/lib/python3.7/site-packages/pluggy/manager.py", line 68,
in _hookexec
return self._inner_hookexec(hook, methods, kwargs)
File "/usr/lib/python3.7/site-packages/pluggy/hooks.py", line 289, in __call__
return self._hookexec(self, self.get_hookimpls(), kwargs)
File "/usr/lib/python3.7/site-packages/_pytest/config/__init__.py",
line 79, in main
return config.hook.pytest_cmdline_main(config=config)
File "/usr/bin/py.test-3", line 11, in <module>
load_entry_point('pytest==4.4.1', 'console_scripts', 'py.test')()

Hopefully that helps. Regards,

Re: s390x: hanging koji build

By Felix Schwarz at 05/12/2019 - 15:25

Am 12.05.19 um 00:25 schrieb Jerry James:
I'm afraid I still don't have a clue but I'd like to thank everyone who spent
time on this. As you can imagine I'm not eager spending time to debug some
failing test on a niche platform - but as my initial email led three devs to
spend some time I'll try a bit harder.

I think I saw the same test failing on x86_64 in a specific configuration.
Maybe I can get some insights from investigating that test failure on a more
common platform.

thanks again,
Felix

Re: s390x: hanging koji build

By Dan =?ISO-8859-... at 05/11/2019 - 04:56

On Sat, 11 May 2019 10:41:37 +0200

looks like a bug (data endianness?) in borgbackup or its test-suite. I
can look more or give you access to a s390x machine for debugging after
the weekend. Please ping me on Monday :-)

Dan

Re: s390x: hanging koji build

By Felix Schwarz at 05/11/2019 - 15:47

Am 11.05.19 um 10:56 schrieb Dan HorĂ¡k:
That would be great - I don't have any experience with s390x so I'm glad for
any help.

Just to note: The same version of borgbackup (1.1.9) did complete successfully
on F29 which is a bit puzzling (but of course I know that there is a million
things which can trigger subtle bugs in C code):
<a href="https://koji.fedoraproject.org/koji/buildinfo?buildID=1228236" title="https://koji.fedoraproject.org/koji/buildinfo?buildID=1228236">https://koji.fedoraproject.org/koji/buildinfo?buildID=1228236</a>

Also there was some debugging by Thomas Moschny
<a href="https://bugzilla.redhat.com/show_bug.cgi?id=1669083#c38" title="https://bugzilla.redhat.com/show_bug.cgi?id=1669083#c38">https://bugzilla.redhat.com/show_bug.cgi?id=1669083#c38</a>

Will do :-)

Felix