DevHeads.net

are the ppc64le builders healthy?

I built the latest ceph-14 (14.2.2) on rawhide successfully two days ago.

Two different builds on f30 built or are building fine on x86_64, i686, and
aarch64, but failed with different errors on ppc64le at different places in
the build. One looks like it ran out of space in the file system. The
other may have been OOM killed (?).

<a href="https://kojipkgs.fedoraproject.org//work/tasks/2448/36422448/build.log" title="https://kojipkgs.fedoraproject.org//work/tasks/2448/36422448/build.log">https://kojipkgs.fedoraproject.org//work/tasks/2448/36422448/build.log</a>

<a href="https://kojipkgs.fedoraproject.org//work/tasks/4819/36444819/build.log" title="https://kojipkgs.fedoraproject.org//work/tasks/4819/36444819/build.log">https://kojipkgs.fedoraproject.org//work/tasks/4819/36444819/build.log</a>

Thanks,

Comments

Re: are the ppc64le builders healthy?

By Jason L Tibbitts III at 07/23/2019 - 14:36

KK> I built the latest ceph-14 (14.2.2) on rawhide successfully two days
KK> ago. Two different builds on f30 built or are building fine on
KK> x86_64, i686, and aarch64, but failed with different errors on
KK> ppc64le at different places in the build. One looks like it ran out
KK> of space in the file system. The other may have been OOM killed (?).

There was just a bit of talk about this in IRC. The issue seems to be
that the CPU count of the PPC64le builders was bumped from 4 to 12, but
the amount of memory was unchanged at 10GB RAM/2GB swap. This could
potentially cause resource exhaustion.

Seems they've now been bumped to 22GB of RAM, which should help with OOM
issues but probably not with disk space issues.

- J<

Re: are the ppc64le builders healthy?

By Dan =?utf-8?B?x... at 07/26/2019 - 03:02

Jason L Tibbitts III < ... at math dot uh.edu> writes:

Although not directly related, but I thought I'll throw this in:
openSUSE has a handy package called memory-constraints which features a
single macro: %limit_build.

You give this macro the amount of RAM you expect
make/ninja/$my_build_cmd to consume at peak per thread and it sets the
number of jobs via overriding _smp_mflags so that you won't OOM.

Source:
<a href="https://build.opensuse.org/package/view_file/network:chromium/memory-constraints/memory-constraints.macros?expand=1" title="https://build.opensuse.org/package/view_file/network:chromium/memory-constraints/memory-constraints.macros?expand=1">https://build.opensuse.org/package/view_file/network:chromium/memory-con...</a>

That could be useful for arches which have builders with a lot of cores
but not a lot of RAM or builds which consume excessive amounts.

Cheers,

Dan

Re: are the ppc64le builders healthy?

By Dan =?ISO-8859-... at 07/26/2019 - 03:21

On Fri, 26 Jul 2019 09:02:57 +0200

long time ago I filed a bug/RFE to compute the "-j" value from the
number of cpus and available memory, still waiting on me to actually
implement it ;-)

Dan

Re: are the ppc64le builders healthy?

By Tom Stellard at 07/24/2019 - 18:53

On 07/23/2019 11:36 AM, Jason L Tibbitts III wrote:
Are these updated builders only used for f30? Because I'm still getting
builders with 4 CPU/ 10GB RAM/2GB swap on rawhide. For example:
<a href="https://koji.fedoraproject.org/koji/taskinfo?taskID=36476090" title="https://koji.fedoraproject.org/koji/taskinfo?taskID=36476090">https://koji.fedoraproject.org/koji/taskinfo?taskID=36476090</a>

-Tom

Re: are the ppc64le builders healthy?

By Jason L Tibbitts III at 07/24/2019 - 19:38

TS> Are these updated builders only used for f30?

It appears that there are 29 PPC64le builders configured currently:
<a href="https://koji.fedoraproject.org/koji/hosts?start=80&amp;state=enabled&amp;order=name" title="https://koji.fedoraproject.org/koji/hosts?start=80&amp;state=enabled&amp;order=name">https://koji.fedoraproject.org/koji/hosts?start=80&amp;state=enabled&amp;order=name</a>

They don't all have the same "capacity" rating.

TS> Because I'm still getting builders with 4 CPU/ 10GB RAM/2GB swap on
TS> rawhide.

I imagine that there is some randomness in play. The build you list ran
on buildvm-ppc64le-21.ppc.fedoraproject.org which has a capacity rating
of 2.0. Some of the builders have a rating of 4.0. (Which I guess
doesn't correspond to the increase in core count, but I don't know how
it's calculated.)

- J<

Re: are the ppc64le builders healthy?

By Kevin Fenzi at 07/24/2019 - 18:58

On 7/24/19 3:53 PM, Tom Stellard wrote:
It's not all ppc64le builders. Only the ones on the power9 virthosts for
now (01-19). I'm planning on redoing the rest (20-29) (which are on
power8 vhosts), but I ran out of time before the mass rebuild. I'll do
them as soon as it's over, likely next week.

kevin

Re: are the ppc64le builders healthy?

By Kevin Fenzi at 07/23/2019 - 18:59

On 7/23/19 11:36 AM, Jason L Tibbitts III wrote:
Right. Please file a ticket or let us know if you hit the disk issue again.

kevin