Due to a HDD swap/replacement, we'll have to shut down a node, proceed
with maintenance, and restart the server

Migration is scheduled for """"Tuesday 9th October, 4:00 pm UTC time"""".
You can convert to local time with $(date -d '2018-10-9 16:00 UTC')

The expected "downtime" is estimated to ~20 minutes.

Affected services:
- some internal services for centos infra
- <a href="https://paste.centos.org" title="https://paste.centos.org">https://paste.centos.org</a>

Thanks for your comprehending and patience.

on behalf of the Infra team,

<paste>

Recently I had to update the existing code running behind
mirrorlist.centos.org (the service that returns you a list of validated
mirrors for yum, see the /etc/yum.repos.d/CentOS*.repo file) as it was
still using the Maxmind GeoIP Legacy country database. As you can
probably know, Maxmind announced that they're discontinuing the Legacy
DB, so that was one reason to update the code.

When we consolidated all CentOS Distro builders in a new centralized
setup, covering all arches (so basically x86_64, i386, ppc64le, ppc64,
aarch64 and armhfp those days), we wanted also to add redundancy where
it was possible to.

The interesting "SecureBoot" corner case came on the table and we had to
find a different way to build the following packages:
- shim (both signed and unsigned)
- grub2
- fwupdate
- kernel

The other reason why we considered rebuilding it is that the cert we
were using has expired :

curl --location --silent
<a href="https://github.com/CentOS/sig-core-SecureBoot/raw/mast" title="https://github.com/CentOS/sig-core-SecureBoot/raw/mast">https://github.com/CentOS/sig-core-SecureBoot/raw/mast</a>

Due to a hardware replacement, we'll have to power down the hypervisor
hosting several VMs, including the ones used for the following impacted
services :

* <a href="https://bugs.centops.org" title="https://bugs.centops.org">https://bugs.centops.org</a>
* <a href="https://status.centos.org" title="https://status.centos.org">https://status.centos.org</a>
* <a href="https://fr.centos.org" title="https://fr.centos.org">https://fr.centos.org</a>

Hardware maintenance is scheduled for """"Thursay May 24th, 12:00 pm UTC
time"""".
You can convert to local time with $(date -d '2018-05-24 12:00 UTC')

Root cause : a HDD in the array used to host the VMs failed, and there
is no hot-swap capabilities, so full "power off" is required to replace
the failed HDD and then start the rebuild operation.

We have a major service disruption with our DevCloud RDO environment
(<a href="https://wiki.centos.org/DevCloud" title="https://wiki.centos.org/DevCloud">https://wiki.centos.org/DevCloud</a>) right now so it's currently unusable.

The underlying storage node (over infiniband) used by the openstack
compute nodes is actually powered off, and in unusable state.

Hi,

As you're all aware, RHEL 7.5 was released yesterday, and at the CentOS
side, the massive rebuild started for all arches (x86_64, i386, ppc64,
ppc64le, aarch64 and armhfp).

We'd like to have your opinion about the following proposal (TBD) :
- we freeze the signing process for now, while we have focus on the
distro rebuild.
- we can also eventually freeze the CBS builders, so that all SIGs are
now waiting for 7.5.1804 content to be available in CBS/koji and so
rebuild against those pkgs

We hope (but can't have any estimate) that for this release, all arches
should be in "sync", so normal

This notification to let you know that we'll migrate our
<a href="https://bugs.centos.org" title="https://bugs.centos.org">https://bugs.centos.org</a> bug tracker service to a different node, and
also updated to a new version of MantisBT (version 2.x)

This migration will give us more possibilities, including soon trying to
get SSO working even on our bug tracker (that part still need to be done
though)

Migration is scheduled for """"Tuesday February 20th, 8:00 am UTC time"""".
You can convert to local time with $(date -d '2018-2-20 8:00 UTC')

The expected "downtime" is estimated to ~10 minutes , time needed to
update/propagate updated dns A/AAAA records + las

Last week I tried to build a pkg for infra, that has all the arches
enabled (so x86_64, ppc64, ppc64le and aarch64)

That package has "BuildRequires: golang >= 1.8" , which is in the base
distro but the --scratch build failed starting with aarch64 and
complaining with
"DEBUG util.py:417: Error: No Package found for golang >= 1.8 "

So, while there is a golang 1.8.3 package in [base], there is still an
old version in [extras] :
golang.aarch64 1.6.3-2.el7
extras

And because koji will not pick automatically the higher ENVR, but rather
a pkg bas

Due to important security updates that we need to apply within the whole
CentOS infra, please be aware that some nodes/services will be
unresponsive during that maintenance window.

As it targets both CentOS 6 and CentOS 7, we'll probably apply those
updates ASAP but we'll not notify for each impacted service, so this
announce will cover all impacted services during the next hours/days.

Thanks for your comprehending and patience.

on behalf of the Infra team,

Hi,

Our monitoring platform informed us that the ppc64{le} builders used in
our Koji environment are actually unreachable (same for the underlying
IBM P8 hypervisor node)
After investigation it seems that it's hardware issue

I'll work with the DC people next monday to see how we can try to fix
the issue and I'll also investigate some options to have at least one
builder per arch asap (so ppc64 and ppc64le)

That means that people building for multiple arches will be impacted as
long as those builders will not be back online (or some equivalent node
that can take those $arch jobs)

I'll keep

<paste>
Package Signing
- SIG chairs should request feedback/insight into the package signing
process --> KB
- sometimes, there is a delay in package signing/sync to mirror.centos.org
- have keys been generated securely (known bugs in package versions that
make less secure keys?)
</paste>

Just a copy/paste from the meeting minutes here, to start a dedicated
thread.
I can't answer that, but I'll let involved people participate in that
thread.

<paste>
sigs would like to use centpkg / lookaside, build direct through git to koji
authentication requirements to accounts.centos.org
Fabian to evaluate git solutions and report back to sig chairs.
mrunge has volunteered to be the "guinea pig" of the new system
</paste>

So let's start the thread, to be sure that all involved people would be
able to comment.
SIGs would like to start building from git, and not from SRPMs they have
to create/upload themselves.

For GIT itself, several options exist :
- using git.centos.org : that would mean SIG would need access to
specific repositories and al

Here are some notes taken from the CERN pre-dojo meeting from last week :

<paste>
Allow SIGs to have separate accounts for build bots
- separate user accounts from "bot" accounts for security reasons
- [proposal] have an email alias (not list) per sig for the bots, like
sig-<bla>@centos.org pointing to the SIG's chair
- [proposal] SIG chair must request or approve email alias requests/
ACO account creation sent to CentOS Board chairman
</paste>

So, (as also discussed yesterday in the CBS meeting -
<a href="https://www.centos.org/minutes/2017/October/centos-devel.2017-10-23-14.01.log.html" title="https://www.centos.org/minutes/2017/October/centos-devel.2017-10-23-14.01.log.html">https://www.centos.org/minutes/2017/October/centos-devel.2017-10-23-14.0...</a>)

The pro

When we started to expose the minutes of the meetings held in
#centos-devel (on irc.freenode.net) on the website some time ago, it was
mandatory for someone from infra team to pull/organize/push the minutes
files to <a href="https://www.centos.org/minutes" title="https://www.centos.org/minutes">https://www.centos.org/minutes</a>.

Thanks to the work that was done by John R Dennison (Bahumbug nick name
on irc) we automated the work behind the scene so that after each
meeting, when a SIG chair terminates the meetbot meeting, the full link
to the meeting minutes will be displayed in #centos-devel and those will
be directly available under https://www.centos.org/minutes/$path

Hi,

Some of you know that we deployed initially an RDO/Openstack cloud in
the CI environment, so that it would be possible to get cloud
instances/VMs instead of bare metal when having to test something.
Such VMs could/can also be used to add multiple jenkins slaves when needed.

The needed work in Duffy wasn't yet merged, so the CI projects couldn't
already request a VM (yet).

With the fact the it was based on Newton (now EOL) and that also we're
slowly adding other arches (like aarch64, ppc64, ppc64le) in CI, we
discussed the best way to add those arches to existing cloud, but that's
not po

It was already discussed several times during CBS meetings [1] but in
the last one held yesterday [2] we decided to enforce the following rules :

- People with active account on <a href="https://accounts.centos.org" title="https://accounts.centos.org">https://accounts.centos.org</a> who had a
TLS cert expiring in the following two weeks (now 3 weeks) will get a
weekly mail notification (already in place for some months now) to
invite them to renew their TLS cert

= New
- People still marked as "active" in ACO but who ignored the weekly
reminder will get a different reminder when their cert expired.

Hi,

As discussed on the list, we wanted to push 7.4.1708 first to CBS/Koji
and at the same time to CI (so that what's built against a specific tree
can be tested against the same set of pkgs in CI)

We'll so make it 7.4.7108 available within CBS in the next hours, and
for *all* arches actually covered within CBS.

That means that we'll switch (before it's pushed to external mirrors and
so be announced) the buildroots to start using the following repositories:

7.4.1708/{os,updates,extras}

That will be so done in parallel for x86_64,aarch64,ppc64,ppc64le .

We're just launching some sanity te

Hi,

Some of you actually use our DevCloud (<a href="https://wiki.centos.org/DevCloud" title="https://wiki.centos.org/DevCloud">https://wiki.centos.org/DevCloud</a>)
as a playground/sandbox environment when willing to test some
deployments/scratch operations.

We want you to notice that we'll completely refresh that environment
next week, so DevCloud itself will be unavailable during the reinstall
phase, but also all VMs currently running in that environment will also
be destroyed (it's a "cloud" and for dev/testing so everything could be
considered ephemeral )

Details of the migration:
- setting up two new dedicated storage boxes (each using SSD disks for
faster storage backend

Hi,

While technically speaking a ConfigManagement SIG was approved a long
time ago
(<a href="https://wiki.centos.org/SpecialInterestGroup/ConfigManagementSIG" title="https://wiki.centos.org/SpecialInterestGroup/ConfigManagementSIG">https://wiki.centos.org/SpecialInterestGroup/ConfigManagementSIG</a>) it
seems that nothing was built at all, and that initial people probably
lost interest in that idea (?)

The interesting part is that other SIGs that were relying on such config
management tools (like puppet or ansible) built it themselves in their
own tags :
- puppet : <a href="https://cbs.centos.org/koji/packageinfo?packageID=390" title="https://cbs.centos.org/koji/packageinfo?packageID=390">https://cbs.centos.org/koji/packageinfo?packageID=390</a>
- ansible : <a href="https://cbs.centos.org/koji/packageinfo?packageID=1947" title="https://cbs.centos.org/koji/packageinfo?packageID=1947">https://cbs.centos.org/koji/packageinfo?packageID=1947</a>

I was myself just the "sponsor" for other community peop

Hi

As discussed in CBS meeting today
(<a href="https://www.centos.org/minutes/2017/may/centos-devel.2017-05-22-14.00.html" title="https://www.centos.org/minutes/2017/may/centos-devel.2017-05-22-14.00.html">https://www.centos.org/minutes/2017/may/centos-devel.2017-05-22-14.00.html</a>),
and also reflected on <a href="https://status.centos.org" title="https://status.centos.org">https://status.centos.org</a> , we currently suffer
from a hardware issue that impacts our connectivity to some nodes,
including the ppc64/ppc64le builders.

That means that SIG people willing to build against/for those arches
will not be able to see their build jobs/tasks running as long as
connectivity isn't restored (and no ETA yet)
Worth noting that those builders are disabled at the koji level
(<a href="https://cbs.centos.org/koji/hosts?state=all&amp;order=name" title="https://cbs.centos.org/koji/hosts?state=all&amp;order=name">https://cbs.centos.org/koji/hosts?state=all&amp;order=name</a>)

We'll

As we announced/released 6.9 today
(<a href="https://lists.centos.org/pipermail/centos-announce/2017-April/022351.html" title="https://lists.centos.org/pipermail/centos-announce/2017-April/022351.html">https://lists.centos.org/pipermail/centos-announce/2017-April/022351.html</a>)
, we reflected that in the internal mirror used by koji/cbs.centos.org
so from now on, all built content will be built against 6.9.

We regenerated also all the metadata from within koji to reflect this.

Should you have issues, feel free to discuss those on this list, or
#centos-devel on irc.freenode.net

Due to some reorganization at the DC/Cage level, we'll have to
shutdown/move/reconfigure a big part of our hosted infra for the
following services :
- cbs.centos.org (Koji)
- accounts.centos.org (auth backend)
- ci.centos.org (jenkins-driven CI environment)

We're working on a plan to minimize the downtime/reconfiguration part,
but at first sight, due to the hardware move of the racks/recabling
parts/etc, the announced downtime will be probably ~48h.

What does that mean ?

Hi,

Last week we enabled https for buildlogs.centos.org and
cloud.centos.org, but we haven't (yet) enforced the redirection.

So both are available over http:// and https://

For buildlogs, as we already have redirection in place (RewriteRules)
for content to the backend (see
<a href="https://lists.centos.org/pipermail/centos-devel/2016-March/014552.html" title="https://lists.centos.org/pipermail/centos-devel/2016-March/014552.html">https://lists.centos.org/pipermail/centos-devel/2016-March/014552.html</a>)
, we can enforce the redirection without any issue (and good news is
that the CDN backend also support https so it will be https end-to-end),
so I'll implement the additional http=>https redirect soon

My only concern is about doing the same for clou

There was a discussion earlier today in #centos-devel about a way to
build live iso images on cbs.centos.org.

Koji seems to permit to do that through "koji spin-livecd"

Documentation is available here : <a href="https://docs.pagure.org/koji/image_build/" title="https://docs.pagure.org/koji/image_build/">https://docs.pagure.org/koji/image_build/</a>

One thing to keep in mind though is that one needs to have livecd-tools
pkg installed, and corresponding to the os distro/version the livecd
will be based on : you need a centos 7 host to build a centos 7 liveCD
and that can't work to build a centos 6 liveCD (see as an example
<a href="https://bugzilla.redhat.com/show_bug.cgi?id=1035276" title="https://bugzilla.redhat.com/show_bug.cgi?id=1035276">https://bugzilla.redhat.com/show_bug.cgi?id=1035276</a>)

Depending on the t

As announced some time ago (
<a href="https://lists.centos.org/pipermail/centos-announce/2016-September/022065.html" title="https://lists.centos.org/pipermail/centos-announce/2016-September/022065.html">https://lists.centos.org/pipermail/centos-announce/2016-September/022065...</a>
) , there was a big hardware migration/DC relocation happening yesterday
that impacted several of our public services.
Here is a small report of that migration, that finally took less time
that previously announced.

* 2016-10-10 - 1pm UTC : we started to preconfigure all changes, and
powered down all the services in correct order
* 2016-10-10 - 2pm => 6pm UTC : hardware was moved/migrated
* 2016-10-10 - 6pm => 8pm UTC : hardware was (in serial order) plugged
back in and ipmi/oob interfac

As announced some time ago (
<a href="https://lists.centos.org/pipermail/centos-announce/2016-September/022065.html" title="https://lists.centos.org/pipermail/centos-announce/2016-September/022065.html">https://lists.centos.org/pipermail/centos-announce/2016-September/022065...</a>
) , there was a big hardware migration/DC relocation happening yesterday
that impacted several of our public services.
Here is a small report of that migration, that finally took less time
that previously announced.

* 2016-10-10 - 3pm UTC : we started to preconfigure all changes, and
powered down all the services in correct order
* 2016-10-10 - 3pm UTC
* 2016-10-10 - 3pm UTC
* 2016-10-10 - 3pm UTC
* 2016-10-10 - 3pm UTC

We have to upgrade our current MantisBT instance for the CentOS bug
tracker service (aka <a href="https://bugs.centos.org" title="https://bugs.centos.org">https://bugs.centos.org</a>)

Migration is scheduled for """"Friday october 7th, 12:15 pm UTC time"""".
You can convert to local time with $(date -d '2016-10-7 12:15 UTC'')

The expected "downtime" is estimated to ~15 minutes , time needed to
update MantisBT code, run the mysql schema update, and then put the node
back in production mode.

Thanks for your comprehending and patience.

on behalf of the Infra team,

As pre-announced earlier this year
(<a href="https://lists.centos.org/pipermail/centos-devel/2016-May/014792.html" title="https://lists.centos.org/pipermail/centos-devel/2016-May/014792.html">https://lists.centos.org/pipermail/centos-devel/2016-May/014792.html</a>)
we'll have to move a part of our existing hardware to a new DC.
That means that the following public services will be powered off and
unreachable :

- <a href="https://cbs.centos.org" title="https://cbs.centos.org">https://cbs.centos.org</a> (Koji build farm front end and also
builders/storage nodes behind)
- <a href="https://accounts.centos.org" title="https://accounts.centos.org">https://accounts.centos.org</a> (auth backend)
- <a href="https://ci.centos.org" title="https://ci.centos.org">https://ci.centos.org</a> (jenkins-driven CI environment and all nodes in
that dedicated/isolated environment)

What does that mean ?
- Nobody from the SIGs (<a href="https://wiki.centos.org/SpecialInterestGroup" title="https://wiki.centos.org/SpecialInterestGroup">https://wiki.centos.org/SpecialInterestGroup</a>)
will b

We'll have to shutdown our DevCloud [https://wiki.centos.org/DevCloud]
for maintenance reasons :
- upgrade from gluster 3.6.x to gluster 3.8.x
- change the gluster bricks layout for better performances (adding a
second disk in each node and using underlying striped LV for each brick
in the gluster setup)
- moving the files between gluster volumes

Important notice : if you are a CentOS contributor/developer having some
Virtual Machines in DevCloud, ensure that everything is correctly setup
on your side, as we'll only (from an hypervisor PoV) issue acpi shutdown
to the (still running) VMs.
So i

I just had a look at CBS and was wondering how one SIG (so not
ConfigMgmt SIG specific, but let's use that as an example) can interact
with other SIGs.

One example is Ansible : it seems some other SIGs are relying on it and
so actually the ConfigMgmgt SIG isn't able to build it as it's already
built with the same ENVR but in a different target/tag :
<a href="https://cbs.centos.org/koji/packageinfo?packageID=1947" title="https://cbs.centos.org/koji/packageinfo?packageID=1947">https://cbs.centos.org/koji/packageinfo?packageID=1947</a>

What would be the option for this ?
Actually the direct option is for the ConfigMgmt SIG to just tag that
build (for example for ansible-2.1.0.0-1.el7) so that it will appear in
the correct