DevHeads.net

Postings by Ben Cooksley

Gitlab Update

Hi all,

This afternoon we completed the move of Gitlab from it's initial
server (which was for the evaluation) to the production server that
will hopefully serve us well for many years to come.

Changes to CI System and Binary Factory

Hi all,

Over the past few days we have completed a series of changes to how
the Binary Factory and CI system operate, with a view to improving
scalability and security of the systems going forward.

As a consequence of these changes, we should now be able to scale
Flatpak builds across to multiple builders, more easily provision
Android signing keys, provide Android builds in our F-Droid testing
repository more rapidly and have the possibility of providing Snap
builds in the future.

Additionally, it will allow us to increase the number of FreeBSD and
Windows builders (from the existing 2 of

Disabling of 'kdav-davitemfetchjob' test in KDav

Hi everyone,

Currently we have an issue where the test 'kdav-davitemfetchjob' within
kdav causes kdeinit5 to be relaunched, as can be seen in the CI
run log below.

<a href="https://build.kde.org/job/Applications/job/kdav/job/kf5-qt5%20SUSEQt5.12/38/console" title="https://build.kde.org/job/Applications/job/kdav/job/kf5-qt5%20SUSEQt5.12/38/console">https://build.kde.org/job/Applications/job/kdav/job/kf5-qt5%20SUSEQt5.12...</a>

This is behaviour which is not permitted within the CI system, as the
newly spawned 'kdeinit5' processes are treated as being launched by
that test by CTest, meaning it will wait for eternity for them to exit
(which as resident daemon processes, they will not do without manual
intervention).

This in turn blocks the job from completing, and blocks

Disabling akonadi test runner infrastructure across all repositories

Hi all,

For some time now the Akonadi Test Runner infrastructure has had
issues on the CI system where it will fail to properly shutdown the
akonadiserver (and associated subprocesses) that it starts up for
tests.

This leads to jobs on the CI system becoming indefinitely stuck as
CTest will wait indefinitely for subprocesses spawned by tests to
exit. This in turn requires manual cleanup.

Tonight jobs for kdepim-runtime and akonadi itself both required
manual assistance to complete.

Disabling of 'testkioarchive' in kio-extras

Hi all,

Currently we have an issue where the test 'testkioarchive' within
kio-extras causes kdeinit5 to be relaunched, as can be seen in the CI
run log below.

<a href="https://build.kde.org/job/Applications/job/kio-extras/job/kf5-qt5%20SUSEQt5.12/40/console" title="https://build.kde.org/job/Applications/job/kio-extras/job/kf5-qt5%20SUSEQt5.12/40/console">https://build.kde.org/job/Applications/job/kio-extras/job/kf5-qt5%20SUSE...</a>

This is behaviour which is not permitted within the CI system, as the
newly spawned 'kdeinit5' processes are treated as being launched by
that test by CTest, meaning it will wait for eternity for them to exit
(which as resident daemon processes, they will not do without manual
intervention).

This in turn blocks the job from completing, and block

General Infrastructure Maintainability: API and EBN

Hi all,

Over the past number of years one of the tasks the Sysadmin team has
worked on has been improving the overall maintainability of our
systems, with a significant number of specialised cronjobs, exceptions
and hidden linkages being eliminated.

That is with one great exception: api.kde.org and ebn.kde.org.

Both of these are suffering from an extreme amount of digital bitrot
and special casing and in general are now in a condition where I
cannot say for certain whether it would be possible to replicate the
setup on a new system without us experiencing some degree of breakage
(some of wh

Sysadmin Load Reduction: Subversion Infrastructure

Hi all,

When KDE committed to performing a migration to Git back in 2010, one
of the things that was agreed at the time was that translators could
remain on Subversion to avoid disrupting their workflows.

This however has led to a certain amount of additional infrastructure
which Sysadmin needs to continue to maintain.

In recognition of the fact that with few exceptions, everything has
now migrated from Subversion aside from Translations, i'd like to
reduce the level of infrastructure supporting our Subversion
repository to the bare minimum necessary.

This would include the shutdown of Web

Sysadmin Load Reduction: Code Related Services

Hi all,

In the category of code related services, Sysadmin currently supports
a wide-ranging number of services (which makes sense given the nature
of the community). Some of these however may no longer be in use or
will be duplicative of other services following the transition to
Gitlab.

In the category of duplicative, we have our two CGit instances at
cgit.kde.org and packaging.neon.kde.org.

Work Branches

Hi all,

Recently we had a discussion (which I think may have ended up spread
over a couple of mailing lists in the end) concerning branches and the
ability to force push to them.

Current policy forbids force pushing to branches except in very
limited circumstances (essentially where it is the only option to fix
a branch)

The discussion ended with two ways forward, but no 100% clear
consensus on which way we want to go forward.

Replacement of notes.kde.org

Good morning everyone,

As previously discussed on the KDE Community mailing list, we've now
retired notes.kde.org from service in favour of the Nextcloud realtime
editor and it's associated functionality.

To assist in making this functionality as easy to work with as
possible, Sysadmin has provisioned a community shared folder to host
these documents.

Binary Factory & CI System Maintenance

Hi all,

As part of changes to help improve the capacity, capability and
security of the Binary Factory and CI system, i'm going to be
performing a rebuild of several of the physical hosts responsible for
performing builds on the CI system.

Following these changes, signing operations for Flatpak and Android
builds will have been shifted away from the machines responsible for
handling the actual builds, and the system will have an additional
machine added to handle builds.

This will also pave the way for the build process for Snaps being
brought to the Binary Factory (although there are other

Retirement of notes.kde.org

Good evening all,

Currently we're in the situation where the software we use to run
notes.kde.org is both difficult to maintain, as well as support (for
things like document confidentiality for various groups within KDE).

We'd therefore like to retire the service, replacing it with the
Realtime Text Editor within Nextcloud (share.kde.org).

Eliminating the last CI Failures

Hi all,

For a while now we've had a couple of projects which have been
persistently failing on all platforms, which it would be nice to get
fixed.

If someone would like to take a look into one of the following, it
would be appreciated, and would help to eliminate the last couple of
items from the Failing tab at <a href="https://build.kde.org/" title="https://build.kde.org/">https://build.kde.org/</a>

- Kajongg: Fails with a CMake error due to the build path getting
wrapped up in part of the target name.

Issues with Jenkins Builds

Hi all,

Recently we were affected by a regression within Jenkins, the effect
of which meant that in some circumstances builds would not be
triggered when new commits were introduced.

In addition, the views showing the list of all builds would also not
show the last successfully completed build in some circumstances.

This fault has now been worked around on build.kde.org, and rebuilds
have been initiated of a number of jobs which have been identified as
either affected or potentially affected, which unfortunately will take
the better part of the next day or so for the system to process.

For

Downtime Announcement: Transactional Email Infrastructure

Hi all,

We've been informed by one of our hosting providers that they need to
schedule some downtime in order to allow for a server move.

This will take place this coming Monday, from approximately 4pm UTC
for up to 4 hours.

As a result, transactional email services will be unavailable during
this time meaning email notifications from Bugzilla, Phabricator,
Gitlab, Jenkins and other services hosted on KDE Infrastructure will
be held and delivered once normal service has resumed.

IRC Bots supported by email notifications, including both commits and
bugs may also be impacted and will catchup

CI system maintainability

Hi all,

We currently have a rather substantial issue, in that the CI system
has been once again left in a position where it isn't possible to make
any changes to the system.

This means we can't update to newer versions of packages, add new
packages or correct for binary incompatible changes which periodically
get introduced to non-Frameworks.

This issue has arisen because currently we have a recurring failure to
build from source, within KDE PIM. Specifically, KContacts fails due
to broken CMake logic.

Concluding the Gitlab Discussion

Hi all,

Over the past few weeks we've had a discussion on whether we'd like to
migrate from Phabricator to Gitlab, for handling both our code reviews
as well as internal tasks (user facing bug reports are explicitly out
of context at this time)

Based on the comments the overall consensus seems to be at this stage
to favour switching to Gitlab.

This however is subject to a caveat around multiple task boards, which
would be needed for larger projects to effectively coordinate amongst
the various sub-projects.

As part of the transition we will also arrange for the email interface
to be enable

Gitlab Evaluation & Migration

Hi all,

Over the past few months a small group of KDE projects have been
evaluating Gitlab as a replacement for Phabricator, and we’ve now
reached the point where we’re ready to have a community wide
discussion regarding whether we’d like to migrate to Gitlab.

Upcoming Service Downtime

Hi all,

Sysadmin is currently in the process of planning server migrations for
a large number of services which have significant visibility, and
which are often substantially involved in the release process for
projects as well as general community operation.

As it may take several hours to perform these migrations, we are
asking people to please let us know when they are planning to do
things for the next month or so to allow us to schedule the migrations
accordingly.

Bugzilla Server Move

Hi all,

We've just completed the migration of Bugzilla from a previous system
to a newer system. While this hasn't changed Bugzilla itself much, it
did involve an update from Ubuntu 14.04 to Ubuntu 18.04.

From our testing everything appears to be operating correctly.

Should anyone see anything unusual or broken, please let Sysadmin know.

Thanks,
Ben

CI system disruption

Hi all,

Due to a massive translation change which took place in the last 24
hours, the CI system is currently in the process of rebuilding the
whole of KDE for all the platforms it covers (both branch sets)

This is a process that has already been underway for several hours,
and at this stage is expected to take several more. Normal CI service
will unfortunately be unavailable until this completes.

Unfortunately, as a matter of bad timing, the network card in one of
the main CI builders has also glitched and as a consequence is now
intermittent from time to time.

Transitioning CI builds of all non-Frameworks from Qt 5.9

Hi all,

I've been informed by the PIM developers that they would like to bump
the dependency of the Qt version they use, from Qt 5.9 which it's on
currently, to Qt 5.10.

As a consequence, due to many KDE projects using PIM libraries, their
dependency on Qt will also be effectively bumped.

Downtime announcement: www.kde.org

Hi all,

In order to allow for hardware maintenance, one of our physical
hardware hosts will need to be shutdown for a few hours on Monday.
This downtime will commence around 9:30am UTC and may take several
hours.

During this time a number of sites will be inaccessible, including:
- <a href="http://www.kde.org" title="www.kde.org">www.kde.org</a>
- autoconfig.kde.org
- docs.kde.org
- ev.kde.org
- freebsd.kde.org
- hig.kde.org
- kdesrc-build.kde.org
- neon.kde.org
- releases.neon.kde.org
- networkcheck.kde.org
- planet.kde.org

Other websites within KDE.org that are dependent on resources hosted
on those sites may also experience delayed loading

Emergency downtime notice

Hi all,

We've just received notice that both disks in the server hosting
Phabricator, along with our Git and Subversion repositories have
failed their SMART self-assessment tests.

I have now requested replacement of one of those disks, and once the
array has been rebuilt will request the second be replaced.

To preserve our data in the meantime I have shutdown the container
hosting Phabricator/Git/Subversion, so they will be unavailable for
the next few hours.

My apologies for the inconvenience.

Regards,
Ben Cooksley
KDE Sysadmin

CI System Reorganisation

Hi all,

As a followup to my earlier mail sent about 3 weeks ago, i've now
transitioned all builds on the CI system over to the folder layout
previously described.

Builds can now be found in folders following the following structure
on Jenkins:
<Product> / <Project / Repository>

For those with access to run DSL Scripts: Please do not run them.
While I have completed the necessary adaptions to them, I have yet to
test them and would like to have a full backup of the current state of
Jenkins before I proceed with doing so (due to the colossal number of
files and data involved, this takes quite

Upcoming reorganisation of the CI system

Hi all,

Currently CI jobs are all created within a flat namespace, meaning
that it is quite difficult to view the overall status of an individual
project. Additionally, it creates the issue that the main default view
can take a significant amount of time to load.

To resolve this we intend to shift everything within Folders in
Jenkins.

Mail system switchover complete

Hi everyone,

I'm happy to announce that our transition from our old mail system
(Postbox) to our new mail system (Letterbox) has been made
successfully.

The transition was made with minimal impact on service as far as we
can tell, aside from a slight delay in mail delivery as the DNS change
propagated. At this time all major providers appear to have picked up
this change and are now delivering to Letterbox successfully.

As part of this changeover, we started from scratch with our Bayesian
database.

Upcoming change to mail infrastructure

Hi all,

We've recently completed configuration of a new mail server which will
be replacing the current system which handles kde.org mail.

Binary Factory & Windows CI builds temporarily offline

Hi everyone,

Due to some issues with MSVC we are currently encountering, the Binary
Factory is currently unable to successfully complete Windows builds.
As part of diagnosing these, reinstallation of MSVC has been required,
which has required Windows CI builds to be disabled temporarily.

It is likely that CI coverage will be restored sometime in the next 24
hours, however Binary Factory capability will take longer.

Unfortunately due to the nature of these failures it is possible we
will not be able to restore full service until Microsoft releases the
3rd preview release of Visual Studio 15.

Windows CI KO until further notice

Hi all,

Due to a regression of unknown origin which results in meinproc being
unusable, all Windows builds which are dependent on kdoctools are
currently KO on Windows.

This breakage was introduced sometime shortly after the new builders
became operational (but was working at that time).

Until someone has the time to investigate this, they'll remain broken
i'm afraid.

Regards,
Ben