Postings by Ben Cooksley

CI System Reorganisation

Hi all,

As a followup to my earlier mail sent about 3 weeks ago, i've now
transitioned all builds on the CI system over to the folder layout
previously described.

Builds can now be found in folders following the following structure
on Jenkins:
<Product> / <Project / Repository>

For those with access to run DSL Scripts: Please do not run them.
While I have completed the necessary adaptions to them, I have yet to
test them and would like to have a full backup of the current state of
Jenkins before I proceed with doing so (due to the colossal number of
files and data involved, this takes quite

Upcoming reorganisation of the CI system

Hi all,

Currently CI jobs are all created within a flat namespace, meaning
that it is quite difficult to view the overall status of an individual
project. Additionally, it creates the issue that the main default view
can take a significant amount of time to load.

To resolve this we intend to shift everything within Folders in

Mail system switchover complete

Hi everyone,

I'm happy to announce that our transition from our old mail system
(Postbox) to our new mail system (Letterbox) has been made

The transition was made with minimal impact on service as far as we
can tell, aside from a slight delay in mail delivery as the DNS change
propagated. At this time all major providers appear to have picked up
this change and are now delivering to Letterbox successfully.

As part of this changeover, we started from scratch with our Bayesian

Upcoming change to mail infrastructure

Hi all,

We've recently completed configuration of a new mail server which will
be replacing the current system which handles mail.

Binary Factory & Windows CI builds temporarily offline

Hi everyone,

Due to some issues with MSVC we are currently encountering, the Binary
Factory is currently unable to successfully complete Windows builds.
As part of diagnosing these, reinstallation of MSVC has been required,
which has required Windows CI builds to be disabled temporarily.

It is likely that CI coverage will be restored sometime in the next 24
hours, however Binary Factory capability will take longer.

Unfortunately due to the nature of these failures it is possible we
will not be able to restore full service until Microsoft releases the
3rd preview release of Visual Studio 15.

Windows CI KO until further notice

Hi all,

Due to a regression of unknown origin which results in meinproc being
unusable, all Windows builds which are dependent on kdoctools are
currently KO on Windows.

This breakage was introduced sometime shortly after the new builders
became operational (but was working at that time).

Until someone has the time to investigate this, they'll remain broken
i'm afraid.


Upcoming CI changes - service disruption

Hi all,

In order to allow for two replacement physical build hosts to be
rotated in and the old ones to be decommissioned, i'm scheduling some
downtime for both the CI system and the Binary Factory tomorrow.

Assuming all goes well, this downtime should be fairly short. During
the downtime Jenkins will still be accessible, however builds will not
trigger on push as they usually do.

Phabricator mail disruption

Hi all,

Some of you may have noticed that the delivery of Phabricator mail has
been disrupted over the past few hours. This is due to Spamcop
blacklisting the relays used by Mailgun to send our Phabricator email.

List moderators, i'm afraid you will need to approve any mail which
has been automatically held for your approval.

I have now resolved this issue by terminating our use of Spamcop.

Phabricator Notification Mails

Hi all,

It has been brought to my attention that some find that they are
receiving too much mail or other notifications from Phabricator.

Global Dependency and Release Freeze

Hi all,

Due to a snowball of various dependency bumps and other tickets which
have been submitted over the past few days, i'm imposing a global
dependency freeze upon all KDE projects.

In addition to this, due to the workload these also create, I am also
imposing a total release freeze on all Extragear projects. Requests
for tarballs to be released will not be actioned while the freeze is

CI & Binary Factory maintenance

Hi all,

To permit extensive maintenance on the underlying nodes which support
the CI system & Binary Factory, I have temporarily disabled the
mechanism which triggers builds on the CI and have brought the Binary
Factory into a halted state.

During this time builds will not be triggered as they normally would.

Following the maintenance the system will need to perform a complete
rebuild of all Windows based projects, and will also need to catch up
on the builds it has missed. As such it may take up to 24-48 hours
before the system is fully functional again.

Ben Cooksley
KDE Sysadmin

Mirror of Qt being discontinued

Hi all,

As part of a recent review of our systems it has been determined that
the Qt mirror we were hosting was not being used significantly.

Advance Downtime Notification

Hi all,

Over the past 24 hours we have received initial warning notifications
that a disk in our bulk storage server has begun to show signs of

Significant CI Maintenance

Hi all,

As some of you will know, we're currently unable to build our Fedora
image on the CI system, and have effectively been unable to do so for
some time.

Retirement of Reviewboard - Transition to Phabricator

Hi all,

The following is Sysadmin's suggested plan for the retirement of
Reviewboard now that Phabricator is fully up and running for hosting
of code reviews.

Phase 1: Commences September 2: All repositories are closed for
accepting new reviews on Reviewboard. A notice is added to the top of
the main page indicating that reviews should now be done on

Phase 2: Commences September 16: Login to Reviewboard is disabled, and
final backups are taken.

Notice of significant adjustment to KWin

Hi all,

At the request of the KWin maintainer the KWin repository master
branch has been forcibly rewound to c5861b99 (per T6301)

A log of the changes which have been thrown away is attached.

Developers may need to reinstate these changes (assuming they have
been properly reviewed) to ensure no fixes are lost.

Apologies for the inconvenience.

Ben Cooksley
KDE Sysadmin

Next Gen CI Will Be Moving to Production Shortly: Upcoming Changes

Hi everyone,

This is going to be quite a long email, my apologies in advance for that.
If you use the CI system regularly as part of your development flow it
is important you read this email in it's entirety as your action will
probably be required. If you are aware of a list I have missed in the
above, please feel free to forward it there.

As some will have been aware of for some time we currently have a
problem where changing the system base is an incredibly disruptive

Communication with Phabricator upstream

Hi everyone,

Just repeating my last email on this subject as it seems some folks
have missed the previous memo.

All issues with Phabricator should be logged at
<a href="" title=""></a> - not with upstream.

This is being done to avoid duplicate tasks, and to allow us to
communicate on a whole of project basis the things we consider
important. This helps simplify things for upstream.

Ben Cooksley
KDE Sysadmin

Migration of Download Infrastructure: Please check your applications

Hi all,

Sysadmin have now completed the migration of our file distribution
systems, and, to a new server.

Project Releases: Infrastructure Migration

Hi everyone,

To allow for Sysadmin to migrate the primary host for
and to another server (with significantly more disk
space) we'll need to make both of those read only for a period of time
this weekend.

During this period, it will not be possible to add/remove files to
those trees - which means projects will not be able to make releases.

To the knowledge of Sysadmin, this does not affect the release of any
project, however anyone who had plans should please contact us

Ben Cooksley
KDE Sysadmin

Phabricator updated

Hi all,

Over the weekend we've updated our Phabricator instance to the latest
stable version. This has brought in a few changes which you may wish
to check out - including the ability to customise the global home menu
and setup a favourites menu for yourself to access frequently used

As the global items setup by Sysadmin cannot be overridden by users,
i've only setup a very basic set of items for everyone to use.

For those who use Arcanist, now would be a good time for you to ensure
you are running the latest stable version as well.

Communication with Phabricator upstream

Hi all,

To help simplify things for Phabricator upstream, they've asked that
only a couple of people bring things to their attention (including
filing tasks). This helps them with figuring out our priorities as a
project in regards to usage of Phabricator.

Please therefore file all issues / etc you hit regarding Phabricator
on our own instance, tagging them with the project "Phabricator".

At the moment i'm the one liasing with them primarily, although i'll
be looking to bring one or two others on board soonish.


Phabricator: All repositories registered - upcoming workflow changes

Hi everyone,

We've just completed the registration of all mainline repositories
(not including Websites or Sysadmin namespaced ones) on Phabricator.
Thanks go to Luigi Toscano for providing significant assistance with
this process.

From this point forward, communities should be moving away from
Reviewboard to Phabricator for conducting code review.

CI Requirements - Lessons Not Learnt?

Hi all,

It seems that my previous vocal complaints about system level /
serious impact dependency bumps on the CI system have gone completely
unnoticed by (some) members of our Community.

This was demonstrated earlier this week when components of Plasma
bumped their version requirements for XKBCommon and Appstream-Qt -
without even a thought about notifying Sysadmin or checking which
version the CI had, until their builds broke.

Neither of these is easy to fix at this stage, as the system base is
now too old to receive updates such as these.

Replacement of

Hi all,

Sysadmin is currently in the process of testing and evaluating
Collabora Online (CODE) as a replacement for We've
enabled this on

It would be appreciated if people please test it out - anyone who is a
Developer, member of the e.V, or Visual Design Group should be able to
login on Please send your feedback to this thread.

We've already figured out a process for exporting all data from, and will look to do this once initial testing is
finished - ideally this would happen a day or two after New Years.


Scheduled downtime:

Hi all,

I'm scheduling downtime of a few hours for tomorrow (28th December),
from 9am NZDT (UTC+13) for the server This is to allow
for a system upgrade to Ubuntu 16.04 LTS.

Please cleanup your scratch and clone repositories

Hi all,

To help Sysadmin assess how these types of repositories would be used
under Phabricator, it would be appreciated if everyone could please
cleanup the scratch and clone repositories they're no longer using and
have no need to keep.

You can get a list of repositories you have by visiting
<a href="" title=""></a> or by running the below command and filtering
appropriately using your favourite tools.

ssh <a href="mailto: ... at git dot"> ... at git dot</a> info

Once you've determined the repositories to remove, they can be removed
by running the following two commands:

ssh <a href="mailto: ... at git dot"> ... at git dot</a> D unlock scratch/username/reponame

Discontinuing repository tarballs

HI all,

As of late Sysadmin has been assessing what components of our
infrastructure are in active use and should be maintained going

As part of this it has come to our attention that the repository
tarballs offered by our anongit network (tar.gz files of our
repositories essentially) are effectively unused - with only 19 hits
being received by one mirror in the space of a week.

These tarballs are generated by cronjob once per week, and are
effectively a duplicate copy of the repositories which must be held by
each anongit.

Discontinuing Qt mirror

Hi all,

For sometime now, Sysadmin has maintained a mirror of Qt on It has been some time now however since Qt's
infrastructure was hosted on the partly unreliable Gitorious systems,
and therefore the need for the mirror has passed.

Therefore we'd like to discontinue the mirror - in part because
Phabricator does not support directories for repositories, and
requires everything be at top level.

Bugzilla Update: Antispam modification deployed

Hi all,

As some of you will already be aware, Bugzilla has recently been
experiencing some issues with spam - courtesy of the same group that
were previously attacking our Wikis and Forums.

I've now deployed antispam countermeasures on Bugzilla, which should
mitigate many forms of the spam which were previously affecting us.

If you see any instances of legitimate comments being blocked, please
let us know.

Ben Cooksley
KDE Sysadmin