DevHeads.net

Sysadmin report on the modernization of our infrastructure

Hi all,

As promised in the earlier thread, i'd like to present the sysadmin
report on the state of the infrastructure surrounding our code.

It contains a detailed summary of what is broken with our existing
systems, why change is necessary and an evaluation of the options we
considered. We have also made a proposal based on our evaluations and
the wishlist of functionality drawn up the community.

Please find it attached - and let us know what you think. Feedback is welcome.

Cheers,
Ben Cooksley
KDE Sysadmin

Comments

Re: Sysadmin report on the modernization of our infrastructure

By =?iso-8859-1?Q?... at 01/21/2015 - 06:44

Hi Ben,
thanks for your proposal. A couple of points which I'm missing from the
text, and a few further questions as well:

1) A detailed list of the issues which a competing proposal would have to
address. Some glimpse of this is in the last paragraph, but that's just a
very high-level description. Could you please provide a list of
functionality that has to be supported by a replacement, so that we can
work out how to get there?

2) It is unclear to me whether you plan to use Gitolite in future or not.
At first, the proposal says that there are inherent scaling issues with it
and that the replication is beyond its scaling limits, yet at the end of
the document you say that a replacement has to support Gitolite metadata
generation. I do not understand that.

3) The idea of rolling out Phabricator is not specified, so it's hard to
get a proper understanding of what exactly will have to be done. What
changes in workflow are going to be needed? What custom tooling will have
to be written? How is Phabricator going to play with the rest of the
infrastructure? What pieces of the infrastructure will actually remain?

4) You have indicated some (pretty important to me) limitations of
Phabricator with a remark that "it's a fast moving software, we might get
there eventually". I think that SW should be evaluated based on
functionality present right now in the development version and not based on
opened feature requests. We've been promising e.g. support for multiple
accounts in Trojita for many years already, and it didn't make it happen.

5) You're indicating a requirement for scratch repos to be present from the
very beginning, yet you acknowledge that Phabricator won't have it for at
least six months.

6) The discussion focuses in highlighting Phabricator's benefits, which is
understandable from your point of view. However, much of the same things
can be said about Gerrit as well, especially its backing by a well-known
player, adoption by multiple companies and multinational corporations
(SonyErisccon, SAP, KitWare,...) as well as by many FLOSS projects (Qt,
Andorid, LibreOffice, Eclipse, OpenStack, WikiMedia, Typo3, Cyanogenmod,
Tizen, and a ton of others). Its scaleability has also been verified by
real-world use over many years (for far longer than Phabricator).

Now coming to Gerrit and its analyzis:

7) It is possible to have scratch repositories with Gerrit, but it's true
that it will require a simple tool which verifies user's request and
executes an API call with proper credentials. We're speaking about one
trivial webapp or a shell script here.

8) There is no need for any modifications to Gerrit as your text implies.
What is running and integrated into KDE right now is an unpatched release
straight from upstream, with no custom plugins.

9) I do not know what effort for integration with KDE Indentity you're
referring to. It's a simple LDAP server, and the entire "integration" was
trivial. It's really just a matter of configuring Gerrit to talk to LDAP,
which I expect to be also the case with Phabricator.

10) Re Phabricator's support for submitting and retrieving changes via pure
git, I believe you're overly optimistic in this regard. This is a pretty
fundamental design decision which can only be retroactively fitted with
asignificant amount of pain.

11) While the Gerrit trial has been running for a few months, being used by
real KDE projects and generated actual feedback and tweaks, there's been no
trial of Phabricator so far. In my opinion, it is premature to have a plan
for migration to a particular tool prior to having verified said tool in
production environment. In this regard, your proposal effectively discusses
throwing away the results we got with Gerrit, and fails to provide some
rationale for that. Indeed, the question is "where is Phabricator better
than Gerrit", and I propose to focus on this aspect in future.

12) WikiMedia uses Phabricator for issue tracking and nothing else -- it's
just a Bugzilla replacement for them. According to publicly available
information, their reasons for choosing Phabricator had everything to do
with the PHP as a language known to all of their developers. They still use
Gerrit for git repo management and code review.

So given that we're about to rebuild the whole Git infrastructure anyway,
the counter-proposal is to build it around Gerrit this time. Based on what
I know about KDE's infrastructure, Gerrit can fill in these roles without
any problem. I would like to work with you towards that goal; are you
interested?

Finally, I would like to say that I appreciate the work of the sysadmin
team. In future though, I'd love to see a bit more transparency of the
entire process. Right now it isn't clear to me whether investing a few
additional man-months of my time towards working with Gerrit has any merit,
or whether it's been already decided that the KDE's future is with
Phabricator. I don't know who will make the final decision -- is it going
to be Ben and Jeff, or are the contestants going to be judged by the wider
community and how and using what criteria?

The way I see it, both Ben and Jeff (was the report written by anyone
else?) have expressed their intention to go with Phabricator despite my
long-standing and thorough work on evaluating Gerrit, and I'm a bit
disappointed by that. It would be fine to reject Gerrit on technical
rounds, but I haven't seen much of technical arguments here. So let's make
this simple -- is this going to be a decision of the sysadmin team, or is
there an actual potential of offering alternatives?

With kind regards,
Jan

Re: Sysadmin report on the modernization of our infrastructure

By Milian Wolff at 01/21/2015 - 10:54

On Wednesday 21 January 2015 11:44:00 Jan Kundrát wrote:
Yes, a simple bullet point list would be useful. I'll list some stuff below
also.

Agreed.

<snip>

Agreed.

Imo, quite the contrary. It concentrates on the issues the admins have with
their current setup, and then shows how Phabricator could help with that.

<snip>

I don't see where this is mentioned in regard to Gerrit. I can only find LDAP
being mentioned when talking about the status quo for KDE, which does not
include Gerrit.

This belongs to the point below, imo. Or what are you referring to? If you do
mean the git integration, then yes, maybe it's important to you, but it's
really not important in the big picture, imo. And do note, I'd personally also
prefer gerrit for reviewing.

As far as I understand it, we (KDE) won't jump to Phabricator in a matter of
days, rather it will take a few more months before we even start (correct?).
There are still some other blockers upstream, which will be resolved. Lets see
how this works out. Generally, I don't think its impossible to add this to
Phabricator. So lets not be overly pessimistic, nor optimistic. Maybe this
feature is added, maybe not. While I personally would also love to have it,
it's a minor detail compared to the rest of the feature set, imo.

<snip>

See <a href="http://www.mediawiki.org/wiki/Phabricator" title="http://www.mediawiki.org/wiki/Phabricator">http://www.mediawiki.org/wiki/Phabricator</a>, what Jan says is true. But
WikiMedia is in the process of migrating away from Gerrit, so what Ben says is
also kinda true ;-)

Now to the following:

To my knowledge, here are some things that Gerrit does not provide, but
Phabricator potentially provides:

- a project overview with a code browser, project meta data etc. pp. and a
list of commits inside a repository. Qt still uses gitorious for this, afaik
- the ability to get notified about new commits in a project. (this is
different from new reviews)
- Apparently the anongit problem, but Ben would need to fill in more details
here.

then, there are some other things that are really nice to have in Phabricator,
which are not available in Gerrit. But, imo, they are not as important as the
above:

- post-commit review
- project management, todo / kanban board
- issue tracker (only if we ever port the bugzilla content over)
- paste (only if that would replace paste.kde.org)
- wiki (only if that could be used to replace the existing wiki structure)
- various tools which would help with release management, e.g. calendars /
countdown timers / release branches
- overall, the app framework phabricator provides seems to indicate that its
easy to extend, contrary to what I heard from gerrit

Please correct me if I'm wrong. I only know Gerrit from its usage in Qt.

Bye

Re: Sysadmin report on the modernization of our infrastructure

By Ben Cooksley at 01/21/2015 - 17:49

On Thu, Jan 22, 2015 at 3:54 AM, Milian Wolff < ... at milianw dot de> wrote:
That is correct - we have to evaluate it first, and no migration would
be rushed through in a matter of days.
Things have to be planned for, prepared, tested and pre-flight
migrations run before the final production migration is done.

Thanks,
Ben

Re: Sysadmin report on the modernization of our infrastructure

By =?iso-8859-1?Q?... at 01/21/2015 - 12:53

On Wednesday, 21 January 2015 15:54:52 CEST, Milian Wolff wrote:
Right, I should have said "the proposal to use Phabricator and the reasons
for using that particular tool focus on highlighting Phabricator benefits
...". Sorry for not being clear, I appreciate the value of describing the
pain points of the legacy setup (and I woud appreciate even more details to
be able to offer a better alternative).

The part which I was commenting on is the following paragraph:

There was no extra work to integrate Gerrit with KDE's Identity, and I
expect most of other tools which we might need to use to have LDAP support
out of box because that's an industry standard for identity management.

"KDE Identity" == "LDAP", in this context.

The paragraph above also assumes that Gerrit would not be used as a primary
code hosting place. There's no reason for that, so the raised conclusions
("this will require syncing") are not true.

When I read the proposal, I see some enthusiasm for Phabricator's swift
development pace. That's good, but at the same time it isn't an answer to a
lack of features. Here are some of the relevant bits:

- CI integration
- scratch repos
- clustering for scaleability
- preserving author metadata
- direct git access to patches under review
- patch upload via git

What I'm saying is that an opened feature request and a rapid speed of
development are not something to gurantee that these will be ready in a
month, or even in a couple of years.

Yes, Gerrit doesn't include wiki pages, issue tracker, Kanban planner and
various calendars. I don't necessarily see that as a drawback. Do we want
to migrate wikis and Bugzilla? If yes, I can understand that Phabricator
might be a compeling tool, but so far the proposal was limited to just
revamping the git hosting and code review.

Right, we will have to pick one (or multiple) of the WWW git browsers out
there. If I was designing such a service, I would let it run from a
dedicated VM and have Gerrit replicate stuff to these servers. Scaling up a
stateless, read-only service such as cgit/gitweb/... is very easy.

Gerrit has hooks for triggering this ("ref-updated"), and it's easy to send
e-mails from that context. There are scripts for that in git's contrib/.

@sysadmins, how is that tackled by Phabricator?

Also, if a proper code review was enforced, there would be no need for
this.

We are already using a stock upstream plugin called "replication" which can
push refs around, create and even delete remote repos based on local
events. Our Gerrit instance is using it for propagation of changes to
git.kde.org.

I do not understand where the problem could possibly be; Gerrit is 100%
ready to replicate to as many anongit nodes and/or GitHub repos as needed.
Many people are doing this, and many people are even using various Gerrit
instances together for even better scaleability. I seriously doubt that KDE
will need *that* level of scaling in the next five years, though.

One can comment on closed review requests, and the result is available as
any other review, e-mailed out to the review participants, and accessible
to search engines.

It's true that there's no git browser where you could attach notes to a
particular line and "open a ticket" for that. Do we need such a feature,
and if so, how do we intend to use it?

"Project management" can mean multiple things, so I assume that you're
referring to release planning, issue prioritization etc here. Gerrit indeed
doesn't doesn't have that. The plan will have to offer some tool for this.
What are your (community's) favorite tools for TODOs etc?

See above; I'm afraid that we have to draw a line somewhere on what we
would like to replace and what to preserve. A Gerrit proposal should
probably say "we're happy with curent wikis because they work well enough",
sure.

I just don't see Phabricator's feature set as a huge benefit here.

My understanding is that other tools are expected to interact with Gerrit
through its rich set of APIs. There are REST APIs for everything, there's
an SSH-based notification channel and we already have an IRC bot notifying
us about reviews which uses this. There are patches under review upstream
which add support for a real message bus (AMQP IIRC). I've deployed a full
CI system which talks to Gerrit through these APIs and provides feedback
about changes succeeding/failing to build on multiple platforms. Other
people are tying into these events to automatically create and upload
release tarballs. I'm sure that this demonstrates that Gerrit is quite
extensible.

Cheers,
Jan

Re: Sysadmin report on the modernization of our infrastructure

By Milian Wolff at 01/21/2015 - 13:50

On Wednesday 21 January 2015 17:53:51 Jan Kundrát wrote:

<snip>

OK, I see. Maybe Ben and Jeff where not aware of this. Or they mean something
different - lets wait for them to clarify.

I agree with you. At the same time, many things are missing in Gerrit - so
it's not different there.

You are misinterpreting this. Kanban, wiki and issue tracker are optional
things that are nice to have, but they are not the important stuff (see
below).

Please read Bens article again. We do this currently and its not working. This
is what needs to be replaced. Phabricator seems to support this, or so they
say, **and** is does what Gerrit does. So why not use that and have everything
integrated? It's not as simple as picking a WWW git browser, it must be
integrated into the rest of the system. And it must be easy to maintain etc.
pp. Really, just reread the report.

Scripting stuff means its not configurable to users, which is extremely
important here. And no, writing a script, maintaining it, and adding a config
UI on top of it if is explicitly **not** what the sysadmins are looking for.
They want to cut down on tools and maintenance burden.

Herald. Trivial there, no scripting required whatsoever. User configurable
with a simple three-level ACL which seems sufficient to me.

The KDE community won't enforce code review anytime soon.

As I said, no clue about this. So lets wait for Ben and Jeff to clarify this.

Note: You removed my marker here, that the below is "nice to have". Please
keep it in, it's important. I never said that this stuff is of utmost
importance.

So everything that goes into the CI without review will be "dead". See above,
KDE will keep this as-is for the foreseeable future.

We currently have this in the form of the kde-commits mailing list. It is an
extremely useful "feature" of the KDE community. What we get with Phabricator
is that, just so much better.

Having an external tool like our current kanban board on todo.kde.org means
it's not integrated with the rest. No easy way to link to reviews, branches,
issues, etc. pp. Phabricator gives us all of this, for free!

Well, and I disagree here. Having it all integrated will mean we eventually
have a GitHub clone which makes it trivial to close issues or reference stuff
from the wiki and have stuff updated there as needed. And remember, I said
that this stuff is *nice to have*. It's not the important reason why we should
use Phabricator, it's just additional sugar on top which you won't have with
Gerrit.

OK, cool - thanks for the clarification. I was not aware of this, only of the
SSH "API".

Bye

Re: Sysadmin report on the modernization of our infrastructure

By =?iso-8859-1?Q?... at 01/21/2015 - 15:05

The report mentions problems with the current Git replication. I'm also
aware of the technical nature of these things beyond what was said in the
report because I did talk to Nicolas and Ben about them. The current setup
is slow, pull-based, unreliable, and can only work once an hour on an
overseas node due to network latencies and systems' throughput. Fair
enough.

Compared to that, I know about people using Gerrit's replication with a
world-wide, distributed network of Git mirrors on multiple continents. What
I'm proposing is simply using what works well for these people and use that
for scaleable repository browsing as well. There is no need to write any
custom scripts. Just deploy as many mirrors as we might need, and add an
arbitrary read-only WWW git browser to them, with no extra configuration
required -- "just serve all the repos in a read-only manner".

Gerrit's replication is not stupid, it does understand the concept of
queues and automated retries, so it actually *works* and handles
bandwidth-limited or latency-bound nodes fine.

If one was to use Phabricator, there will still be the same set of problems
with git browser having to scale to sustain automated web crawlers and what
not. We will also still need N machines to bring actual content close to
the users. Nothing changes by using a single application here, really. The
problem is that replication needs improvements, so let's improve the
replication. Gerrit can do that. What does Phabricator use for replication,
and how does it deal with crappy network?

OK, I agree that a perfect way is to add this to Gerrit's existing
notifications and contributing that patch upstream -- that already includes
the whole infrastructure, up to and including the pretty UI. Let's do this;
I'll write that patch and push it upstream once I know that this is not
going to be a wasted work (see my first mail in this thread).

I remove those parts of the mail which I mostly agree with; I find that
better for readability. Sorry for confusion. As you said, it's important to
assign a proper importance to various features; we're in total agreement
here.

Fair enough, the problem with what we have right now is that nobody is
guaranteed to read these and to follow up on them, to paraphrase someone
(you, maybe?) from the previous iteration of this conversation. Improving
that is a nice bonus, so it's indeed cool to be able to create
tickets/issues by a single click when one browses a git repo. It's nice
that Phabricator can do that. Do we however actually have some people doing
this project-wide code auditing *and* not sending patches at the same time?

I'm just wondering whether this fancy feature would be used that often, and
whether it could be better to do this as regular code review instead. And I
also understand that you've listed it in the "nice bonuses" section.

If something is bundled, it doesn't mean that it's free. One of the costs
that you pay is that you cannot switch to a better alternative because
everything and everybody's workflow assumes that you're using the built-in
solution.

(I don't understand how you close issues from a wiki, though, but I hope
that I understand the general idea.)

Fair enough, I see that it's cute to e.g. have a wiki engine that can show
you a status of a patch under code review right from that hyperlink. That
is of course possible with Gerrit as well, but it requires some custom
scripting. If the idea is to just use tools from a single vendor and to not
consider anything that requires any sort of integration effort or a 3rd
party script, than a monolithic tool will by definition win.

As a counter example of how separate instances can be integrated, try
pushing a change to KDE's Gerrit. You will immediately notice a
live-updated status table about the CI build being started, changing to a
passive notification about an updated change when the build finishes. Some
of that is custom code that was developed by the OpenStack project which
was trivial to adapt for our purposes. It's all client-side JavaScript and
it fully integrates two separate systems (Gerrit for patch review and Zuul
for CI job scheduling in this case). A user sees an integrated result, and
that's the only thing that they care about.

Another example is Bugzilla integration in Gerrit; I have not enabled it on
our instance yet because it's done by git.k.o hooks right now, so we only
have some hyperlinks for now. However, there's an upstream plugin for a
proper integration. It is possible to have Bugzilla (or Jira, or
Phabricator, or ...) status updates based on changes
proposed/approved/rejected in Gerrit. The plugin is capable of that just
fine. Want to enforce mandatory reference to a bug# for some projects?
Supported out of box. Want to ignore commits to non-release branches? Sure
thing. Want to get rid of that fragile hooks which use e-mail for bug
closing? Just use that Gerrit plugin and rely on Bugzilla's XMLRPC
interface.

Let's evaluate the tools and scripts which are available. It shouldn't
matter whether e.g. a code review system and a repo browser are written by
the same software vendor. As long as both offer APIs for integration *and*
the integration is taken care of and well-maintained, there should be no
difference in our evaluation process.

Thank you for these e-mails. IMHO, it's important to make sure that we, as
a community, identify what we are actually looking for, and having a polite
and civilized discussion certainly helps get there.

With kind regards,
Jan

Re: Sysadmin report on the modernization of our infrastructure

By Ben Cooksley at 01/21/2015 - 18:02

On Thu, Jan 22, 2015 at 8:05 AM, Jan Kundrát < ... at kde dot org> wrote:
Automated web crawlers are told to get lost.
We publish robots.txt files which block bots, and for bots caught
ignoring that follow up with various forms of blocks.

Phabricator itself doesn't provide replication - we'll be building
that to match our needs (which includes repository removal and
addition, which I imagine you'll need to add to Gerrit as well).

Once Phabricator is CI enabled (we'd need to wire it up to Jenkins,
code already exists for that, we just have to tweak it to fit our
model) it can offer similar integration there.

Cheers,
Ben

Re: Sysadmin report on the modernization of our infrastructure

By Ben Cooksley at 01/21/2015 - 17:47

On Thu, Jan 22, 2015 at 5:53 AM, Jan Kundrát < ... at kde dot org> wrote:
What sort of information are you after exactly? The detailed part of
the report lays out the problems fairly well.

Please see my earlier mail. You need to draw SSH keys from Identity.

We've addressed that. It can be done with little fuss.

Also doable, once again with minimal fuss.

Not sure what you mean here. The folks behind Phabricator are building
a hosted solution called Phacility, so it will be scalable.
Components of it, given the right setup, can run on multiple machines.

Being worked on upstream. Only a problem when you use "arc land" and
will likely be resolved by the time the testers are finished with it.

I appreciate this is important to you, but it isn't as critical to others.

Others who are interested in the trial have indicated an interest in
using these components to an extent.

Herald can do that. Please do take a look at it's documentation - it
is very powerful and gives quite a bit of control over who gets
notified about what, and can subscribe people to reviews.

That is a workflow matter, set by individual projects.

I see one likely issue here: removal and creation of repositories on the nodes.
How does Gerrit handle that?

Our anongit scripts look after that at the moment.

Phabricator doesn't offer any replication functionality - as we
mentioned in our report, this would be built by us as an improvement
on our existing systems.

Also - git.kde.org at the moment does support replicating over SSH.
This is functionality provided by our hooks, and is used to maintain
some Necessitas repositories on Gitorious at the moment.

The audit tool in Phabricator can do that.
We currently use kde-commits emails for this purpose, but it is
difficult to raise a review if you don't have the mail at hand, and
there is no way to track whether a review is resolved or not.

Thanks,
Ben

Re: Sysadmin report on the modernization of our infrastructure

By =?utf-8?Q?Thoma... at 01/21/2015 - 18:39

I googled for CI and saw quite some "we need such" comments over the last two years and also some homebrew script to integrate travis, but it didn't sound like "put this 10 LOC sippet here and be happy" or so and apparently is not a trivial issue.
Would Scarletts efforts (you're pointing them?) be upstreamed to Phabricator?

That looks scary.
On how many machines do we run atm? (combined, ie. git, RB, and bugzilla, in case)

Cheers,
Thomas

Re: Sysadmin report on the modernization of our infrastructure

By Ben Cooksley at 01/21/2015 - 19:13

On Thu, Jan 22, 2015 at 11:39 AM, Thomas Lübking
<thomas. ... at gmail dot com> wrote:
Reviewboard does not make it trivial to create such an integration at
this time. Please read the report where this is explained in full.

Phabricator does make it easier, please see the links i've posted to
the mailing list.

Scarlett's efforts are KDE specific and relate exclusively to Jenkins.
We would provide information to the Phabricator community on how we
achieved our integration of course.

Git runs on the physical host Kater (i7-2600 w/ 32gb RAM). It shares
this with Identity, CiviCRM and Jenkins, as well as our IRC services.
The machine isn't heavily utilised except at certain times - but that
is caused by scaling issues in the anongit network. At all other times
the machine is fairly idle.

Reviewboard runs on the physical host Tesla (i7-950 w/ 8gb RAM). It
shares this with reports.kde.org, an Owncloud instance and some other
minor services.
Once again, the machine isn't heavily utilised except when someone
makes a large number of commits or sends a stack of emails, which
generates a bit of load as reports.kde.org processes them (design is
slightly suboptimal here).

Bugzilla runs on the physical host Katze (i7-3770 w/ 32gb RAM). It
shares this with the Dot, Forum, Wikis as well as all our other CMS
based sites including krita.org, calligra.org, kdevelop.org and
kdenlive.org. It also hosts todo.kde.org and paste.kde.org. This
machine gets a little busy during the EU daytime due to these other
sites - Bugzilla doesn't present a load issue (it is a memory hog due
to the Perl and MySQL caching needed to keep it performant when
queried, but is itself a very low hit rate site).

Quickgit runs on a donated container, on the anongit node Serenity. It
shares this with nothing else. There are no load issues here.

Chiliproject is extremely suboptimal, and a massive resource hog. It
runs on a separate machine again.

I don't expect there to be any issues requiring us to cluster or move
beyond one server hosting Phabricator.

What was in production? The hack allowing for reviews to be posted
using Git itself? It isn't at this time...

Thanks,
Ben

Re: Sysadmin report on the modernization of our infrastructure

By Ben Cooksley at 01/21/2015 - 15:07

On Wed, Jan 21, 2015 at 11:44 PM, Jan Kundrát < ... at kde dot org> wrote:
Hi Jan,

Certainly.

The list in question was drawn from my summary mail which was sent to
the previous thread.
Not everything in that list is a requirement (as no software exists
which can meet all of them).

Phabricator will be replacing Gitolite in our proposed setup.
There are no scaling issues with Gitolite (it handles things very
well), the scaling issues were with Gitorious, and are with
Chiliproject and our anongit scripts.

With regards to the metadata - I think this was an unintentional
reference to the Gitolite configuration generator. If your solution
uses something else of course, you would need to build the appropriate
logic for that instead.

There would be no changes in workflow as such.
The only way to properly test a tool though is to actually put it into
use for a while - and thus find out whether it is able to fit our
workflows.

In terms of custom tooling, we will need to integrate it with Jenkins
and set it up to receive keys from Identity.
Due to the way it works with SSH keys this will be a fairly trivial
process as our existing systems which auto-sync keys for svn.kde.org
and git.kde.org can be almost entirely reused.

Our evaluation was conducted on the basis of functionality currently
present in it.
Meeting the requirements of the list was done on a best-effort basis.

There are two paths for scratch repositories:

1) A solution we can implement right now with a minor (20 lines of
code) change which will permit developers to create repositories with
a certain naming schema.

2) Wait for namespaces, which is what is 6 months off at the earliest.

We're likely to go for #1, and possibly migrate to #2 when upstream
implements that.

This is a separate application, and would therefore require separate
maintenance correct? (Plus people have to find it)
How would developers delete no longer used scratch repositories?

Thiago indicated that for sane mail handling one wants the patches Qt
currently has.
Is this not the case?

The integration in question I'm referring to are SSH keys, and
ensuring details are automatically updated from LDAP when they change.
If i'm not wrong, your current solution requires keys to be uploaded a
second time.

Upstream has already demonstrated that one could upload a patch using
just Git with some modifications.
The code in question was only a proof of concept, but it certainly worked.

I'm certainly confident that it could be implemented without too much hassle.
The only painful part would be accepting patches via the web - and
getting them into Git commit form.

I don't see a migration plan there. I see an invitation for people to
test it out as well as an estimate of how long a migration would take
if the test was successful. No judgement has been passed and the
results from the Gerrit evaluation certainly haven't been discarded.
(Feedback from participants on this proposal is welcome).

I look forward to seeing the answer to that question.

They are currently in the process of evaluating it's code review
capabilities to replace Gerrit.
See <a href="http://www.mediawiki.org/wiki/Phabricator/Plan" title="http://www.mediawiki.org/wiki/Phabricator/Plan">http://www.mediawiki.org/wiki/Phabricator/Plan</a>

In terms of what is chosen, it is ultimately up to the community.

No decision has yet been made.
The final call is essentially made by community consensus - there is
certainly no voting.
In terms of criteria, they're up to each person in question who
participates in forming that consensus.

There is the potential of offering alternatives.
We certainly do appreciate the work you have put into Gerrit and the
evaluation of it.

The decision will be made by the community as above.

Cheers,
Ben

Re: Sysadmin report on the modernization of our infrastructure

By =?iso-8859-1?Q?... at 01/21/2015 - 17:39

On Wednesday, 21 January 2015 20:07:21 CEST, Ben Cooksley wrote:
Ben, it just isn't clear to me what parts of the system are free for
replacement and which parts are set in stone to remain as-is. Maybe we
don't know yet; that would be fine as well. However, as I'm reading the
rest of the text, it seems that some people are looking forward to migrate
away from Buzgilla, or to use Phabricator for task management as well.
There are also options for getting away with the current set of git hooks,
or at least some of them.

Let me ask in a different way, then:

- How are you to integrate with CI?
- How are you to plug into Bugzilla?
- How are you going to tie with Jenkins in order to automatically create
build jobs/profiles?
- How are you going to send commit e-mails? How are people going to filter
them or to subscribe to various projects of interest?
- How are IRC notifications going to be supported?

I'm sure there are more; the above is just to show what I'm asking about
when I speak about integration.

Here's the script that I'm talking about, in pseudocode:

if not check_ldap_group($user, "developers"):
die "you are not a KDE developer"

if not regexp_match($proj, '[-a-zA-Z0-9_]':
die "invalid project name"

if operation == "delete":
return `ssh bot@gerrit delete --yes-really-delete --force \
scratch/$user/$proj`

if operation != "create":
die "invalid operation"

return `ssh bot@gerrit create-project --owner $user \
--parent sysadmin/gerrit-user-scratch-projects \
scratch/$user/$proj`

That's 9 lines prior to line-wrapping for mail, including error handling.
As of maintenance, what's your estimate of the required time to maintain
this over the next 10 years?

As a nice bonus, Gerrit supports enforcing quotas for number of per-user
repositories as well as the consumed disk space; there's a plugin for that.

I'll be happy to include a nice page in Gerrit's top menu saying "Create
project" which will explain how to request official repositories as well as
how to request regular projects from sysadmins :).

In the same manner as creating them, see above. It's also possible to have
a cronjob querying the date of last change, etc.

I don't know what is or is not sane, but [1] is a random example of what we
have right now. Is that sane enough?

E-mails that I'm receiving from Qt's own Gerrit look the same, but maybe
I'm missing some crucial difference.

Yes. This is a matter of one command for each event:

`cat $key | ssh bot@gerrit set-account $user --add-ssh-key -`

...and an appropriate version in case a key got removed.

I believe that this is an equivalent to what you're already doing and what
you plan to continue with Phabricator. Is that right?

With kind regards,
Jan

[1] <a href="http://lists.kde.org/?l=kde-frameworks-devel&amp;m=141992935613350&amp;w=2" title="http://lists.kde.org/?l=kde-frameworks-devel&amp;m=141992935613350&amp;w=2">http://lists.kde.org/?l=kde-frameworks-devel&amp;m=141992935613350&amp;w=2</a>

Re: Sysadmin report on the modernization of our infrastructure

By Ben Cooksley at 01/21/2015 - 18:57

On Thu, Jan 22, 2015 at 10:39 AM, Jan Kundrát < ... at kde dot org> wrote:
We don't entirely know yet. Part of the fun of considering migrations
like this really.
The less we have to maintain though, the better.

Using either <a href="http://www.guywarner.com/2014/06/part-2-integrating-phabricator-and.html" title="http://www.guywarner.com/2014/06/part-2-integrating-phabricator-and.html">http://www.guywarner.com/2014/06/part-2-integrating-phabricator-and.html</a>
or <a href="http://www.dctrwatson.com/2013/01/jenkins-and-phabricator/" title="http://www.dctrwatson.com/2013/01/jenkins-and-phabricator/">http://www.dctrwatson.com/2013/01/jenkins-and-phabricator/</a> or a
variation thereof.

Yet to be determined. People have linked Phabricator with JIRA, so
similar integration should be achievable for Bugzilla.

Jenkins will handle this, using the Job DSL plugin in all likelihood.
Jobs would be created based on information provided by the Phabricator
Conduit API.

Commit emails could either be sent by our existing hooks, or we could
migrate to Herald and customise it's template to fit what we need if
necessary.
People would filter them / subscribe to them through Herald.

For commits or reviews?
Probably the same mechanism we have at the moment to be honest -
commit hooks feeding a Irker bot.
I do know that Phabricator does have an IRC bot around which does
integrate with it using the Conduit API.

Doesn't seem too high, although I don't see how that would be made web
accessible - which might be the hard and costly part maintenance wise.
(You have to deal with security issues too as you are in a separate
web application, so you need to authenticate the developer first).

The community would be the judge of that, not me.

Nope.

Our existing solution is triggered on change events in LDAP and causes
all SSH keys to be re-read and a new ~/.ssh/authorized_keys file to be
written out. You can't rely on OpenLDAP stating the addition/removals
properly when using the syncrepl interface, at least in my experience.
In this way we avoid dependence on the Identity web application.

Cheers,
Ben

Re: Sysadmin report on the modernization of our infrastructure

By =?iso-8859-1?Q?... at 01/21/2015 - 22:03

On Wednesday, 21 January 2015 23:57:07 CEST, Ben Cooksley wrote:
That is quite some custom code that one has to maintain, though.

How would they subcribe via Herald if it was done via the existing hooks?

Well, Apache's mod_authnz_ldap and a "Require group developers" stanza
makes this really easy. Just look up $user from an appropriate env var
provided by the web server. Where is the problem?

A quick & dirty approach:

`ssh bot@gerrit set-account $user --remove-ssh-keys ALL`
`ssh bot@gerrit set-account $user --add-ssh-key - < authorized_keys`

A better and race-free code would have to invoke `comm` in addition to
that, and only add/remove keys which has to be removed or added. That's
left as an excercise for the reader, it's easy enough. Or, to avoid relying
on a local state altogether, just issue a REST call for SSH key retrieval
and base a decision on that. It's gonna be like 10 lines of custom code.

Cheers,
Jan

Re: Sysadmin report on the modernization of our infrastructure

By Ben Cooksley at 01/22/2015 - 01:55

On Thu, Jan 22, 2015 at 3:03 PM, Jan Kundrát < ... at kde dot org> wrote:
Doesn't look like an excessive amount to me, once you strip out the
comments (which all code should have anyway).

Some of it also governs how Jenkins reports back to Phabricator -
allowing to provide fine grained commentary on test success or
failure. We could probably extend it to report on cppcheck or code
coverage differences if someone wanted to even.

I see our control over that as a plus point - so developers don't have
to go look somewhere else to see the exact nature of the failure
(whether build or test related).

Herald can generate mails itself and send them.
While you wouldn't get the same commit mails unless we change the
template Herald uses, you will get the commit mails.

It's a separate web application to maintain. I'll admit that does
solve the authentication issue nicely, except for the part where
developers can't log out unless they close their browser.

Regards,
Ben

Re: Sysadmin report on the modernization of our infrastructure

By Kevin Ottens at 01/21/2015 - 13:08

Hello,

On Wednesday 21 January 2015 17:12:07 Ben Cooksley wrote:
At last some movement in that space that's really nice. I think I've been
annoying the sysadmin team with our current tools for a while now, so I can
only propose Zanshin as one of the test projects.

I especially appreciate the global approach you took, indeed we have to go the
forge approach not focus on code hosting + review only. It looks like
Phabricator is a nice move in that direction to have something really
consolidated and not plenty of disjoint tools.

BTW, since we didn't open a kanboard for Zanshin yet, we're totally up to test
the task tracking of Phabricator. In fact the more tools we get enabled for
our test the better so that we can cover wide on the features.

Regards.

Re: Sysadmin report on the modernization of our infrastructure

By Milian Wolff at 01/22/2015 - 10:52

On Wednesday 21 January 2015 17:12:07 Ben Cooksley wrote:
Another question regarding Phabricator:

Currently, Chili offers a very nice tree structure of projects, which follows
our KDE structure with frameworks, playground, extragear etc. pp. - nice!

Can Phabricator do the same? I assumed so, from the fact that you hinted at it
being able to generate the metadata XML file which is used by kdesrc-build to
replicate the tree structure e.g.

But <a href="https://showoff.phab.io/project/" title="https://showoff.phab.io/project/">https://showoff.phab.io/project/</a> only shows a list of projects. Same for
<a href="https://showoff.phab.io/diffusion/" title="https://showoff.phab.io/diffusion/">https://showoff.phab.io/diffusion/</a>. I tried to create a new project and a new
repository, and could not figure out how to create a tree with it.

Bye

Re: Sysadmin report on the modernization of our infrastructure

By Ben Cooksley at 01/22/2015 - 22:40

On Fri, Jan 23, 2015 at 3:52 AM, Milian Wolff < ... at milianw dot de> wrote:
We can probably use the tag feature of Phabricator to associate
repositories to projects.
This would allow us to tag projects as #extragear #multimedia as an
example - giving us a different form of structure.
It doesn't directly support the tree structure.

(I will add that the tree structure support in Chiliproject is part
sysadmin-hack, and is a big part of the upgrade problems associated
with it as the changes needed to support it are both highly invasive
and in an area which is regularly changed).

Projects can't contain other projects with Phabricator i'm afraid.

Thanks,
Ben

Re: Sysadmin report on the modernization of our infrastructure

By Milian Wolff at 01/22/2015 - 23:10

On Friday 23 January 2015 15:40:40 Ben Cooksley wrote:
So this means kdesrc-build will break, no? At least for those who use the
default layout which mimicks the tree structure on projects.kde.org.

And I'd bet that tags or similar metadata can also be attached to gerrit
repos. I hoped to see Phabricator have a clear lead here, but apparently not
so.

Bye

Re: Sysadmin report on the modernization of our infrastructure

By Ben Cooksley at 01/22/2015 - 23:17

On Fri, Jan 23, 2015 at 4:10 PM, Milian Wolff < ... at milianw dot de> wrote:
We will be putting in place other mechanisms to map
repositories/projects to the tree structure.
Because the kde_projects.xml file is expensive to generate anyway,
it'll be rebuilt by a script whenever changes have been made.

Note: the kde_projects.xml file is unique to us - everything requires
some level of custom work here to build this file.

Cheers,
Ben

Re: Sysadmin report on the modernization of our infrastructure

By Boudewijn Rempt at 01/21/2015 - 05:39

Hi,

I am all for the proposal -- though I would like to use Phabricator for issue tracking as well, actually. I would also like to propose Calligra/Krita as one of the test projects.

On Wednesday 21 January 2015 Jan 17:12:07 Ben Cooksley wrote:

Re: Sysadmin report on the modernization of our infrastructure

By Martin Klapetek at 01/21/2015 - 07:07

Thanks a lot Ben for this, very comprehensive document.

One question about Phabricator - in the discussions before the RB web UI
and uploading patches via web was mentioned a lot, will uploading diff
through
web interface still be possible via Phabricator or is the Arcanist thing a
must?
This isn't clear from the report.

Cheers

Re: Sysadmin report on the modernization of our infrastructure

By Ben Cooksley at 01/21/2015 - 07:15

On Thu, Jan 22, 2015 at 12:07 AM, Martin Klapetek
<martin. ... at gmail dot com> wrote:
Sorry about the confusion there.

You can still upload patches through the web interface using Phabricator.

Thanks,
Ben

Re: Sysadmin report on the modernization of our infrastructure

By Helio Chissini ... at 01/21/2015 - 07:57

Thanks Ben for the right proposal.

During the process when the original thread becomes a hell of bickering
people defending his point instead of a concise approach, i almost loose
the faith on results.

I personally tested several of the tools on the last job, where i managed
internally the buildsystem, so i vow for phabricator as a good decision.
I really don't like gerrit, and my experience with it sometimes remove me
from want to do things.

Phabricator is the obvious system to go. We trusted our sysadmins decisions
for years, would not be different now.

Helio

On Wed, Jan 21, 2015 at 9:07 AM, Martin Klapetek <martin. ... at gmail dot com>
wrote:

Re: Sysadmin report on the modernization of our infrastructure

By Luca Beltrame at 01/21/2015 - 02:50

Hello Ben,

I know I'm much less into coding than other people here (which are probably
far more knowledgeable), but in general the proposal seems very sane to me.

Assuming this proposal is accepted by the community at large, how can the
community help as well? Meaning, is there anything that can be done, given
that sysadmin resources are thin and already spread out?

That question aside, I have a question on the proposal itself: KDE uses both
Git and SVN, so my question is what will be done to ensure smooth transition
also of the SVN side (as the hooks are very old, IIRC, and do a lot of
things).

Also, can Phabricator be configured to do what our "hooklib.py" does here?
In particular license checks,source checks such as eol and so on?

Lastly, I assume that there is no plan to use the issue management of
Phabricator for now, with the exact rationale of when Redmine was chosen
(need to import existing database, size of said database...). I have no
issue with it, but perhaps it should still be pointed out.

Thanks for all your work.

Re: Sysadmin report on the modernization of our infrastructure

By Ben Cooksley at 01/21/2015 - 05:51

On Wed, Jan 21, 2015 at 7:50 PM, Luca Beltrame < ... at kde dot org> wrote:
Hi Luca,

We can probably come up with some items I think.

Items like determining the rewrite rules for things like Quickgit,
WebSVN and Chiliproject would definitely help ease matters.
We'd also need to consider a migration path for existing Git project
subscriptions into Herald.

This isn't a complete list of course, I imagine there will be more.

We should be able to use our existing Subversion hooks, much like our
existing Git ones should also be usable from what I understand.

I've yet to look into the complete power of Herald and it's pre-commit
powers, but it is certainly possible that it is capable of doing some
of this, yes.

At least in terms of bugs.kde.org - there isn't a plan to use that at
the moment.
We would need to consider how to prepare Dr Konqi for that, and how to
ease the migration there which is probably another project of work.

The Wikimedia migration of their instance (about 80,000 bugs) took
quite a while from what I understand - and we have ~337,000 bugs in
our system at the moment (composed of 1.49 million comments).

We do have Kanboard (todo.kde.org) though, for which migration might
make sense and should be fairly easy to achieve.

Thanks,
Ben

Re: Sysadmin report on the modernization of our infrastructure

By Milian Wolff at 01/21/2015 - 10:33

On Wednesday 21 January 2015 17:12:07 Ben Cooksley wrote:
Hello Ben,

thanks for this thorough explanation, it fills in quite some gaps that I was
not aware of.

Phabricator does look interesting, and I'd be willing to experiment with it as
well. But, and this is a big but, there are some issues I have with the
proposal:

a) the scratch repo functionality not being present yet.
b) <a href="https://secure.phabricator.com/T4333" title="https://secure.phabricator.com/T4333">https://secure.phabricator.com/T4333</a> <-- this is a serious blocker

Does this mean we will wait until upstream has this implemented before we
start moving KDE? I'd dislike doing a move, or even a test-run, before all
required functionality exists.

For those who want to try out Phabricator, I found <a href="https://showoff.phab.io/" title="https://showoff.phab.io/">https://showoff.phab.io/</a>

Furthermore, some open questions from my side, regarding Phabricator:

- is it really trivial to implement commit hooks, such as closing bugs in
bugzilla, with herald? With the demo above, it is not clear how to do this. I
can only "Block Change With Message" in a commit filter.
- is it possible, and if so - how easy is it, to add bots to the review
system? i.e. something like a nitpicker bot similar to the Qt Sanity Bot. The
herald demo above does not seem to support running scripts which go beyond the
simple point-and-click GUI.
- Could, eventually, our CI be integrated? Such that, when a change is landed
which breaks the build, it's automatically reverted and/or a comment added to
the review page? There seems to be a "build plan" for phabricator - what is
that?
- will reviews be enforced, or will they stay opt-in as we have it in the
current KDE workflow?
- again a herald-related question: will the IRC bots continue to work which
announce new commits?
- is there still an ATOM/RSS feed of commits? I could not find that in the
phabricator demo (note: must work without prior login)
- can "common" herald rules be offered to users to save them some work? I.e. a
simple button to get an email for every "kdevelop" or "frameworks" or "kdepim"
or ... related review/commit/issue/... the more git repos we add, the bigger
this issue becomes
- you said projects.kde.org required custom patches, e.g. for adding support
for selecting the i18n branches. how is this going to be different in
Phabricator? what existing functionality there will supplement this, thereby
obsoleting custom patches?
- you also mentioned the issues around Gitolite metadata generation,
kde_projects.xml etc. - how will Phabricator address these issues?

Overall, I'd welcome if a simple dummy phabricator instance could be set up
such that we all can play around with the features it provides, including
pushing changes and trying out arc.

Bye

Re: Sysadmin report on the modernization of our infrastructure

By Ben Cooksley at 01/21/2015 - 15:25

On Thu, Jan 22, 2015 at 3:33 AM, Milian Wolff < ... at milianw dot de> wrote:
Hi Milian,

Not a problem.

Of course.

Sorry, need to clarify here - scratch repository functionality can be
provided using a ~20-30 line patch which would allow developers to
freely create repositories with a given name.

In terms of the 'arc land' issue we plan to work with the necessary
folks upstream to get that implemented as soon as possible.

We'll work on providing a patch to upstream should need be to get that
issue out of the way.

Phabrictor supports using regular Git hooks as well, so our existing
ones can simply be dropped in place.

It is certainly possible, assuming the bot exists.
Please see <a href="http://www.dctrwatson.com/2013/01/jenkins-and-phabricator/" title="http://www.dctrwatson.com/2013/01/jenkins-and-phabricator/">http://www.dctrwatson.com/2013/01/jenkins-and-phabricator/</a>

Build plans are part of it's integrated Harbormaster CI system.
We could either utilise it to trigger Jenkins (see
<a href="http://www.guywarner.com/2014/05/integrating-jenkins-and-phabricator.html" title="http://www.guywarner.com/2014/05/integrating-jenkins-and-phabricator.html">http://www.guywarner.com/2014/05/integrating-jenkins-and-phabricator.html</a>)
or use the method documented above to trigger Jenkins.

They will remain opt-in.

Our existing IRC Bot, pursuivant, will be unaffected as it is fed by
our current commit hooks - which we can continue to use.

I don't think so - please see <a href="https://secure.phabricator.com/T1928" title="https://secure.phabricator.com/T1928">https://secure.phabricator.com/T1928</a>
People could replace this with Herald rules and receive the notices by
email instead though.

You mean some kind of template?
It does support global rules, but they're quite different in nature.

It is quite likely users would have to add the repositories they're
interested one by one to their own Herald rule.

We will be shifting the generation of kde_projects.xml off to a
separate script which doesn't need to be updated as aggressively.
This will allow us to only regenerate the file when necessary and will
make it independent except for retrieving a list of repositories
(which can be done using the Conduit API).

As for i18n branches, i'll be discussing this with Albert, etc. to
determine the future of this functionality.

Of course.

Thanks,
Ben

Re: Sysadmin report on the modernization of our infrastructure

By Milian Wolff at 01/22/2015 - 08:27

Hey, thanks for the clarification. I'll remove everything which I think was
sufficiently answered by you.

<snip>

OK, but I don't understand half of the article.

"You will need to install Arcanist wherever you plan to have Jenkins run these
Phabricator builds" - on the server nodes that run Jenkins? Or on the
developer machines?

And the libphutil stuff as well - where would that be put? On the Phabricator
server?

And the phabricator python API bindings means we need to write a script,
right? I don't see it linked from the blog post, what am I missing? Maybe this
is clear to you as an admin who looked into jenkins and phabricator. To me, it
looks quite complex to get up and running. Anyhow, the blog post below seems
to be simpler, from my layman POV.

From <a href="http://phabricator.org/applications/:" title="http://phabricator.org/applications/:">http://phabricator.org/applications/:</a> "Harbormaster (alpha)
Continous build, not for the faint of heart."

That does not give me good confidence... But the two blog posts do seem to
indicate that it can be made to work, somehow. And hopefully it will be made
stable over time.

<snip>

This is bad for me. Emails cannot easily be embedded into websites. On
KDevelop.org e.g. we show a list of recent commits, which we get from
aggregated RSS commit feeds.

But from skimming the bug report, it seems as if they do have Atom feeds, just
not RSS? That would be sufficient. Can you confirm that this exists?

Yes, like a template. The global rules cannot be disabled-by-default and
enabled on a per-user level, right? So that is not option. I think this is
definitely something that we should report upstream. People keep saying KDE is
loosing community cohesion. Having the ability to easily track all KDE repos,
or all frameworks repos, or all kdepim repos or... is something extremely
important for us.

<snip>

Thanks again!

Re: Sysadmin report on the modernization of our infrastructure

By Ben Cooksley at 01/22/2015 - 22:34

On Fri, Jan 23, 2015 at 1:27 AM, Milian Wolff < ... at milianw dot de> wrote:
No problem.

These blog posts concern solely the setup on the CI nodes and Jenkins master.
No changes would be needed on developer systems to support initiating
CI builds for revisions.

For that particular post - we would need to install Arcanist on all of
the build nodes behind build.kde.org.
The libphutil part you see there goes on the Phabricator instance
itself - and is used to inform Jenkins that it needs to perform a
build.

The script in question is included in that post, and is also available
at <a href="https://gist.github.com/dctrwatson/4669113/raw/phabricator-post.py" title="https://gist.github.com/dctrwatson/4669113/raw/phabricator-post.py">https://gist.github.com/dctrwatson/4669113/raw/phabricator-post.py</a>
We would need to install the appropriate Phabricator Python API though.

Harbormaster is eventually intended to be a full blown CI solution -
like Jenkins.
At the moment it is only capable of doing some very basic operations,
hence the Alpha label.

I see. Always helps to have a use case :)

It has JSON format feeds available from it's Conduit API, but from
what I can see ATOM/RSS aren't available.

Another way of doing this would be to associate repositories with projects.
People can then use Herald to subscribe to commits made to a
repository's which belong to those projects.

This would be fairly easy for someone to setup if it works how I think it works.

Thanks,
Ben

Re: Sysadmin report on the modernization of our infrastructure

By =?utf-8?Q?Thoma... at 01/23/2015 - 11:05

Would that boil down to "For the moment, we patch CI onto Phabricator with
a custom script based on arcanist event listeners and ad some point later,
it (Jenkins) will be replaced by Harbormaster, the CI supposed to be
integrated into Phabricator"?

Cheers,
Thomas

Re: Sysadmin report on the modernization of our infrastructure

By Ben Cooksley at 01/23/2015 - 17:16

On Sat, Jan 24, 2015 at 4:05 AM, Thomas Lübking
<thomas. ... at gmail dot com> wrote:
Assuming Harbormaster meets our needs and we decide to migrate to it,
then that is approximately correct yes.
We would probably follow the method in
<a href="http://www.guywarner.com/2014/05/integrating-jenkins-and-phabricator.html" title="http://www.guywarner.com/2014/05/integrating-jenkins-and-phabricator.html">http://www.guywarner.com/2014/05/integrating-jenkins-and-phabricator.html</a>
though as it is cleaner implementation wise - and also provides better
integration for the CI system (Jenkins) into Phabricator.

Thanks,
Ben

Re: Sysadmin report on the modernization of our infrastructure

By Matthew Dawson at 01/24/2015 - 19:23

On January 21, 2015 05:12:07 PM Ben Cooksley wrote:
Thanks for putting this report together! I think the outline of the current
issues and the listed requirements are very useful, and as outlined
Phabricator does seem like a good contender.

I do have one important question regarding its review system, how does it
handle a series of commits? For more complicated changes, there may be
several commits to get from point A to B that I'd like to get reviewed.
ReviewBoard doesn't currently handles this, and instead squashes them all into
one patch (see for instance: <a href="https://git.reviewboard.kde.org/r/117010/" title="https://git.reviewboard.kde.org/r/117010/">https://git.reviewboard.kde.org/r/117010/</a> , which
was originally made up of 5 different commits). One of the plus points of
Gerrit for me was that it would have shown each of these commits separately
(though having each become a change isn't ideal to me). How would Phabricator
handle a set of commits like this?

Re: Sysadmin report on the modernization of our infrastructure

By Ben Cooksley at 01/25/2015 - 06:51

On Sun, Jan 25, 2015 at 12:23 PM, Matthew Dawson < ... at mjdsystems dot ca> wrote:
Not a problem.

The actual diff itself is shown as a single merged entity ala Reviewboard.
However the chain of commits used to generate the change (assuming you
use Arcanist to submit it) is shown in the user interface, complete
with SHA's and commit messages.

Cheers,
Ben

Re: Sysadmin report on the modernization of our infrastructure

By Thomas Friedric... at 01/24/2015 - 10:24

Hi!

On Wed, 21 Jan 2015 17:12:07 +1300
Ben Cooksley < ... at kde dot org> wrote:
Thanks a lot for all your hard work. I think Phabricator definitely
looks much more newcomer-friendly on the surface than Gerrit, although,
knowing there is a web-form available for uploading diffs to Gerrit
(will that handle follow-up patches, too?) goes quite some way in
addressing the entry barrier issue.

But beyond review functionality, I think moving towards a more
integrated solution is clearly a step in the right direction, and this
is what makes the choice of Phabricator over Gerrit a clear case to me.

Also, definitely a huge thank you to Jan for investing so much work in
the Gerrit setup. Clearly that was in the hope of making Gerrit the
solution of choice, and I think we can all feel your pain, if that is
not going to be the case. But I don't think it would be the right way to
go for KDE.

Regards
Thomas

Re: Sysadmin report on the modernization of our infrastructure

By =?utf-8?Q?Thoma... at 01/26/2015 - 22:30

I assume we all agree in wanting to have a more integrated solution than
the status quo (which is basically not integrated at all), but keep in mind
that "integration" does not require a "vendor lock-in", eg. not two items
in my HiFi system are from the same vendor.

A fully integrated pre-solution might look nice on a glimpse, but the
pre-integration does not necessarily compensate for defects in the
components.

So far, this entire thread has actually only ever addressed the review
process or maybe SCM entirely (unless I missed something) and the
pre-integration has been brought up as a benefit (and from sysadming POV, I
do certainly understand why), but I wonder whether the other required
components have been checked for feature completeness.

Eg. I do not see the Phabricator Task tracker approach to be fundamentally
different from (read: "superior to") bugzilla (notably the comments seem to
be just as linear and there also does not seem to be a "summary" comment to
be edited by maintainers anytime - two things I would actually like in a
bugtracker), but even a short research suggested it neither does suggest
duplicates, not is there intention to add such.
To me, this raises the question whether it's actually capable (or will ever
be) of replacing (let alone "surpassing") the current bugzilla solution.

For another example (and vastly discussed) the integrated CI is "when it's
done", nobody has actually seen it in process, so we might end up with the
patched-on Jenkins forever.

Also the definitive blocker of scratching on-behalf-commit meta-info (for
the SCM) has been brought up.

I'm not familiar enough with either Jenkins nor Zuul to say which one's
"better" and I cannot judge about the Phabricator bugtracker feature
completeness, but if we end up with having to use external components
because the pre-integrated ones do not fit our requirements (for an
undefined time), the pre-integration argument is simply void.

Bottom line is:
And to asset the advance of the pre-integrated solution, that solution has
to be evaluated in its completeness - as one has to presume that it might
be much harder to replace components of a solution that is meant to run en
bloc, than it would be to plug together components which provide a public
API for that very designed purpose.

Cheers,
Thomas

Re: Sysadmin report on the modernization of our infrastructure

By Thomas Friedric... at 01/27/2015 - 07:19

Hi,

On Tue, 27 Jan 2015 03:30:48 +0100
Thomas Lübking <thomas. ... at gmail dot com> wrote:
I think vendor lock-in is just the wrong term, here. It's not like it's
not possible to keep using bugzilla with phabricator, for instance.
It's just that you don't get some integration features, or only at
considerably more effort. Arguably, at this level, "nothing" has kept
us from achieving similar integration with the current setup.

Naturally. And clearly a "pick tool A from X, B from Y"-kind of
approach will - at least on paper - allow to satisfy more wishes.
Unless you take integration and maintenance effort into account. (Which
is still not actually saying that Gerrit would be the solution to
satisfy "all wishes" in the review department; clearly there are
"defects" of debatable severity to be found in this solution, too).

Ideally that would be done in much more detail, true. There does seem
to be a certain level of impatience about actually starting to move in
some direction, though (and that may make a lot of sense). So the
options are
1. spend some more months on open evaluation and testing
2. commit to solution X (barring "real" blockers to be discovered), and
deal with issues as they come up
3. split the problem into smaller portions, by looking at one tool at a
time. Sounds good on paper, but - again - adds unnecessary prejudice
against any integrated solution.

Well, as far as I see it, the key points of "superiority" would be:
- automatic cross-linking with review requests and tasks
- cross-linking with project(s) (and actually, lack of this is the
number one issue that keeps driving me mad with b.k.o, and its
drop-down list of one-gazillion projects+subcomponents)
I.e., once again: Not fundamentally different, but integrated.

Regarding the dupes issue, the comment you refer to, is the developer's
explanation, why the feature does not currently exist. The issue
also has a patch for review.

Regarding the summary comment: It's called "description", in
phabricator.

A risk sysadmins seem willing to take.

To be honest, I think terms like "definite blocker" should be used with
more care (on both sides of the argument, of course). Esp. as this is
not available in reviewboard, either (for all I am aware), and there is
a workaround available (requiring three
commands instead of one).

Bad: yes. Blocker: Come on, now.

True. And I can't give you answers to all your questions, either. But I
actually do trust that sysadmins that done the _basic_ checking well
enough.

My bottom line: Potentially integrated is just not the same thing as
pre-integrated. Discarding the pre-integrated solution, because you have
not yet managed to evaluate all features and defects of all potentially
interesting components, is not going anywhere.

I also don't think your presumption holds water. At least for the
review-component (the one we're talking about, initially), an API
clearly exists. Further, upgrading parts or all of a manually integrated
solution might be much harder than upgrading a pre-integrated solution.

Regards
Thomas

Re: Sysadmin report on the modernization of our infrastructure

By Ben Cooksley at 01/27/2015 - 04:51

On Tue, Jan 27, 2015 at 3:30 PM, Thomas Lübking
<thomas. ... at gmail dot com> wrote:
Agreed and agreed.

Jenkins provides rich tracking of tests, code coverage and code
quality (eg: cppcheck) in addition to checking if it builds.
Zuul is designed to determine if it builds and if tests fail -
providing a binary pass/fail response.
Harbormaster at this point at least looks like it will be similar in
nature to Zuul.

I'll also note that Jenkins provides scheduling and can bring nodes up
and down as needed (when equipped with access to a cloud/cluster).
For this reason Openstack is still relying on Jenkins in part as Zuul
can't do this.

Jenkins also permits us to track jobs unrelated to our code reviews,
which is necessary for
Hmm? It has a description section on tasks. See
<a href="https://secure.phabricator.com/T1315" title="https://secure.phabricator.com/T1315">https://secure.phabricator.com/T1315</a> for an example.
This can also be freely edited.

I'm not aware of any threaded discussion issue tracker, but i'm happy
to be corrected.

It is mainly intended as a task tracker rather than a bug tracker I
believe, hence this position.
The more suitable item to compare it to would therefore be todo.kde.org.

I concur here that pre-integration is void if the pre-integrated
components aren't useful.

For those that are worried - people have integrated JIRA and
Phabricator together, so it is more than possible to connect
Phabricator with external task trackers.

Please recall that no change of bug tracker or CI system is being
planned at this time - such a change would be for future discussion.

Thanks,
Ben

Re: Sysadmin report on the modernization of our infrastructure

By Ben Cooksley at 01/27/2015 - 05:00

Hi all,

Based on the feedback we'll now begin putting together a test instance
so Phabricator can be seriously evaluated by the community.

I've noted that the following projects have expressed interest in using it:
- Calligra/Krita.
- Zanshin.
- KActivities.
- Konversation.

If any other projects would also like to evaluate it, please let us
know - more are welcome to evaluate it as well.
We'll be in touch with you all later in the week to make the
preparations to begin the test.

Once a sufficient period of testing has been completed with
Phabricator we can then come back and make a final decision on what
will be implemented.

Thanks,
Ben

Re: Sysadmin report on the modernization of our infrastructure

By =?iso-8859-1?Q?... at 01/27/2015 - 05:21

On Tuesday, 27 January 2015 09:51:46 CEST, Ben Cooksley wrote:
This is not true. Please read Zuul's documentation at [1] (or the
modernization proposal I'll be sending later today) for a short overview.

This is not true.

- OpenStack uses Nodepool for node management (VM teardown & bringing up
new nodes, and image building), not Jenkins.
- The reason why OpenStack uses Jenkins *with* Zuul rather than just Zuul +
Turbo-Hipster is purely due to inertia and less work required due to their
history.
- OpenStack's use of Jenkins is limited to acting as a conduit for
launching jobs.
- Zuul knows how many resources are online at any time, so Nodepool can be
(and is) used with it just fine.

So does Zuul; the OpenStack project is building release tarballs and have
nightly QA processes on stable branches, all with Zuul.

Fair enough, but then the argument of a "fully integrated solution" should
not be advertised as Phabricator's advantage, IMHO.

Cheers,
Jan

[1] <a href="http://ci.openstack.org/zuul/" title="http://ci.openstack.org/zuul/">http://ci.openstack.org/zuul/</a>

Re: Sysadmin report on the modernization of our infrastructure

By Inge Wallin at 01/27/2015 - 05:33

On Tuesday, January 27, 2015 03:30:48 Thomas Lübking wrote:
And this is of course the reason why the sysadmins have asked for projects
that would volunteer as guinea pigs for the new system. And several have
volunteered so we might soon know.

Since one of the sysadmin's complaints of the competing solutions was that
they didn't well integrate without lots of hacks, I suggest using the wording
"If you can..." instead.

And I think that this is the point that has gotten lost in the discussion at
large. Gerrit may well be just as good as Phabricator or perhaps even better
at code review; I simply don't know. But for overall integration with the
rest of the infrastructure it's just one piece of the puzzle that needs to be
interfaced to - again.

The git hosting needs to be switched to something faster, the bug tracker
needs to be integrated, projects would like to have their own mini-websites
close to but not inside KDE. And so on.

Can we please put a little more emphasis on this part of the issues too?

Re: Sysadmin report on the modernization of our infrastructure

By Marco Martin at 01/22/2015 - 14:41

On Wednesday 21 January 2015 17:12:07 Ben Cooksley wrote:
thanks for the analysis, work much appreciated :)

as my just two cents, i'm quite happy with gerrit, has an horrible interface,
but i like what it does.

about phabricator, can't say anything, never tried, if any project want to
test it as it seems from the thread would be happy to try it.

but in the end, I would be quite happy even if we go with just gerrit.

Re: Sysadmin report on the modernization of our infrastructure

By Eike Hein at 01/21/2015 - 17:20

On 01/21/2015 05:12 AM, Ben Cooksley wrote:
As someone on the original git infra team, I'm impressed - thanks
for your hard work.

Cheers,
Eike

Re: Sysadmin report on the modernization of our infrastructure

By Dominik Haumann at 01/22/2015 - 16:04

On Wednesday 21 January 2015 22:20:18 Eike Hein wrote:
As someone not involved in sysadmin tasks, I'm also implressed :-)
I trust the thorough research and I'm all for just giving it a try. There
are already plenty of volunteers to test this.

One tiny question (or rather proposal): Will we be able to have a 1:1 mapping
of the bug numbers of our ~377000 bugs on bko? That would help immensely also
to find issues again even if bugzilla is completely dropped.

Keep it up! Thanks,
Dominik

Re: Sysadmin report on the modernization of our infrastructure

By Ben Cooksley at 01/22/2015 - 22:08

On Fri, Jan 23, 2015 at 9:04 AM, Dominik Haumann < ... at kde dot org> wrote:
This will depend on when we run the migration. If we start using it
for task tracking before we migrate from Bugzilla, then we might have
to increase the bug numbers by a specified amount like Wikimedia had
to. Either way, the bug numbering should be kept intact in some form
or another.

If the Phabricator test is successful and is selected by the community
then we'll probably migrate our Git infrastructure first, before
conducting another round of consultation for Bug tracking.

Thanks,
Ben

Re: Sysadmin report on the modernization of our infrastructure

By Scarlett Clark at 01/21/2015 - 18:00

When the test system is in place I would like to start my Jenkins integration.
So let me know!
Thanks
Scarlett

Re: Sysadmin report on the modernization of our infrastructure

By =?utf-8?Q?Thoma... at 01/21/2015 - 11:28

Some open questions:

- Do we have access to Qt's (ie. KDE's major upstream) decision process (reasoning) towards gerrit?

- The major concern about gerrit seems about scratch repos and I think I found a quite old bug/request on this which might cross "trivial" approaches w/ scripts? [1]
Otoh, it seems Phabricator doesn't support scratch repos right now either.

- The lack of non-repo driven (web interface) patch supplies in gerrit does not seem to have been addressed in the document, but there've been many calls for such. So what's the state on this?

- Phabricator is suggested to have integration w/ a bugtracker
-> Can gerrit be easily integrated w/ bugzilla (or other trackers), notably by cross-referencing the review (once! ;-) for the "BUG" keyword? (link and tag that this bug is being worked on - tag must be removed in case the patch is abandoned) - and of course closing as well.

Cheers,
Thomas

[1] <a href="https://code.google.com/p/gerrit/issues/detail?id=1142" title="https://code.google.com/p/gerrit/issues/detail?id=1142">https://code.google.com/p/gerrit/issues/detail?id=1142</a>

Re: Sysadmin report on the modernization of our infrastructure

By =?iso-8859-1?Q?... at 01/21/2015 - 15:55

On Wednesday, 21 January 2015 16:28:20 CEST, Thomas Lübking wrote:
The bug you linked to is about something else than scratch repos as far as
I see.

There's a tool for that, <a href="https://tools.wmflabs.org/gerrit-patch-uploader/" title="https://tools.wmflabs.org/gerrit-patch-uploader/">https://tools.wmflabs.org/gerrit-patch-uploader/</a> .

Yes, see my other mail once it clears the moderation queue.

Cheers,
Jan

Re: Sysadmin report on the modernization of our infrastructure

By =?utf-8?Q?Thoma... at 01/21/2015 - 18:12

My uninformed guess would be to handle the change-id smarter, ie. bind it to a branch (and pot. repo)

Yupp, thanks. Sounds feature complete (and we're not bound to one bugtracker - though i don't know how good Pahbricator is on this)

Cheers,
Thomas

Re: Sysadmin report on the modernization of our infrastructure

By =?iso-8859-1?Q?... at 01/21/2015 - 21:40

On Wednesday, 21 January 2015 23:12:33 CEST, Thomas Lübking wrote:
IMHO, a pretty straightforward option is to close bugzilla entries once a
change is approved and lands in an appropriate branch, and that's easy with
Gerrit's its-bugzilla plugin. It is possible to specify which branches are
appropriate, of course.

Yes. Their instance doesn't keep a "secondary index" which limits
searching, but a proper query would be something like
"owner: ... at example dot org OR comment:'uploaded by ... at example dot org'". That
works on our Gerrit.

Jan