DevHeads.net

Git Worflow, branch creation.

Hello,

Not sure this was discussed before but I'd like to bring the topic for
discussion.

Now that KDE moved to git it seems we still have old habits like
"Freezing period" on trunk (or should I say master branches of each
git repos). That sounds like weird to me when one of the main feature
of git is branch management. Unlike SVN it's easy for anyone to switch
from a branch to another one, making back-porting patches very easy,
checkouting being 10 times more easy.

I know it's a wider announcement so that people focus on bug fix
rather than feature (which is great). Now I don't understand why the
branching of let say 4.7 does not happen before on each repo (when the
freeze is decided) and we could make the policy on branches way more
clear and defined what could go on the branch and what can't go (just
using the procedure we have today). It has always been annoying the
fact that you couldn't hack features in the official trunk, people had
to use personal svn branches and now teams like plasma think about
setup integration repos. That sounds weird, you usually use these
extra branches for huge stuff you don't want to put in trunk right now
(like big refactoring, or research) not for every little features or
improvements projects want to push. To me the term "trunk" means it's
always open to features. I'm sure the Plasma need is not alone and we
will see these integration repos pop for not particular reason and
will mess up the history (with merge commit), make the checkout
complicated for people to use vanilla and make the KDE project
complicated to understand.

I looked around and all projects using git I've tried (and I'm not
saying I covered all :D) have a trunk always open to features. Ok it's
not as complicated/big as KDE but still.

Perhaps I miss a point here so if it's the case, could you please tell me?

I'm trying to see if our workflow could be improved.

Thanks.

Comments

Re: Git Worflow, branch creation.

By Aaron J. Seigo at 05/18/2011 - 07:06

On Tuesday, May 17, 2011 11:40:30 Alexis Ménard wrote:
personally, i don't care whether there is a separate repository for
integration or its all done in branches in one repository. there are benefits,
and challenges, to doing it either way.

what we want to move away from, however, is a master branch that is open to
all and any changes at any point in time. we want a branch where things are
integrated and tested by the main group of people working on the software and
those who follow it closely. that branch will have feature branches merged
into it, bug fixes applied to it, etc. then, at regular intervals that allow
for stabilization, that branch will be merged into master.

we want the ability to use Feature A with Feature B without running that
experiment in master. we want master to be as stable as possible at all times
without removing the ability to run experiments and confirm that thing work
well together, which means moving what we do today in master to another
branch.

the ability for developers to delete branches would also be very helpful in
that regard.

but the real problem goes a lot deeper here, imho. the reason for a feature
freeze in master is that it assumes we're all working on features *in* master
(or, previously, in trunk). we may wish to rethink that approach in general.
if all new feature devel is done in branches, then a "feature freeze" would
mean "no feature branches that haven't been merged by now can be merged". how
to manage that process, including translatation updates and multiple merges
over time from a given feature branch, is what needs planning.

i don't want to start this discussion here on this mailing list as i doubt it
will result in anything useful, given the constraints on bandwidth and
opportunity to easily diverge into a thousand sub-topics. let's ensure this
gets attention at up-coming face-to-face meetings and work out a solution
there. Platform 11 in a couple of weeks could look at this for kdelibs, and as
a broader community we could take it on at the Berlin Desktop Summit.

Re: Git Worflow, branch creation.

By Alexander Neundorf at 05/18/2011 - 16:08

On Wednesday 18 May 2011, Aaron J. Seigo wrote:
...
From my side, I would strongly prefer to have one policy which is used at
least for all of KDE SC, (what was trunk/KDE/ before), if possible also for
kdesupport.
For somebody who from time to time looks at all projects in that scope (at the
cmake stuff) it would make things very complicated if different policies are
used by different repositories/projects.

Alex

Re: Git Worflow, branch creation.

By Aaron J. Seigo at 05/19/2011 - 03:13

On Wednesday, May 18, 2011 22:08:00 Alexander Neundorf wrote:
agreed. and in working on this:

<a href="http://techbase.kde.org/Development/Tutorials/Git/Feature_Development_Workflow" title="http://techbase.kde.org/Development/Tutorials/Git/Feature_Development_Workflow">http://techbase.kde.org/Development/Tutorials/Git/Feature_Development_Wo...</a>

that is our goal. to make something that we can propose to all of KDE. the
above workflow documentation will change a lot before then as people offer
input, we discuss at it face-to-face meetings (the above was the result of
such a small gathering at Tokamak 5) and we actually try it out.

what we do need and are still lacking is a writte nlist of requirements that
all our stakeholders have so that we can craft a solution that meets them.
right now the requirements are being passed around "orally" and that's
probably not good. to that end, i've added a (perhaps temporary) Requirements
section to the above wiki page:

<a href="http://techbase.kde.org/Development/Tutorials/Git/Feature_Development_Workflow#Requirements" title="http://techbase.kde.org/Development/Tutorials/Git/Feature_Development_Workflow#Requirements">http://techbase.kde.org/Development/Tutorials/Git/Feature_Development_Wo...</a>

please feel free to add items to it that may be missing.

Re: Git Worflow, branch creation.

By Ben Cooksley at 05/19/2011 - 04:20

2011/5/19 Aaron J. Seigo < ... at kde dot org>:
I see quite a few problems with that workflow:

First you are using a repository outside the main repository. Scratch
and clone repositories do not have email or CIA notification provided
to them due to both configuration issues and to the need to permit
those whose personal workflows depend on rebasing to work.

This means those who would normally do code reviews as it takes place,
as well as those who monitor the state of KDE such as the Commit
Digest are effectively kept out of the loop until it is pushed to the
final repository.

Second you are heavily advocating rebasing. This shouldn't be done in
public repositories as it:
- Inflates the size of the repository.
- Requires some Git magic to recover if the branch you are currently
using gets force pushed.
- Is detected as new commits by the hooks, which if you were to
operate in the main repository would cause re-notification of commits.

If you were to continue with your current workflow, then you would
likely need the assistance of Sysadmin everytime you wanted to merge a
significant feature which comprises of more than 100 commits into the
main repository. The hooks forbid pushes of more than 100 commits in
order to protect our infrastructure - and that of others such as CIA.

It also makes it harder to perform in place reviews as the code may
have changed significantly, demotivating those who would normally
perform in place review (these reviews predate Reviewboard existing -
and still continue to this day)

Please reconsider your proposed workflow.

Regards,
Ben Cooksley
KDE Sysadmin

Re: Git Worflow, branch creation.

By Stephen Kelly at 05/19/2011 - 05:44

I thought git gc which runs periodically would mean that the repo size is
not inflated?

Do wee have any information about feature branches of 100 commits? When KDE
developers push local branches how many commits are on them? Do we have
numbers about that?

Re: Git Worflow, branch creation.

By Thiago Macieira at 05/19/2011 - 06:22

On Thursday, 19 de May de 2011 11:44:59 Stephen Kelly wrote:
Correct.

100-commit feature branches probably will do merges and thus cannot be
rebased.

A rebasing branch is usually done by a single developer and should have in
average up to 10 commits. 20 at the outset, then it starts to get really hard
to maintain.

Re: Git Worflow, branch creation.

By Ben Cooksley at 05/19/2011 - 06:30

On Thu, May 19, 2011 at 10:22 PM, Thiago Macieira < ... at kde dot org> wrote:
Doesn't apply to KDE repositories, as performing a rebase involves a
force push, which initiates the damage prevention area of our hooks.
This triggers creation of a "backup ref" protecting the contents of
the old ref from being affected by a gc, and ensures they are always
reachable.
It is a protective measure to guard against malicious force pushes or
branch deletions. (Note that git doesn't fetch backup refs normally)

Further, if the code you rebased is still accessible in it's original
form in another branch people will still need to download those
commits when they clone the repo.

Re: Git Worflow, branch creation.

By Stephen Kelly at 05/19/2011 - 12:48

This is interesting. Has this always been the case? I don't remember this
coming up in previous discussions about rebases and force pushes. Can this
feature be limited to release branches (master, x.y.z), just out of
curiosity?

Re: Git Worflow, branch creation.

By Ben Cooksley at 05/19/2011 - 16:09

On Fri, May 20, 2011 at 4:48 AM, Stephen Kelly < ... at gmail dot com> wrote:
Ever since we got the new hooks. The backups don't affect developers
as such, it only causes an increase of the repository size on the
server. Git doesn't retrieve the contents of backup refs by default.

It needs to run on all branches/tags/refs to ensure that abuse is prevented.

The more developer focused side of repo growth with rebasing usually
occurs when a branch is rebased on top of another branch - and the
original isn't deleted. Therefore everyone ends up downloading those
commits effectively twice.

Regards,
Ben

Re: Git Worflow, branch creation.

By Thiago Macieira at 05/19/2011 - 06:39

On Thursday, 19 de May de 2011 22:30:35 Ben Cooksley wrote:
Right, but those backups should be deleted after some time.

Re: Git Worflow, branch creation.

By Aaron J. Seigo at 05/19/2011 - 04:56

On Thursday, May 19, 2011 20:20:13 Ben Cooksley wrote:
thanks for the input; i've added some of the core issues you raise to the
"Requirements" section of the wiki page. this will help us craft something
that fits the actual needs of us all.

i'd like this to become "our" workflow rather than "your" workflow. we need to
start somewhere, and here is as good a place as any. if nothing else, it ends
the stalemate of "nobody moves, nobody gets hurt" around defining a git based
workflow for our efforts. but it will only produce good results if we, as a
community, take it on as "ours". not "yours", not "mine", not "theirs".
"ours". because then we'll participate with eagerness and expectation. :)

those of us involved thus far are doing so because no one has really stepped
up yet. however, we are not interested in dictating anything. we want
participation, we want discussion, we want to come up with something that
Works(tm) which means being completely open to adjusting any parts that need
adjusting until we have something that does work for us all (as much as that
is possible, of course :)

Re: Git Worflow, branch creation.

By Thiago Macieira at 05/19/2011 - 04:44

On Thursday, 19 de May de 2011 20:20:13 Ben Cooksley wrote:
Advocating rebasing is a good thing, so I have to interject here. However, it
only works as long as people are not basing their work on their branch, or are
properly informed of it.

Personal branches should be rebased. Git's own "pu" branch is rebased every
now and then and people know what to expect if they use that.

Re: Git Worflow, branch creation.

By Cornelius Schumacher at 05/19/2011 - 04:05

On Thursday 19 May 2011 Aaron J. Seigo wrote:
Wow. This is a pretty complex workflow, and it's breaking with quite some
practices KDE used to use for a very long time. We have to make sure that we
don't leave developers behind with such a drastic change.

The approach of having one central repository and all committers being equal
has served us well. Maybe it's time to move forward to a different model, but
I think this should be done with care, and without changing more than needed.
A lot of this is about semantics and how to name things, not necessarily about
technical processes. For example, if master is the stable branch or a release
branch doesn't make a big difference technically, but it might affect our
development culture quite a bit.

This needs to be discussed. I'm looking forward to the upcoming face-to-face
meetings, but we should also have a wider discussion, as this is affecting
many more people than those who will have the opportunity to be at these
meetings.

Re: Git Worflow, branch creation.

By Aaron J. Seigo at 05/19/2011 - 05:14

On Thursday, May 19, 2011 10:05:27 Cornelius Schumacher wrote:
this is why we need to have work on it together. while a few of us have taken
first steps in terms of drafting something, basically because someone has to
otherwise we all stay in limbo with nothing being done, this is more of a
provocative call to participate than anything even close to a final proposal.

if we don't participate as if we care about this, we will get a workflow that
doesn't serve us very well. which is what we have now. ;) so consider this a
process in creating our shared future workflow with everything on the table to
be modified, changed and improved as we see fit.

note that there wasn't unanimous agreement on this workflow at Tokamak, but it
at least gave us a starting point. we've openly documented it, brought the
discussion to k-c-d and want input. there's a reason for that: we don't want
to leave anyone behind and we want a workflow that, well, works. :) with a few
iterations of this approach, i think we can have something pretty damn good
put together.

i don't think that will change much. what is likely to change is that we won't
be developing in a one mainline branch that we expect to become stable. it's
nearly magical thinking that this is possible, in fact. in the goal of
increasing the predictability of master, we can (and should) all still be
equal in our interactions.

yes, hopefully we change only what is needed and no more. :)

agreed; note that the suggestion is not to make master _the_ stable branch
(release branches would still maintain that position) but to make master as
stable as possible over any given period of time.

right now we have unpredictable levels of stability in master depending on the
phase of the moon and where in the release cycle we are. this makes many
things more difficult (including making time based releases) than it probably
needs to be. at times it results in us shipping releases with more defects
than we ought to.

increasing the day-to-day stability and predictability of master by using
git's strength in merging could probably do wonders for our releases. not to
mention it would be very welcome by people working on device targets for which
the release date of a given KDE SC event is less important compared to always
having a place to pull from that is reasonably safe and not feature-stale.

let's just be careful with the definition of "wider". we need a representative
sampling of people from across KDE's community in terms of skill levels, needs
and perspectives (e.g. casual developer, core developer, sys admin, writers of
the commit digest ..). we don't need every possible individual involved,
however. as such, if we do these discussions at Platform 11 and then again at
BDS, and document results as we go, we probably will reach that level of
coverage.

we don't need everyone involved, we just need the various points of view and
needs represented. as such, i do think the F2F meetings are likely to suffice.

Re: Git Worflow, branch creation.

By Boudewijn Rempt at 05/19/2011 - 04:11

On Thursday 19 May 2011 May, Cornelius Schumacher wrote:
In Calligra, we sort of discussed that we might call the staging branch where everyone can commit "master", and that we'll try to get an automated system to copy commits that didn't break unittests to the release branch, which only the automated system can commit to. That keeps everyone in the loop on "master", at the expense of having a release branch that nobody really runs. We're not there yet, obviously :-)

Re: Git Worflow, branch creation.

By Thiago Macieira at 05/19/2011 - 04:42

On Thursday, 19 de May de 2011 10:11:04 Boudewijn Rempt wrote:
That's what we do in Qt and the result is less than stellar.

The trick is in your "copy commits" part. If you meant cherry-picking each
commit, that means you need to do it one by one, which in turn means having a
lot of computing power to test each and every commit. Moreover, it also stops
dead in the water any changes that require more than one commit. Making atomic
commits, that do each their changes is preferable, but this may introduce
broken states in the intermediary steps. That is, two commits together may not
break anything, but each separately would and your script would discard both.

If you meant merging everything periodically, that's how we do it in Qt. It
works, as long as someone is always fixing the problems. What happens to us is
that some of the "staging areas" end up broken due to a commit that does break
a unit test. Then every single commit there, past and new, is prevented from
being merged into the official "master" branch because there's one test failing.

The new solution we're working on, to be deployed with Open Governance, is to
have the CI system pull the commits directly from the code review tool. It
will take each and every approved commit / commit-set and test it. If it
works, that's great, it's integrated. If it fails, it's rejected with no side-
effects: the next approved commit-set will be taken instead.

Re: Git Worflow, branch creation.

By John Layt at 05/18/2011 - 15:27

On Wednesday 18 May 2011 12:06:18 Aaron J. Seigo wrote:
+1. I've added it to the agenda for Randa. I also want to shame people to
use a spare couple of hours while there to help update TechBase.

John.

Re: Git Worflow, branch creation.

By Maksim Orlovich at 05/17/2011 - 11:42

Testing.

If everyone is working on branches on cool new stuff, the release will be crap.

On 5/17/11, Alexis Ménard < ... at kde dot org> wrote:

Re: Git Worflow, branch creation.

By Ivan Cukic at 05/17/2011 - 11:52

On Tuesday, 17. May 2011. 11.42.41 Maksim Orlovich wrote:
I don't agree. Freezes just stop the development - it doesn't improve the
bugfixing as much.

The best way (IMO) would be to have all done in parallel.

Cheerio,
Ivan

Re: Git Worflow, branch creation.

By Andrea Diamantini at 05/17/2011 - 12:36

On 05/17/2011 05:52 PM, Ivan Cukic wrote:

Just my 0.02 cents.

Re: Git Worflow, branch creation.

By Fredrik =?utf-8... at 05/17/2011 - 12:16

On Tuesday 17 May 2011, Ivan Cukic wrote:
It's not just about keeping people focused on fixing bugs, it's about
developers actually running and testing the branch that we're about
to release.

Regards,
Fredrik

Re: Git Worflow, branch creation.

By =?ISO-8859-1?Q?... at 05/17/2011 - 19:21

On Tue, May 17, 2011 at 1:16 PM, Fredrik Höglund < ... at kde dot org> wrote:
Well it's the same you could run 4.7 branch (supposedly more
mature/tested than the trunk). And you could also point out that
branch to early adopters rather than trunk.
Also for distribution it's easier, they pick a branch (4.7) and make
nightly builds. It's more your responsibility as a KDE developer to
switch to that branch rather than sticking with trunk if you really
want to help to fix the stability when the release is about to happen.
I fail to see why 4.7 branch could be less tested than trunk if you
communicate it properly. Btw still on the old svn, you were sticking
with trunk or after the branching you were running the release branch?
To me it's the same but in the other hand you let the development to
continue (a bit like Firefox new approach/WebKit).

Re: Git Worflow, branch creation.

By Ivan Cukic at 05/17/2011 - 12:27

With that I agree. The idea in plasma-world was to keep an always
releasable branch. That is, things get in from the development branches
only after they are finished and reviewed.

This way the things get tested, and there is no cooling period for the
development.

Ch