DevHeads.net

dist-git project update

It's been a while since I last updated folks on dist-git, and in reality
it's been a while since I last worked on it. Fedora 13 took up all my
time.

Since my last update we've made great progress on fedpkg, the new tool
that will replace the make system. It is packaged up with
fedora-packager and has the ability to do many tasks that our Make
system handled. Here is a quick list:

build
chainbuild
clean
clog
clone
compile
gimmespec
install
lint
local
mockbuild
new
new_sources
prep
scratchbuild
sources
srpm
unusedpatches
verrel

Many of these targets take optional arguments which extend their
functionality and replace some other specific Make targets. This list
is enough to get us checking code out and in, and building in koji.

On the koji front we've recently discovered the changes necessary to
build from dist-git style repos, and those changes are being polished up
and committed upstream. We have done multiple builds successfully from
dist-git repos.

Where do we go from here?

We're ready for more wide scale testing, and to facilitate that I am
refreshing the git repos from current CVS (people with existing clones
will have to blow them away and re-clone), and getting a koji stage
instance up that we can build against (with limited builders). We'll
then push out another fedora-packager update that has the right URLs to
build against this stage Koji and announce that it is ready for
building.

Based on this testing, and some decisions around git tagging and branch
usage, we stand a good chance at being able to roll this out prior to
the F14 branch event. I hope you are all as excited as I am about this!

Comments

Re: dist-git project update

By Andreas Schwab at 06/14/2010 - 03:57

Jesse Keating < ... at redhat dot com> writes:

Doing git fetch and git reset --hard should be enough.

Andreas.

Re: dist-git project update

By Rahul Sundaram at 06/14/2010 - 03:31

Is the effort to make it easy to build RPM's directly from git tags
related to this or is that a separate project?

Rahul

Re: dist-git project update

By Jesse Keating at 06/14/2010 - 17:57

That's a different project, and even then we will likely need to
construct a source tarball at some point in the process (behind the
scenes) or make even more significant changes to Koji.

Re: dist-git project update

By Toshio Kuratomi at 06/14/2010 - 17:11

If you mean the Source0: is a git URL, then separate.

-Toshio

Re: dist-git project update

By Kevin Kofler at 06/13/2010 - 12:54

Count me as not excited.

As I already pointed out several times, I don't see anything obviously wrong
with our CVS setup, so I don't see what we have to gain from switching to
one of the hardest to use SCM systems out there.

Kevin Kofler

Re: dist-git project update

By Michael Cronenworth at 06/17/2010 - 09:17

Linus can tell you everything that is wrong with CVS.

<a href="http://www.youtube.com/watch?v=4XpnKHJAok8" title="http://www.youtube.com/watch?v=4XpnKHJAok8">http://www.youtube.com/watch?v=4XpnKHJAok8</a>

Re: dist-git project update

By Jaroslav Reznik at 06/14/2010 - 03:29

Fedpkg should hide the GIT details from you.

Jaroslav

Re: dist-git project update

By Kevin Kofler at 06/18/2010 - 19:08

But it's a command-line tool. Cervisia (what I use on the CVS repos) is a
GUI.

Kevin Kofler

Re: dist-git project update

By Jesse Keating at 06/18/2010 - 19:53

fedpkg is also a library, where all the interesting things are done in
the library, so that somebody could write a gui (or even web) frontend
for it. I'm making a strong effort to keep the separation between
library and frontend as clean as possible.

Re: dist-git project update

By Jarod Wilson at 06/18/2010 - 23:42

Plus, cervisia is a *cvs* gui, not a Fedora Makefile gui, no? There
are a fair number of git guis too.

Re: dist-git project update

By stefan riemens at 06/19/2010 - 02:54

TeamGit is a really nice gui to git

2010/6/19, Jarod Wilson < ... at wilsonet dot com>:

Re: dist-git project update

By Rakesh Pandit at 06/13/2010 - 15:35

I second that, unless there are some obvious advantages which I cannot
see. In case there are some it would be great if they get mentioned in
<a href="https://fedoraproject.org/wiki/Dist_Git_Proposal" title="https://fedoraproject.org/wiki/Dist_Git_Proposal">https://fedoraproject.org/wiki/Dist_Git_Proposal</a>

Re: dist-git project update

By Matthew Miller at 06/13/2010 - 15:59

Linked from there:

Current Pain Points

* No atomic commits
* Not being able to work offline (cvs add needs a server, wtf.)
* Adding sources can be "weird", can easily clobber existing sources
* Can't handle big files well
* CVS bogons/bonghits/grimlins/websuckage
* Prohibitively expensive to reconstruct infrastructure outside our
environment
* Better handling of force-tag
* Commits are SLOW!!!
* Common dir, wtf.
* Really really unreliable (especially with a lot of actions or
continuous actions)
* Prep work to get into package source control is done outside of source
control. No opportunity to learn the tools

Re: dist-git project update

By Karel Zak at 06/17/2010 - 07:43

* changes are not content-addressed
* difficult to reliable compare two trees (repositories)
* branches are useless, difficult to maintain, merge, ...
* does not support distributed development (how I can clone my
fedora pkg SCM at fedorapeople?)
* does not allow to track more remote repositories in one local
repository
* does not support GPG-signed tags
* impossible to read patches from e-mail, send well formatted
patches by email, ...
* commit messages are very poor and not integrated to patches
* does not differentiate between patch author and committer
* browse project history is difficult and SLOW
* cvs log/status is horrible, unreadable and unformattable
* brain dead web interface
* bad documentation
* no active development of CVS
* many developer use already git and Fedora is the last place where
they have to fight with CVS

Karel

Re: dist-git project update

By Ryan Rix at 06/11/2010 - 15:46

How will this affect current packagers' workflow? Will the changes necessary
be documented on the wiki? It doesn't seem straight off the bat that this
would be a drop-in replacement without packagers needing to tweak their
workflow.

Ryan

Re: dist-git project update

By Jesse Keating at 06/11/2010 - 17:09

Packager workflow will indeed be affected. First and foremost they will
need to make use of 'fedpkg'. Fedpkg is replacing the Make system, and
adding a few things on top of it. Today to check out a package and
build it for rawhide you'd essentially do:

cvs <junk here> checkout <module>

cd <module>/devel/

edit junk

cvs add/commit

make tag

make build

With dist-git it's slightly different:

fedpkg clone <module>

cd <module>

edit junk

fedpkg commit (this target is missing right now, alternatively "git
commit")

fedpkg build

We've kept the target names the same, so "make foo" largely becomes
"fedpkg foo". But since fedpkg can take options for the targets much
easier than make can, some targets grew options, like "fedpkg build
--scratch" which essentially does "fedpkg scratch-build". Also, where
with the Make system we'd have to pass arguments as env variables (make
new-sources FILES=blah..) we can do it as arguments: fedpkg new-sources
file1 <file2> <file3>.

We also are able to put in a better help system, so that fedpkg --help
and fedpkg <target> --help provide better help to end users trying to
get things done.

Re: dist-git project update

By Simo Sorce at 06/11/2010 - 17:21

On Fri, 11 Jun 2010 14:09:41 -0700

Is it necessary to do fedpkg clone ?
Or can regular git be used ?

Simo.

Re: dist-git project update

By Jesse Keating at 06/11/2010 - 17:57

Not necessary at all. The current url is
pkgs.stg.fedoraproject.org/<package> and that works with git:// and
ssh://. One advantage to using fedpkg clone is that if you like the
current directory layout where each release is a subdir, you can do
'fedpkg clone --branches' and you'll get that layout. You can also do
fedpkg clone -b <branch> and get a checkout of the specific branch
listed. By no means is fedpkg required for the clones and commits, but
it is a convenience that some will like to use, and will likely be the
"documented" way of doing things.

Re: dist-git project update

By Christof Damian at 06/14/2010 - 03:23

I am looking forward to git too.

One question though: will it be possible to create a git repository
local for packages which are going through the review process (or even
before that for private packages) and then push this to fedora once
the review is approved?

That would make it possible to use fedpkg locally already to do local
and mockbuilds and also keep the history, which otherwise gets lost on
import.

Cheers,
Christof

Re: dist-git project update

By Andreas Schwab at 06/14/2010 - 04:03

Christof Damian < ... at damian dot net> writes:

You can probably just merge your private history.

Andreas.

Re: dist-git project update

By Christopher Brown at 06/13/2010 - 12:26

Hi Jesse,

I'm really keen to see this come off as I think pretty much everyone
is and look forward to the day when we all bid cvs goodbye. Would it
not be saner to extend git through a plugin to provide fedora-specific
stuff rather than use a new, separate command like fedpkg? Might help
the migration if people understand that they are essentially using git
with a few fedora options instead.

Just a thought.

Regards

Re: dist-git project update

By Jesse Keating at 06/14/2010 - 01:37

fedpkg is more than just an interaction with the source control, it's
also how you interact with the koji build system and with bodhi, and
potentially with other things in the future. Bundling those all into
git doesn't necessarily make sense. For those used to git and prefer to
use git directly to manipulate the sources, they can do that just fine.

Re: dist-git project update

By Jochen Schmitt at 06/12/2010 - 15:39

Am 11.06.2010 23:57, schrieb Jesse Keating:

It's seems, that this feature is not implemented in the current
fedora-packager package.

when I try to make a fedpkg cloone --branches I will get only
a message which describe the function for this command without
any visible result.

Best Regards:

Jochen Schmitt

Re: dist-git project update

By Jesse Keating at 06/15/2010 - 19:50

Oops, looks like that's another "to be done later" items (:

Re: dist-git project update

By Iain Arnell at 06/12/2010 - 02:22

Any chance of making that work with http:// and https:// (for pushes) too?

Re: dist-git project update

By Jesse Keating at 06/14/2010 - 01:35

I hadn't planned on it. Is there really a need for this? Read only via
http is do-able, but undesirable. Write via https is going to be a long
shot.

Re: dist-git project update

By Peter Robinson at 06/14/2010 - 04:58

I think anon git via http is essential. I use the feature all the time
to verify things on packaged I don't maintain. I don't want to have to
git clone every time I need to do that.

Peter

Re: dist-git project update

By Till Maas at 06/14/2010 - 06:44

I guess you are confusing using "git clone via http" with a web
interface to access the repository contents like gitweb. I am pretty
sure the web interface will be there anyhow.

Nevertheless imho both should only be accessible via https to avoid
potential security problems with maintainers git-cloning manipulated
data and then pushing it into the repo.

Regards
Till

Re: dist-git project update

By Mamoru Tasaka at 06/14/2010 - 05:34

Peter Robinson wrote, at 06/14/2010 05:58 PM +9:00:

And http access is useful when showing Fedora's patches to others
(e.g. to the upstream or to other distros) and discuss them.

Mamoru

Re: dist-git project update

By Andreas Schwab at 06/14/2010 - 04:00

Jesse Keating < ... at redhat dot com> writes:

Recent git supports the smart protocol over http/https, both for fetch
and push.

Andreas.

Re: dist-git project update

By Iain Arnell at 06/14/2010 - 03:28

It would be incredibly useful for those of us that spend a significant
amount of time stuck behind a restrictive firewall/proxy. Enabling
http would be a good start with minimal effort involved (at least I
can pull and prepare patches). I can understand that https is far
trickier, but would be an awesome feature (especially if it uses the
same client certificate as koji, and even more especially if bodhi can
be taught to use the same certificate too).

Re: dist-git project update

By Jarod Wilson at 06/10/2010 - 21:41

...

Oh hell yeah. Looking forward to the day when none of the projects I
work on are using cvs any longer (this includes you, Red Hat internal
cvs, and you, lirc upstream)...