Projects/MovetoGit

< Projects
Revision as of 22:48, 2 December 2009 by Zander (Talk | contribs)

Jump to: navigation, search

This is the page for co-ordinating KDE's move to Git.

If you're interested in helping, you should join the scm-interest@kde.org mailinglist and #kde-git on freenode.

Contents

The Plan

KDE is, eventually, moving to Git. We will be using gitorious.org servers, with funding from Nokia. We will also have our own mirrors using existing KDE servers.

We are working with the Gitorious people to ensure their server will meet all our needs as well as everyone's privacy requirements. The distributed nature of Git will make it easy for us to migrate off gitorious.org at any time should the need arise (but that's unlikely :).

In the summer of 2009, Amarok moved to Gitorious to test the waters and find problems that would affect KDE. Once those problems have been solved, all of KDE will be able to switch.

Why?

Git offers many advantages over svn, including offline commits and much easier to keep a feature branch up-to-date. Many KDE developers are already using git-svn, but this tool has its limitations. We want to have the full power of Git available, and we have people willing to do the work necessary to migrate.

How?

When we move, KDE's svn repository will be migrated into several Git repos, all on gitorious.org. Main modules such as kdelibs and kdebase will each become one repository. Projects in extragear will each have their own repository. The kde.gitorious.org site will have a list (lists?) of all these repositories using the builtin project wiki. Scripts will be provided for downloading, say, all of extragear, so "moving" a project from kdereview to extragear would simply involve editing a file kept online that defined the location of projects.

A few things will stay in subversion - currently websites, translations and manuals. It's possible they could move to Git later, but they won't be part of the mass migration.

On Gitorious, all KDE developers will be part of the kde-developers group. Developers in this group are required to set their "full name" for their Gitorious account to their real name. If you are not in this group or do not have an account, a system will be set up to simplify the process. Or you can just go to Gitorious yourself, create an account, and ask to be added to the group. :)

Blockers

Tasks that need to get done before we can migrate

SLA for gitorious.org

20% completed (estimate)

  

Owner: aseigo, TZander

The SLA terms need to be documented as well as who will be footing the bill, if any. TZander will be taking the discussion to people internal @ Qt to discuss this as there was a tentative understanding that cooperation is forthcoming there.

Gitorious Needs a feature to disable merge request emails for certain repos

10% completed (estimate)

  

Owner: Gitorious

Everyone added to the kde-developers group gets an e-mail when anyone makes a merge request to any of our projects. This is undesirable (as most KDE devs are not Amarok devs and many other permutations). A solution for this will have to be developed on Gitorious before all of KDE goes to Gitorious.
Johan will be working on this at the next Gitorious sprint.
Requirements:
  • be able to emulate commit-fitler features, or have it feed commits into the kde-commits and the current commit-filter can be used. This req could also be fulfilled by the 'post-commit' todo item.
  • allow sysadmins to email all KDE account holders
  • be able to subscribe to specific projects and their merge requests

Discussion

At a minimum, users need the ability to choose which projects to get email from. It would be really awesome to have more fine-grained control, though, like the commitfilter. konqueror and kwin and plasma will all be in the same git repo but I only want merge requests for one of them. --Chani
One solution there is to have Gitorious send all email to kde-commits and we can continue using commit-filter. Probably a good idea to set that up to keep continuity regardless. --eean 16:16, 12 November 2009 (UTC)
keeping commitfilter working is good, but merge-requests aren't commits... can commit-filter be upgraded to handle them? Chani 23:19, 12 November 2009 (UTC)
put progress to 10 since Gitorious folks are working on it. :) --eean 16:21, 12 November 2009 (UTC)


Write / update importing rules for svn2git

0% completed (estimate)

  

Owner: morice, tzander - volunteers wanted!

The importer is on gitorious.org as svn2git we have a set of rules to tell the importer what svn dirs turn into which git repos and those need constant updating whenever a new branch or tag or project is created. This has not been done for quite some time and so someone should rsync the svn repo run svn2git and fix the rules and importer whenever the import stops.

This process will need to be repeated every so often until we're ready to do the migration.

Script for downloading virtual KDE hierarchies

5% completed (estimate)

  

Owner: Michael Jansen (mjansen)

One goal for the whole project is to support how people use KDE SVN as much as possible. One issue is that people often check out 'extragear/multimedia' or 'kdesupport' as a whole. However with git extragear, playground, kdesupport etc projects are going to be split up and are essentially going to exist as a flat list on Gitorious.
The proposed solution is have a computer-readable file (eg XML) that only sysadmins could edit. It would define a tree of projects mirroring the current SVN tree. A script would then be able to easily perform operations like 'remote update', 'pull', 'clone' on multiple repos at once, in the same way users are able to checkout or update 'extragear' all at once.
More info is at the scripts development page. The tool itself is hosted at: http://rubyforge.org/projects/build-tool/.
Morice-net will help mjansen with testing once the extensions have been finished.

Discussion

Progress 5 since we have a spec. :) --eean
Ok now the plan is basically that mjansen will extend his build script to allow clone-only and to update its recipes from an online source. The recipes have the advantage of defining all the build deps as well, so we would end up with official documentation of build dependencies, something thats often been missing. --eean 17:18, 12 November 2009 (UTC)

Link http://techbase.kde.org/Projects/MovetoGit/MassCloneScript

Push log

0% completed (estimate)

  

Owner: nobody!!

we need a record of who pushed what, because commit ids can be anything (and there are legitimate reasons to push commits with another person's name). could whoever remember how that discussion ended please fill in the details here?

post-commit hooks

25% completed (estimate)

  

Owner: (morice, dario, johan) maybe?? volunteers needed!!

List of scripts needed:
  • BUG/CCMAIL
  • email/CIA
  • license header checks
  • EBN (dario)
Gitorious needs to provide a way for hooks to be called; KDE needs to write said hooks.

Discussion

There is a branch of gitorious called web-hooks http://gitorious.org/gitorious/mainline/commits/web-hooks --Panagiotis Papadopoulos 1 November 2009
Same situation as commit emails. I can do it but it doesn't scale well and a Gitorious-supported solution would be nicer. --eean 16:07, 12 November 2009 (UTC)

Get rid of svn:externals

40% completed (estimate)

  

Owner: David Faure

not possible with git, broken by design.

Discussion

Is there a list of instances of svn::externals anywhere?- -Yokem55 19:27, 1 November 2009 (UTC)
        &lt;dfaure&gt; argonel: kdesupport is clean. kdebase still has an ugly one, nuno and others agreed to duplicate the little bit of shared code iirc, but it wasn't done.
        &lt;argonel&gt; dfaure: ok, what is the ugly one in kdebase?
        &lt;dfaure&gt; argonel: oxygenhelper stuff
        &lt;argonel&gt; dfaure: ok, can i also have a number to stick on the progress bar?
        &lt;dfaure&gt; Med: indeed.
        &lt;dfaure&gt; argonel: you sound like a manager :-)
        &lt;dfaure&gt; "I want numbers!" :-)
        &lt;argonel&gt; dfaure: lol
        &lt;argonel&gt; dfaure: i have two d10 here, i'll just roll them :p\
        &lt;argonel&gt; dfaure: and 22% it is :p
        &lt;dfaure&gt; argonel: actually most externals are in extragear and playground. the main modules have only 3 left AFAICS.
        &lt;dfaure&gt; argonel: so for the main modules, I solved 2/5, which is 40% :-)
--Argonel 03:04, 14 November 2009 (UTC)

Nice to have before the migration

Snapshot to read-only svn

0% completed (estimate)

  

Owner:

It's work, but maybe some people would like it. NEEDED for documentation, in order to get it back into SVN for the translators/scripty/?

Discussion

Could be done with a git-svn gateway presumably? -Mike Arthur 19/10/2009 16:04
if we leave the docbook stuff in svn, we can avoid this a bit longer. --Chani 23:21, 12 November 2009 (UTC)

Techbase Documentation

Owner: Chani, greeneg, - please help out!

10% completed (estimate)

  

At least minimal documentation about how to checkout, how to get a Gitorious account, how to request a merge needed, other git documentation and links to other git information would be very useful also.
see the Git Tutorial Page. help wanted!!

Discussion


Setup git mirrors for cloning

0% completed (estimate)

  

Owner: No one (help!)

Re-purpose the anonsvn servers. This item might be a blocker.

Discussion

Talk to people using other distros about git

0% completed (estimate)

  

Owner: Sebas, Eike

Discussion

  • Gentoo: They seem to be prepared for moving their live SVN packages to git; their package manager has easily-reusable classes to fetch from an SCM and moving the ebuilds to using the git class rather than the SVN class should be easy. Positive comments to that end from people in #gentoo-kde.
  • Fedora: Some unhappyness about git because SVN allows them to remotely produce a diff between two SVN URLs, without making a checkout / git clone first. Kevin Kofler (IRC nick Kevin_Kofler, #fedora-kde) says this will make their packager work harder.


Post-migration Issues

Website Branding

2% (initial ideas on the table)

  

Owner: ruphy

KDE Gitorious should be branded accordingly, and should be reachable from git.kde.org as well as kde.gitorious.org

Discussion

Unscheduled & Open

Allow tagging without involving sysadmins

0% completed (estimate)

  

Owner: johan

Pushing a tag currently requires permissions to do a force push, which is a repository-wide checkbox that can only be toggled by a kde-developers admin. Thus the workflow for a dev wanting to do a release tag for his app is to ask an admin to enable force pushing, then to push his tag, and then to tell the admin he can disable force pushing again. This doesn't scale, is insecure, and at odds with KDE's open access policy when it comes to managing the repos (right now in SVN, you need sysadmin to create an app dir in /tags for you, but don't have to ask permission for every individual tag). Johan has promised a solution for this.
Notable discussion points on kde-scm-interest:
http://mail.kde.org/pipermail/kde-scm-interest/2009-November/000782.html
http://mail.kde.org/pipermail/kde-scm-interest/2009-November/000784.html

Discussion

Account setup on Gitorious

0% completed (estimate)

  

Owner: eean, johan, boud, dario

Creating an account on Gitorious isn't hard, but asking to be added to the KDE group is inconvenient. For the migration we should set up a system (via email or wiki?) where developers can ask to have an account autocreated for them, or add their existing Gitorious account to a list to be added to the group. Once this is in place an announcement should be sent to all svn accounts explaining the process, and privacy information.
Basically the currently method of using Bugzilla works fine now and works fine in the longterm. But in the transition month when hundreds of accounts must be created or added, we need a better system. Its important to make it as easy as possible so that we don't lose anyone in the shuffle.

Discussion

Update bugs.kde.org's account-request page

0% completed (estimate)

  

Owner: ???

When people get new svn accounts it would make sense for them to join the kde-developers group too.
We also need a standard way for existing svn users to request to join kde-developers. This must include an explanation of the requirements for joining the group, (iirc, just that your gitorious display name is set to your real name)

Completed Tasks

Opt-in privacy exception required for kde-developers

0% completed (estimate)

  

Owner: Gitorious, KDE e.V. Board, eean

KDE sysadmins need access to some information that Shortcut could not give them due to their privacy policy. Examples include an email list of all the developers and SQL-level access to information about all the repos in KDE (since it stores who pushes what, information not stored in the git repo itself).
Such requirements will likely be put into the contract with Shortcut.


Discussion

So the e.V. Board is an owner since this is a legal/contract/money issue. Added myself only because I'm shepherding the issue. --eean 16:16, 12 November 2009 (UTC)


We will not get sql access to the information. For the alternative solution see the Push Log issue

Reviewboard

100% completed (estimate)

  

Owner: darktears

This should be easily done with Gitorious web interface and merge requests actually.

Discussion

but reviewboard has features gitorious (right now) doesn't, like commenting on specific lines and not having to set up a merge request. --chani
Also email notifications when someone reviews are needed --thomasz
We're working on this for someone else right now, so pretty soon --johan-s
I consider the latest changes to gitorious to finish this. If more reviewboard features are still needed, and git supports reviewboard, I think this is something we can look at doing post-conversion. --Ian Monroe


SSH blocked in corporations and universities.

100% completed (estimate)

  

Owner: Unknown

Some universities tend to block the SSH port. There should be a workaround to use SSH on some different port. github.com already runs a SSH server on port 443. But that assumes you are using a proxy. It has been found that this hasn't worked with a lot of people, especially those who have a direct connection to the internet ( so some transparent blocking by the ISP ). It would be great if (almost) every KDE developer were to be asked to check if other ports work before KDE made the switch. Otherwise there could be an automated email where the git patches could be sent, and appropriately patched to the right location too.

Discussion

http://blog.gitorious.org/2009/10/20/stuck-behind-a-firewall/, and there's always been HTTP cloning (although the current impl. in Git is a bit on the slow side) --johan-s


Talk to windows guys about git.

100% completed (estimate)

  

Owner: aseigo

Discussion

They aren't huge fans of git, but are using it. They require a single mainline and can't cope with multiple branches. Otherwise, it's workable, even if it will take an adjustment period.

pre-commit hooks

100% completed (estimate)

  

Owner: (unknown)

acltest, docbook, EOL/UTF-8
A web hook isn't good enough for these because they have to run and return whether to allow the push, for every single push to every KDE repo.


Discussion

gitorious guys said they *might* be willing to allow a few scripts on their server for KDE as a special exception, iirc. --chani
Yes, at least for basic things, heavier things like doc building would probably have to be mirrored (goes for pre/post) --johan-s
It turns out that acl and docbook might not be needed so long as web and docs/ stuff stays in svn.
Here's where to find the current scripts - http://websvn.kde.org/trunk/kde-common/svn/hooks/ --Argonel 23:06, 11 November 2009 (UTC)
So: this is actually done because it needs no longer to be done? (boud)
Apparently, so; moving to complete. (aseigo)

other notes

Random

convert all accounts to SSH?? multiple git repos joined?? - use repo tool from android?


KDE accounts file is no longer necessary---used for mapping svn ID -> email, but we have that now from Gitorious.

http://mail.kde.org/pipermail/dot-stories/2005-May/000509.html might be a good guide on what docs we need.


some of this stuff was from the list from GCDS that was in this email http://markmail.org/message/u6eqfjece7fibfyo


IRC Meetings

jobs

TODO merge this with the todolists above

michael jansen: talking to kdesvn-build/mpyne

--Done? -> http://kdesvn-build.kde.org/releases/kdesvn-build-1.10.php -- Panagiotis Papadopoulos 1 November 2009
Yes, but the __kdesvn-build-remote used in the impl isn't pleasant for users already on git so it still needs more work for them. Mpyne 20:32, 11 November 2009 (UTC)


thiago and sebas: funding

jonas: domain name

thiago: actually convert :D (ThomasZ put some work in the converter tool to make it easier to use; see his 'svn2git' project on gitorious.org)

ML: convert to SSH

chani: techbase docs for scripty

thiago: pre/post-commit hooks

sebas/lydia/leo: communication with teams! tell people! keeping track that everything is being done.


KDE® and the K Desktop Environment® logo are registered trademarks of KDE e.V.Legal