Difference between revisions of "Development/Tools/svnmerge.py"

Jump to: navigation, search
(Commit and merge policy for the KDEPIM branches: add yet another scenario)
(Commit and merge policy for the KDEPIM branches: add heading)
Line 146: Line 146:
 
Manual merges and backports should be avoided because it doesn't update the merge tracking information in the SVN properties. This means that the merge guy later has to find out which commits were merged to which places, which usually involves digging through mail archives and websvn.kde.org, which is very time consuming.
 
Manual merges and backports should be avoided because it doesn't update the merge tracking information in the SVN properties. This means that the merge guy later has to find out which commits were merged to which places, which usually involves digging through mail archives and websvn.kde.org, which is very time consuming.
  
 +
=== Ideal Lifecycle ===
 
The ideal lifecycle of a commit is the following:
 
The ideal lifecycle of a commit is the following:
 
* The initial commit is done into the enterprise3 branch
 
* The initial commit is done into the enterprise3 branch
Line 158: Line 159:
 
One exception to this is when the commit is needed only in enterprise3 or only in the KDE4 branches (enterprise4, trunk, 4.2).
 
One exception to this is when the commit is needed only in enterprise3 or only in the KDE4 branches (enterprise4, trunk, 4.2).
  
 +
=== Enterprise3-only lifecycle ===
 
The lifecycle for a commit that is only needed in the enterprise3 branch is:
 
The lifecycle for a commit that is only needed in the enterprise3 branch is:
 
* The initial commit is done into the enterprise3 branch
 
* The initial commit is done into the enterprise3 branch
 
* The commit is blocked with svnmerge.py in the enterprise3 -> enterprise4 direction
 
* The commit is blocked with svnmerge.py in the enterprise3 -> enterprise4 direction
  
 
+
=== KDE 4-only lifecycle ===
 
The lifecycle for a commit that is only needed in the KDE 4 branches is:
 
The lifecycle for a commit that is only needed in the KDE 4 branches is:
 
* The initial commit is done into the enterprise4 branch
 
* The initial commit is done into the enterprise4 branch
Line 169: Line 171:
 
* If the commit is backported to KDE 4.2, it needs to be blocked with svnmerge.py in the 4.2 -> enterprise4 merge direction
 
* If the commit is backported to KDE 4.2, it needs to be blocked with svnmerge.py in the 4.2 -> enterprise4 merge direction
  
Alternatively, the lifecycle for commits that are only needed in the KDE4 branches can also be like this, where the commit is started in trunk:
+
=== Alternative KDE 4-only lifecycle for bugfixes ===
 +
Alternatively, the lifecycle for commits that are only needed in the KDE4 branches can also be like this, where the commit is started in trunk. This only applies for bugfixes.
 
* The initial commit is done in trunk
 
* The initial commit is done in trunk
 
* The commit is manually backported from trunk into the 4.2 branch. See notes above.
 
* The commit is manually backported from trunk into the 4.2 branch. See notes above.
Line 175: Line 178:
 
* The commit is blocked with svnmerge.py in the enterprise4 -> trunk merge direction
 
* The commit is blocked with svnmerge.py in the enterprise4 -> trunk merge direction
  
If you prefer to develop features in trunk, refer to the following lifecycle. Developing in enterprise4 first is usually better, since then you don't need to do manual merges.
+
=== Lifecycle of features started in trunk (should be avoided) ===
 +
If you prefer to develop features in trunk, refer to the following lifecycle. Developing in enterprise3 or enterprise4 first is usually better, since then you don't need to do manual merges.
 
* Commit the initial feature to trunk. Note that it can't be backported to KDE 4.2, since it is a new feature.
 
* Commit the initial feature to trunk. Note that it can't be backported to KDE 4.2, since it is a new feature.
 
* Manual merge the feature to the enterprise4 branch. Please make sure that the log messages includes the original revision and the fact that the commit is a manual merge that originated from trunk.
 
* Manual merge the feature to the enterprise4 branch. Please make sure that the log messages includes the original revision and the fact that the commit is a manual merge that originated from trunk.

Revision as of 10:51, 10 April 2009

Contents

What is svnmerge and why use it?

When you have multiple branches of the same project in SVN, you probably want to keep these branches in sync, at least to a certain extend.

One example of this would be feature branches, i.e. a branch were a bigger feature is developed. You usually want to keep that feature branch in sync with trunk by merging all commits from trunk into that feature branch. Once the feature branch is ready to be merged back to trunk, you want to do just that with as little work as possible.

Another example are the KDEPIM enterprise branches. All changes from the enterprise branches are merged to trunk, but only some changes from trunk are merged back to the enterprise branches.

In both situation, svnmerge.py will help you by keeping track of the merges and assisting you in the process of merging.

It has the following useful features:

  • Keeping a list of merged and unmerged commits, so you never forget to merge a single commit
  • Ability to block commits from merging
  • Ability to automatically generate commit messages
  • Support for merging many commits at once
  • Support for multiple branches as merge source
  • Support for bidirectional merges

See also the svnmerge wiki page.

Note that svnmerge.py should not be confused with svn merge. The later is a normal SVN command, which is used, among others, by svnmerge.py. When refering to svnmerge on this page, I always mean svnmerge.py. Also, when referring to svn merge, I mean the built-in subversion command.

How does it work internally?

svnmerge somehow needs to keep track of which commits are merged and which are not. For this, it uses the SVN properties of the top-level project directory. The properties contain a list with revisions, and that list is updated each time you merge something with svnmerge.

You can view that list with

svn propedit svnmerge-blocked .

and with

svn propedit svnmerge-integrated .

Never change those properties manually!

Installing svnmerge

Download svnmerge.py from the svnmerge home page. The trunk version should be fine.

Then, put it into /usr/local/bin and make sure it has executable permissions by issuing

chmod +x /usr/local/bin/svnmerge.py

You need to have Python installed, probably with some add-on modules.

Initializing merge tracking for a branch

After creating a new work branch, you need to tell svnmerge about this, so it can set up the initial revision list in the properties. You can skip this section if somebody else already did set up merge tracking for your branches.

To get help in doing so, type

svnmerge.py help init

After you executed the correct svnmerge.py init command, make sure you commit your changes. Only the SVN properties should have changed, type svn status to confirm this.

Afterwards, commit it with

svn ci -F svnmerge-commit-message.txt

Merging Changes

All the svnmerge commands below should be issued in the top-level project working directory, i.e. the directory svnmerge.py init was run in. This directory is what I mean with working directory below.

Getting the list of available merges

First of all, you probably want a list of revisions that are not yet merged. This list can be retrieved with svnmerge avail. The avail command supports many options, to get help, type:

svnmerge.py help avail

Sometimes you already know the revision numbers of the commits you want to merge, in this case you don't need to run svnmerge avail. But be careful, it is easy to forget commits when you don't run this command.

A good practice is get the list of revisions to be merged together with their log message, which often comes in handy. Run the following to store that list in a file:

svnmerge.py avail --log -b > avail.txt &

Note that the command can take a long time, so we run it in the background with &. Also, svnmerge issues a lot of SVN command in the background, so be sure your passphrase is cached when using svn+ssh, by using ssh-add and ssh-agent.

When you have got multiple branches to merge from, you need to specify the target branch with the -S option. This is the case in the enterprise4 branch, which can merge from the enterprise35 branch or the KDE 4.2.x branch.

svnmerge.py avail --log -b -S enterprise > avail.txt &

Now, open avail.txt in a text editor. Your goal is now to deal with all revisions in this list, either by merging them, blocking them or marking them as already merged.

Merging commits

Merging a commit is simple:

svnmerge.py merge [-S enterprise] -r900000

Here, 900000 is a revision number you got from avail.txt, and the name after -S is the branch you want to merge from. You can omit -S if you only have a single branch with merge tracking.

Before typing that command, make sure your working directory is clean, neither the SVN properties nor any files in sub-directories should have any pending changes. Normally, svnmerge will abort if the directories are not clean. Also, your working branch should be up-to-date.

After svnmerge is finished, it will have modified the properties of the working directory, because the properties contain the list of merged revisions, and the revision 900000 was just added to that list. Also, the command has modified your source files, as it merged all changes from the original revision. Nothing is committed yet.

You should now test that the changes actually compile and work as expected, and fix any problems. Sometimes, you'll get conflicts, which you have to resolve manually and then mark as resolved by using svn resolved. After that is done, you can check your changes in.

svnmerge has helpfully created a file named svnmerge-commit-message.txt. You can use that file directly as commit message, or modify its contents. Check in your changes with

svn ci -F svnmerge-commit-message.txt

Congratulations, you have just merged your first commit!

You can also merge multiple revisions at once. But use this with care, it will appear as a single commit, and therefore make svn annotate less useful. Only use this for closely related changes. For this, you can specify a revision list after -r, like:

svnmerge.py merge [-S enterprise] -r900000,9000002,900004-900010

Blocking commits

Sometimes, you don't want to merge a certain revision, and want that revision to not appear in the avail list at all, for example if the original revision is not useful for the current branch. This can be done by blocking a commit.

Blocking commits has the same syntax as merging:

svnmerge.py block [-S enterprise] -r900099,9000101,900110-900120

Unlike merging, you should block all revisions you want to block in one go, so you don't create many unnecessary commits.

You should edit svnmerge-commit-message.txt before committing, adding a very brief explanation why you block the commits. Also, add the SVN_SILENT keyword.

Now you can commit your changes with

svn ci -F svnmerge-commit-message.txt

Blocking a commit only changes the SVN properties, no source files are changed (since the goal of blocking the revision is to not merge the source file changes of that revision).

The next time you run svnmerge.py avail, the revisions you blocked will not show up there.

Recording merges

Sometimes, it happens that a revision is already merged, but svnmerge doesn't know this. One situation where this happens is if somebody else, who didn't use svnmerge, merged that revision manually. Because the revision was merged manually, the list of merged revisions in the SVN properties was not updated. This means that the commit will show up in svnmerge.py avail, although it was already merged.

To fix this situation, you can mark a revision as merged by adding -M to the svnmerge.py merge command:

svnmerge.py merge -M [-S enterprise] -r900099,9000201,900210-900220

This command will add the given revisions to the list of merged commits in the list of the SVN properties.

As with blocking commits, try to record as many merges in a single go as possible. Also, add a short explanation to the commit message, as well as the SVN_SILENT keyword.

To check in your changes, type

svn ci -F svnmerge-commit-message.txt

Commit and merge policy for the KDEPIM branches

This section is only about the commits to the KDEPIM enterprise branches and for those who work on it.

There are 4 branches in total which need to be kept in sync: enterprise3, enterprise4, trunk and 4.2.

Merge tracking is available in the following directions:

(3.5 -> enterprise3)
enterprise3 -> enterprise4
enterprise4 -> trunk
4.2 -> enterprise4

It is important to use svnmerge.py as much as possible. Avoid manual merging and crossporting! Manual merges and backports should be avoided because it doesn't update the merge tracking information in the SVN properties. This means that the merge guy later has to find out which commits were merged to which places, which usually involves digging through mail archives and websvn.kde.org, which is very time consuming.

Ideal Lifecycle

The ideal lifecycle of a commit is the following:

  • The initial commit is done into the enterprise3 branch
  • The commit is merged with svnmerge.py from enterprise3 to enterprise4
  • The commit is merged with svnmerge.py from enterprise4 to trunk
  • The commit is manually backported from trunk into the 4.2 branch, if it is an important bugfix and does not contain string changes or new features. While manually backporting, the original svn log message (which includes the auto-generated svnmerge log messages) should be included, so that it is still known where the commit originated from.
  • If the commit is backported to KDE 4.2, it needs to be blocked with svnmerge.py in the 4.2 -> enterprise4 merge direction, since it is already in enterprise4 and therefore a cyclic merge.

Try to stick to this ideal lifecycle, it makes the life of the merge guy much easier! This ideal lifecycle also means that commits should never be started in trunk or the KDE 4.2 branch, since then you need to do manual merging, for example to the enterprise3 branch. One exception are commits that are only needed in the KDE 4 branches, but not in the enterprise3 branch. These commits can also start in trunk, there is a scenario somewhere below for that.

One exception to this is when the commit is needed only in enterprise3 or only in the KDE4 branches (enterprise4, trunk, 4.2).

Enterprise3-only lifecycle

The lifecycle for a commit that is only needed in the enterprise3 branch is:

  • The initial commit is done into the enterprise3 branch
  • The commit is blocked with svnmerge.py in the enterprise3 -> enterprise4 direction

KDE 4-only lifecycle

The lifecycle for a commit that is only needed in the KDE 4 branches is:

  • The initial commit is done into the enterprise4 branch
  • The commit is merged with svnmerge.py from enterprise4 to trunk
  • The commit is manually backported from trunk into the 4.2 branch. See notes above.
  • If the commit is backported to KDE 4.2, it needs to be blocked with svnmerge.py in the 4.2 -> enterprise4 merge direction

Alternative KDE 4-only lifecycle for bugfixes

Alternatively, the lifecycle for commits that are only needed in the KDE4 branches can also be like this, where the commit is started in trunk. This only applies for bugfixes.

  • The initial commit is done in trunk
  • The commit is manually backported from trunk into the 4.2 branch. See notes above.
  • The commit is merged with svnmerge.py from the 4.2 branch to enterprise4
  • The commit is blocked with svnmerge.py in the enterprise4 -> trunk merge direction

Lifecycle of features started in trunk (should be avoided)

If you prefer to develop features in trunk, refer to the following lifecycle. Developing in enterprise3 or enterprise4 first is usually better, since then you don't need to do manual merges.

  • Commit the initial feature to trunk. Note that it can't be backported to KDE 4.2, since it is a new feature.
  • Manual merge the feature to the enterprise4 branch. Please make sure that the log messages includes the original revision and the fact that the commit is a manual merge that originated from trunk.
  • With svnmerge, block the commit in the enterprise4 -> trunk direction, since it is a cyclic merge now.

If you also want this new feature in enterprise3, do the following:

  • Manually merge the feature to the enterprise3 branch. Like above, use a detailed log message.
  • With svnmerge, block the commit in the enterprise3 -> enterprise4 direction, since it is a cyclic merge now.

If you encounter a situation where no of the above 3 lifecycles would apply, please ask Thomas McGuire.

Problems & Solutions

Common Error Messages

  • "xyz" is neither a valid URL, nor an unambiguous substring of a repository path, nor a working directory
This can happen when using the -S parameter. Make sure you run svnmerge on the top-level project directory, not a subdirectory. Also make sure you didn't misspell the branch name after the -S parameter.
  • svnmerge: no integration info available
Run svnmerge in the top-level project directory, not a subdirectory. Also, someone should have run svnmerge.py init on the top-level project directory once.

Files that have been moved will be skipped

One problem of svnmerge is that it can't deal well with moved files. As an example, take merging from the KDE 4.2 branch to the enterprise4 branch. In KDE 4.2, some files of KOrganizer have been moved to the views sub-directory, while in the enterprise4 branch, they are still in the top-level KOrganizer folder.

When this happens, svnmerge will say something like Skipped missing target xyz.cpp. It is now your job to merge the changes to the moved files manually.

This can be done by running svn merge. Example: cd korganizer svn merge $SVNBASE/branches/KDE/4.2/kdepim/korganizer/views/agendaview/ -c934407 In this example, the exact directory where the moved files, in the branch to merge from, are, is specified (korganizer/views/agendaview). If this command is run in the directory (korganizer) where the files are, in the branch to merge to, the changes will be applied.

After you merged the changes to the skipped and moved files manually, you can commit like you normally would do.

Files have been moved to a different project

The above case deals with files that have been moved within the project. But what to do if the files have been moved to a totally different project, which is not covered by the merge tracking?

One example where this happens is when merging from the enterprise35 branch to the enterprise4 branch. Say there was a commit in enterprise35, which changed both the KCal library libkcal and KOrganizer. The enterprise 35 branch is KDE 3 based. In KDE 3, the kdepim directory had both the libkcal and the korganizer sub-directories. In the KDE 4 based enterprise4 branch, the KCal library has been moved to a new module, kdepimlibs, and was renamed from libkcal to kcal. But kdepimlibs is not covered by merge tracking, only the two kdepim directories are. What now?

First, you start by normally merging the revision, but not yet committing it.

svnmerge.py merge -S enterprise -r678876

You'll see some messages about skipped files, since we know they have been moved. In our example, it would be something like Skipped missing target libkcal/incidence.h. Only the KOrganizer changes are not skipped.

You need to merge the moved files manually with svn merge. In our example above, you would first change into the directory where the new files are, in this case the KCal library that has been moved to kdepimlibs:

cd /path/to/kdepimlibs/kcal

Then, use svn merge, like so:

svn merge $SVNBASE/branches/kdepim/enterprise/kdepim/libkcal -c678876

Now, you can commit the changes to kdepimlibs. The easiest way is to reuse the commit message svnmerge created in the kdepim directory:

svn ci -F /path/to/kdepim/svnmerge-commit-message.txt

You can also copy the commit message file and add something like First part of the merge to it, so others will know that there will be more commits for this merge.

So now, you have done the first part of the merge. You have merged the changes of the files that have been moved outside of the project, and you have not changed any SVN properties yet that keep track of the merges.

Time for the second part of the merge. Change back into the original project directory where merge tracking information is available, in our case the kdepim directory:

cd /path/to/kdepim

Then simply commit the changes svnmerge has made in the first step. Those are the changes to the files that have not been moved.

svn ci -F svnmerge-commit-message.txt

Now, both the changes to moved files and the changes to the non-moved files have been made. Also, the SVN properties have been updated in the second step to record the merge. The merge is now complete.


As always, you should make sure your changes compile and work before checking anything in.

Conflicts on .

When you have a merge chain, it can sometimes happen that there are conflicts on the SVN properties, which will show up as conflicts on .. One place where this can happen is when merging a commit that was itself a merge of another commit, for example when merging from enterprise35 to enterprise4, and than merging that commit to trunk. This also happens when backporting a merge.

This conflict normally is bogus. To resolve it, type

svn resolved .

Afterwards, double-check that the SVN properties changes are indeed correct, by using the mergediff script from below. You do not want your SVN properties to become corrupted!

Helpful Scripts

Place those functions into your ~/.bashrc.

mergediff

The mergediff function shows the changes on the SVN properties in a nice way, because the normal output of svn diff is unreadable.

function mergediff {

 svn diff -N | {    
   read; read; read; read # skip headers}
   read line;                            
   echo $line | tr ' ' '\n' | tr ',:' '\n' | grep -v "^-$" > /tmp/svnmerge.pre
   while read line; do                                                        
     if ! [ -z "$line" ]; then break; fi
   done
   echo $line | tr ' ' '\n' | tr ',:' '\n' | grep -v "^+$" > /tmp/svnmerge.post
 }
 diff -u /tmp/svnmerge.pre /tmp/svnmerge.post
 rm /tmp/svnmerge.pre /tmp/svnmerge.post

}

automerge

The automerge function is helpful if you want to automatically merge a list of revisions, but do not want to use svnmerge.py merge because it wouldn't preserve history for svn annotate correctly.

Normally, you would do

svnmerge.py merge -r1,2,3,4,5

However, that places all 5 revisions into a single commit, so you'll loose history information.

Instead, you can do

automerge_e4 1,2,3,4,5

This will use svnmerge to automatically merge all 5 revisions, but makes sure it uses a separate commit for each revision, thus producing 5 distinct commits. This preserves history much better. The _e4 suffix is just from a custom one-liner function to merge from the enterprise4 branch, it is easy to create your own functions. See the code below for more details. Also notice the lack of -r.

Warning: This automatically commits stuff, without any opportunity for reviews. The script will only stop in case of conflicts, even compile errors are not tested! Only use this when you know what you are doing!! A good way is to first use svnmerge.py merge for your revisions, test if that compiles and works, then use svn revert -R . * to revert those changes again. Afterwards, you can use automerge, because you already know that those revisions work correctly, since you just tried that. For a large list of commits that can be merged without conflicts, this is still faster than manually merging every single commit.

You need to export $SVNBASE before using this function.

The ssh-add is only needed when using svn+ssh, remove that is you use https or have found a better way to cache the key.

function revlist_sort {

 sed 's/,/\n/g' | sort

}

function automerge_general {

 ssh-add -t 99999999
 REPO=$1
 while read line
 do
   rm -f svnmerge-commit-message.txt
   echo "Merging revision $line from $REPO"
   echo "  Running svnmerge..."
   echo "    command: svnmerge.py merge -S $REPO -r$line"
   svnmerge.py merge -S $REPO -r$line
   if test -e svnmerge-commit-message.txt; then
     echo "  Committing..."
     echo "    command: svn ci -F svnmerge-commit-message.txt"
     svn ci -F svnmerge-commit-message.txt
   else
     echo "    Commit file missing! Something went wrong! Aborting!"
     return 1
   fi
 done
 return 0

}

function automerge_e4 {

 echo $1 | revlist_sort | automerge_general $SVNBASE/branches/kdepim/enterprise4/kdepim

}

Further Reading


KDE® and the K Desktop Environment® logo are registered trademarks of KDE e.V.Legal