Difference between revisions of "Development/Tools/svnmerge.py"

Jump to: navigation, search
m (Text replace - "<code bash>" to "<syntaxhighlight lang="bash">")
m (Text replace - "</code>" to "</syntaxhighlight>")
Line 197: Line 197:
cd korganizer
cd korganizer
svn merge $SVNBASE/branches/KDE/4.2/kdepim/korganizer/views/agendaview/ -c934407
svn merge $SVNBASE/branches/KDE/4.2/kdepim/korganizer/views/agendaview/ -c934407
In this example, the exact directory where the moved files, in the branch to merge from, are, is specified (''korganizer/views/agendaview''). If this command is run in the directory (''korganizer'')  where the files are, in the branch to merge to, the changes will be applied.
In this example, the exact directory where the moved files, in the branch to merge from, are, is specified (''korganizer/views/agendaview''). If this command is run in the directory (''korganizer'')  where the files are, in the branch to merge to, the changes will be applied.
Line 275: Line 275:
   rm /tmp/svnmerge.pre /tmp/svnmerge.post
   rm /tmp/svnmerge.pre /tmp/svnmerge.post
=== automerge ===
=== automerge ===
Line 328: Line 328:
== Further Reading ==
== Further Reading ==

Latest revision as of 20:50, 29 June 2011

What is svnmerge and why use it?

When you have multiple branches of the same project in SVN, you probably want to keep these branches in sync, at least to a certain extent.

One example of this would be feature branches, i.e. a branch where a bigger feature is developed. You usually want to keep that feature branch in sync with trunk by merging all commits from trunk into that feature branch. Once the feature branch is ready to be merged back to trunk, you want to do just that with as little work as possible.

Another example are the KDEPIM enterprise branches. All changes from the enterprise branches are merged to trunk, but only some changes from trunk are merged back to the enterprise branches.

In both situation, svnmerge.py will help you by keeping track of the merges and assisting you in the process of merging.

It has the following useful features:

  • Keeping a list of merged and unmerged commits, so you never forget to merge a single commit
  • Ability to block commits from merging
  • Ability to automatically generate commit messages
  • Support for merging many commits at once
  • Support for multiple branches as merge source
  • Support for bidirectional merges

See also the svnmerge wiki page.

Note that svnmerge.py should not be confused with svn merge. The later is a normal SVN command, which is used, among others, by svnmerge.py. When refering to svnmerge on this page, I always mean svnmerge.py. Also, when referring to svn merge, I mean the built-in subversion command.

How does it work internally?

svnmerge somehow needs to keep track of which commits are merged and which are not. For this, it uses the SVN properties of the top-level project directory. The properties contain a list with revisions, and that list is updated each time you merge something with svnmerge.

You can view that list with

svn propedit svnmerge-blocked .

and with

svn propedit svnmerge-integrated .

Never change those properties manually!

Installing svnmerge

Download svnmerge.py from the svnmerge home page. The trunk version should be fine.

Then, put it into /usr/local/bin and make sure it has executable permissions by issuing

chmod +x /usr/local/bin/svnmerge.py

You need to have Python installed, probably with some add-on modules.

Initializing merge tracking for a branch

After creating a new work branch, you need to tell svnmerge about this, so it can set up the initial revision list in the properties. You can skip this section if somebody else already did set up merge tracking for your branches.

To get help in doing so, type

svnmerge.py help init

After you executed the correct svnmerge.py init command, make sure you commit your changes. Only the SVN properties should have changed, type svn status to confirm this.

Afterwards, commit it with

svn ci -F svnmerge-commit-message.txt

Merging Changes

All the svnmerge commands below should be issued in the top-level project working directory, i.e. the directory svnmerge.py init was run in. This directory is what I mean with working directory below.

Getting the list of available merges

First of all, you probably want a list of revisions that are not yet merged. This list can be retrieved with svnmerge avail. The avail command supports many options, to get help, type:

svnmerge.py help avail

Sometimes you already know the revision numbers of the commits you want to merge, in this case you don't need to run svnmerge avail. But be careful, it is easy to forget commits when you don't run this command.

A good practice is get the list of revisions to be merged together with their log message, which often comes in handy. Run the following to store that list in a file:

svnmerge.py avail --log -b > avail.txt &

Note that the command can take a long time, so we run it in the background with &. Also, svnmerge issues a lot of SVN command in the background, so be sure your passphrase is cached when using svn+ssh, by using ssh-add and ssh-agent.

When you have got multiple branches to merge from, you need to specify the target branch with the -S option. This is the case in the enterprise4 branch, which can merge from the enterprise35 branch or the KDE 4.2.x branch.

svnmerge.py avail --log -b -S enterprise/kdepim > avail.txt &

Now, open avail.txt in a text editor. Your goal is now to deal with all revisions in this list, either by merging them, blocking them or marking them as already merged.

Since getting the list can take really long, KDAB has a cron job that runs each night and generates the logs for all KDEPIM branches. You can find the logs here. If you have a branch that you like to include in the automatic generation of logs, ask me (Thomas McGuire) on IRC and I'll set it up for you.

Merging commits

Merging a commit is simple:

svnmerge.py merge [-S enterprise/kdepim] -r900000

Here, 900000 is a revision number you got from avail.txt, and the name after -S is the branch you want to merge from. You can omit -S if you only have a single branch with merge tracking.

Before typing that command, make sure your working directory is clean, neither the SVN properties nor any files in sub-directories should have any pending changes. Normally, svnmerge will abort if the directories are not clean. Also, your working branch should be up-to-date.

After svnmerge is finished, it will have modified the properties of the working directory, because the properties contain the list of merged revisions, and the revision 900000 was just added to that list. Also, the command has modified your source files, as it merged all changes from the original revision. Nothing is committed yet.

You should now test that the changes actually compile and work as expected, and fix any problems. Sometimes, you'll get conflicts, which you have to resolve manually and then mark as resolved by using svn resolved. After that is done, you can check your changes in.

svnmerge has helpfully created a file named svnmerge-commit-message.txt. You can use that file directly as commit message, or modify its contents. Check in your changes with

svn ci -F svnmerge-commit-message.txt

Congratulations, you have just merged your first commit!

You can also merge multiple revisions at once. But use this with care, it will appear as a single commit, and therefore make svn annotate less useful. Only use this for closely related changes. For this, you can specify a revision list after -r, like:

svnmerge.py merge [-S enterprise/kdepim] -r900000,9000002,900004-900010

Blocking commits

Sometimes, you don't want to merge a certain revision, and want that revision to not appear in the avail list at all, for example if the original revision is not useful for the current branch. This can be done by blocking a commit.

Blocking commits has the same syntax as merging:

svnmerge.py block [-S enterprise/kdepim] -r900099,9000101,900110-900120

Unlike merging, you should block all revisions you want to block in one go, so you don't create many unnecessary commits.

You should edit svnmerge-commit-message.txt before committing, adding a very brief explanation why you block the commits. Also, add the SVN_SILENT keyword.

Now you can commit your changes with

svn ci -F svnmerge-commit-message.txt

Blocking a commit only changes the SVN properties, no source files are changed (since the goal of blocking the revision is to not merge the source file changes of that revision).

The next time you run svnmerge.py avail, the revisions you blocked will not show up there.

Recording merges

Sometimes, it happens that a revision is already merged, but svnmerge doesn't know this. One situation where this happens is if somebody else, who didn't use svnmerge, merged that revision manually. Because the revision was merged manually, the list of merged revisions in the SVN properties was not updated. This means that the commit will show up in svnmerge.py avail, although it was already merged.

To fix this situation, you can mark a revision as merged by adding -M to the svnmerge.py merge command:

svnmerge.py merge -M [-S enterprise/kdepim] -r900099,9000201,900210-900220

This command will add the given revisions to the list of merged commits in the list of the SVN properties.

As with blocking commits, try to record as many merges in a single go as possible. Also, add a short explanation to the commit message, as well as the SVN_SILENT keyword.

To check in your changes, type

svn ci -F svnmerge-commit-message.txt

Commit Keywords

  • SVN_SILENT: As written in the above sections, use this keyword when blocking or recording commits. This will cause the commit message to be filtered out for most users, avoiding unnecessary noise.
  • SVN_MERGE: Add this keyword to all commit messages that involve merging with svnmerge.py. With this keyword, the changes in the SVN properties are not added to the commit mail, which makes it possible to see the actual diff of the commit. This is very useful, as otherwise reviewing a merge from the enterprise35 to the enterprise4 branch would not be possible without launching an external diff viewer. Additionally, it lets users filter out merge commit notifications if they are not interested in those.
  • MERGE: This keyword is a reminder, and should contain a list of branches that the commit should be merged. This is purely informational. For example, a commit with the line MERGE: trunk, 4.5 in the commit message would indicate that the commit should be merged to trunk and to the 4.5 branch. This is useful information in case that the person doing the merge is someone else than the person doing the initial commit. If there are no branches that the commit should be merged to, the keyword should be MERGE: None, which simply indicates that the commit can be blocked by the person doing the merges with svnmerge.py.

Commit and merge policy for the KDEPIM branches

This section is only about the commits to the KDEPIM branches and for those who work on it.

There are 4 branches in total which need to be kept in sync: enterprise3, enterprise4, trunk and the 4.5 branch.

Merge tracking is available in the following directions:

(3.5 -> enterprise3)
enterprise3 -> trunk
enterprise4 -> trunk
4.5 -> trunk
trunk -> 4.5

The enterprise4 branch is discontinued, only Kleopatra-related commits happen there. Don't commit anything else to that branch.

It is important to use svnmerge.py as much as possible. Avoid manual merging and crossporting! Manual merges and backports should be avoided because it doesn't update the merge tracking information in the SVN properties. This means that the merge guy later has to find out which commits were merged to which places, which usually involves digging through mail archives and websvn.kde.org, which is very time consuming.

Lifecycle of a commit

trunk & 4.5

All bugfixes should go to the 4.5 branch first, and then be merged to trunk. This way we can provide a stable 4.5 branch that includes all bugfixes, which will become the first Akonadi-based PIM release. For details, see this thread on the PIM mailinglist. Please commit to the 4.5 branch first if possible.

If you for some reason committed a bugfix to trunk first, it can be backported to the 4.5 branch with svnmerge.py. Since svnmerge.py is set up in both directions and since svnmerge.py detects bidirectional merges, the backported commit should not appear in the list of available commits in the 4.5 -> trunk direction, so everything should work. If however you manually backported a bugfix from trunk to 4.5 without using svnmerge.py, then that commits needs to be marked as merged in the 4.5 -> trunk direction.

New features should go to trunk and shouldn't be merged anywhere.


All commits to the enterprise3 branch should be merged into trunk. All commits should go to enterprise3 first. A commit might need backporting to the KDE 4.5 branch, which should also be done with svnmerge.py (using trunk as the merge source). The commit does not need to be merged to the enterprise4 branch.


Only Kleopatra-related commits should go into the enterprise4 branch at all. All commits to the enterprise4 branch should be merged into trunk. All commits should go to enterprise4 first. A commit might need backporting to the KDE 4.5 branch, which should also be done with svnmerge.py (using trunk as the merge source).

Problems & Solutions

Common Error Messages

  • "xyz" is neither a valid URL, nor an unambiguous substring of a repository path, nor a working directory
This can happen when using the -S parameter. Make sure you run svnmerge on the top-level project directory, not a subdirectory. Also make sure you didn't misspell the branch name after the -S parameter.
  • svnmerge: no integration info available
Run svnmerge in the top-level project directory, not a subdirectory. Also, someone should have run svnmerge.py init on the top-level project directory once.

Files that have been moved will be skipped

One problem of svnmerge is that it can't deal well with moved files. As an example, take merging from the KDE 4.2 branch to the enterprise4 branch. In KDE 4.2, some files of KOrganizer have been moved to the views sub-directory, while in the enterprise4 branch, they are still in the top-level KOrganizer folder.

When this happens, svnmerge will say something like Skipped missing target xyz.cpp. It is now your job to merge the changes to the moved files manually.

This can be done by running svn merge. Example:

cd korganizer
svn merge $SVNBASE/branches/KDE/4.2/kdepim/korganizer/views/agendaview/ -c934407

In this example, the exact directory where the moved files, in the branch to merge from, are, is specified (korganizer/views/agendaview). If this command is run in the directory (korganizer) where the files are, in the branch to merge to, the changes will be applied.

After you merged the changes to the skipped and moved files manually, you can commit like you normally would do.

Files have been moved to a different project

The above case deals with files that have been moved within the project. But what to do if the files have been moved to a totally different project, which is not covered by the merge tracking?

One example where this happens is when merging from the enterprise35 branch to the enterprise4 branch. Say there was a commit in enterprise35, which changed both the KCal library libkcal and KOrganizer. The enterprise 35 branch is KDE 3 based. In KDE 3, the kdepim directory had both the libkcal and the korganizer sub-directories. In the KDE 4 based enterprise4 branch, the KCal library has been moved to a new module, kdepimlibs, and was renamed from libkcal to kcal. But kdepimlibs is not covered by merge tracking, only the two kdepim directories are. What now?

First, you start by normally merging the revision, but not yet committing it.

svnmerge.py merge -S enterprise/kdepim -r678876

You'll see some messages about skipped files, since we know they have been moved. In our example, it would be something like Skipped missing target libkcal/incidence.h. Only the KOrganizer changes are not skipped.

You need to merge the moved files manually with svn merge. In our example above, you would first change into the directory where the new files are, in this case the KCal library that has been moved to kdepimlibs:

cd /path/to/kdepimlibs/kcal

Then, use svn merge, like so:

svn merge $SVNBASE/branches/kdepim/enterprise/kdepim/libkcal -c678876

Now, you can commit the changes to kdepimlibs. The easiest way is to reuse the commit message svnmerge created in the kdepim directory:

svn ci -F /path/to/kdepim/svnmerge-commit-message.txt

You can also copy the commit message file and add something like First part of the merge to it, so others will know that there will be more commits for this merge.

So now, you have done the first part of the merge. You have merged the changes of the files that have been moved outside of the project, and you have not changed any SVN properties yet that keep track of the merges.

Time for the second part of the merge. Change back into the original project directory where merge tracking information is available, in our case the kdepim directory:

cd /path/to/kdepim

Then simply commit the changes svnmerge has made in the first step. Those are the changes to the files that have not been moved.

svn ci -F svnmerge-commit-message.txt

Now, both the changes to moved files and the changes to the non-moved files have been made. Also, the SVN properties have been updated in the second step to record the merge. The merge is now complete.

As always, you should make sure your changes compile and work before checking anything in.

Conflicts on .

When you have a merge chain, it can sometimes happen that there are conflicts on the SVN properties, which will show up as conflicts on .. One place where this can happen is when merging a commit that was itself a merge of another commit, for example when merging from enterprise35 to enterprise4, and than merging that commit to trunk. This also happens when backporting a merge.

ONLY in the above cases the conflict is bogus. To resolve it, type

svn resolved .

Remember, only use that command in the cases described above, but not when you get a normal conflict!

NEVER USE svn resolved . WHEN YOUR WORKING COPY IS OUTDATED, this will CORRUPT the merge information!!

If you get a conflict on . that is not one of the above cases, revert your work with svn revert -R . * and do a svn update, then try the svnmerge.py operation again. Do not attempt to fix the conflict on . manually, revert! If you have important changes, save them with svn diff > somefile before reverting, and then afterwards cherry-pick those changes again.

Afterwards, double-check that the SVN properties changes are indeed correct, by using the mergediff script from below. You do not want your SVN properties to become corrupted!

Helpful Scripts

Place those functions into your ~/.bashrc.


The mergediff function shows the changes on the SVN properties in a nice way, because the normal output of svn diff is unreadable.

function mergediff { 
  svn diff -N | {    
    read; read; read; read # skip headers}
    read line;                            
    echo $line | tr ' ' '\n' | tr ',:' '\n' | grep -v "^-$" > /tmp/svnmerge.pre
    while read line; do                                                        
      if ! [ -z "$line" ]; then break; fi
    echo $line | tr ' ' '\n' | tr ',:' '\n' | grep -v "^+$" > /tmp/svnmerge.post
  diff -u /tmp/svnmerge.pre /tmp/svnmerge.post
  rm /tmp/svnmerge.pre /tmp/svnmerge.post


The automerge function is helpful if you want to automatically merge a list of revisions, but do not want to use svnmerge.py merge because it wouldn't preserve history for svn annotate correctly.

Normally, you would do

svnmerge.py merge -r1,2,3,4,5

However, that places all 5 revisions into a single commit, so you'll loose history information.

Instead, you can do

automerge_e4 1,2,3,4,5

This will use svnmerge to automatically merge all 5 revisions, but makes sure it uses a separate commit for each revision, thus producing 5 distinct commits. This preserves history much better. The _e4 suffix is just from a custom one-liner function to merge from the enterprise4 branch, it is easy to create your own functions. See the code below for more details. Also notice the lack of -r.

Warning: This automatically commits stuff, without any opportunity for reviews. The script will only stop in case of conflicts, even compile errors are not tested! Only use this when you know what you are doing!! A good way is to first use svnmerge.py merge for your revisions, test if that compiles and works, then use svn revert -R . * to revert those changes again. Afterwards, you can use automerge, because you already know that those revisions work correctly, since you just tried that. For a large list of commits that can be merged without conflicts, this is still faster than manually merging every single commit.

You need to export $SVNBASE before using this function.

The ssh-add is only needed when using svn+ssh, remove that is you use https or have found a better way to cache the key.

function revlist_sort {
  sed 's/,/\n/g' | sort

function automerge_general {
  ssh-add -t 99999999
  while read line
    rm -f svnmerge-commit-message.txt
    echo "Merging revision $line from $REPO"
    echo "  Running svnmerge..."
    echo "    command: svnmerge.py merge -S $REPO -r$line"
    svnmerge.py merge -S $REPO -r$line
    if test -e svnmerge-commit-message.txt; then
      echo "  Committing..."
      echo "    command: svn ci -F svnmerge-commit-message.txt"
      svn ci -F svnmerge-commit-message.txt
      echo "    Commit file missing! Something went wrong! Aborting!"
      return 1
  return 0

function automerge_e4 {
  echo $1 | revlist_sort | automerge_general $SVNBASE/branches/kdepim/enterprise4/kdepim

Further Reading

This page was last modified on 29 June 2011, at 20:50. Content is available under Creative Commons License SA 3.0 as well as the GNU Free Documentation License 1.2 unless otherwise noted.