KDE TechBase - User contributions [en]

Localization/Tools/Lbundle Check

2013-02-02T16:05:04Z

Ilic: Removed extra empty lines.

{{LocalizationBrowser|

section=[[Localization#Tools|Tools]]|

name=Lbundle Checker|

prereqs=[[Localization/Tools/Subversion|Subversion Ops]], [[Localization/Concepts/Non_Text_Resources|Localizing Non-Text Resources]]|

}}

== About ==

The <tt>lbundle_check.py</tt> script checks and records the state of localized non-text resources organized in lbundles, allowing translators to track their relation to original resources. Its latest version can be found in <tt>trunk/l10n-support/scripts/</tt> directory in the KDE repository.

<tt>lbundle_check.py</tt> is in no way tied to KDE Translation Project, but can be used in any other environment. Only the instructions in this article are specifically about using <tt>lbundle_check.py</tt> in context of KDE TP.

== Checking Out and Updating Sources ==

To be able to track lbundle states, the translator (one of the coordinators) needs to have all the sources to which localized resources correspond. This is done by checking out once from the KDE repositories, and then regularly updating local checkouts. This is hard to do and maintain manually for selected sources only.

Instead, the easiest is to check out and update all localization-relevant KDE sources, using a single command. This command is <tt>populate_source.sh</tt>, found in <tt>trunk/l10n-kde4/scripts/</tt> and <tt>branches/stable/l10n-kde4/scripts/</tt> directories; each knows how to check out and update sources corresponding to the given translation branch (trunk or stable). It is run simply like this:
<syntaxhighlight lang="bash">
$ cd $KDEREPO/trunk/l10n-kde4
$ scripts/populate_source.sh
$ cd $KDEREPO/branches/stable/l10n-kde4
$ scripts/populate_source.sh
</syntaxhighlight>
Running <tt>populate_source.sh</tt> which will take quite some time for the first run, but subsequent runs will be much faster. After the run is complete, in current working directory there will be the <tt>source/</tt> directory, with all the checkouts in it. The local directory tree should then look like this:
<syntaxhighlight lang="text">
$KDEREPO/
trunk/
l10n-kde4/
scripts/
source/
...
branches/
stable/
l10n-kde4/
scripts/
source/
...
</syntaxhighlight>
At the moment of this writing, the <tt>source/</tt> in trunk occupies ~10 GiB of disk space, and in stable ~6 GiB. While this is not little space, it should not be a significant problem given contemporary typical disk sizes.

To quickly check out or update only one or few modules, their names can be given as arguments to <tt>populate_source.sh</tt>:
<syntaxhighlight lang="bash">
$ scripts/populate_source.sh extragear-network_konversation kdegames_konquest
</syntaxhighlight>
Module names for use as arguments can be found in <tt>source/modules</tt> file. This file is generated anew whenever <tt>populate_source.sh</tt> is run without arguments, i.e. to update everything.

The <tt>source/</tt> directory contains the <tt>repo/</tt> subdirectory with actual local checkouts, and the <tt>link/</tt> subdirectory, with links to checkout directories suitable for later operations. If the imaginary KDingus application is part of kdeutils top module, and is kept in a Git repository, it will be located like this in <tt>source/</tt> in trunk:
<syntaxhighlight lang="text">
source/
repo/
git-unstable/
kdeutils_kdingus
link/
kdeutils/
kdingus --> ../../repo/kdeutils_kdingus
</syntaxhighlight>
And if KDingus were in the Subversion repository:
<syntaxhighlight lang="text">
source/
repo/
trunk/
KDE/
kdeutils/
kdingus
link/
kdeutils/
kdingus --> ../../repo/trunk/KDE/kdeutils/kdingus
</syntaxhighlight>
Note that in both cases the link location stays the same, <tt>source/link/kdeutils/kdingus</tt>. It will also stay the same in <tt>source/</tt> of stable branch, where <tt>repo/</tt> will instead contain <tt>git-stable/</tt>, <tt>branches/KDE/4.x/</tt>, etc.

There is a special case when a top module name represents Git repository of its own, such as <tt>kde-baseapps</tt> or <tt>kdepim</tt>. The link then has to repeat the module name:
<syntaxhighlight lang="text">
source/
link/
kde-baseapps/
kde-baseapps --> ../../repo/git-unstable/kde-baseapps
kdepim/
kdepim --> ../../repo/git-unstable/kdepim
</syntaxhighlight>
<tt>lbundle_check.py</tt> is capable of gracefully handling this case as well.

== Setup ==

Before going through the setup examples, refresh the examples on organization of [[Localization/Concepts/Non_Text_Resources#Inside The Repository|lbundles in the repository]], which dealt with localizing the splash screen of the imaginary KDingus application.

<tt>lbundle_check.py</tt> can be used to track both out-of-source and in-source lbundles, but the setup needed for these two modes is somewhat different. The setup is explained for both modes, followed by the details of operation.

=== Out-of-Source Lbundles ===

When it was assumed that KDingus was part of kdeutils top module, its out-of-source lbundle for the language <tt>aa</tt> was organized like this:
<syntaxhighlight lang="text">
$KDEREPO/trunk/l10n-kde4/aa/data/kdeutils/
CMakeLists.txt
kdingus/
CMakeLists.txt
pics/
CMakeLists.txt
l10n-spec
l10n/
aa/
kdingus-splash.png
</syntaxhighlight>
except that this listing includes one new file, the <tt>l10n-spec</tt>. This is the setup file for tracking the lbundle, or rather, all lbundles on its level and beneath. The files outside of <tt>l10n/</tt> subdirs are going to be called ''unbundled'', as they are not considered a part of the lbundle proper.

<tt>l10n-spec</tt> files are composed of <tt>key = value</tt> fields per line. For the example above the content of <tt>l10n-spec</tt> would be:
<syntaxhighlight lang="text">
# l10n-spec for KDingus' out-of-source bundle.
source-root = aa/data:source/link;^1
source-vcs = auto
bundle-vcs = auto
languages = aa
track-unbundled = CMakeLists.txt
</syntaxhighlight>

The fields used in the example are:

; source-root: the path to the root directory of the KDingus' sources. Every path-valued field can specify the path in several ways, which will be explained shortly. In this example, <tt>aa/data:source/link</tt> means to construct the path to original file corresponding to localized file by replacing <tt>aa/data</tt> in the absolute localized path with <tt>source/link</tt>; if sources were [[#Checking Out and Updating Sources|checked out as explained earlier]], this will do exactly the right thing. Then there is the second path specification element separated by semi-colon, <tt>^1</tt>, which means to assume that the original file may have one extra parent directory inserted somewhere in its path; this is used to cover the special case of top module name being equal to submodule name (e.g. <tt>source/link/kdepim/kdepim</tt>). Note that this path specification nowhere refers to KDingus in particular: it should be applicable to all out-of-source bundles.

; source-vcs, bundle-vcs: the version control systems used by the source code and the lbundle, respectively. These fields are needed so that <tt>lbundle-check.py</tt> knows how to perform VCS operations when necessary, as well as to avoid trying to track version control bookkeeping files in the lbundle (e.g. <tt>.svn</tt> subdirs for Subversiong). Currently the two known VCS are <tt>svn</tt> and <tt>git</tt>, and <tt>auto</tt> can be set to let <tt>lbundle-check.py</tt> which one it is.

; languages: space-delimited list of languages expected in <tt>l10n</tt> subdirs. This is an optional field, which serves to check for naming errors with language subdirs. It may be left out, preventing such checks, but there is no reason to do so (especially for in-source bundles described below).

; track-unbundled: space-delimited list of relative file paths, or pairs of file paths, of unbundled files which should be tracked nevertheless. In this example, <tt>CMakeLists.txt</tt> should be tracked because the installation instructions for the lbundle may need to change when instructions for the original resources change. A pair of file paths, as two space-separated paths within parenthesis (e.g. <tt>(foo.txt bar.txt)</tt>), can be given instead of a single file path when the local relative path needs to be different from that of the tracked original. A shell glob can be specified instead of a particular path (e.g. <tt>*.txt</tt>), but only for single paths, not in pairs. Backslash serves as escape character, e.g. when the file name contains a space.

Several other fields, not used in this example, are available:

; ignore-unbundled: space-delimited list of relative file paths which are to be completely ignored by <tt>lbundle-check.py</tt>. For an out-of-source bundle, a file which is unbundled should be either tracked too (using <tt>track-unbundled</tt>) or ignored using this field, otherwise <tt>lbundle-check.py</tt> will complain about it. Shell globs also possible.

; strict-state: a boolean value stating whether the tracking state is to be strictly imposed. Non-strict means that out-of-sync states between localized and original resources are recorded only in the track file, whereas strict state means that the localized files names will be changed too to reflect the out-of-sync state. More details in section on [[#Operation|operation]].

; ignore-substr: space-delimited list of substrings to ignore in names of bundled files when determining their original counterparts. This is never needed in KDE, but in other environments some localized files may be handled differently than a plain substitute for the original file, based on a substring in their name (e.g. a localized image may be overlayed over the original, rather than replacing it).

; track-by-subdir: by default there is only one tracking file per <tt>l10n-spec</tt> file (see [[#Operation|operation]] for tracking files); when this option is enabled (a boolean), there will be instead one tracking file per subdirectory (excluding <tt>l10n/</tt> directories) starting from the level of <tt>l10n-spec</tt> file and below. Usefull when frequent moving of subdirectories within the repository is expected, and there are a lot of localized resources to track.

Paths in path-valued fields (such as <tt>source-root</tt>) can be specified in several modes. When it makes sense, to make up one path field value two or more modes can be chained, separated with semi-colon (<tt>;</tt>). Each subsequent mode refers to the path established up to that point. Path specification modes are as follows:

* Relative to a top root directory. This is represented by ordinary relative path. When running <tt>lbundle-check.py</tt>, the top root directory will have to be given through <tt>-s</tt> option.

* Relative to the directory of current lbundle (i.e. to the parent directory of <tt>l10n-spec</tt> file). This is done by prefixing a relative path with exclamation mark (<tt>!</tt>).

* By replacing a part of the absolute path of the current lbundle directory. The replacement specification is of the form <tt>findstr:replstr</tt>.

* With allowed insertion of one or more consecutive parent directories when trying to find the path. This is given by <tt>^N</tt>, where N is the maximum number of inserted parents. This mode cannot be the first in mode chain.

The usual #-comments are allowed in <tt>l10n-spec</tt> files, as well as line continuation by trailing backslashes (e.g. when several file paths are needed in <tt>track-unbundled</tt> field).

=== In-Source Lbundles ===

If KDingus is an extragear application, its directory structure with the in-source lbundle having <tt>aa</tt> and <tt>bb</tt> languages, and an <tt>l10n-spec</tt> file, would be this:
<syntaxhighlight lang="text">
/source/link/
kdingus/
CMakeLists.txt
pics/
CMakeLists.txt
kdingus-splash.png
l10n-spec
l10n/
aa/
kdingus-splash.png
bb/
kdingus-splash.png
</syntaxhighlight>
The contents of <tt>l10n-spec</tt> is simpler than that of out-of-source lbundles:
<syntaxhighlight lang="text">
# l10n-spec for KDingus' in-source bundle.
source-vcs = auto
languages = aa bb
</syntaxhighlight>

Since the lbundle is kept together with the application, sharing same root directory and version control system, there is no need for the <tt>source-root</tt> and <tt>bundle-vcs</tt> fields. In fact, the presence or lack of the <tt>source-root</tt> field identifies the lbundle as out-of-source or in-source.

Field <tt>track-unbundled</tt> (or <tt>ignore-unbundled</tt>) are not present either. All files which are unbundled are silently ignored. It is assumed that the KDingus' maintainer will modify and test installation instructions such as not to break installation of localized resources.

<tt>languages</tt> field could be omitted here as well, but for in-source lbundles it is even more important to keep tight check of wrongly named language subdirs.

=== Greedy Bundling ===

The default assumption behind lbundles is that only a small part of all resources need to be localized. It is upon the translator to spot the resources which need localization, make their localized versions, and add them to an lbundle where they will get tracked.

When instead it would be better to track all of the original resources, lbundle tracking can be set to ''greedy'' mode: any original resource that does not have a localized counterpart yet will be tracked as missing. When a new original resource gets added, the translator will be notified of that by <tt>lbundle-check.py</tt>.

Fields in <tt>l10n-spec</tt> used in greedy mode are as follows:

; greedy-bundling: the option to engage greedy mode (a boolean value, set to true). Other fields controlling greedy bundling have no effect if this option is not set.

; greedy-monolingual: states whether the lbundle is monolingual (boolean). This should make sense only for out-of-source bundles, where there may be one per language. Greedy mode needs this piece of information in order to know how to report and track missing localized files, as bundled (multilingual lbundle) or unbundled (monolingual).

; greedy-from-level: if the lbundle covers a tree of original subdirectories, this field can be used to limit the greedy collection and reporting of missing files only to subdirectories at this level and below. 0 is the level of the <tt>l10n-spec</tt> file itself, 1 is one level below, etc.

; greedy-only-started: normally all original resources missing in localization are reported as such (except the ignored ones), no matter where they reside in the original subdirectory tree covered by the lbundle. This field (a boolean) is used to limit greedy bundling only to those subdirectories already existing in the lbundle. Useful in several scenarios, e.g. when starting the resource localization to avoid being engulfed in listings of missing localized resources. Automatically enabled when <tt>track-by-subdir</tt> is in effect.

== Operation ==

To check all lbundles for the language <tt>aa</tt> in trunk, <tt>lbundle-check.py</tt> can be run like this:
<syntaxhighlight lang="bash">
$ cd $KDEREPO/trunk/l10n-kde4/
$ lbundle-check.py aa/ # check all out-of-source
$ lbundle-check.py source/link/ -l aa # check all in-source
</syntaxhighlight>
If <tt>-l aa</tt> option weren't specified in the second invocation, <tt>lbundle-check.py</tt> would check ''all'' languages in <tt>source/link/</tt> instead of only <tt>aa</tt>. In the first invocation <tt>-l aa</tt> was omitted because out-of-source bundles contain only their own language, but it wouldn't cause any problem if it were present. Only particular lbundles can be checked by giving their subdirectories as arguments.

So, what does <tt>lbundle-check.py</tt> actually do, and what does it report? For brevity, let's limit to the out-of-source bundle for the KDingus as kdeutils application. After setting up the <tt>l10n-spec</tt> for that case (as shown [[#Out-of-Source Lbundles|earlier]]), running <tt>lbundle-check.py</tt> for the first time will output:
<syntaxhighlight lang="bash">
$ cd $KDEREPO/trunk/l10n-kde4
$ lbundle-check.py aa/data/
! aa/data/kdeutils/kdingus/pics/l10n-track
--------------------
Added to tracking: 2
aa/data/kdeutils/kdingus/pics/CMakeLists.txt
aa/data/kdeutils/kdingus/pics/l10n/aa/kdingus-splash.png
$
</syntaxhighlight>
The file <tt>aa/data/kdeutils/kdingus/pics/l10n-track</tt> will be created, with the following contents:
<syntaxhighlight lang="text">
# Do not edit manually, except to remove complete lines.

# -
ok ¦CMakeLists.txt¦ 865f7...e926d 755260

# aa
ok ¦l10n/aa/kdingus-splash.png¦ 37c4a7...8fe3a 755647
</syntaxhighlight>

For each <tt>l10n-spec</tt> file encountered, <tt>lbundle-check.py</tt> will create one of these <tt>l10n-track</tt> files. <tt>l10n-track</tt> file, in each non-comment, non-empty line, contains four fields, in order: the state of the localized resource against the original, the relative path to the bundled resource, the checksum of the ''original'' resource, and the VCS revision string of the original file which has this checksum.

At this point, when <tt>l10n-spec</tt> file has been manually written, and <tt>l10n-track</tt> created by running <tt>lbundle-check.py</tt>, both should be added and committed to version control.

So long as the original resource does not change, rerunning <tt>lbundle-check.py</tt> in the same way will do nothing, as everything is in sync. Obviously, the original resources should be updated to the latest repository version prior to checking, and <tt>lbundle-check.py</tt> can do that itself if started with <tt>-u</tt> option:
<syntaxhighlight lang="bash">
$ lbundle-check.py -u aa/data/
svn up .../kdeutils/kdingus/pics/CMakeLists.txt
At revision 762512.
svn up .../kdeutils/kdingus/pics/kdingus-splash.png
At revision 762512.
$
</syntaxhighlight>

Once the original resource is modified, e.g. KDingus' splash screen is tweaked up, <tt>lbundle-check.py</tt> will report the following:
<syntaxhighlight lang="bash">
$ lbundle-check.py aa/data/
! aa/data/kdeutils/kdingus/pics/l10n-track
--------------------
Newly fuzzied: 1
aa/data/kdeutils/kdingus/pics/l10n/aa/kdingus-splash.png
</syntaxhighlight>
and its entry in <tt>l10n-track</tt> will state:
<syntaxhighlight lang="text">
fuzzy ¦l10n/aa/kdingus-splash.png¦ 37c4a7...8fe3a 755647
</syntaxhighlight>
i.e. only the status will have changed from <tt>ok</tt> to <tt>fuzzy</tt>. Now the translator has the information needed to compare what has changed in the original splash image: this entry still states the revision of the previous original file on which the localized file was based.

If the original resource is renamed, moved, or deleted, the report will be slightly different:
<syntaxhighlight lang="bash">
$ lbundle-check.py aa/data/
! aa/data/kdeutils/kdingus/pics/l10n-track
--------------------
Newly obsoleted: 1
aa/data/kdeutils/kdingus/pics/l10n/aa/kdingus-splash.png
</syntaxhighlight>
and the entry in <tt>l10n-track</tt>:
<syntaxhighlight lang="text">
obsolete ¦l10n/aa/kdingus-splash.png¦ 37c4a7...8fe3a 755647
</syntaxhighlight>
Now the translator must find out, using the old revision string, what exactly happened to the original resource, and do the same for the localized resource.

When the fuzzy or obsolete state has been resolved, localized resource updated to reflect changes in the original, it should be recorded as such. To do this, the entry for the resource is first manually deleted from <tt>l10n-track</tt> file (the whole line is removed), and then <tt>lbundle-check.py</tt> is rerun. This will recreate the entry with a fresh <tt>ok</tt> state.

If <tt>strict-state</tt> field has been set to true in <tt>l10n-spec</tt>, then not only the state of the entry in <tt>l10n-track</tt> is changed, but the tracked file itself is renamed to contain <tt>~fuzzy</tt> or <tt>~obsolete</tt> marker just before the extension. This mode is called strict because unless the out-of-sync state is corrected, the file is "misnamed" from the point of view of the runtime system, and will not be used instead of the original (it may even not get installed, depending on the exact installation instructions).

To resolve the fuzzy or obsolete state in strict mode, it is not necessary to manually remove the entry from <tt>l10n-track</tt>. Instead, it is enough to put an updated version of the file (without the state maker in its name) next to the out-of-sync version, and rerun <tt>lbundle-check.py</tt>. It will remove the old version and update the entry in <tt>l10n-track</tt>. (The renaming/removal are version control aware, i.e. proper version control commands will be used for these operations if <tt>*-vcs</tt> fields have been set in the spec file.)

When [[#Greedy Bundling|greedy bundling]] is engaged, each original file which doesn't have a localized counterpart will be reported in the track file (either as bundled or unbundled, depending on <tt>greedy-monolingual</tt> option), with it's state set to <tt>missing</tt> (except for the ignored files, given by the <tt>ignore-unbundled</tt> option). When the localized file is placed at the appropriate location, next run of <tt>lbundle-check.py</tt> will change the state to <tt>ok</tt>.

== Special Case: Tracking KDE Documentation Resources ==

{{note|This section should be mostly self-contained, providing all the information needed for resource tracking in documentation, for two reasons. Firstly, it can be reasonably assumed that many more KDE translation teams will want to track localized documentation resources, rather than some other, special resources. Secondly, structure of documentation resources is significantly different from that of normal lbundles, so having separate examples helps.}}

KDE documentation consists of two types of resources: the text, which is provided through Docbook files, and other data, mostly screenshots from applications. The text from original Docbook files is extracted into PO templates, translated through PO files, and localized Docbook files created out of them. Thus, when the original text changes, translators are automatically made aware of that through new and fuzzy entries in PO files. This is not so for screenshots. When an original screenshot changes, gets removed or added, translators get no notice of it. However, the tracking mechanism provided by <tt>lbundle-check.py</tt> can be used to help rectify this.

Localized resources of KDE documentation are not really organized into lbundles; their organization far precedes the lbundling system. Still, every language's documentation directory (<tt>aa/docs/</tt>) can be understood as one ''greedy monolingual'' lbundle composed solely of ''unbundled'' resources, and as such tracked by the <tt>lbundle-check.py</tt>. Docbook files are excluded from tracking, as they are kept up-to-date through Docbook-PO-Docbook roundtrip. At the moment, tracking is set up and operated by the coordinator of each language team that wants to have it.

To set up tracking, the coordinator of the language <tt>aa</tt> should maintain the local checkout of several modules, with same structure as KDE repository, as follows:
<syntaxhighlight lang="text">
$KDEREPO/
trunk/
l10n-kde4/
scripts/
templates/
documentation/
aa/
l10n-support/
scripts/
branches/
stable/
l10n-kde4/
scripts/
templates/
documentation/
aa/
</syntaxhighlight>
In fact, it is a good idea anyway for a language coordinator to maintain such a checkout (the total space requirement is between 500 MB and 1 GB).

The <tt>documentation/</tt> subdirectory is not actually found in the Subversion repository; it is created and periodically updated by running the <tt>scripts/populate_documentation.sh</tt> script, in stable and trunk:
<syntaxhighlight lang="bash">
$ cd $KDEREPO/trunk/l10n-kde4
$ scripts/populate_documentation.sh
$ cd $KDEREPO/branches/stable/l10n-kde4
$ scripts/populate_documentation.sh
</syntaxhighlight>
<tt>populate_documentation.sh</tt> from trunk will get exactly the documentation modules' branches translated as trunk, and <tt>populate_documentation.sh</tt> from stable those translated as stable.

After this file structure is established, create the files:
<syntaxhighlight lang="text">
$KDEREPO/trunk/l10n-kde4/aa/docs/l10n-spec
$KDEREPO/branches/stable/l10n-kde4/aa/docs/l10n-spec
</syntaxhighlight>
both with the following, branch-independent content:
<syntaxhighlight lang="text">
source-root = aa/docs:documentation
source-vcs = file
bundle-vcs = auto
track-by-subdir = yes

greedy-bundling = yes
greedy-monolingual = yes
greedy-from-level = 2

ignore-unbundled = \
CMakeLists.txt */CMakeLists.txt */*/CMakeLists.txt */*/*/CMakeLists.txt \
*/*/*.docbook */*/*/*.docbook \
</syntaxhighlight>
Note the language code <tt>aa</tt> in <tt>aa/docs:...</tt> at the top, and modify it accordingly.

Finally, run the <tt>lbundle-check.py</tt> script from <tt>trunk/l10n-support/scripts/</tt> on the documentation directories of your language, e.g. for the trunk:
<syntaxhighlight lang="bash">
$ lbundle-check.py $KDEREPO/trunk/l10n-kde4/aa/docs/
</syntaxhighlight>
After some time, depending on how many localized screenshots you have made already, a whole lot of output may appear -- listing of files having been added to tracking. More importantly, each subdirectory in <tt>aa/docs/</tt> which has (or could have) at least one localized screenshot, will get a ''tracking file'', named <tt>l10n-track</tt>. Tracking files will be automatically added to version control as well, so you can commit <tt>aa/docs/</tt> right away.

What does a tracking file look like? For example, the original documentation for KRuler contains these files:
<syntaxhighlight lang="text">
documentation/kdegraphics/kruler/
CMakeLists.txt
index.docbook
kruler.png
kruler-settings.png
</syntaxhighlight>
Assume that your language's localized documentation contains:
<syntaxhighlight lang="text">
aa/docs/kdegraphics/kruler/
CMakeLists.txt
index.docbook
kruler.png
</syntaxhighlight>
that is, the localized variant of <tt>kruler-settings.png</tt> is missing. Then, the <tt>aa/docs/kdegraphics/kruler/l10n-track</tt> that got created will have this contents:
<syntaxhighlight lang="text">
# Do not edit manually, except to remove complete lines.

ok ¦kruler.png¦ 58ba...872a 58ba...872a
missing ¦kruler-settings.png¦ 4dd2...5ebf 4dd2...5ebf
</syntaxhighlight>
Each line contains four pieces of data: the sync state and the name of the localized file, and the twice repeated checksum of the ''original'' file. (Instead of the second checksum normally there would be the repository revision of the original file, but <tt>populate_documentation.sh</tt> does not make proper repository checkouts). Since this is the first run of <tt>lbundle-check.py</tt>, it assumed that the present localized <tt>kruler.png</tt> is up-to-date (more later on this assumption), and so its state is set to <tt>ok</tt>. For <tt>kruler-settings.png</tt>, which is not there in the localized directory, the state is set as <tt>missing</tt>.

The day-to-day operation constitutes running <tt>lbundle-check.py</tt> from time to time, in the same way as the first time. When a new original screenshot is added, the appropriate <tt>l10n-track</tt> file will get a new <tt>missing</tt> entry. When a localized screenshot is produced and put into correct place, the <tt>missing</tt> state will change into <tt>ok</tt>:
<syntaxhighlight lang="bash">
$ cd $KDEREPO/trunk/l10n-kde4/
$ (...put localized kruler-settings.png into aa/docs/...)
$ lbundle-check.py aa/docs/
A (bin) aa/docs/kdegraphics/kruler/kruler-settings.png
M aa/docs/kdegraphics/kruler/l10n-track
--------------------
New 'ok': 1
aa/docs/kdegraphics/kruler/kruler-settings.png (...)
$
</syntaxhighlight>
Note that version control operations are done automatically too.

Assume now that, in the above example, after some time the original <tt>kruler.png</tt> is modified, and the original <tt>kruler-settings.png</tt> is removed. Then, running the tracker produces:
<syntaxhighlight lang="bash">
$ cd $KDEREPO/trunk/l10n-kde4/
$ lbundle-check.py aa/docs/
M aa/docs/kdegraphics/kruler/l10n-track
--------------------
New 'fuzzy': 1
aa/docs/kdegraphics/kruler/kruler.png (...)
New 'obsolete': 1
aa/docs/kdegraphics/kruler/kruler-settings.png (...)
$
</syntaxhighlight>
and the content of tracking file becomes:
<syntaxhighlight lang="text">
# Do not edit manually, except to remove complete lines.

fuzzy ¦kruler.png¦ 58ba...872a 3d94...2c6f
obsolete ¦kruler-settings.png¦ 4dd2...5ebf ae31...b561
</syntaxhighlight>
Now the translator should remove the obsoleted localized <tt>kruler-settings.png</tt>, and check the new original <tt>kruler.png</tt> to see what to do with the localized <tt>kruler.png</tt>. When the fuzzy state of localized <tt>kruler.png</tt> is resolved -- by making a new localized screenshot, or deciding that the existing screenshot is still fine -- the translator edits <tt>l10n-track</tt> to remove its line. After rerunning <tt>lbundle-check.py</tt>, new localized <tt>kruler.png</tt> will be picked up with the default <tt>ok</tt> state, and the line for <tt>kruler-settings.png</tt> will be automatically removed.

{{warning|As the top comment in <tt>l10n-track</tt> files states, never manually edit these files except to unfuzzy entries by removing their complete lines. E.g. do not change the state manually, which would leave the original checksum and revision wrong.}}

Instead of running <tt>lbundle-check.py</tt> on the whole <tt>aa/docs/</tt> tree, it can also be run on any given subdirectory of it. It will then check only the files in that subdirectory and below.

=== Many Localized Screenshots at Start ===

There is a small issue with the fact that <tt>lbundle-check.py</tt> will set state of new files to <tt>ok</tt>. If the documentation directory already has a lot of screenshots with unknown state compared to the original, it is not appropriate to consider them <tt>ok</tt>, but <tt>fuzzy</tt>. To have this, after the ''first'' run of <tt>lbundle-check.py</tt> simply postprocess all <tt>l10n-track</tt> files:
<syntaxhighlight lang="bash">
$ find aa/docs/ -iname l10n-track | xargs perl -pi -e 's/^ok /fuzzy/'
$ find aa/docs/ -iname l10n-track | xargs perl -pi -e 's/[0-9a-z]{32}/0/'
</syntaxhighlight>
(The second command replaces the original checksums with all zeros, as otherwise the files would be again recognized as up-to-date on the next run.) Afterwards, in due time, existing screenshots can be inspected one by one, and their fuzzy state removed as explained previously.

Localization/Tools/Lbundle Check

2013-02-02T16:04:05Z

Ilic: Updated to new Git/Subversion state of affairs.

Localization/Workflows/PO Summit

2013-01-31T00:13:24Z

Ilic: Put back placeholder language.

{{LocalizationBrowser|
section=[[Localization#Workflows|Workflows]]|
name=Translating in Summit|
prereqs=[[Localization/Workflows/Coordinator|Language Coordinator]]|
related=[[Localization/Tools/Pology|Pology]]|
}}

== Software Branches and Translation ==

The obvious approach to releasing software is for developers to focus on one central body of code, a single "branch", fixing bugs and adding new features to it, and from time to time taking a "snapshot" of the branch and packing it as next release. One approach to make the release process more robust, is for development to proceed in two parallel branches. From the "stable" branch the actual releases are made, mostly with bugs fixed and only very important features added. The other, "unstable" branch, is used to develop new and redesign existing features, and at some point will become the next stable branch. Depending on the project, releases may be made from the unstable branch as well, in parallel with stable releases, in order for eager users to help testing the novelties.

In KDE, all three release models may be present at any given moment. Core KDE modules, which are together labeled as "the KDE" and released in unison, follow the two branch model, with stable and trunk branch, and releases from stable branch only. KDE extragear applications may do the same, but they are not required to; they may instead make releases from the trunk branch only, or also use stable and trunk branch, but make releases from both. This presents translators with the workflow as on the following figure.

[[Image:Translating_in_branches.png|center|Classic translation by branches]]

The KDE [[Localization/Workflows/Repository_Automation|repository automation]] (aka "Scripty") is preparing template POs and merging them with language POs. From the point of view of language teams, real people who perform global actions (e.g. moving around POs for all languages when an application moves from module to another) can also be grouped here. However, the translation team is still presented with two branches of POs to translate.

In general, this means that, just like programmers, at times translators have to work on both branches, and propagate fixes from one branch to another ("backport" from trunk to stable, or "forwardport" from stable to trunk). This may be prone to coordinatorial confusion, and demand extra effort and attention when updating translations. Keeping up with two branches is easier in core KDE modules, as they are released from stable branch only and with singular schedule, but could be more taxing with extragear applications.

The exact amount of effort spent due to parallel translation, porting of fixes, and coordination, depends on the translation team. For example, a well-manned and well-coordinated team, with established style and terminology guides, custom automation to support these, may be able to keep all POs in both branches fully translated at all times with ease. Or, a team may have worked out well-defined schedules, such that any given member of the team at one point switches from translating one branch to another, without looking back, thus effectively having everyone always working on one branch (even if not the same one).

If, however, you as the team coordinator have said and thought "Should do that in trunk too, bugger", "Didn't we fix that already?", "No, you should take it from stable", "It was released from WHAT branch?", more often than you would have liked, the following presents one possibility for sidestepping such issues.

== Translating in Summit ==

For a PO catalog which exists in both stable and trunk branch, a new same-named catalog can be made by ''gathering'' all the unique messages from branch PO files, i.e. the ''summit'' of branch POs. Assuming that branch POs were not all that different, since they are two versions of the same catalog, the summit PO shouldn't have much more messages than either. Translators at all times work on the summit PO, from which the messages are periodically ''scattered'' back to original branch POs. Thus, translators can always work on summit POs, not having to do any parallel translation, branch switching or porting of fixes. The summit workflow is presented by the following figure.

[[Image:Translating_in_summit.png|center|Translation in summit]]

In the summit mode, repository automation gathers summit PO templates from branch templates, and stores them separately from real branches, at {{path|trunk/l10n-support/templates/summit/}}. The language team also has a collection summit POs, at {{path|trunk/l10n-support/LANG/summit/}}, which is the sole location where translation happens. As before, repository automation handles merging of templates in branches, but merging of summit POs is done by the team coordinator; team coordinator can opt to manually merge branch POs as well (more on why-and-how of this later). From time to time, the team coordinator fills out branch POs by scattering from summit. Not to worry, each of these special actions upon the team coordinator is done with a single command.

While it is in principle obvious that two branch POs can be made into one summit PO with the union of their messages, there are some important details that should be handled the right way by the summitting system:

* What if a PO changes modules in one branch, so that it no longer belongs to the same module in both branches (e.g. application is moved from one to another module in trunk)?

* What if a PO changes its name in one branch, but not in the other (e.g. application is renamed in trunk)?

* What if a PO in one branch is split into two POs in another (e.g. extracting a library out of monolithic application in trunk)?

* Where to place messages unique to one branch in the summit PO? The original file context of a message, which messages precede it and which follow it, should be kept as much as possible.

* If source references are used to achieve good ordering of messages in summit PO, what to do if some source file paths change in one branch (e.g. application gets restructured in trunk)?

* How to handle messages with different plurality across branches (since messages are identified only by their <tt>msgctxt</tt> and <tt>msgid</tt> field, an not <tt>msgid_plural</tt>)?

And handle these details it does, the present summitting system. This also means that teams working in the summit need not take care of the first three issues above, which are affecting manual branch handling too.

Summit POs are normal, fully valid POs in their own right. A message in a summit PO is different from branch PO only by being equipped with another comment, <tt>#. +> ...</tt>, showing in which branches the message exists:

<syntaxhighlight lang="text">
#. +> trunk
#: kdeui/jobs/kwidgetjobtracker.cpp:469
msgctxt "The destination url of a job"
msgid "Destination:"
msgstr ""
⁠
#. +> stable
#: kdeui/jobs/kwidgetjobtracker.cpp:469
msgid "Destination:"
msgstr ""
⁠
#. +> trunk stable
#: kdeui/jobs/kwidgetjobtracker.cpp:517
msgid "&Keep this window open after transfer is complete"
msgstr ""
</syntaxhighlight>

The first message above thus exists in trunk only, the second in stable only, and the third in both branches. The source reference always points to the source file in the first listed branch. Any extracted comments (<tt>#.</tt>) other than the branch list are also taken from the first listed branch.

Note that the two messages above are different only by context; the context was added in trunk, but not in stable, in order not to break message freeze. However, due to careful ordering of messages in summit POs, these two messages appear together, allowing translator to immediately make correction in stable branch too if the new context in trunk shows it to be necessary.

=== Setting Up and Daily Operation ===

Before initializing language summit, the team coordinator has to have all the necessary paths checked out from the KDE repository, and structured on the local machine exactly as in the repository. If the path to the root of KDE repository on the local machine is <tt>$KDEREPO</tt>, and the language code <tt>LANG</tt>, then the structure should be as follows, with leaf directories checked out in full:

<syntaxhighlight lang="text">
$KDEREPO/
trunk/
l10n-kde4/
scripts/
templates/
LANG/
l10n-support/
scripts/
pology/
templates/
LANG/
branches/
stable/
l10n-kde4/
scripts/
templates/
LANG/
</syntaxhighlight>

It is important that this structure is fully under version control (rather than e.g. top directories being manually created), so if that is not the case, the following commands will fetch everything anew in proper order:

<syntaxhighlight lang="bash">
cd $KDEREPO
svn co --depth=empty svn+ssh://svn.kde.org/home/kde .
svn up --depth=empty branches branches/stable branches/stable/l10n-kde4
svn up --depth=empty trunk trunk/l10n-support trunk/l10n-kde4
svn up branches/stable/l10n-kde4/{scripts,templates,LANG}
svn up trunk/l10n-kde4/{scripts,templates,LANG}
svn up trunk/l10n-support/{pology,scripts,templates,LANG}
</syntaxhighlight>

Note the trailing dot in the first <tt>svn</tt> command, and do not miss to substitute LANG for the real language code in the last three commands. <tt>--depth=empty</tt> option prevents recursive fetching, so that e.g. the first <tt>svn</tt> command does not fetch the complete repository tree. Later you can regularly update this tree with single command:

<syntaxhighlight lang="bash">
svn up $KDEREPO/{trunk,branches/stable}/l10n-kde4/{scripts,templates,LANG} \
$KDEREPO/trunk/l10n-support/{pology,scripts,templates,LANG}
</syntaxhighlight>

Summit operations are performed using the <tt>posummit</tt> command, which is part of [[Localization/Tools/Pology|Pology]], residing in {{path|trunk/l10n-support/pology/}}. Therefore the first thing to do is to setup Pology, which amounts only to setting the proper path:

<syntaxhighlight lang="bash">
$ export PATH=$KDEREPO/trunk/l10n-support/pology/bin:$PATH
</syntaxhighlight>

To initialize the summit, by gathering from existing translation in branches, the team coordinator executes:

<syntaxhighlight lang="bash">
$ cd $KDEREPO/trunk/l10n-support
$ posummit scripts/messages.summit LANG gather --create --force
</syntaxhighlight>

Depending on the amount of translation, after some minutes the initial gathering will have been completed, and language summit located under {{path|$KDEREPO/trunk/l10n-support/LANG/summit/messages/}}. This is the only time when the coordinator performs the gather operation on language POs; it is daily done only on templates by repository automation. Then, the created language summit should be merged with current summit templates:

<syntaxhighlight lang="bash">
$ posummit scripts/messages.summit LANG merge
</syntaxhighlight>

Merging the summit is something that the coordinator does periodically, with frequency of own desire. For example, it can be done daily, or with increasing frequency as the last day for translation for the next release approaches.

After the first merging, language summit is ready for active translation. The coordinator should now commit {{path|$KDEREPO/trunk/l10n-support/LANG/}}, and, importantly, notify team members to stop working on branch POs and focus exclusively on summit POs.

To scatter the summit, i.e. fill out POs in stable and trunk branch from the summit POs, the coordinator periodically executes:

<syntaxhighlight lang="bash">
$ posummit scripts/messages.summit LANG scatter
</syntaxhighlight>

As with merging, there is no fixed schedule when scattering should be done. Of course, it must necessarily be done before the next release is tagged, and in between it is useful to scatter for runtime testing, or to have translation statistics by branches on l10n.kde.org up to date.

Periodic scattering and merging of the complete summit are basically all that a language team coordinator needs to do specifically to operate the summit. Also, since {{path|l10n-support/scripts/}} and {{path|l10n-support/pology/}} contain scripts and settings critical for proper functioning of summit operations, and may be tweaked at any time, they should always be updated from the repository together with PO files and templates (in fact, it is best to always update at once the whole tree as outlined above).

{{note|For documentation POs, summit setup and operation is the same, only replacing every <tt>messages</tt> with <tt>docmessages</tt> in the command lines above. The user interface and documentation summits are fully independent, so for a trial period it is reasonable to work with the interface summit only, and engage documentation summit once the trial has been deemed successfull.}}

=== Operation Targets ===

Sometimes it is advantageous to merge or scatter just a single catalog, a single module, a single branch, or any combination thereof. To this end, scatter and merge operations accept any number of ''operation targets'' after the operation keyword, specified as one of <tt>''CATALOG''</tt>, <tt>''BRANCH'':''CATALOG''</tt>, <tt>''MODULE''/</tt>, <tt>''BRANCH'':''MODULE''/</tt>, and <tt>''BRANCH'':</tt>. For example, to scatter just to Dolphin's PO in stable branch, in order to test translation at runtime, one would execute:

<syntaxhighlight lang="bash">
$ cd $KDEREPO/trunk/l10n-support
$ posummit scripts/messages.summit LANG scatter stable:dolphin
</syntaxhighlight>

(note no <tt>.po</tt> ending on catalog name). Or, to scatter to every PO in <tt>kdeplasma-addons</tt> module in stable branch:

<syntaxhighlight lang="bash">
$ cd $KDEREPO/trunk/l10n-support
$ posummit scripts/messages.summit LANG scatter stable:kdeplasma-addons/
</syntaxhighlight>

(the trailing slash is mandatory, or else <tt>posummit</tt> would think that <tt>kdeplasma-addons</tt> is a catalog name). Finally, to scatter to all catalogs in the stable branch (with the trailing colon for the same reason as earlier):

<syntaxhighlight lang="bash">
$ cd $KDEREPO/trunk/l10n-support
$ posummit scripts/messages.summit LANG scatter stable:
</syntaxhighlight>

The command line arguments of <tt>posummit</tt> operations are intentionally arranged such that all the arguments fixed for one language are placed first. If operation targets are used frequently, then it is convenient to place the immutable part of the command line under a shell alias, with absolute path to the summit setup file:

<syntaxhighlight lang="bash">
$ alias posummit-kde-LANG="posummit $KDEREPO/trunk/l10n-support/scripts/messages.summit LANG"
$ cd ANYWHERE
$ posummit-kde-LANG scatter stable:dolphin
</syntaxhighlight>

=== Externally Injected Branch Catalogs ===

Once the summit is in operation, normally the only way for a catalog to appear in branches (trunk or stable) is through scattering from the summit. However, sometimes an application which had seen considerable development outside of the infrastructure of KDE project gets moved in, drawing in any of its existing translations into branches. Such externally injected branch catalogs cause warnings to be issued on summit scattering:

<syntaxhighlight lang="bash">
$ cd $KDEREPO/trunk/l10n-support
$ touch ../l10n-kde4/LANG/messages/kdetoys/foobar.po # dummy injection
$ touch ../l10n-kde4/LANG/messages/kdetoys/libfoobar.po # dummy injection
$
$ posummit scripts/messages.summit LANG scatter
posummit: [warning] Some branch catalogs have no associated summit catalog (expected summit path given):
... ../l10n-kde4/LANG/messages/kdetoys/foobar.po --> LANG/summit/messages/kdetoys/foobar.po
... ../l10n-kde4/LANG/messages/kdetoys/libfoobar.po --> LANG/summit/messages/kdetoys/libfoobar.po
$
</syntaxhighlight>

When this happens, the injected catalogs need to be gathered into the summit. This is done just like the initial gathering, only giving injected catalogs' names as operation targets:

<syntaxhighlight lang="bash">
$ posummit scripts/messages.summit LANG gather --create --force foobar libfoobar
</syntaxhighlight>

=== Summit Customization ===

File {{path|scripts/messages.summit}} (i.e. {{path|$KDEREPO/trunk/l10n-support/scripts/messages.summit}}), given as first argument to <tt>posummit</tt>, contains the general summit setup for all languages. This file is set to include customization file per language, if it exists at {{path|$KDEREPO/trunk/l10n-support/LANG/summit/messages.extras.summit}}, which contains additions and overrides to the general setup as desired by the language team. Same as the general setup file, the customization file is a Python source. It uses an external object named <tt>S</tt> to define summit settings, and can, of course, contain any helper Python code (the file is executed only once, at the beginning of a <tt>posummit</tt> run). It has the following general layout:

<syntaxhighlight lang="python">
# -*- coding: UTF-8 -*-
# kate: syntax Python;
# This file is included by scripts/messages.summit
# for language-specific additions/overrides.
#
# ...
# ... operations with object S and other Python code ...
# ...
</syntaxhighlight>

Object <tt>S</tt> provides both data attributes and methods to configure different aspects of summit operation. For example, while the general setup specifies that text fields in summit catalogs should not be wrapped on column but should be wrapped on logical breaks (like some markup tags), this can be overriden using <tt>summit_wrap</tt> and <tt>summit_fine_wrap</tt> attributes:

<syntaxhighlight lang="python">
S.summit_wrap = True
S.summit_fine_wrap = False
</syntaxhighlight>

Or, to get the path to a file relative to the summit customization file itself (rather than to current working directory, where <tt>posummit</tt> was executed), method <tt>resolve_path_rooted</tt> can be used:

<syntaxhighlight lang="python">
relpath = S.resolve_path_rooted("../whatever.txt")
</syntaxhighlight>

The following sections will present some typical customization possibilities.

=== Fully Local Merging ===

When scattering from the summit, sometimes there will be reports of "messages missing in the summit". This happens because of time rift created by the Scripty merging branch POs, gathering summit templates, and a team coordinator merging the language summit, thus making some messages in branch POs not always present in the summit. This condition is benign, as such warnings will start to disappear with the message freeze approaching, but can be annoying. For this reason, team coordinator can stop Scripty from merging branch POs, and have the <tt>posummit ... merge</tt> command alone merge not only summit POs, but stable and trunk POs as well, such that summit and branches are always in perfect sync.

First, to stop Scripty from merging branch POs, a file named <tt>no-auto-merge</tt> (with arbitrary content) should be committed to the roots of respective trees, e.g.:

<syntaxhighlight lang="text">
$ touch $KDEREPO/trunk/l10n-kde4/messages/LANG/no-auto-merge
$ touch $KDEREPO/branches/stable/l10n-kde4/LANG/messages/no-auto-merge
</syntaxhighlight>

Then, to make summit <tt>posummit ... merge</tt> merge everything, the following lines should be added into the summit customization file:

<syntaxhighlight lang="python">
# Set local merging for all branches.
for branch in S.branches:
branch["merge_locally"] = True
</syntaxhighlight>

Once local merging of all branches is set, the coordinator can also use operation targets for selective merging, e.g. to merge only stable branch:

<syntaxhighlight lang="bash">
$ cd $KDEREPO/trunk/l10n-support
$ posummit scripts/messages.summit LANG merge stable:
</syntaxhighlight>

=== Merging with Compendium ===

By default, <tt>msgmerge</tt> takes into account only the catalog itself to fill out near-match translations to new messages introduced by merging. i.e. to produce fuzzy messages. However, an arbitrary PO file can be given to <tt>msgmerge</tt> as another possible source of earlier translations, through the <tt>--compendium</tt> option. For maximum effect, this PO file is usually constructed as the collection of messages from all PO files project-wide, and hence called the ''compendium''.

Compendium can be used in summit merge operations simply by setting <tt>compendium_on_merge</tt> attribute in summit customization file. If the compendium is located at {{path|$KDEREPO/trunk/l10n-support/LANG/summit/messages-compendium.po}}, then:

<syntaxhighlight lang="python">
# Compendium to use when merging summit catalogs.
S.compendium_on_merge = S.resolve_path_rooted("messages-compendium.po")
</syntaxhighlight>

Note that if not located as above, compendium should not be placed within {{path|.../LANG/summit/messages/}}, as then it would be considered a summit catalog itself. In case fully local merging is engaged, only summit POs will be merged with compendium; it makes no sense for branch POs, since they are not being directly translated.

See [http://www.gnu.org/software/automake/manual/gettext/Creating-Compendia.html "Creating Compendia"] section of Gettext manual for details on how to create a compendium out of current body of summit translations. Once you establish the precise commands to create the compendium (whether to collect fuzzies too, whether to include old compendium as one of sources for the new, etc.), you would periodically refresh and commit the updated compendium.

=== Vivifying Summit Catalogs ===

Translation of a summit catalog normally starts the same way as it did for branch catalogs, by copying it over from template and initializing the header. This can be done manually, but it can also be quite automatic when using project manager as sometimes provided by dedicated PO editors. However, relying on the editor's project manager can be disadvantageous at times. For example, aside from the PO editor, other, frequently command line tools may be used to process the body of translation, and these tools might need to consider non-started templates as if they were empty POs (e.g. statistics). Team members who do not have full local checkout, or no project set up in the editor, may need to jump between language and template directories when looking for files to translate.

Therefore, summit setup can be customized to automatically create ("vivify") summit catalogs for every new summit template, so that there is never the need (for a tool or human) to specifically treat templates as empty catalogs. This is done by adding the following lines into customization file:

<syntaxhighlight lang="python">
# Create empty summit catalog for every new summit template.
S.vivify_on_merge = True
S.vivify_w_translator = "Noone Noonian <noone.noonian@nowhere.null>"
S.vivify_w_langteam = "Neverneese <neverteam@nowhere.null>"
S.vivify_w_plurals = "nplurals=2; plural=n != 1;"
# Minimum translation state to create branch catalog on scatter.
S.scatter_min_completeness = 0.9
</syntaxhighlight>

The <tt>vivify_w_*</tt> attributes set the necessary data to initialize headers of newly created summit catalogs. Aside from those listed above, there is also the <tt>vivify_w_charset</tt> attribute, which is by default <tt>"UTF-8"</tt>, and very probably should not be changed.

The <tt>scatter_min_completeness</tt> attribute does not refer to vivification of summit catalogs, but should invariably be set in this context. When scattering from summit, normally the branch catalog is automatically created if there is the corresponding summit catalog. When vivification is engaged, this would result in creation of empty branch catalogs too, which is not desired for two reasons. Firstly, empty branch catalogs would become part of the released language pack, for which there is no practical reason. Secondly, KDE's i18n system regards an application with installed catalog as translated, so messages coming from basic system catalogs (e.g. <tt>kdelibs4</tt>) would show through, resulting in mostly untranslated user interface with specks of translation -- many users consider this rather ungainly. Therefore, <tt>scatter_min_completeness</tt> sets how complete the translation of currently non-existing branch catalog should be after scatter (0.0 empty, 1.0 fully complete), for the branch catalog to actually be created. If the <tt>--force</tt> option is given to <tt>posummit</tt>, the branch catalog will be created regardless of its translation state, which may be handy in combination with [[#Operation Targets|operation target]] when wanting to check work in progress in running application.

If [[#Merging with Compendium|the compendium has been set]] for merging, every vivified summit catalog will also be merged against the compendium, to fill out as many of messages with approximate translations.

=== Scatter Hooks ===

((TODO. Checks, modifications...))

=== Merge Hooks ===

((TODO. Special header fields...))

== Disadvantages to Summit and Remedies ==

Although hopefully shadowed by the advantages, working in summit is not without its disadvantages. These should be weighed when deciding of whether to try out the summit workflow.

Obviously, while summit operations are made to be quite automatic, some extra aptitude is asked of the team coordinator. Reasonable shell handling, understanding of version control operations, feeling the pulse of repository automation, are all prerequisites, and some scripting ability advantageous.

After the summit is put in operation, any changes made manually in branch POs will not propagate to summit, and will be soon lost to scattering -- summit translations override everything in branches. This means that the whole team must work in the summit, it is not possible for some members to use the summit, and some not.

A summit PO file will necessarily have more messages than either of the branch files. For example, in the KDE 4.0/4.1 and 4.1/4.2 cycle, summit POs of core KDE modules had on average less than 5% more words than their stable counterparts. However, the said percent is the top, never approached limit of wasted workload due to trunk messages coming and going, given that as the next feature KDE release approaches, more and more trunk messages will find their way into it.

Another, more pressing issue with increased size of summit POs is the following scenario: a stable release is around the corner, and the team has no time to update summit POs fully, but could update only stable messages in them. E.g. there are 1000 untranslated and fuzzy messages, out of which only 100 are from the stable branch. A clever dedicated PO editor could allow jumping only through untranslated and fuzzy messages which also satisfy a general search criteria, which in this case would be that a comment matches <tt>#\.\+>.*stable</tt> regular expression. On the other hand, with some external help, it is enough if the PO editor can merely search through comments. Then, the <tt>posieve</tt> command (ready to use next to <tt>posummit</tt>) can equip untranslated and fuzzy stable messages with <tt>untranslated</tt> flag (producing <tt>#, ..., untranslated</tt> comment), and this flag can be searched for in the PO editor:

<syntaxhighlight lang="bash">
$ posieve tag-untranslated -sbranch:stable -swfuzzy PATHS_TO_PO_FILES_OR_DIRS
</syntaxhighlight>

The <tt>untranslated</tt> flag does not need to be manually removed when the message is updated. It will automatically disappear on the next merge, as it is not among flags known to Gettext.

If an automatic source lookup mechanism (to check the message context by following its source references) was available for branches, it may not entirely work for summit. Already the branch POs in KDE are kept separate from the code, which means that paths given by source references cannot be followed directly, but appropriate root paths have to be added in some way (a dedicated PO editor may provide a way to do this); this should continue to work with summit for those messages which exist both in trunk and stable branch, since their source references correspond to trunk code. But for messages found only in stable branch, whose source references correspond to stable code, existing source lookup mechanism will likely assume trunk roots again, and try to fetch wrong sources. To fix this, the source lookup mechanism should be flexible enough to select roots based on branch keywords given in summit messages' <tt>#. +> ...</tt> comment.

There is also the organizational issue with starting to use the summit, and, if it does not help as expected, stopping to use it. Team members have to be reminded to not send in branch POs at start, and then to be sent back to branch POs if summit is disbanded. On the plus side, disbanding summit is technically simple: just remove from the repository {{path|l10n-supprot/LANG/summit}}, possibly also <tt>no-auto-merge</tt> files if [[#Fully Local Merging|fully local merging]] was set up, and that is it.

== Another Way to Improve Branch Handling ==

If summit seems a lot to digest, or is simply an overkill for team's needs, but still some improvement to manual handling of branches would be welcomed, KDE's dedicated PO editor [[Localization/Tools/Lokalize|Lokalize]] offers a ''branch sync'' mode. It works as follows.

In Lokalize project definition, the local paths of trunk and stable PO roots are set in ''Translation directory:'' and ''Branch directory:'' fields. Then, when a trunk PO file is opened, if it has a stable counterpart with same name and location as in the trunk, this stable PO is also going to be opened. For each trunk message in the main editing pane, if such a message exists in stable PO too, the stable message will be shown in the ''Secondary Sync'' pane; changes in the translation of trunk message will reflect to the stable message, and stable PO file will also be saved when the trunk is saved.

Furthermore, any team member can personally choose to work like this, there is no need to change the workflow of the language team as whole. When sending modifications to the coordinator, team members who rely on this feature of Lokalize simply send both trunk and stable POs that got modified.

Localization/Workflows/PO Summit

2013-01-31T00:11:23Z

Ilic: Updated to warning instead of error behavior on externally injected branch catalogs.

{{LocalizationBrowser|
section=[[Localization#Workflows|Workflows]]|
name=Translating in Summit|
prereqs=[[Localization/Workflows/Coordinator|Language Coordinator]]|
related=[[Localization/Tools/Pology|Pology]]|
}}

== Software Branches and Translation ==

The obvious approach to releasing software is for developers to focus on one central body of code, a single "branch", fixing bugs and adding new features to it, and from time to time taking a "snapshot" of the branch and packing it as next release. One approach to make the release process more robust, is for development to proceed in two parallel branches. From the "stable" branch the actual releases are made, mostly with bugs fixed and only very important features added. The other, "unstable" branch, is used to develop new and redesign existing features, and at some point will become the next stable branch. Depending on the project, releases may be made from the unstable branch as well, in parallel with stable releases, in order for eager users to help testing the novelties.

In KDE, all three release models may be present at any given moment. Core KDE modules, which are together labeled as "the KDE" and released in unison, follow the two branch model, with stable and trunk branch, and releases from stable branch only. KDE extragear applications may do the same, but they are not required to; they may instead make releases from the trunk branch only, or also use stable and trunk branch, but make releases from both. This presents translators with the workflow as on the following figure.

[[Image:Translating_in_branches.png|center|Classic translation by branches]]

The KDE [[Localization/Workflows/Repository_Automation|repository automation]] (aka "Scripty") is preparing template POs and merging them with language POs. From the point of view of language teams, real people who perform global actions (e.g. moving around POs for all languages when an application moves from module to another) can also be grouped here. However, the translation team is still presented with two branches of POs to translate.

In general, this means that, just like programmers, at times translators have to work on both branches, and propagate fixes from one branch to another ("backport" from trunk to stable, or "forwardport" from stable to trunk). This may be prone to coordinatorial confusion, and demand extra effort and attention when updating translations. Keeping up with two branches is easier in core KDE modules, as they are released from stable branch only and with singular schedule, but could be more taxing with extragear applications.

The exact amount of effort spent due to parallel translation, porting of fixes, and coordination, depends on the translation team. For example, a well-manned and well-coordinated team, with established style and terminology guides, custom automation to support these, may be able to keep all POs in both branches fully translated at all times with ease. Or, a team may have worked out well-defined schedules, such that any given member of the team at one point switches from translating one branch to another, without looking back, thus effectively having everyone always working on one branch (even if not the same one).

If, however, you as the team coordinator have said and thought "Should do that in trunk too, bugger", "Didn't we fix that already?", "No, you should take it from stable", "It was released from WHAT branch?", more often than you would have liked, the following presents one possibility for sidestepping such issues.

== Translating in Summit ==

For a PO catalog which exists in both stable and trunk branch, a new same-named catalog can be made by ''gathering'' all the unique messages from branch PO files, i.e. the ''summit'' of branch POs. Assuming that branch POs were not all that different, since they are two versions of the same catalog, the summit PO shouldn't have much more messages than either. Translators at all times work on the summit PO, from which the messages are periodically ''scattered'' back to original branch POs. Thus, translators can always work on summit POs, not having to do any parallel translation, branch switching or porting of fixes. The summit workflow is presented by the following figure.

[[Image:Translating_in_summit.png|center|Translation in summit]]

In the summit mode, repository automation gathers summit PO templates from branch templates, and stores them separately from real branches, at {{path|trunk/l10n-support/templates/summit/}}. The language team also has a collection summit POs, at {{path|trunk/l10n-support/LANG/summit/}}, which is the sole location where translation happens. As before, repository automation handles merging of templates in branches, but merging of summit POs is done by the team coordinator; team coordinator can opt to manually merge branch POs as well (more on why-and-how of this later). From time to time, the team coordinator fills out branch POs by scattering from summit. Not to worry, each of these special actions upon the team coordinator is done with a single command.

While it is in principle obvious that two branch POs can be made into one summit PO with the union of their messages, there are some important details that should be handled the right way by the summitting system:

* What if a PO changes modules in one branch, so that it no longer belongs to the same module in both branches (e.g. application is moved from one to another module in trunk)?

* What if a PO changes its name in one branch, but not in the other (e.g. application is renamed in trunk)?

* What if a PO in one branch is split into two POs in another (e.g. extracting a library out of monolithic application in trunk)?

* Where to place messages unique to one branch in the summit PO? The original file context of a message, which messages precede it and which follow it, should be kept as much as possible.

* If source references are used to achieve good ordering of messages in summit PO, what to do if some source file paths change in one branch (e.g. application gets restructured in trunk)?

* How to handle messages with different plurality across branches (since messages are identified only by their <tt>msgctxt</tt> and <tt>msgid</tt> field, an not <tt>msgid_plural</tt>)?

And handle these details it does, the present summitting system. This also means that teams working in the summit need not take care of the first three issues above, which are affecting manual branch handling too.

Summit POs are normal, fully valid POs in their own right. A message in a summit PO is different from branch PO only by being equipped with another comment, <tt>#. +> ...</tt>, showing in which branches the message exists:

<syntaxhighlight lang="text">
#. +> trunk
#: kdeui/jobs/kwidgetjobtracker.cpp:469
msgctxt "The destination url of a job"
msgid "Destination:"
msgstr ""
⁠
#. +> stable
#: kdeui/jobs/kwidgetjobtracker.cpp:469
msgid "Destination:"
msgstr ""
⁠
#. +> trunk stable
#: kdeui/jobs/kwidgetjobtracker.cpp:517
msgid "&Keep this window open after transfer is complete"
msgstr ""
</syntaxhighlight>

The first message above thus exists in trunk only, the second in stable only, and the third in both branches. The source reference always points to the source file in the first listed branch. Any extracted comments (<tt>#.</tt>) other than the branch list are also taken from the first listed branch.

Note that the two messages above are different only by context; the context was added in trunk, but not in stable, in order not to break message freeze. However, due to careful ordering of messages in summit POs, these two messages appear together, allowing translator to immediately make correction in stable branch too if the new context in trunk shows it to be necessary.

=== Setting Up and Daily Operation ===

Before initializing language summit, the team coordinator has to have all the necessary paths checked out from the KDE repository, and structured on the local machine exactly as in the repository. If the path to the root of KDE repository on the local machine is <tt>$KDEREPO</tt>, and the language code <tt>LANG</tt>, then the structure should be as follows, with leaf directories checked out in full:

<syntaxhighlight lang="text">
$KDEREPO/
trunk/
l10n-kde4/
scripts/
templates/
LANG/
l10n-support/
scripts/
pology/
templates/
LANG/
branches/
stable/
l10n-kde4/
scripts/
templates/
LANG/
</syntaxhighlight>

It is important that this structure is fully under version control (rather than e.g. top directories being manually created), so if that is not the case, the following commands will fetch everything anew in proper order:

<syntaxhighlight lang="bash">
cd $KDEREPO
svn co --depth=empty svn+ssh://svn.kde.org/home/kde .
svn up --depth=empty branches branches/stable branches/stable/l10n-kde4
svn up --depth=empty trunk trunk/l10n-support trunk/l10n-kde4
svn up branches/stable/l10n-kde4/{scripts,templates,LANG}
svn up trunk/l10n-kde4/{scripts,templates,LANG}
svn up trunk/l10n-support/{pology,scripts,templates,LANG}
</syntaxhighlight>

Note the trailing dot in the first <tt>svn</tt> command, and do not miss to substitute LANG for the real language code in the last three commands. <tt>--depth=empty</tt> option prevents recursive fetching, so that e.g. the first <tt>svn</tt> command does not fetch the complete repository tree. Later you can regularly update this tree with single command:

<syntaxhighlight lang="bash">
svn up $KDEREPO/{trunk,branches/stable}/l10n-kde4/{scripts,templates,LANG} \
$KDEREPO/trunk/l10n-support/{pology,scripts,templates,LANG}
</syntaxhighlight>

Summit operations are performed using the <tt>posummit</tt> command, which is part of [[Localization/Tools/Pology|Pology]], residing in {{path|trunk/l10n-support/pology/}}. Therefore the first thing to do is to setup Pology, which amounts only to setting the proper path:

<syntaxhighlight lang="bash">
$ export PATH=$KDEREPO/trunk/l10n-support/pology/bin:$PATH
</syntaxhighlight>

To initialize the summit, by gathering from existing translation in branches, the team coordinator executes:

<syntaxhighlight lang="bash">
$ cd $KDEREPO/trunk/l10n-support
$ posummit scripts/messages.summit LANG gather --create --force
</syntaxhighlight>

Depending on the amount of translation, after some minutes the initial gathering will have been completed, and language summit located under {{path|$KDEREPO/trunk/l10n-support/LANG/summit/messages/}}. This is the only time when the coordinator performs the gather operation on language POs; it is daily done only on templates by repository automation. Then, the created language summit should be merged with current summit templates:

<syntaxhighlight lang="bash">
$ posummit scripts/messages.summit LANG merge
</syntaxhighlight>

Merging the summit is something that the coordinator does periodically, with frequency of own desire. For example, it can be done daily, or with increasing frequency as the last day for translation for the next release approaches.

After the first merging, language summit is ready for active translation. The coordinator should now commit {{path|$KDEREPO/trunk/l10n-support/LANG/}}, and, importantly, notify team members to stop working on branch POs and focus exclusively on summit POs.

To scatter the summit, i.e. fill out POs in stable and trunk branch from the summit POs, the coordinator periodically executes:

<syntaxhighlight lang="bash">
$ posummit scripts/messages.summit LANG scatter
</syntaxhighlight>

As with merging, there is no fixed schedule when scattering should be done. Of course, it must necessarily be done before the next release is tagged, and in between it is useful to scatter for runtime testing, or to have translation statistics by branches on l10n.kde.org up to date.

Periodic scattering and merging of the complete summit are basically all that a language team coordinator needs to do specifically to operate the summit. Also, since {{path|l10n-support/scripts/}} and {{path|l10n-support/pology/}} contain scripts and settings critical for proper functioning of summit operations, and may be tweaked at any time, they should always be updated from the repository together with PO files and templates (in fact, it is best to always update at once the whole tree as outlined above).

{{note|For documentation POs, summit setup and operation is the same, only replacing every <tt>messages</tt> with <tt>docmessages</tt> in the command lines above. The user interface and documentation summits are fully independent, so for a trial period it is reasonable to work with the interface summit only, and engage documentation summit once the trial has been deemed successfull.}}

=== Operation Targets ===

Sometimes it is advantageous to merge or scatter just a single catalog, a single module, a single branch, or any combination thereof. To this end, scatter and merge operations accept any number of ''operation targets'' after the operation keyword, specified as one of <tt>''CATALOG''</tt>, <tt>''BRANCH'':''CATALOG''</tt>, <tt>''MODULE''/</tt>, <tt>''BRANCH'':''MODULE''/</tt>, and <tt>''BRANCH'':</tt>. For example, to scatter just to Dolphin's PO in stable branch, in order to test translation at runtime, one would execute:

<syntaxhighlight lang="bash">
$ cd $KDEREPO/trunk/l10n-support
$ posummit scripts/messages.summit LANG scatter stable:dolphin
</syntaxhighlight>

(note no <tt>.po</tt> ending on catalog name). Or, to scatter to every PO in <tt>kdeplasma-addons</tt> module in stable branch:

<syntaxhighlight lang="bash">
$ cd $KDEREPO/trunk/l10n-support
$ posummit scripts/messages.summit LANG scatter stable:kdeplasma-addons/
</syntaxhighlight>

(the trailing slash is mandatory, or else <tt>posummit</tt> would think that <tt>kdeplasma-addons</tt> is a catalog name). Finally, to scatter to all catalogs in the stable branch (with the trailing colon for the same reason as earlier):

<syntaxhighlight lang="bash">
$ cd $KDEREPO/trunk/l10n-support
$ posummit scripts/messages.summit LANG scatter stable:
</syntaxhighlight>

The command line arguments of <tt>posummit</tt> operations are intentionally arranged such that all the arguments fixed for one language are placed first. If operation targets are used frequently, then it is convenient to place the immutable part of the command line under a shell alias, with absolute path to the summit setup file:

<syntaxhighlight lang="bash">
$ alias posummit-kde-LANG="posummit $KDEREPO/trunk/l10n-support/scripts/messages.summit LANG"
$ cd ANYWHERE
$ posummit-kde-LANG scatter stable:dolphin
</syntaxhighlight>

=== Externally Injected Branch Catalogs ===

Once the summit is in operation, normally the only way for a catalog to appear in branches (trunk or stable) is through scattering from the summit. However, sometimes an application which had seen considerable development outside of the infrastructure of KDE project gets moved in, drawing in any of its existing translations into branches. Such externally injected branch catalogs cause warnings to be issued on summit scattering:

<syntaxhighlight lang="bash">
$ cd $KDEREPO/trunk/l10n-support
$ touch ../l10n-kde4/LANG/messages/kdetoys/foobar.po # dummy injection
$ touch ../l10n-kde4/LANG/messages/kdetoys/libfoobar.po # dummy injection
$
$ posummit scripts/messages.summit LANG scatter
posummit: [warning] Some branch catalogs have no associated summit catalog (expected summit path given):
... ../l10n-kde4/sr/messages/kdetoys/foobar.po --> sr/summit/messages/kdetoys/foobar.po
... ../l10n-kde4/sr/messages/kdetoys/libfoobar.po --> sr/summit/messages/kdetoys/libfoobar.po
$
</syntaxhighlight>

When this happens, the injected catalogs need to be gathered into the summit. This is done just like the initial gathering, only giving injected catalogs' names as operation targets:

<syntaxhighlight lang="bash">
$ posummit scripts/messages.summit LANG gather --create --force foobar libfoobar
</syntaxhighlight>

=== Summit Customization ===

File {{path|scripts/messages.summit}} (i.e. {{path|$KDEREPO/trunk/l10n-support/scripts/messages.summit}}), given as first argument to <tt>posummit</tt>, contains the general summit setup for all languages. This file is set to include customization file per language, if it exists at {{path|$KDEREPO/trunk/l10n-support/LANG/summit/messages.extras.summit}}, which contains additions and overrides to the general setup as desired by the language team. Same as the general setup file, the customization file is a Python source. It uses an external object named <tt>S</tt> to define summit settings, and can, of course, contain any helper Python code (the file is executed only once, at the beginning of a <tt>posummit</tt> run). It has the following general layout:

<syntaxhighlight lang="python">
# -*- coding: UTF-8 -*-
# kate: syntax Python;
# This file is included by scripts/messages.summit
# for language-specific additions/overrides.
#
# ...
# ... operations with object S and other Python code ...
# ...
</syntaxhighlight>

Object <tt>S</tt> provides both data attributes and methods to configure different aspects of summit operation. For example, while the general setup specifies that text fields in summit catalogs should not be wrapped on column but should be wrapped on logical breaks (like some markup tags), this can be overriden using <tt>summit_wrap</tt> and <tt>summit_fine_wrap</tt> attributes:

<syntaxhighlight lang="python">
S.summit_wrap = True
S.summit_fine_wrap = False
</syntaxhighlight>

Or, to get the path to a file relative to the summit customization file itself (rather than to current working directory, where <tt>posummit</tt> was executed), method <tt>resolve_path_rooted</tt> can be used:

<syntaxhighlight lang="python">
relpath = S.resolve_path_rooted("../whatever.txt")
</syntaxhighlight>

The following sections will present some typical customization possibilities.

=== Fully Local Merging ===

When scattering from the summit, sometimes there will be reports of "messages missing in the summit". This happens because of time rift created by the Scripty merging branch POs, gathering summit templates, and a team coordinator merging the language summit, thus making some messages in branch POs not always present in the summit. This condition is benign, as such warnings will start to disappear with the message freeze approaching, but can be annoying. For this reason, team coordinator can stop Scripty from merging branch POs, and have the <tt>posummit ... merge</tt> command alone merge not only summit POs, but stable and trunk POs as well, such that summit and branches are always in perfect sync.

First, to stop Scripty from merging branch POs, a file named <tt>no-auto-merge</tt> (with arbitrary content) should be committed to the roots of respective trees, e.g.:

<syntaxhighlight lang="text">
$ touch $KDEREPO/trunk/l10n-kde4/messages/LANG/no-auto-merge
$ touch $KDEREPO/branches/stable/l10n-kde4/LANG/messages/no-auto-merge
</syntaxhighlight>

Then, to make summit <tt>posummit ... merge</tt> merge everything, the following lines should be added into the summit customization file:

<syntaxhighlight lang="python">
# Set local merging for all branches.
for branch in S.branches:
branch["merge_locally"] = True
</syntaxhighlight>

Once local merging of all branches is set, the coordinator can also use operation targets for selective merging, e.g. to merge only stable branch:

<syntaxhighlight lang="bash">
$ cd $KDEREPO/trunk/l10n-support
$ posummit scripts/messages.summit LANG merge stable:
</syntaxhighlight>

=== Merging with Compendium ===

By default, <tt>msgmerge</tt> takes into account only the catalog itself to fill out near-match translations to new messages introduced by merging. i.e. to produce fuzzy messages. However, an arbitrary PO file can be given to <tt>msgmerge</tt> as another possible source of earlier translations, through the <tt>--compendium</tt> option. For maximum effect, this PO file is usually constructed as the collection of messages from all PO files project-wide, and hence called the ''compendium''.

Compendium can be used in summit merge operations simply by setting <tt>compendium_on_merge</tt> attribute in summit customization file. If the compendium is located at {{path|$KDEREPO/trunk/l10n-support/LANG/summit/messages-compendium.po}}, then:

<syntaxhighlight lang="python">
# Compendium to use when merging summit catalogs.
S.compendium_on_merge = S.resolve_path_rooted("messages-compendium.po")
</syntaxhighlight>

Note that if not located as above, compendium should not be placed within {{path|.../LANG/summit/messages/}}, as then it would be considered a summit catalog itself. In case fully local merging is engaged, only summit POs will be merged with compendium; it makes no sense for branch POs, since they are not being directly translated.

See [http://www.gnu.org/software/automake/manual/gettext/Creating-Compendia.html "Creating Compendia"] section of Gettext manual for details on how to create a compendium out of current body of summit translations. Once you establish the precise commands to create the compendium (whether to collect fuzzies too, whether to include old compendium as one of sources for the new, etc.), you would periodically refresh and commit the updated compendium.

=== Vivifying Summit Catalogs ===

Translation of a summit catalog normally starts the same way as it did for branch catalogs, by copying it over from template and initializing the header. This can be done manually, but it can also be quite automatic when using project manager as sometimes provided by dedicated PO editors. However, relying on the editor's project manager can be disadvantageous at times. For example, aside from the PO editor, other, frequently command line tools may be used to process the body of translation, and these tools might need to consider non-started templates as if they were empty POs (e.g. statistics). Team members who do not have full local checkout, or no project set up in the editor, may need to jump between language and template directories when looking for files to translate.

Therefore, summit setup can be customized to automatically create ("vivify") summit catalogs for every new summit template, so that there is never the need (for a tool or human) to specifically treat templates as empty catalogs. This is done by adding the following lines into customization file:

<syntaxhighlight lang="python">
# Create empty summit catalog for every new summit template.
S.vivify_on_merge = True
S.vivify_w_translator = "Noone Noonian <noone.noonian@nowhere.null>"
S.vivify_w_langteam = "Neverneese <neverteam@nowhere.null>"
S.vivify_w_plurals = "nplurals=2; plural=n != 1;"
# Minimum translation state to create branch catalog on scatter.
S.scatter_min_completeness = 0.9
</syntaxhighlight>

The <tt>vivify_w_*</tt> attributes set the necessary data to initialize headers of newly created summit catalogs. Aside from those listed above, there is also the <tt>vivify_w_charset</tt> attribute, which is by default <tt>"UTF-8"</tt>, and very probably should not be changed.

The <tt>scatter_min_completeness</tt> attribute does not refer to vivification of summit catalogs, but should invariably be set in this context. When scattering from summit, normally the branch catalog is automatically created if there is the corresponding summit catalog. When vivification is engaged, this would result in creation of empty branch catalogs too, which is not desired for two reasons. Firstly, empty branch catalogs would become part of the released language pack, for which there is no practical reason. Secondly, KDE's i18n system regards an application with installed catalog as translated, so messages coming from basic system catalogs (e.g. <tt>kdelibs4</tt>) would show through, resulting in mostly untranslated user interface with specks of translation -- many users consider this rather ungainly. Therefore, <tt>scatter_min_completeness</tt> sets how complete the translation of currently non-existing branch catalog should be after scatter (0.0 empty, 1.0 fully complete), for the branch catalog to actually be created. If the <tt>--force</tt> option is given to <tt>posummit</tt>, the branch catalog will be created regardless of its translation state, which may be handy in combination with [[#Operation Targets|operation target]] when wanting to check work in progress in running application.

If [[#Merging with Compendium|the compendium has been set]] for merging, every vivified summit catalog will also be merged against the compendium, to fill out as many of messages with approximate translations.

=== Scatter Hooks ===

((TODO. Checks, modifications...))

=== Merge Hooks ===

((TODO. Special header fields...))

== Disadvantages to Summit and Remedies ==

Although hopefully shadowed by the advantages, working in summit is not without its disadvantages. These should be weighed when deciding of whether to try out the summit workflow.

Obviously, while summit operations are made to be quite automatic, some extra aptitude is asked of the team coordinator. Reasonable shell handling, understanding of version control operations, feeling the pulse of repository automation, are all prerequisites, and some scripting ability advantageous.

After the summit is put in operation, any changes made manually in branch POs will not propagate to summit, and will be soon lost to scattering -- summit translations override everything in branches. This means that the whole team must work in the summit, it is not possible for some members to use the summit, and some not.

A summit PO file will necessarily have more messages than either of the branch files. For example, in the KDE 4.0/4.1 and 4.1/4.2 cycle, summit POs of core KDE modules had on average less than 5% more words than their stable counterparts. However, the said percent is the top, never approached limit of wasted workload due to trunk messages coming and going, given that as the next feature KDE release approaches, more and more trunk messages will find their way into it.

Another, more pressing issue with increased size of summit POs is the following scenario: a stable release is around the corner, and the team has no time to update summit POs fully, but could update only stable messages in them. E.g. there are 1000 untranslated and fuzzy messages, out of which only 100 are from the stable branch. A clever dedicated PO editor could allow jumping only through untranslated and fuzzy messages which also satisfy a general search criteria, which in this case would be that a comment matches <tt>#\.\+>.*stable</tt> regular expression. On the other hand, with some external help, it is enough if the PO editor can merely search through comments. Then, the <tt>posieve</tt> command (ready to use next to <tt>posummit</tt>) can equip untranslated and fuzzy stable messages with <tt>untranslated</tt> flag (producing <tt>#, ..., untranslated</tt> comment), and this flag can be searched for in the PO editor:

<syntaxhighlight lang="bash">
$ posieve tag-untranslated -sbranch:stable -swfuzzy PATHS_TO_PO_FILES_OR_DIRS
</syntaxhighlight>

The <tt>untranslated</tt> flag does not need to be manually removed when the message is updated. It will automatically disappear on the next merge, as it is not among flags known to Gettext.

If an automatic source lookup mechanism (to check the message context by following its source references) was available for branches, it may not entirely work for summit. Already the branch POs in KDE are kept separate from the code, which means that paths given by source references cannot be followed directly, but appropriate root paths have to be added in some way (a dedicated PO editor may provide a way to do this); this should continue to work with summit for those messages which exist both in trunk and stable branch, since their source references correspond to trunk code. But for messages found only in stable branch, whose source references correspond to stable code, existing source lookup mechanism will likely assume trunk roots again, and try to fetch wrong sources. To fix this, the source lookup mechanism should be flexible enough to select roots based on branch keywords given in summit messages' <tt>#. +> ...</tt> comment.

There is also the organizational issue with starting to use the summit, and, if it does not help as expected, stopping to use it. Team members have to be reminded to not send in branch POs at start, and then to be sent back to branch POs if summit is disbanded. On the plus side, disbanding summit is technically simple: just remove from the repository {{path|l10n-supprot/LANG/summit}}, possibly also <tt>no-auto-merge</tt> files if [[#Fully Local Merging|fully local merging]] was set up, and that is it.

== Another Way to Improve Branch Handling ==

If summit seems a lot to digest, or is simply an overkill for team's needs, but still some improvement to manual handling of branches would be welcomed, KDE's dedicated PO editor [[Localization/Tools/Lokalize|Lokalize]] offers a ''branch sync'' mode. It works as follows.

In Lokalize project definition, the local paths of trunk and stable PO roots are set in ''Translation directory:'' and ''Branch directory:'' fields. Then, when a trunk PO file is opened, if it has a stable counterpart with same name and location as in the trunk, this stable PO is also going to be opened. For each trunk message in the main editing pane, if such a message exists in stable PO too, the stable message will be shown in the ''Secondary Sync'' pane; changes in the translation of trunk message will reflect to the stable message, and stable PO file will also be saved when the trunk is saved.

Furthermore, any team member can personally choose to work like this, there is no need to change the workflow of the language team as whole. When sending modifications to the coordinator, team members who rely on this feature of Lokalize simply send both trunk and stable POs that got modified.

Localization

2011-11-26T12:35:57Z

Ilic: Modified description of Pology.

{{Template:I18n/Language Navigation Bar|Localization}}

''Welcome, adventurer, to the universe of KDE localization! Brought along a generous reserve of breathing air?''

{{warning|This assembly of articles is intended to become the central source on localization of KDE, but is far from being there yet. In the meantime, the authoritative body of documentation (albeit in parts outdated) is the [http://l10n.kde.org/docs/translation-howto KDE Translation Howto].}}

== The Milky Way ==

This page is the hub of articles on all things concerning localization of KDE. The topics covered are intended to detail facts, issue advice and satisfy curiosity about KDE localization ways. And to do so for "all audiences": from newly interested in free software localization, over those more experienced seeking to get involved into KDE localization, to veteran KDE translators curious about new developments.

To achieve this aim, the topics are by necessity not structured linearly. When you read a textbook, in the introduction you can frequently find authors' instructions on how to "follow" the book: possible coherent chains of chapters, which chapter is a prerequisite for another, and which optional. So is with the topics presented here, taken to the extreme, with many entry and exit points depending on individual interests.

The hub currently connects three spiralling arms of topics:

; [[#Concepts|Concepts]] : Expositions of widespread concepts in free software localization in general and KDE localization in particular. General mechanics and formats, advanced technical possibilities, organization and communication processes.

; [[#Tools|Tools]] : Descriptions of tools which may benefit the localization process, from command line scripts to full-featured GUI applications. Their possible roles in support of localization concepts.

; [[#Workflows|Workflows]] : Instructions and procedures on how to contribute to KDE localization, on various levels of engagement. What are the need-to-know concepts in different scenarios, and which tools are appropriate to carry them through.

{{tip|If you are fresh in the trade, do not feel intimidated by the expanse. Not even crack KDE translators are supposed to be intimately acquainted with, or need all this stuff. Instead, if you are eager to start churning out results, head to the [[Localization/Workflows/Rookie|rookie workflow]] and follow the leads therein.}}

=== Note to Editors ===

It is important to correctly place certain bits of information in the localization universe. The reader should be made aware of what is a local KDE convention, what a special feature of the tool they use, and what a general concept and its embodiment in specific KDE context.

While this is obviously a KDE-focused resource, it is nevertheless useful to provide examples of how some elements are handled outside of KDE. Through contrast and comparison, the reader may better understand the whys and hows of presented material. Likewise, it may help those already familiar with other localization environments.

== Concepts ==

For better or for worse, there is no lack of frameworks, formats, procedures and other vague notions that a KDE translator may stumble upon along the way. They may be a burden of sorts -- many things to keep in mind -- but also a source of fun, challenge, and deep satisfaction when creatively combined towards great efficiency of everyday work and mirror-perfect polish of the final output. Thus, expect the articles here not to withold much.

; [[Localization/Concepts/Encodings|Text Encoding]]
: ''Text is the most basic object of localization. However, to handle it at low level -- to encode text -- such that languages of the world are smoothly supported, was historically not trivial. Read about the current standards, proper setups and errors due to text encoding which may pop up.''


; [[Localization/Concepts/PO_Odyssey|The PO Format]]
: ''The PO format is the mainstay of free software translation. Ragardless of the actual workflows and tools used, translators should maintain a good measure of familiarity with the underlying PO format. This article thoroughly describes the elements of the PO format and various uses of PO catalog files which embody it.''


; [[Localization/Concepts/XML_Markup|XML Markup]]
: ''Parts of text are sometimes presented to the user in special way: bold or italic, title sized, etc. XML-like text markup is a popular way of specifying such presentation, and translators will frequently find it embedded in the source texts. This article deals with XML markup from translators' viewpoint.''


; [[Localization/Concepts/Version_Control|Version Control]]
: ''KDE evolves by integrating a lot of work contributed by a lot of people scattered around the planet, and that along parallel lines of development. To prevent information collapse into the gravity well of unhindered creativity, programmers employ version control systems -- and so do the translators.''


; [[Localization/Concepts/Communication|Rings of Communication]]
: ''Translators have many communication outlets at their disposal. Beyond the circle of one's own language team, translators from other teams and programmers are orbiting close at reach. Find out which issues are best dealt with at which level, for the time spent in constructive discussion to achieve most positive impact.''


; [[Localization/Concepts/Auto_Assists|Automatic Translation Assists]]
: ''Automatic assists help translators to speed up the work, avoid common errors, and achieve consistency in style and terminology. They may be provided as standalone tools, or as features of specialized translation tools. These assists include spell checkers, glossaries, translation memories, etc.''


; [[Localization/Concepts/Statistics|Collecting and Using Statistics]]
: ''Statistics provide overall indicators of the past localization progress, and means to extrapolate for the future. However, as always with statistics, for meaningful conclusions they should be used with care. Read about the sources of statistics, things to count, and effort estimates.''


; [[Localization/Concepts/Doc_Translation|Translating Documentation]]
: ''Compared to localizing application interfaces, translating their documentation both throws new elements into the fray, and casts others in a different light. The need to keep the interfaces and documentation in sync, demands another layer of attention. This article covers those peculiarities.''


; [[Localization/Concepts/Non_Latin_Alpha_LTR|Writing Systems Distinctions]]
: ''User interfaces have been historically centered and developed around alphabetic writing system, using Latin alphabet, and written from left to right. Drastic changes to any of these assumptions, such as right to left writing or using ideographs instead of alphabet, requires some consideration.''


; [[Localization/Concepts/Locales|Language/Country Specific Formats]]
: ''Applications frequently process and display pieces of information which, while universal in concept, are presented differently accros cultures: time and dates, numbers, calendars, and so on. KDE apps gracefully handle such differences, relying on the language/country-based specifications in the KDE core.''


; [[Localization/Concepts/KDE_Special_Messages|Special Entries in Translation Catalogs]]
: ''Most messages in translation catalogs are ordinary text intended for the user, but some are not. Programmers may use messages which let translators choose behavior for their language, add language-specific data, or state translation credits. Such special entries typically found in KDE catalogs are described here.''


; [[Localization/Concepts/Transcript|Translation Scripting with Transcript]]
: ''Traditional UI translation frameworks are based on English as the pivotal language. This frequently leads to technical problems in target languages, where translators may be forced to choose between bad and worse. To alleviate such issues, KDE provides a way to act on translation at runtime -- the Transcript engine.''

; [[Localization/Concepts/Non_Text_Resources|Localizing Non-Text Resources]]
: ''While the first thought of localization is that of text translation, text is not the sole resource for localization. Any content presented to the user -- an icon, image, sound -- may require localization in certain cultural contexts. Learn how to localize non-text resources in KDE as the need arises.''


== Tools ==

Great many tools exist to support the localization process. Some may be quite general, and other tightly coupled with KDE localization process. Tools are thus presented in different ways. More general tools typically have referent documentation of their own, and here it is explained how they relate to concepts and workflows used in KDE. Custom, KDE-specific tools are explained in greater detail, sometimes to the point of these articles being their referent documentation.

; [[Localization/Tools/Gettext_Tools|Gettext Tools]]
: ''GNU Gettext is the de-facto referent implementation of the PO format. It is used to extract templates from the code and update PO catalogs, and has many tools for processing PO files. Its compiler of PO format into binary format for application use, is the final arbiter of validity of a PO file.''

; [http://userbase.kde.org/Lokalize Lokalize]
: ''[http://userbase.kde.org/Lokalize Lokalize] is a computer-aided translation (CAT) tool, a full-featured GUI application for translators, written from scratch using KDE4 framework. Aside from basic editing of PO files with nifty auxiliary details, it integrates support for glossary, translation memory, diff-modes for QA, project managing, etc.''

; [[Localization/Tools/Emacs_and_Vim|Emacs & Vim]]
: ''Emacs and Vim are ubiquitous Unix text editors, in continuous use and development from times immemorial. Both very different from today's typical editors, as well as between each other, powerfull and extendable, they have been pressed into many roles. One is power-assisted editing of PO files.''

; [[Localization/Tools/Translate_Toolkit|Translate Toolkit]]
: ''Translate Toolkit is a host of command-line utilities, written mostly in Python, that expands and extends on Gettext's tools. They provide advanced search, selection and merging of PO files, and environment-specific validity checks. Also included are converters between various non-PO formats.''

; [[Localization/Tools/Subversion|Subversion]]
: ''Subversion, SVN for short, is the version control system currently used by the KDE project as whole. Same as the code, KDE localization data are stored in the central SVN repository. This article describes the use cases of Subversion for translators, as well as repository organization of localized data.''

; [[Localization/Tools/Pology|Pology]]
: ''Pology is a Python library and a set of command-line tools for processing PO files. The tools perform various specialized operations on PO files, some of which have explicit support for PO files within the KDE Translation Project. The library is geared towards quick and robust assembly of "field" scripts for processing PO files, in a version-controlled environment.''

; [[Localization/Tools/Lbundle_Check|Lbundle Checker]]
: ''For text resources translated through PO files there are well-established means of tracking changes as the underling code evolves. This is the script to provide a degree of such support for localized non-text resources, when organized as localization bundles.''

; [[Localization/Tools/Misc_Scripts|Miscellaneous Scripts]]
: ''KDE repository contains many standalone scripts to check and process localized data, of various degrees of specificity -- many even tied to the exact repository organization. Collected here are the descriptions of such scripts which may be generally useful to translators.''

== Workflows ==

There is no single way to participate in KDE localization. Contributors will differ by the amount and direction of effort they put in, and the workflow articles are here to provide guidance for the frequently observed roles. Also presented are the technical and organizational details which have tidal influence on everyday translator's workflow.

; [[Localization/Workflows/Rookie|Rookie Translator]]
: ''This is your first forray into localization and you're looking for the sign saying "Don't Panic", in large friendly letters? Find about the essential prerequisites to start translating early, but productively.''


; [[Localization/Workflows/Translator|Seasoned Translator]]
: ''Translators that have surmounted the difficulties and acquired the concepts and tools for the work to become daily routine, may start looking into new directions. How to better coordinate effort, cooperate with other language teams, test the localization quality in live environment, etc.''


; [[Localization/Workflows/Coordinator|Language Coordinator]]
: ''Each language team in KDE needs one or few persons tasked additionally to their basic work on localization. They take care that the team effort progresses smoothly, and voice their teams in global matters. Language coordinators have write-access to KDE repository, and the responsibility to boot.''


; [[Localization/Workflows/Overlord|Global Coordinator]]
: ''While the cooperation can and does take place between translators from different language teams, more focused attention is needed to handle some global issues. To that end, some translators engage in maintenance tasks on wider scale, and one of them is appointed the KDE Localization Coordinator.''


; [[Localization/Workflows/New_Language|Introducing a New Language]]
: ''Great many languages are already being localized into within KDE project. Sometimes, however, a new language is to be introduced, which requires coordination between its translators and core KDE team. Also, to ship a language as part of official KDE release, some essentials must be satisfied.''


; [[Localization/Workflows/Repository_Automation|Repository Automation]]
: ''Daily in the KDE repository, the lumbering machinery scrutinizes code, updates translations to reflect changes, performs checks, serving the results to translators. Learn to follow its hum, and know where to grease when the gears clog.''


; [[Localization/Workflows/PO_Summit|Translating in Summit]]
: ''Handling two branches for translation, stable and trunk, can be tedious. Porting fixes from one to another branch, making sure team members work on correct branch, etc. Especially so when non-core modules are considered, like extragear. Read about one possibility to curb this overhead.''

; [[Localization/Workflows/PO_Ascription|Reviewing by Ascriptions]]
: ''It is essential for quality that translations are reviewed: for context, terminology, grammar, style, etc. Given text is normally reviewed by persons other than its translator, which prompts the question of how to coordinate and track reviewing. This article presents the ascription system of reviews, building on the summit workflow.''

[[Category:Localization]]

Localization/Tools/Pology

2011-11-26T12:31:26Z

Ilic: Links to other Pology-related articles.

{{Template:I18n/Language Navigation Bar|Localization/Tools/Pology}}

{{LocalizationBrowser|
section=[[Localization#Tools|Tools]]|
name=Pology|
prereqs=[[Localization/Tools/Gettext_Tools|Gettext Tools]]|
related=[[Localization/Tools/Pology/PO_Embedded_Diffing|PO Diffing]], [[Localization/Workflows/PO_Summit|Summiting Translation Branches]], [[Localization/Workflows/PO_Ascription|Review by Ascription]]|
}}

== About ==

Pology is a Python library and a set of command-line tools for processing PO files. The library aims to enable easy, fast and robust creation of scripts for tackling problems encountered in the "field", in everyday translation work. The tools perform various specialized operations, beyond that of other PO-handling software. Some of the tools are designed to (or have provisions to) treat a collection of PO files as an entity unto itself.

In Pology, no attempt is made to handle any other translation file format but PO. All of the end-user tools and library programming interfaces are geared towards the technical aspects, conventions and workflows around the PO format. For example, some tools explicitly take into account that PO files are frequently kept under version control, providing the functionality to support that workflow.

Language- and project-specific support is present throughout Pology, and it is designed to be easily expanded in the future. In the context of KDE, for example, there is a tool for validating PO files within the KDE Translation Project: it will recognize the "role" of the particular PO file (e.g. a native KDE code PO file, a .desktop PO file, Docbook PO file...) and then apply checks appropriate for that role. (This tool is run weekly on KDE servers, on PO files of all languages, and results announced to <tt>kde-i18n-doc</tt> mailing list.)

Pology source distribution can be fetched from its home page at:

http://pology.nedohodnik.net

The home page also links to on-line documentation (user manual and library API), and provides instructions for browsing or fetching the current development code.

Localization/Tools/Pology

2011-11-26T12:24:23Z

Ilic: The content of this page is no longer actual. Instead, brief overview is given and links to Pology home page.

Localization/Concepts/Transcript

2011-08-18T07:56:25Z

Ilic: New method: warnputs().

{{LocalizationBrowser|
section=[[Localization#Concepts|Concepts]]|
name=Translation Scripting with Transcript|
prereqs=[[Localization/Concepts/PO_Odyssey|The PO Format]]|
extread=[http://developer.mozilla.org/en/docs/JavaScript JavaScript Guide and Reference]|
}}

== Translation Scripting? ==

The current state of affairs in the localization of user-visible strings (messages) in application interfaces is such that the translator is sometimes forced to supply an inadequate translation. This problem typically occurs when a message contains a placeholder to be substituted at runtime, or when two unrelated strings become related by placement in the interface. In either case, the modest requirements of the English language on the congruence of words in a sentence allow the original string to remain grammatically correct, while in many other languages this is not the case.

Translators and programmers sometimes try to work out a change in the code which would provide a more workable alternative. However, this process is difficult, and worse yet, the outcome is still language-dependent: solving the problem for a few languages does not necessarily solve it for all of them.

One way to overcome these problems in a more general and compartmentalized manner is to provide translators with a way to modify translated strings at runtime, depending on the context (eg. particular placeholder substitutes). In other words, to script translation. The translator should be able to operate on any interface string he wishes, while the programmer should not bear any extra burden (or know about translation scripting at all).

== The Transcript Engine ==

KDE4 comes ready with a translation scripting system, called ''Transcript''. Several strategic choices were made in its design:

* programmers are unaware of scripting, which means that translator can script any message without outside coordination

* unless the translator wants to script a message, he is faced with the familiar standard Gettext PO environment (i.e. translators, too, can ignore scripting)

* scripting is a low-level bolt-on to Gettext environment, to keep existing PO tools in the game

* in order to be powerful enough for unforeseen needs, a general-purpose scripting language is provided: JavaScript (with extensions for interfacing with Transcript)

{{note|For comparison, another translation scripting approach taking some different decisions (new translation environment down to the file format, scripting facilities more specialized, etc.) is brewing at http://wiki.mozilla.org/L20n}}

To script a particular message, the translator writes short scripting calls into msgstr in the PO file which expand into parts of msgstr (''interpolations''), and JavaScript code which defines these calls into an accompanying Transcript module file (eg. {{path|foo.po}} can be augmented with {{path|foo.js}}).

In case where the application draws translations from several PO files, scripting calls defined in one of the Transcript modules are available in all used PO files. Since every KDE app uses {{path|kdelibs.po}}, calls defined in {{path|kdelibs.js}} are available everywhere.

The scripting process is illustrated by several examples. More detailed explanations of the elements are given in following sections.

=== A Useless Example ===

In Nevernessian it is impolite to speak a greeting with the same tone of voice throughout; instead, the name of the person must be shouted out. Hence, the translator wants to capitalize the placeholder substitute in the following login greeting in {{path|neverness.po}}:

<syntaxhighlight lang="text">
#: neverness_login.cpp:10
msgid "Hello, %1!"
msgstr "Heelyy, %1!"</syntaxhighlight>

So the translator adds a scripted msgstr, with an interpolation:

<syntaxhighlight lang="text">
#: neverness_login.cpp:10
msgid "Hello, %1!"
msgstr "Heelyy, %1!"
"|/|"
"Heelyy, $[shout %1]!"</syntaxhighlight>

The first thing to note is that, while a bit longer, msgstr is still a proper PO msgstr, which means that it can be edited and processed by the usual PO tools.

The first part of msgstr is same as before, and called the ''fallback'' in this context: if the scripted part happens to fail in some way, the fallback translation is used. The fallback is followed by the ''fence'' <tt>|/|</tt>, which separates the fallback and scripted translation (and also indicates that this message is scripted).

Finally, there is the scripted translation after the fence. It is different from the fallback in that it contains the interpolation <tt>$[shout %1]</tt>, which is supposed to evaluate to a capitalized version of the placeholder substitute. It is composed of the call name, <tt>shout</tt>, and one argument to it, the <tt>%1</tt> placeholder which will be replaced by its substitute. The syntax and expansion rules for interpolations are similar to Unix shell.

The call <tt>shout</tt> itself is defined in the Transcript module {{path|neverness.js}}, which contains only these lines:

<syntaxhighlight lang="javascript">
function capitalize (str) {
return str.toUpperCase();
}

Ts.setcall("shout", capitalize);</syntaxhighlight>

Here the function <tt>capitalize</tt> is an ordinary JavaScript function which takes a string argument and returns all-caps version of it.

The link with the PO file is established by the call to <tt>Ts.setcall()</tt> -- the Transcript interface is represented by the property functions of the <tt>Ts</tt> object. In this variant, the <tt>Ts.setcall()</tt> takes the name of the call for the interpolations in the PO messages (a string), and the JavaScript function which will actually be invoked (''bound'' to the call).

That's it: now the fair Nevernesse folks are greeted properly.

=== Basic Case Resolution ===

One problem frequently encountered is use of the wrong noun case when the placeholder is substituted in the msgstr. For example, in many languages every KDE app has such a problem in the Help menu, with one or both of "About %1..." and "%1 &Handbook". This can be scripted in {{path|kdelibs.po}} like this:

<syntaxhighlight lang="text">
msgid "&About %1"
msgstr "&O %1"
"|/|"
"&O $[get-case dative %1]"</syntaxhighlight>

The <tt>get-case</tt> interpolation is supposed to get the <tt>dative</tt> case of whatever app name the <tt>%1</tt> happens to be. The Transcript module {{path|kdelibs.js}} contains the definition of <tt>get-case</tt>, as well as the dictionary of cases:

<syntaxhighlight lang="javascript">
function getProperty (prop, key) {
return _dict_[key][prop];
}
Ts.setcall("get-case", getProperty);

_dict_ = {};
function addDictCases (key, gen, dat, acc, ins) {
if (!_dict_[key])
_dict_[key] = {};
_dict_[key]["genitive"] = gen;
_dict_[key]["dative"] = dat;
_dict_[key]["accusative"] = acc;
_dict_[key]["instrumental"] = ins;
}

// dictionary entries follow:
addDictCases("KWrite", "KWritea", "KWriteu", "KWrite", "KWriteom");
addDictCases("Konsole", "Konsole", "Konsoli", "Konsolu", "Konsolom");
...</syntaxhighlight>

Function <tt>getProperty</tt>, bound to <tt>get-case</tt> call, simply returns the entry from the dictionary of forms. Function <tt>addDictCases</tt> is responsible for adding the static entries (name and its cases) into the dictionary, which is done in the final few lines for all apps of interest.

This completes the example, but for better modularization, it is also possible split out the dictionary insertion in a separate file, eg. {{path|appdict.js}}:

<syntaxhighlight lang="javascript">
// appdict.js
addDictCases("KWrite", "KWritea", "KWriteu", "KWrite", "KWriteom");
addDictCases("Konsole", "Konsole", "Konsoli", "Konsolu", "Konsolom");
...</syntaxhighlight>

and use Transcript interface to load this file in the {{path|kdelibs.js}}:

<syntaxhighlight lang="javascript">
// kdelibs.js
...
...
...
Ts.load("appdict");</syntaxhighlight>

Note that <tt>Ts.load()</tt> takes filename without extension, and assumes its location is relative to the folder of the parent file (ie. in this case {{path|kdelibs.js}} and {{path|appdict.js}} should be in the same folder).

=== Dynamic Case Setting ===

The previous scripted example solves the original problem, but introduces the burden of maintaining the dictionary insertion file. There is no way around this when the placeholder substitutes are "dead" strings from outside (eg. from {{path|.desktop}} files), but when they are coming from KDE's PO files at runtime, this burden can be removed.

The app name in KDE's Help menu indeed comes from the app PO file, and it is of course encountered at runtime before the menu strings come into focus. This allows setting the cases of app name in the PO msgstr which contains it. For example, {{path|katepart.po}} contains the "KWrite" string, and the forms could be set at that point:

<syntaxhighlight lang="text">
msgid "KWrite"
msgstr "KWrite"
"|/|"
"$[set-cases KWritea KWriteu KWrite KWriteom]"</syntaxhighlight>

The <tt>set-cases</tt> is a ''side-effect'' interpolation: it should set the dictionary entries, but this particular message should in any case use the ordinary translation. Assuming all the definitions from previous example are still in effect, here is how <tt>set-cases</tt> could be defined in {{path|kdelibs.js}}:

<syntaxhighlight lang="javascript">
function dynamicSetCases (gen, dat, acc, ins) {
addDictCases(Ts.msgstrf(), gen, dat, acc, ins);
Ts.fallback();
}
Ts.setcall("set-cases", dynamicSetCases);</syntaxhighlight>

In other words, this is little more than a wrapper to "static" <tt>addDictCases</tt> from previous example, but two new elements of Transcript interface appear. First is the <tt>Ts.msgstrf()</tt> function, which returns the finalized ordinary translation (placeholders substituted), and which is needed in this case as the dictionary key. Second is the <tt>Ts.fallback()</tt> function, which signalizes the Transcript engine to disregard the result of the scripted part of msgstr and use the ordinary translation.

Admittedly, the use of <tt>Ts.fallback()</tt> in this case is not necessary, but given for introductory purpose; <tt>dynamicSetCases</tt> might as well ''return'' the ordinary translation via <tt>Ts.msgstrf()</tt>.

== The PO Shell ==

This section gives the details of how the interpolations in the PO msgstr are expanded before evaluation.

The interpolations are parts of msgstr between <tt>$[...]</tt>, and are parsed into a number of strings. The first string is the name of the call registered in the scripting module via <tt>Ts.setcall()</tt>, and the rest are the arguments to bound JavaScript function. The arguments are typically passed as JavaScript type <tt>String</tt>, except in a special case detailed below.

The special characters in the interpolation are whitespace, single quote (') and backslash (\). Whitespace separates arguments, whereas single quote can be used for text which contains whitespaces. The backslash is used as escape; it can escape whitespace in non-quoted text, or single quotes in quoted text. This is pretty much like a typical Unix shell.

Double quotes are not special. Single quotes are used instead of double quotes because it makes it easier to edit interpolations in PO files, where double quotes would have to be escaped. This also means that when escape is needed in the interpolation, it must be escaped once itself for the PO msgstr.

The biggest difference from the shell expansion is that unlike with shell variables, the placeholders are expanded such that no characters within them are treated as special. Otherwise, many tricky problems could arise with whitespace or single quotes contained inside.

The call name bound to a JavaScript function using <tt>Ts.setcall()</tt> does not have to be a proper JavaScript identifier, but any Unicode string not containing the interpolation-special characters. This means that more "natural" call names can be used inside the msgstr, like those with dashes or non-Latin1 characters. E.g. call names can be in the language of the PO file, for an aesthetic impression.

Sub-interpolations, <tt>$[... $[...] ...]</tt>, are also possible. If put inside inside single quotes, they will be treated as ordinary text. Thus, if a literal closing square bracket is needed as an argument inside the interpolation, it can be given within single quotes.

In case that the argument is given as <tt>^''number''</tt>, it will evaluate to the value supplied for the corresponding placeholder -- unlike the <tt>%''number''</tt> form, which evaluates to the placeholder substitution string. Also, the argument type as seen by JavaScript need no longer be <tt>String</tt>, but will correspond to the value substituted (e.g. <tt>Number</tt>). This distinction may be important in some situations. Even if the value is originally a string, the substitution string may have extra formatting (padding, tags...), which may not be desirable. In particular, the number substitutes may come tagged or locale-formatted, such that they cannot be easily parsed back into <tt>Number</tt>. In these cases, <tt>^''number''</tt> form can be used to get the raw values.

== The Transcript Interface ==

Transcript provides extensions to the JavaScript, which interface with the PO file and the Transcript environment. They are all function properties of the <tt>Ts</tt> object, accessible as <tt>Ts.''func''(''args'')</tt>.

; <tt>setcall (''name'', ''func'')</tt>
; <tt>setcall (''name'', ''func'', ''obj'')</tt>
: Binds the call name to the JavaScript function, for use in the interpolations inside the PO file.
:: <tt>''name''</tt> name of the call. Can be any Unicode string, for ease of use in the msgstr
:: <tt>''func''</tt> function object
:: <tt>''obj''</tt> object to act as <tt>this</tt> inside the function; if omitted, <tt>this</tt> refers to global object
: Returns <tt>Undefined</tt>.

; <tt>hascall (''name'')</tt>
: Check if the call for use in PO file has been set.
:: <tt>''name''</tt> name of the call
: Returns <tt>true</tt> if set, <tt>false</tt> otherwise.

; <tt>acall (''name'', ''arg''*)</tt>
: Applies previously set PO call to a list of arguments, within the scripting module itself. Indirect calls can be made this way, e.g. by passing a call name as a string within PO file interpolation.
:: <tt>''name''</tt> name of the call
:: <tt>''arg''*</tt> arguments to the call
: Returns what the indirect call would return.

; <tt>load (''file''*)</tt>
: Evaluates the code in the specified files, in the left to right order. File paths are expected to be relative to current module's folder.
:: <tt>''file''</tt> file name without extension
: Returns <tt>Undefined</tt>.

; <tt>fallback ()</tt>
: Forces Transcript to use ordinary translation, regardless of whether the interpolation evaluates successfully or fails. The evaluation of the script is not aborted, any other interpolations will evaluate too.
:  
: Returns <tt>Undefined</tt>.

; <tt>msgid ()</tt>
: Returns msgid of the last message, with placeholders intact.

; <tt>msgstrf ()</tt>
: Returns finalized ordinary translation of the last message, with placeholders substituted.

; <tt>msgctxt ()</tt>
: Returns msgctxt of the last message, with placeholders intact.

; <tt>msgkey ()</tt>
: Returns a <tt>String</tt> which is implementation-dependent combination of msgctxt and msgid with placeholders intact. Used to uniquely identify the message within the PO file, usefull as a dictionary key.

; <tt>nsubs ()</tt>
: Returns the number of substitutes provided for placeholders in the last message. It is equal to the highest-numbered placeholder for a proper i18n call in the application code, but i18n calls do not have to be quite proper.

; <tt>subs (''index'')</tt>
: Used to access values of placeholder substitutes provided to the last message. Numbering is zero-based.
:: <tt>''index''</tt> index of placeholder substitute
: Returns <tt>String</tt>, regardless of the value type substituted in the application code.

; <tt>vals (''index'')</tt>
: Used to access values of placeholder values provided to the last message. Numbering is zero-based.
:: <tt>''index''</tt> index of placeholder value
: Returns the matching JavaScript type to the value type used in the application code. E.g. strings will be <tt>String</tt>, integers and doubles <tt>Number</tt>. May also return <tt>Undefined</tt>, in case the value cannot be represented as a reasonable JavaScript type.

; <tt>dynctxt (''key'')</tt>
: Programmers may append ''dynamic'' context to the message, which is a pair of strings: a context key and its value. Use this function to retrieve such contexts.
:: <tt>''key''</tt> context key string
: Returns <tt>String</tt>, or <tt>Undefined</tt> if dynamic context with the given key does not exist.

; <tt>dbgputs (''msg'')</tt>
: Outputs a debug message in the shell. The message is seen only if KDE libraries have been compiled in debug mode.
:: <tt>''msg''</tt> message string
: Returns <tt>Undefined</tt>.

; <tt>warnputs (''msg'')</tt>
: Outputs a warning message in the shell.
:: <tt>''msg''</tt> message string
: Returns <tt>Undefined</tt>.
: ''Since KDE 4.7.1.''

; <tt>setcallForall (''name'', ''func'', ''obj'')</tt>
; <tt>setcallForall (''name'', ''func'')</tt>
: Similar to <tt>setcall()</tt>, but the function is executed on every message after it has been finalized (if the message was explicitly scripted, the function is executed after the script has been evaluated). The function receives no arguments, and its return value is ignored, i.e. it cannot change the finalized message; the call is used for side-effects. The <tt>''name''</tt> parameter is used only for reporting errors. When several <tt>setcallForall()</tt> have been issued, the functions are invoked in the order of issue.
:  
: More precisely, these calls are made on every message ''after'' the Transcript engine has been initialized, which happens on first explicitly scripted message. Thus, if earlier application of the calls is necessary, Transcript can be jump-started by scripting one early encountered message with only a single empty interpolation: <tt>msgstr "...|/|$[]"</tt>; this will do nothing for that message, but it will start the engine.
:  
: Returns <tt>Undefined</tt>.

; <tt>toUpperFirst (''text'')</tt>
: Converts the first letter in the string into upper case, even if the first character in the string is not a letter. Unicode specification is used to determine what a letter is (i.e. works for non-ASCII letters).
:: <tt>''text''</tt> string to process
: Returns <tt>String</tt>.

; <tt>toLowerFirst (''text'')</tt>
: Converts the first letter in the string into lower case, under same assumptions as for <tt>toUpperFirst</tt>.
:: <tt>''text''</tt> string to process
: Returns <tt>String</tt>.

; <tt>loadProps (''file''*)</tt>
: Loads [[#Property Maps|property maps]] from the specified files. File paths are expected to be relative to current module's folder.
:: <tt>''file''</tt> file name without extension
: Returns <tt>Undefined</tt>.

; <tt>getProp (''phrase'', ''prop'')</tt>
: Fetches the property value of the phrase, as defined by [[#Property Maps|property map]].
:: <tt>''phrase''</tt> text for which the property is requested
:: <tt>''prop''</tt> property key string
: Returns <tt>String</tt>, or <tt>Undefined</tt> if the given text has no such property.

; <tt>setProp (''phrase'', ''prop'', ''value'')</tt>
: Sets the property value of the phrase. The phrase and property key will be automatically normalized, as they would have been when loaded from a [[#Property Maps|property map]] file.
:: <tt>''phrase''</tt> text for which the property is set
:: <tt>''prop''</tt> property key string
:: <tt>''value''</tt> value of the property (string)
: Returns <tt>Undefined</tt>.

; <tt>normKey (''phrase'')</tt>
: Normalizes the text by removing all whitespace, removing the accelerator, and lowercasing it, e.g. for use in near-match lookups.
:: <tt>''phrase''</tt> text to be normalized
: Returns <tt>String</tt>.

; <tt>getConfString (''key'')</tt>
; <tt>getConfString (''key'', ''value'')</tt>
; <tt>getConfBool (''key'')</tt>
; <tt>getConfBool (''key'', ''value'')</tt>
; <tt>getConfNumber (''key'')</tt>
; <tt>getConfNumber (''key'', ''value'')</tt>
: Fetches a value of the requested type from the [[#User Configuration|user-configuration]] file. A default value of the same type can be stated too (or else it defaults to <tt>undefined</tt>).
:: <tt>''key''</tt> the name of the field in the configuration file
:: <tt>''value''</tt> the default value, in case the key is not found
: Returns the configuration value if the key is found, or the default value or <tt>undefined</tt> otherwise. If the configuration string is not a valid representation of the requested type, again the default or <tt>Undefined</tt> is returned instead.

=== Property Maps ===

There is a frequent need to have different properties attached to the particular pieces of text. For example, case, gender, etc. may be considered as properties of a noun, or phrase with a subject/object role in a sentence. Sometimes it may be convenient (or necessary) to define and maintain the phrases and their properties in an external file. ''Property map'' is Transcript's built-in (i.e. efficient) way to handle such definitions.

Property map files are text files in a simple dictionary format for writing down phrases with their properties (filenames must end in <tt>.pmap</tt>). The format is best shown on the example:
<syntaxhighlight lang="text">
# cities.pmap
=:Athens:Atina:nom=Atina:gen=Atine:dat=Atini:acc=Atinu::
=:Paris:Pariz:nom=Pariz:gen=Pariza:dat=Parizu:acc=Pariz::
</syntaxhighlight>
Each entry starts with two characters, <tt>=:</tt> in this example, which may be chosen arbitrarily (but must be non-letters, and non-#), and can change from entry to entry. The first character defines the separator in the property key-value pair, and the second character is the separator of pairs. E.g. the <tt>gen=Atine</tt> segment above is defining the genitive case of the localized form of Athens. The "pairs" which actually lack the property key (first two in the entries above) are considered phrase identifiers. The completely empty pair (<tt>::</tt> in the example) terminates the entry.

The property map is loaded inside the scripting module using the <tt>Ts.loadProps()</tt> call. The map file should normally reside in the same folder as the scripting module that uses it; if so, the map from the above example is loaded simply with <tt>Ts.loadProps("cities")</tt>. The property values are obtained in the scripts using <tt>Ts.getProp()</tt> call. E.g. the genitive form of Athens would be obtained either by <tt>Ts.getProp("Athens", "gen")</tt> or <tt>Ts.getProp("Atina", "gen")</tt> -- because the map defined both as phrase identifiers for Athens. The phrase/key lookup is performed using normalized values; this normalization can be checked or used for own purposes by <tt>Ts.normKey()</tt> call.

The comments are a bit of an issue in property map files. Since the phrases and property values can be anything (hence the possibility to choose separators), fixing any single character as your typical to-the-end-of-line comment would introduce limitations. Instead, the #-comments are allowed only ''between'' the entries, span to the end of line, and # cannot be a separator character.

The whitespace in property values is mostly preserved (for the keys it does not matter, as it gets stripped for lookups). However, if the leading sequence of whitespace contains a newline, all whitespace to the first newline and including it is stripped; symmetrically, in a trailing whitespace sequence, the last newline (if any) and the whitespace following it are removed.

=== User Configuration ===

Using a configuration file located at {{path|$HOME/.transcriptrc}}, the user can be allowed to configure certain aspects of Transcript operation. The configuration file is in ini-style, where group names are language codes to which the selections apply. For example:
<syntaxhighlight lang="text">
# Settings for German.
[de]
serve-sausages = no
wine-instead-of-beer = yes

# Settings for Italian.
[it]
with-mozzarella = no
</syntaxhighlight>
Within scripting modules, the configuration values can be fetched using <tt>Ts.getConf*()</tt> series of calls. These calls will automatically look for the configuration fields under the proper language group, as the scripting modules are language-dependent too.

For boolean-valued keys, <tt>0</tt>, <tt>no</tt>, and <tt>false</tt> can be used to indicate negative (false) values, and everything else is treated as positive (true). Also, the boolean values are not case sensitive.

== Repository Organization ==

For the PO file in the KDE repository {{path|LL/messages/kdemodule/foo.po}}, the corresponding Transcript module should be located at {{path|LL/scripts/kdemodule/foo/foo.js}}. Note the extra subfolder in the module path, named like the basename of the PO/JS file. This subfolder is introduced because the module's main JS file might need other supporting files, which can then be located conveniently in the same folder.

The {{path|autogen.sh}} script, used for some time already to generate build system support for PO files, will also generate build support for Transcript modules. In particular, it must be rerun whenever a new module subfolder is added (e.g. the {{path|LL/scripts/kdemodule/foo}}), but not when particular files within it are added or removed. Every file in the module subfolder will be installed automatically, so put there only what is needed at runtime.

== Real-Life Examples ==

=== CJK Accelerator Keys ===
In CJK environment, accelerator keys are wrapped in the parenthesis. For example, translation of "&File" in Korean is "파일(&F)", and in Japanese, "ファイル(&F)". When it comes to the names of action, it is shared among toolbar icon names, menu bar, and so on.

However, in toolbar icon name, "Configure shortcut" dialog, etc., those access keys are remaining in the name, makes the name little bit awkward. In this situation, we can use transcript to get rid of those accelerator keys. Currently, the code is in Japanese and Korean kdelibs4.js file.

=== Korean Postpositions ===
In Korean language, postpositions are widely used. Among those, 8 postpositions are in pair and they change forms according to the word. Because PO file itself doesn't know what the substituted word(like %1) is, proper postpositions couldn't be added to the string.

Fortunately, the rule is programmable, and proper postpositions could be added via transcript. Right now a little part of postposition rules is programmed into Korean version of kdelibs4.js.

=== Automatic Property Calls ===
Leveraging [[#Property Maps|property maps]] interface, we can make a generalization of the [[#Dynamic Case Setting|dynamic case setting]] example. We want to set named properties by key-value pairs for a given message, and then, to use calls named as property keys to retrieve the values when that message is used as placeholder replacement in another message.

For example, Nevernessians need to match the noun case of "baloon" when it is inserted into sentence "I see a %1", and additionally the adjective form of "red" to baloon in "I see a red %1", so they do:
<syntaxhighlight lang="text">
msgid "baloon"
msgstr ""
"beeluun"
"|/|"
"$[properties see-it beeluuno see-red raado]"

msgid "I see a %1"
msgstr ""
"O sii e %1"
"|/|"
"O sii e $[see-it %1]"

msgid "I see a red %1"
msgstr ""
"O sii e raad %1"
"|/|"
"O sii e $[see-red %1] $[see-it %1]"
</syntaxhighlight>

The <tt>properties</tt> call, which does all the work -- setting properties, setting new calls by property keys -- looks like this:
<syntaxhighlight lang="javascript">
// Set properties of the phrase given by the finalized msgstr in the PO file.
// The arguments to the call are consecutive pairs of keys and values,
// as many as needed (i.e. total number of arguments must be even).
//
// The property keys are registered as PO calls taking single argument,
// which can be used to retrive the property values for this msgstr
// when it is later used as placeholder replacement in another message.
//
// Always signals fallback.
//
function setMsgstrProps (/*KEY1, VALUE1, ...*/)
{
if (arguments.length % 2 != 0)
throw Error("Property setter given odd number of arguments.");

// Collect finalized msgstr.
phrase = Ts.msgstrf()

// Go through all key-value pairs.
for (var i = 0; i < arguments.length; i += 2) {
var pkey = arguments[i];
var pval = arguments[i + 1];

// Set the value of the property for this phrase.
Ts.setProp(phrase, pkey, pval);

// Set the PO call for getting this property, if not already set.
if (!Ts.hascall(pkey)) {
Ts.setcall(pkey,
function (phr) { return Ts.getProp(phr, this.pkey) },
{"pkey" : pkey});
}
}

throw Ts.fallback();
}
Ts.setcall("properties", setMsgstrProps);
// NOTE: You can replace "properties" in the line above with any UTF-8 string,
// e.g. one in your language so that it blends nicely inside POs.
</syntaxhighlight>

=== Labels Dependent on Dynamic Values ===
Sometimes the text of a standalone label should grammatically conform to the a changing value elsewhere in the interface. The typical example of this is when a sentence-like row of different widgets is used to specify an action:
<syntaxhighlight lang="text">
Search in files modified:
( ) ...
(*) during the last {day|week|month|year}
( ) ...
</syntaxhighlight>
<tt>( )</tt> stand for radio buttons, and <tt>{...|...|...}</tt> for a list box with several choices. The label <tt>"during the last"</tt> contains the adjective "last", which in some languages needs to have different forms according to the noun it describes (one of the choices from list box). Whenever the list box value changes, the label should change to match it. But this label is a standalone message in the PO file, so based on what can its translation be scripted?

The solution is to ask the programmer to make the label refresh on every change of the list box value, and to add a ''dynamic context'' to the label's message (using <tt>inContext</tt> method of <tt>KLocalizedString</tt>). The dynamic context is a key-value string pair which indicates the current list box value; the key and possible values should be documented in the message's context. If the programmer did this properly, the translator should now see in the PO file:
<syntaxhighlight lang="text">
msgctxt ""
"Part of logical sentence 'Search in files modified >during the last< "
"day|week|...'; provides dynamic context 'last-what': 'd' for day, "
"'w' for week, 'm' for month, 'y' for year."
msgid "during the last"
</syntaxhighlight>
The translation can now be scripted based on the <tt>last-what</tt> dynamic context:
<syntaxhighlight lang="text">
msgstr ""
"duureeng thee leestee"
"|/|"
"duureeng $[by-context last-what 'd|m' 'thee leestee' 'w|y' 'thaa leestaa']"
</syntaxhighlight>
The <tt>by-context</tt> call takes the context key, followed by pairs of context value matches (regular expressions) and corresponding phrases; in this example, one form is selected for context values <tt>d</tt> and <tt>m</tt>, and another form for <tt>w</tt> and <tt>y</tt>. The definition of this call is:
<syntaxhighlight lang="javascript">
// Select a phrase according to dynamic context.
// The first argument is the context keyword,
// followed by arbitrary numer of pairs of context values and
// corresponding phrases, optionally followed by default phrase.
//
// Context values are actually regular experessions, so that if one phrase
// corresponds to several context values, it does not have to be repeated
// but its context value can be given e.g. as 'foo|bar|...'.
//
// If the context was not matched, default phrase is returned if
// it was given; otherwise fallback is signaled.
//
function selectByContext (/* ctxt_key,
valrx_1, phrase_1, ..., valrx_N, phrase_N
[, default_phrase] */)
{
if (arguments.length < 1) {
throw Error("Selector by context needs at least one argument.");
}

// Collect context value for the given key.
var ctxtkey = arguments[0];
var ctxtval = Ts.dynctxt(ctxtkey);

// Match the context and select the corresponding phrase.
var phrase;
for (var i = 1; i < arguments.length; i += 2) {
if (ctxtval.match(RegExp(arguments[i]))) {
phrase = arguments[i + 1];
break;
}
}

// If context was not matched, select default phrase or signal fallback.
if (phrase == undefined) {
if (arguments.length % 2 == 0) {
phrase = arguments[arguments.length - 1];
} else {
throw Ts.fallback();
}
}

return phrase;
}
Ts.setcall("by-context", selectByContext);
// NOTE: You can replace "by-context" in the line above with any UTF-8 string,
// e.g. one in your language so that it blends nicely inside POs.
</syntaxhighlight>

Localization/Workflows/PO Summit

2011-04-06T12:35:43Z

Ilic: Spacing.

{{Template:I18n/Language Navigation Bar|Localization/Workflows/PO_Summit}}

{{LocalizationBrowser|
section=[[Localization#Workflows|Workflows]]|
name=Translating in Summit|
prereqs=[[Localization/Workflows/Coordinator|Language Coordinator]]|
related=[[Localization/Tools/Pology|Pology]]|
}}

== Software Branches and Translation ==

The obvious approach to releasing software is for developers to focus on one central body of code, a single "branch", fixing bugs and adding new features to it, and from time to time taking a "snapshot" of the branch and packing it as next release. One approach to make the release process more robust, is for development to proceed in two parallel branches. From the "stable" branch the actual releases are made, mostly with bugs fixed and only very important features added. The other, "unstable" branch, is used to develop new and redesign existing features, and at some point will become the next stable branch. Depending on the project, releases may be made from the unstable branch as well, in parallel with stable releases, in order for eager users to help testing the novelties.

In KDE, all three release models may be present at any given moment. Core KDE modules, which are together labeled as "the KDE" and released in unison, follow the two branch model, with stable and trunk branch, and releases from stable branch only. KDE extragear applications may do the same, but they are not required to; they may instead make releases from the trunk branch only, or also use stable and trunk branch, but make releases from both. This presents translators with the workflow as on the following figure.

[[Image:Translating_in_branches.png|center|Classic translation by branches]]

The KDE [[Localization/Workflows/Repository_Automation|repository automation]] (aka "Scripty") is preparing template POs and merging them with language POs. From the point of view of language teams, real people who perform global actions (e.g. moving around POs for all languages when an application moves from module to another) can also be grouped here. However, the translation team is still presented with two branches of POs to translate.

In general, this means that, just like programmers, at times translators have to work on both branches, and propagate fixes from one branch to another ("backport" from trunk to stable, or "forwardport" from stable to trunk). This may be prone to coordinatorial confusion, and demand extra effort and attention when updating translations. Keeping up with two branches is easier in core KDE modules, as they are released from stable branch only and with singular schedule, but could be more taxing with extragear applications.

The exact amount of effort spent due to parallel translation, porting of fixes, and coordination, depends on the translation team. For example, a well-manned and well-coordinated team, with established style and terminology guides, custom automation to support these, may be able to keep all POs in both branches fully translated at all times with ease. Or, a team may have worked out well-defined schedules, such that any given member of the team at one point switches from translating one branch to another, without looking back, thus effectively having everyone always working on one branch (even if not the same one).

If, however, you as the team coordinator have said and thought "Should do that in trunk too, bugger", "Didn't we fix that already?", "No, you should take it from stable", "It was released from WHAT branch?", more often than you would have liked, the following presents one possibility for sidestepping such issues.

== Translating in Summit ==

For a PO catalog which exists in both stable and trunk branch, a new same-named catalog can be made by ''gathering'' all the unique messages from branch PO files, i.e. the ''summit'' of branch POs. Assuming that branch POs were not all that different, since they are two versions of the same catalog, the summit PO shouldn't have much more messages than either. Translators at all times work on the summit PO, from which the messages are periodically ''scattered'' back to original branch POs. Thus, translators can always work on summit POs, not having to do any parallel translation, branch switching or porting of fixes. The summit workflow is presented by the following figure.

[[Image:Translating_in_summit.png|center|Translation in summit]]

In the summit mode, repository automation gathers summit PO templates from branch templates, and stores them separately from real branches, at {{path|trunk/l10n-support/templates/summit/}}. The language team also has a collection summit POs, at {{path|trunk/l10n-support/LANG/summit/}}, which is the sole location where translation happens. As before, repository automation handles merging of templates in branches, but merging of summit POs is done by the team coordinator; team coordinator can opt to manually merge branch POs as well (more on why-and-how of this later). From time to time, the team coordinator fills out branch POs by scattering from summit. Not to worry, each of these special actions upon the team coordinator is done with a single command.

While it is in principle obvious that two branch POs can be made into one summit PO with the union of their messages, there are some important details that should be handled the right way by the summitting system:

* What if a PO changes modules in one branch, so that it no longer belongs to the same module in both branches (e.g. application is moved from one to another module in trunk)?

* What if a PO changes its name in one branch, but not in the other (e.g. application is renamed in trunk)?

* What if a PO in one branch is split into two POs in another (e.g. extracting a library out of monolithic application in trunk)?

* Where to place messages unique to one branch in the summit PO? The original file context of a message, which messages precede it and which follow it, should be kept as much as possible.

* If source references are used to achieve good ordering of messages in summit PO, what to do if some source file paths change in one branch (e.g. application gets restructured in trunk)?

* How to handle messages with different plurality across branches (since messages are identified only by their <tt>msgctxt</tt> and <tt>msgid</tt> field, an not <tt>msgid_plural</tt>)?

And handle these details it does, the present summitting system. This also means that teams working in the summit need not take care of the first three issues above, which are affecting manual branch handling too.

Summit POs are normal, fully valid POs in their own right. A message in a summit PO is different from branch PO only by being equipped with another comment, <tt>#. +> ...</tt>, showing in which branches the message exists:

<code po>
#. +> trunk
#: kdeui/jobs/kwidgetjobtracker.cpp:469
msgctxt "The destination url of a job"
msgid "Destination:"
msgstr ""
⁠
#. +> stable
#: kdeui/jobs/kwidgetjobtracker.cpp:469
msgid "Destination:"
msgstr ""
⁠
#. +> trunk stable
#: kdeui/jobs/kwidgetjobtracker.cpp:517
msgid "&Keep this window open after transfer is complete"
msgstr ""
</code>

The first message above thus exists in trunk only, the second in stable only, and the third in both branches. The source reference always points to the source file in the first listed branch. Any extracted comments (<tt>#.</tt>) other than the branch list are also taken from the first listed branch.

Note that the two messages above are different only by context; the context was added in trunk, but not in stable, in order not to break message freeze. However, due to careful ordering of messages in summit POs, these two messages appear together, allowing translator to immediately make correction in stable branch too if the new context in trunk shows it to be necessary.

=== Setting Up and Daily Operation ===

Before initializing language summit, the team coordinator has to have all the necessary paths checked out from the KDE repository, and structured on the local machine exactly as in the repository. If the path to the root of KDE repository on the local machine is <tt>$KDEREPO</tt>, and the language code <tt>LANG</tt>, then the structure should be as follows, with leaf directories checked out in full:

<code text>
$KDEREPO/
trunk/
l10n-kde4/
scripts/
templates/
LANG/
l10n-support/
scripts/
pology/
templates/
LANG/
branches/
stable/
l10n-kde4/
scripts/
templates/
LANG/
</code>

It is important that this structure is fully under version control (rather than e.g. top directories being manually created), so if that is not the case, the following commands will fetch everything anew in proper order:

<code bash>
cd $KDEREPO
svn co --depth=empty svn+ssh://svn.kde.org/home/kde .
svn up --depth=empty branches branches/stable branches/stable/l10n-kde4
svn up --depth=empty trunk trunk/l10n-support trunk/l10n-kde4
svn up branches/stable/l10n-kde4/{scripts,templates,LANG}
svn up trunk/l10n-kde4/{scripts,templates,LANG}
svn up trunk/l10n-support/{pology,scripts,templates,LANG}
</code>

Note the trailing dot in the first <tt>svn</tt> command, and do not miss to substitute LANG for the real language code in the last three commands. <tt>--depth=empty</tt> option prevents recursive fetching, so that e.g. the first <tt>svn</tt> command does not fetch the complete repository tree. Later you can regularly update this tree with single command:

<code bash>
svn up $KDEREPO/{trunk,branches/stable}/l10n-kde4/{scripts,templates,LANG} \
$KDEREPO/trunk/l10n-support/{pology,scripts,templates,LANG}
</code>

Summit operations are performed using the <tt>posummit</tt> command, which is part of [[Localization/Tools/Pology|Pology]], residing in {{path|trunk/l10n-support/pology/}}. Therefore the first thing to do is to setup Pology, which amounts only to setting the proper path:

<code bash>
$ export PATH=$KDEREPO/trunk/l10n-support/pology/bin:$PATH
</code>

To initialize the summit, by gathering from existing translation in branches, the team coordinator executes:

<code bash>
$ cd $KDEREPO/trunk/l10n-support
$ posummit scripts/messages.summit LANG gather --create --force
</code>

Depending on the amount of translation, after some minutes the initial gathering will have been completed, and language summit located under {{path|$KDEREPO/trunk/l10n-support/LANG/summit/messages/}}. This is the only time when the coordinator performs the gather operation on language POs; it is daily done only on templates by repository automation. Then, the created language summit should be merged with current summit templates:

<code bash>
$ posummit scripts/messages.summit LANG merge
</code>

Merging the summit is something that the coordinator does periodically, with frequency of own desire. For example, it can be done daily, or with increasing frequency as the last day for translation for the next release approaches.

After the first merging, language summit is ready for active translation. The coordinator should now commit {{path|$KDEREPO/trunk/l10n-support/LANG/}}, and, importantly, notify team members to stop working on branch POs and focus exclusively on summit POs.

To scatter the summit, i.e. fill out POs in stable and trunk branch from the summit POs, the coordinator periodically executes:

<code bash>
$ posummit scripts/messages.summit LANG scatter
</code>

As with merging, there is no fixed schedule when scattering should be done. Of course, it must necessarily be done before the next release is tagged, and in between it is useful to scatter for runtime testing, or to have translation statistics by branches on l10n.kde.org up to date.

Periodic scattering and merging of the complete summit are basically all that a language team coordinator needs to do specifically to operate the summit. Also, since {{path|l10n-support/scripts/}} and {{path|l10n-support/pology/}} contain scripts and settings critical for proper functioning of summit operations, and may be tweaked at any time, they should always be updated from the repository together with PO files and templates (in fact, it is best to always update at once the whole tree as outlined above).

{{note|For documentation POs, summit setup and operation is the same, only replacing every <tt>messages</tt> with <tt>docmessages</tt> in the command lines above. The user interface and documentation summits are fully independent, so for a trial period it is reasonable to work with the interface summit only, and engage documentation summit once the trial has been deemed successfull.}}

=== Operation Targets ===

Sometimes it is advantageous to merge or scatter just a single catalog, a single module, a single branch, or any combination thereof. To this end, scatter and merge operations accept any number of ''operation targets'' after the operation keyword, specified as one of <tt>''CATALOG''</tt>, <tt>''BRANCH'':''CATALOG''</tt>, <tt>''MODULE''/</tt>, <tt>''BRANCH'':''MODULE''/</tt>, and <tt>''BRANCH'':</tt>. For example, to scatter just to Dolphin's PO in stable branch, in order to test translation at runtime, one would execute:

<code bash>
$ cd $KDEREPO/trunk/l10n-support
$ posummit scripts/messages.summit LANG scatter stable:dolphin
</code>

(note no <tt>.po</tt> ending on catalog name). Or, to scatter to every PO in <tt>kdeplasma-addons</tt> module in stable branch:

<code bash>
$ cd $KDEREPO/trunk/l10n-support
$ posummit scripts/messages.summit LANG scatter stable:kdeplasma-addons/
</code>

(the trailing slash is mandatory, or else <tt>posummit</tt> would think that <tt>kdeplasma-addons</tt> is a catalog name). Finally, to scatter to all catalogs in the stable branch (with the trailing colon for the same reason as earlier):

<code bash>
$ cd $KDEREPO/trunk/l10n-support
$ posummit scripts/messages.summit LANG scatter stable:
</code>

The command line arguments of <tt>posummit</tt> operations are intentionally arranged such that all the arguments fixed for one language are placed first. If operation targets are used frequently, then it is convenient to place the immutable part of the command line under a shell alias, with absolute path to the summit setup file:

<code bash>
$ alias posummit-kde-LANG="posummit $KDEREPO/trunk/l10n-support/scripts/messages.summit LANG"
$ cd ANYWHERE
$ posummit-kde-LANG scatter stable:dolphin
</code>

=== Externally Injected Branch Catalogs ===

Once the summit is in operation, normally the only way for a catalog to appear in branches (trunk or stable) is through scattering from the summit. However, sometimes an application which had seen considerable development outside of the infrastructure of KDE project gets moved in, drawing in any of its existing translations into branches. Such externally injected branch catalogs cause warnings to be issued on summit merging, and outright failure on scattering:

<code bash>
$ cd $KDEREPO/trunk/l10n-support
$ touch ../l10n-kde4/LANG/messages/kdetoys/foobar.po # dummy injection
$ touch ../l10n-kde4/LANG/messages/kdetoys/libfoobar.po # dummy injection
$
$ posummit scripts/messages.summit LANG merge
...
posummit: warning: No summit catalog for branch catalog
'../l10n-kde4/LANG/messages/kdetoys/foobar.po'.
posummit: warning: No summit catalog for branch catalog
'../l10n-kde4/LANG/messages/kdetoys/libfoobar.po'.
...
$
$ posummit scripts/messages.summit LANG scatter
...
posummit: error: Missing necessary summit catalogs: foobar libfoobar
$
</code>

When this happens, the injected catalogs need to be gathered into the summit. This is done just like the initial gathering, only giving injected catalogs' names as operation targets:

<code bash>
$ posummit scripts/messages.summit LANG gather --create --force foobar libfoobar
</code>

=== Summit Customization ===

File {{path|scripts/messages.summit}} (i.e. {{path|$KDEREPO/trunk/l10n-support/scripts/messages.summit}}), given as first argument to <tt>posummit</tt>, contains the general summit setup for all languages. This file is set to include customization file per language, if it exists at {{path|$KDEREPO/trunk/l10n-support/LANG/summit/messages.extras.summit}}, which contains additions and overrides to the general setup as desired by the language team. Same as the general setup file, the customization file is a Python source. It uses an external object named <tt>S</tt> to define summit settings, and can, of course, contain any helper Python code (the file is executed only once, at the beginning of a <tt>posummit</tt> run). It has the following general layout:

<code python>
# -*- coding: UTF-8 -*-
# kate: syntax Python;
# This file is included by scripts/messages.summit
# for language-specific additions/overrides.
#
# ...
# ... operations with object S and other Python code ...
# ...
</code>

Object <tt>S</tt> provides both data attributes and methods to configure different aspects of summit operation. For example, while the general setup specifies that text fields in summit catalogs should not be wrapped on column but should be wrapped on logical breaks (like some markup tags), this can be overriden using <tt>summit_wrap</tt> and <tt>summit_fine_wrap</tt> attributes:

<code python>
S.summit_wrap = True
S.summit_fine_wrap = False
</code>

Or, to get the path to a file relative to the summit customization file itself (rather than to current working directory, where <tt>posummit</tt> was executed), method <tt>resolve_path_rooted</tt> can be used:

<code python>
relpath = S.resolve_path_rooted("../whatever.txt")
</code>

The following sections will present some typical customization possibilities.

=== Fully Local Merging ===

When scattering from the summit, sometimes there will be reports of "messages missing in the summit". This happens because of time rift created by the Scripty merging branch POs, gathering summit templates, and a team coordinator merging the language summit, thus making some messages in branch POs not always present in the summit. This condition is benign, as such warnings will start to disappear with the message freeze approaching, but can be annoying. For this reason, team coordinator can stop Scripty from merging branch POs, and have the <tt>posummit ... merge</tt> command alone merge not only summit POs, but stable and trunk POs as well, such that summit and branches are always in perfect sync.

First, to stop Scripty from merging branch POs, a file named <tt>no-auto-merge</tt> (with arbitrary content) should be committed to the roots of respective trees, e.g.:

<code text>
$ touch $KDEREPO/trunk/l10n-kde4/messages/LANG/no-auto-merge
$ touch $KDEREPO/branches/stable/l10n-kde4/LANG/messages/no-auto-merge
</code>

Then, to make summit <tt>posummit ... merge</tt> merge everything, the following lines should be added into the summit customization file:

<code python>
# Set local merging for all branches.
for branch in S.branches:
branch["merge_locally"] = True
</code>

Once local merging of all branches is set, the coordinator can also use operation targets for selective merging, e.g. to merge only stable branch:

<code bash>
$ cd $KDEREPO/trunk/l10n-support
$ posummit scripts/messages.summit LANG merge stable:
</code>

=== Merging with Compendium ===

By default, <tt>msgmerge</tt> takes into account only the catalog itself to fill out near-match translations to new messages introduced by merging. i.e. to produce fuzzy messages. However, an arbitrary PO file can be given to <tt>msgmerge</tt> as another possible source of earlier translations, through the <tt>--compendium</tt> option. For maximum effect, this PO file is usually constructed as the collection of messages from all PO files project-wide, and hence called the ''compendium''.

Compendium can be used in summit merge operations simply by setting <tt>compendium_on_merge</tt> attribute in summit customization file. If the compendium is located at {{path|$KDEREPO/trunk/l10n-support/LANG/summit/messages-compendium.po}}, then:

<code python>
# Compendium to use when merging summit catalogs.
S.compendium_on_merge = S.resolve_path_rooted("messages-compendium.po")
</code>

Note that if not located as above, compendium should not be placed within {{path|.../LANG/summit/messages/}}, as then it would be considered a summit catalog itself. In case fully local merging is engaged, only summit POs will be merged with compendium; it makes no sense for branch POs, since they are not being directly translated.

See [http://www.gnu.org/software/automake/manual/gettext/Creating-Compendia.html "Creating Compendia"] section of Gettext manual for details on how to create a compendium out of current body of summit translations. Once you establish the precise commands to create the compendium (whether to collect fuzzies too, whether to include old compendium as one of sources for the new, etc.), you would periodically refresh and commit the updated compendium.

=== Vivifying Summit Catalogs ===

Translation of a summit catalog normally starts the same way as it did for branch catalogs, by copying it over from template and initializing the header. This can be done manually, but it can also be quite automatic when using project manager as sometimes provided by dedicated PO editors. However, relying on the editor's project manager can be disadvantageous at times. For example, aside from the PO editor, other, frequently command line tools may be used to process the body of translation, and these tools might need to consider non-started templates as if they were empty POs (e.g. statistics). Team members who do not have full local checkout, or no project set up in the editor, may need to jump between language and template directories when looking for files to translate.

Therefore, summit setup can be customized to automatically create ("vivify") summit catalogs for every new summit template, so that there is never the need (for a tool or human) to specifically treat templates as empty catalogs. This is done by adding the following lines into customization file:

<code python>
# Create empty summit catalog for every new summit template.
S.vivify_on_merge = True
S.vivify_w_translator = "Noone Noonian <noone.noonian@nowhere.null>"
S.vivify_w_langteam = "Neverneese <neverteam@nowhere.null>"
S.vivify_w_plurals = "nplurals=2; plural=n != 1;"
# Minimum translation state to create branch catalog on scatter.
S.scatter_min_completeness = 0.9
</code>

The <tt>vivify_w_*</tt> attributes set the necessary data to initialize headers of newly created summit catalogs. Aside from those listed above, there is also the <tt>vivify_w_charset</tt> attribute, which is by default <tt>"UTF-8"</tt>, and very probably should not be changed.

The <tt>scatter_min_completeness</tt> attribute does not refer to vivification of summit catalogs, but should invariably be set in this context. When scattering from summit, normally the branch catalog is automatically created if there is the corresponding summit catalog. When vivification is engaged, this would result in creation of empty branch catalogs too, which is not desired for two reasons. Firstly, empty branch catalogs would become part of the released language pack, for which there is no practical reason. Secondly, KDE's i18n system regards an application with installed catalog as translated, so messages coming from basic system catalogs (e.g. <tt>kdelibs4</tt>) would show through, resulting in mostly untranslated user interface with specks of translation -- many users consider this rather ungainly. Therefore, <tt>scatter_min_completeness</tt> sets how complete the translation of currently non-existing branch catalog should be after scatter (0.0 empty, 1.0 fully complete), for the branch catalog to actually be created. If the <tt>--force</tt> option is given to <tt>posummit</tt>, the branch catalog will be created regardless of its translation state, which may be handy in combination with [[#Operation Targets|operation target]] when wanting to check work in progress in running application.

If [[#Merging with Compendium|the compendium has been set]] for merging, every vivified summit catalog will also be merged against the compendium, to fill out as many of messages with approximate translations.

=== Scatter Hooks ===

((TODO. Checks, modifications...))

=== Merge Hooks ===

((TODO. Special header fields...))

== Disadvantages to Summit and Remedies ==

Although hopefully shadowed by the advantages, working in summit is not without its disadvantages. These should be weighed when deciding of whether to try out the summit workflow.

Obviously, while summit operations are made to be quite automatic, some extra aptitude is asked of the team coordinator. Reasonable shell handling, understanding of version control operations, feeling the pulse of repository automation, are all prerequisites, and some scripting ability advantageous.

After the summit is put in operation, any changes made manually in branch POs will not propagate to summit, and will be soon lost to scattering -- summit translations override everything in branches. This means that the whole team must work in the summit, it is not possible for some members to use the summit, and some not.

A summit PO file will necessarily have more messages than either of the branch files. For example, in the KDE 4.0/4.1 and 4.1/4.2 cycle, summit POs of core KDE modules had on average less than 5% more words than their stable counterparts. However, the said percent is the top, never approached limit of wasted workload due to trunk messages coming and going, given that as the next feature KDE release approaches, more and more trunk messages will find their way into it.

Another, more pressing issue with increased size of summit POs is the following scenario: a stable release is around the corner, and the team has no time to update summit POs fully, but could update only stable messages in them. E.g. there are 1000 untranslated and fuzzy messages, out of which only 100 are from the stable branch. A clever dedicated PO editor could allow jumping only through untranslated and fuzzy messages which also satisfy a general search criteria, which in this case would be that a comment matches <tt>#\.\+>.*stable</tt> regular expression. On the other hand, with some external help, it is enough if the PO editor can merely search through comments. Then, the <tt>posieve</tt> command (ready to use next to <tt>posummit</tt>) can equip untranslated and fuzzy stable messages with <tt>untranslated</tt> flag (producing <tt>#, ..., untranslated</tt> comment), and this flag can be searched for in the PO editor:

<code bash>
$ posieve tag-untranslated -sbranch:stable -swfuzzy PATHS_TO_PO_FILES_OR_DIRS
</code>

The <tt>untranslated</tt> flag does not need to be manually removed when the message is updated. It will automatically disappear on the next merge, as it is not among flags known to Gettext.

If an automatic source lookup mechanism (to check the message context by following its source references) was available for branches, it may not entirely work for summit. Already the branch POs in KDE are kept separate from the code, which means that paths given by source references cannot be followed directly, but appropriate root paths have to be added in some way (a dedicated PO editor may provide a way to do this); this should continue to work with summit for those messages which exist both in trunk and stable branch, since their source references correspond to trunk code. But for messages found only in stable branch, whose source references correspond to stable code, existing source lookup mechanism will likely assume trunk roots again, and try to fetch wrong sources. To fix this, the source lookup mechanism should be flexible enough to select roots based on branch keywords given in summit messages' <tt>#. +> ...</tt> comment.

There is also the organizational issue with starting to use the summit, and, if it does not help as expected, stopping to use it. Team members have to be reminded to not send in branch POs at start, and then to be sent back to branch POs if summit is disbanded. On the plus side, disbanding summit is technically simple: just remove from the repository {{path|l10n-supprot/LANG/summit}}, possibly also <tt>no-auto-merge</tt> files if [[#Fully Local Merging|fully local merging]] was set up, and that is it.

== Another Way to Improve Branch Handling ==

If summit seems a lot to digest, or is simply an overkill for team's needs, but still some improvement to manual handling of branches would be welcomed, KDE's dedicated PO editor [[Localization/Tools/Lokalize|Lokalize]] offers a ''branch sync'' mode. It works as follows.

In Lokalize project definition, the local paths of trunk and stable PO roots are set in ''Translation directory:'' and ''Branch directory:'' fields. Then, when a trunk PO file is opened, if it has a stable counterpart with same name and location as in the trunk, this stable PO is also going to be opened. For each trunk message in the main editing pane, if such a message exists in stable PO too, the stable message will be shown in the ''Secondary Sync'' pane; changes in the translation of trunk message will reflect to the stable message, and stable PO file will also be saved when the trunk is saved.

Furthermore, any team member can personally choose to work like this, there is no need to change the workflow of the language team as whole. When sending modifications to the coordinator, team members who rely on this feature of Lokalize simply send both trunk and stable POs that got modified.

Localization/Workflows/PO Summit

2011-04-06T12:35:20Z

Ilic: Note on problem with following source references in summit.

Localization/Workflows/PO Ascription

2010-08-02T19:10:33Z

Ilic: Fixed some leftovers of previous modifications.

{{Template:I18n/Language Navigation Bar|Localization/Workflows/PO_Summit}}

{{LocalizationBrowser|
section=[[Localization#Workflows|Workflows]]|
name=Reviewing by Ascriptions|
prereqs=[[Localization/Workflows/PO_Summit|Translating in Summit]]|
related=[[Localization/Tools/Pology|Pology]], [[Localization/Workflows/Coordinator|Language Coordinator]]|
}}

== Why Review Translations? ==

Especially to new translators, it may not be obvious to which extent the translation needs to be reviewed. If the translator has exercised due diligence, how "wrong" can the translation be? Even if the translator has good command of the source language (English in context of this article), the answer is "very wrong", when all aspects are considered. Here are some of them.

With comparatively simple grammar of English, the meaning of a short English sentence -- as typically encountered in application user interfaces -- is very dependent on the surrounding context. This context may not be obvious when the translator is going through isolated messages in the translation file, so he may commit the worst of errors from the user's viewpoint, the senseless translation. An experienced reviewer will have developed sense for troublesome contexts, and will have several means to decisively determine the context (including, for example, running the development version of the application).

Even if the context is correctly established, the translator may use "wrong" terminology, which is the next worse thing for the user. A term used in translation does not need to be wrong by itself, in fact it may be exactly the correct term -- in another translation project. The reviewer will have more experience with terminology of the present project, and be able to bring the translation in line with it.

Style in the technical sense is a consistent choice between several perfectly valid constructs in target language when applied to text in the given technical context. For example, how to translate menu titles and items, button labels, or tooltips. The choices may include noun or verb forms, particular grammar categories, tone of address, and so on. There may be a style guide to the project which details such choices, and the reviewer will know it well.

Style in the linguistic sense is especially applicable to longer texts, such as tooltips in user interfaces, and passages in documentation. A typical error of a new translator is to closely adhere to English style and grammar. This may produce translation which is semantically and grammatically valid in the target language, but very out of style -- the "translationese". Reviewer is there to naturalize such passages.

Finally, the reviewer may be an experienced translator, but that does not mean that his own translations need no review. Immersion into the source language, distraction, fatigue, will lead the reviewer into any of the above errors in translation, only with less frequency. So reviewers should also review one anothers' translations.

== Classical Reviewing by Stages ==

Classical review workflow by stages seems simple enough. Translator translates a PO file (or updates existing translation), and declares it ready to review. A reviewer reviews it, and declares it ready to "commit". Committing here should be understood generally, as inclusion into the pool from which translations are periodically shipped to end users. A committer finally commits the file. The process is iterative: the reviewer may return the file to the translator, and translator later again declare it as ready for review. There may be several stages of review (such as ''proof-reading'', ''approving''), each of which may return the translation to a previous stage, or forward it to some special stage. The process can also be more finer grained, where each message in the file goes through stages separately.

Regardless of the particularities, workflows of this kind all have the following in common. Members of the translation team are assigned ''roles'' -- such as ''translator'', ''reviewer'', ''approver'', ''committer'' -- by which they enter into the workflow (single person can have more roles). The later review stages must wait for the earlier stages to complete, and the translation cannot be updated again before the current version clears the pipeline (or the pipeline is aborted). Most importantly, once the translation is committed, it becomes part of simply "admitted" translations, with no further qualifiers.

The system of prescribed roles requires that team members assign them between themselves, stick to them, and shuffle them along the way. The prescribed review pipeline requires a tool to enforce and keep track of the stages in which translations are. This makes the review workflow complex and rigid, most probably with choke points for efficiency. Distribution of roles may become disbalanced by people coming and going, or the workflow tool may be prohibitive to some scenarios (e.g. single translator making small adjustments in dozens of files across the project, but having to upload each manually through a web interface).

Of course, "rigid", "complex", "inefficient", are comparative qualifications, so what is it that the classical review by stages can be compared to in this way?

== Reviewing by Ascriptions ==

Reviewing by ascriptions is even simpler conceptually, and yet less rigid, less complex, and much more efficient than the review by stages. It works on the message-level, rather than file-level. Anyone can simply translate some messages and directly commit modified files, without any review, but with ''ascribing'' modifications to own name. Anyone can review any committed messages at any moment, commit the modifications-on-review and ascribe reviews to own name and (possibly) to certain class -- full review, review of context, of terminology, of style, etc. Only when the translation is to be shipped to end users, the ''insufficiently'' reviewed messages are automatically omitted from the package, by evaluating the ''ascription history'' of each message.

Most importantly, based on the ascription history, the reviewer can select only some particular messages, and review only the difference between their historical and current versions. For example, Alice can select to review only messages modified since she or Bob had last reviewed them for style; she could see the difference from that last review to current version, e.g. if in the whole paragraph only a single word has changed by Charlie when he reviewed the terminology. In terms of PO workflow, the ascription history propagates through merges, so the reviewer can compare the change in original and the change in translation since the last review, to judge if one fits the other.

Since everyone just commits, translations can be efficiently kept in a version control repository, with the ascription system added on top. After having done some translating, the team member simply substitutes commit command of the version control system (VCS) with ascribe-modifications command of the ascription system (AS, which calls the underlying VCS internally). After reviewing, the team member uses ascribe-reviews command of the AS to commit reviews to ascription history (as well as modifications made during the review). To select messages for review, the team member issues diff-for-review command of the AS (with suitable parameters to narrow the set) and selected messages are marked in-place in PO files and [[Localization/Tools/Pology/PO_Embedded_Diffing|embedded with differences]], and possibly popped open in a PO editor.

When the translations are to be released, the team coordinator issues filter-for-release command of the AS, which takes the working PO files and creates final PO files with insufficiently reviewed messages removed. "Release time" is used here only figuratively: this should be a fully automatic process, so it can be performed at any interval of convenience.

What constitutes "sufficient review" can be defined in fine detail. It could be specified that messages modified by Alice need to have only review for terminology, but not necessarily for style; Charlie may belong to the group which needs to be reviewed on style, but not necessarily on context; Bob's reviews for style may be nice to have, but never blocking if missing. These decisions do not preclude released messages to be reviewed later on missing points, after higher priority reviews have been completed. The definition of sufficiency may be changed at any point, e.g. as team members get more experienced and require less review, without interfering with direct translation and review work.



In summary, with reviewing by ascriptions the lean efficiency of raw VCS operation is preserved while providing for great flexibility of review. All team members can be given commit access, no web or email detours are needed. There are no prescribed roles, but an equivalent of role assignment happens automatically at last possible moment, and can take into account both translators' and reviewers' abilities. There is no staging between completing and committing the translation, which enables translator to keep on polishing the translation undisturbed until the reviewer comes around. There is no inefficiency in handling small changes throughout many files, since single AS command commits all changes just as single VCS command would. AS in effect abstracts VCS, so general team members do not have to know the particularities of the underlying VCS. On commit operations, AS can also apply checks (e.g. decline to commit syntactically invalid PO files) and modifications (e.g. update translator's data in the PO header).

== Ascription System in Pology ==

[[Localization/Tools/Pology|Pology]] is a collection of various modular tools for supporting translation based on PO files. Among them is the script <tt>poascribe</tt>, which implements an ascription system (AS); at present, it can use Subversion or Git as the underlying VCS. <tt>poascribe</tt> is still in experimental stage, so what follows is a brief description of how to use it in context of the KDE translation project. However, very little is truly specific to KDE; the only major assumption is that there exists a VCS repository with PO files of a given language grouped together, and that the translation team can use it without special restrictions.

Very important for the AS is how branches are handled (in KDE, the rolling trunk and stable branches). AS can in principle be deployed by branch, but then there is the added complexity of porting translations between branches, which ascriptions should follow. Therefore, the AS implemented by <tt>poascribe</tt> is currently limited to assumption that there is a single branch of translations at all times. The article [[Localization/Workflows/PO_Summit|"Translating in Summit"]] explains how a KDE translation team can set up and operate such a single branch, the summit, and this is the prerequisite for the following instructions. (Note that the summit system is useful on its own, and should be conductive to any kind of review workflow.)

== Setting Up ==

The summit branch for the language <tt>LANG</tt> is positioned like this in the KDE repository:

<code text>
$KDEREPO/
trunk/
l10n-support/
LANG/
summit/
messages/
docmessages/
</code>

The team coordinator already has this part of the repository tree locally due to regular summit operations. For the same reason Pology is already set up. Setting up the ascription system is now simple. The file <tt>ascription-config</tt> is created in the parent directory of the summit:

<code text>
...
LANG/
ascription-config
summit/
messages/
docmessages/
</code>

with the following contents:

<code text>
# ---------------------------
# Global ascription settings.

[global]

# Roots of the catalog and ascription trees.
catalog-root = summit
ascript-root = summit-ascript

# The underlying version control system.
version-control = svn

# Data for updating catalog headers.
# - language code
language = LANG
# - full language name
language-team = LANGUAGE
# - email address of the team
team-email = kde-i18n-LANG@kde.org

# Default commit message.
commit-message = Translation updates.

# -----------------------
# Registered translators.

[user-alice]
name = Alice Akmalryn
original-name = Алиса Акмалрин
email = alice.akmalryn@someplacenice.org

[user-bob]
name = Bob Byomkin
original-name = Бобан Бјомкин
email = bob.byomkin@otherplacenice.org

# ...and so on.
</code>

Some notes:

* The <tt>ascript-root</tt> setting should be exactly <tt>summit-ascript</tt>, for the reason mentioned later.

* <tt>commit-message</tt> field, if defined, allows team members to commit without providing a commit message. The value given by this field will be used by default, with translator's user name appended to the end in special syntax. For example: <tt>Translation updates. [>alice]</tt>. (Translator's user name is also appended to manually supplied commit messages.) Translators can still supply a commit message when they wish, as shown later. If this field is not set, the commit message is supplied as usual on committing.

* Team members are defined by <tt>[user-USERNAME]</tt> sections. Ascription user names can be any valid ASCII identifier: ASCII letters, digits and underscores only, digit cannot be the first character. Ascription user names have no technical relation to the underlying VCS accounts, though it is mnemonically convenient if they are the same (in case of SVN). This means that a translator who does not have a VCS account (yet) can and should be added here, with assigned user name (best one suitable as SVN account name later); why this should be done will be explained later.

* <tt>original-name</tt> field in user sections is there in case the preferred renderings of the name in English and in target language are not the same. When this is not the case, <tt>original-name</tt> can be omitted.

As soon as the <tt>ascription-config</tt> file is committed, the ascription system is ready for operation. Only regular modifications to this file are those of adding new team members. (On the other hand, team members should never be removed, because even after they no longer contribute, their ascription records remain in the system.)

=== Initial Ascription ===

The most common situation at start of ascription workflow is that there already exists a body of translations, contributed to by many different people over time. The coordinator should ascribe all existing translations as initial modifications, but to whom? It cannot be said precisely who translated what. The solution is to introduce a generic user in <tt>ascription-config</tt>, suitably known as "Unknown Hero" (or "Lost Translator", you can be inventive):

<code>
[user-uhero]
name = Unknown Hero
original-name = Незнани јунак
</code>

and ascribe all existing translations as modified and reviewed by this user. The coordinator does this with the following command:

<code bash>
$ cd $KDEREPO/trunk/l10n-support
$ poascribe commit -u uhero --all-reviewed -C LANG/summit/
</code>

The argument <tt>commit</tt> is the ascription mode, and the <tt>-u</tt> option provides the user name to which ascriptions are made. This is an important point: ascriptions are made to a user defined in ascription configuration, and have nothing to do with VCS accounts; someone who has the account can commit in the name of someone who does not. It is the <tt>--all-reviewed</tt> option that declares all messages to be reviewed as well (note that it is normally used only this once, and not for normal day to day reviewing). The <tt>-C</tt> option prevents automatic adding and committing to version control, which is useful for this initial step. Finally the paths which contain all summit catalogs are given.

When the <tt>poascribe</tt> command is issued, a progress bar will appear, and the following output will start to unfold:

<code text>
LANG/summit/messages/extragear-base/rellinks.po (50/50)
LANG/summit/messages/extragear-base/autorefresh.po (13/13)
LANG/summit/messages/extragear-base/babelfish.po (38/38)
...
LANG/summit/messages/qt/libphonon.po (13/13)
LANG/summit/messages/qt/phonon-xine.po (24/24)
LANG/summit/messages/qt/phonon_gstreamer.po (12/12)
===== Ascription summary:
- modified reviewed
translated 111775 111775
fuzzy 26943 26943
obsolete/t 2965 2965
obsolete/f 1626 1626
</code>

The number in parenthesis indicates how many messages have been ascribed in the given PO file (modified/reviewed), and at the end the totals are given. Ascribing the complete summit for the first time will take quite some time (on the order of 10-20 minutes).



After the initial ascription has been made, the ascription tree will appear next to the summit tree. This tree will contain one ascription PO file for each summit PO file, with the same name and relative location within the tree:

<code text>
...
LANG/
ascription-config
summit/
messages/
kdelibs/
kcertpart.po
kdelibs4.po
...
...
docmessages/
summit-ascript/
messages/
kdelibs/
kcertpart.po
kdelibs4.po
...
...
docmessages/
</code>

During the ascription some summit PO files may have been modified as well, in that any previous fields (<tt>#| ...</tt>) on translated messages have been removed. (These fields are sometimes erroneously left in by older PO editors.)

The newly created ascription tree (and any modifications to summit tree) can now be committed as usual:

<code bash>
$ svn add LANG/summit-ascript
$ svn commit LANG/summit LANG/summit-ascript -m "Initial ascription."
</code>

== Daily Use for Translators ==

Team members other than the coordinator, whether translators or reviewers, need to keep around only the <tt>trunk/l10n-support/LANG/</tt> directory. But they always need to update this directory fully (rather than just one particular module or file under <tt>.../*messages/</tt>), so that the summit tree and the ascription tree (and configuration) are kept in sync.

In order not to have to issue their own user name (<tt>-u</tt> option to <tt>poascribe</tt>) all the time, translators can set it in Pology user configuration <tt>~/.pologyrc</tt>, in <tt>[poascribe]</tt> section:

<code text>
[poascribe]
user = alice
</code>

Translators can then submit updated PO files simply by substituting <tt>svn commit</tt> (or whatever the VCS commit command is) with <tt>poascribe commit</tt> (<tt>co</tt> or <tt>ci</tt> for short):

<code bash>
$ poascribe ci LANG/summit/messages/kdefoo/*fooapp*.po
LANG/summit/messages/kdefoo/fooapp.po (144)
LANG/summit/messages/kdefoo/libfooapp.po (25)
===== Ascription summary:
- modified
translated 169
>>>>> VCS is committing catalogs:
Sending LANG/summit/messages/kdefoo/fooapp.po
Sending LANG/summit/messages/kdefoo/libfooapp.po
Sending LANG/summit-ascript/messages/kdefoo/fooapp.po
Sending LANG/summit-ascript/messages/kdefoo/libfooapp.po
Transmitting file data ....
Committed revision 1267069.
$
</code>

<tt>poascribe</tt> will add ascription records into ascription catalogs corresponding to summit catalogs to be committed, and commit them all. Like <tt>svn commit</tt>, <tt>poascribe ci</tt> can take any number of file or directory paths, and can be issued from any working directory (it will always find ascription catalogs). If default commit message has not been set in the ascription configuration, <tt>poascribe</tt> will ask for it; or it can be given in command line through <tt>-m</tt> option.

=== Translators Without Commit Access ===

With the ascription system in place, every regular team member should have commit access. But, there may be some period of time before new translators are given accounts, revision control may be too technical for some, and even those with the account may not be able to commit temporarily for some reason.

These translators may send in their work by email, to ''any'' team member with commit access (not necessarily the coordinator or a reviewer); this team member can commit received files without any review, as review can be conducted at any later time. If Bob sends some files to Alice, she can commit them immediately by stating Bob's user name:

<code bash>
$ poascribe ci -u bob ...
</code>

For this to work, the translator who sent in the files has to be defined in the ascription configuration. There are no hidden costs or security issues to this (as opposed to opening a VCS account), so every new translator should be defined there before any work of that person is committed.

== Daily Use for Reviewers ==

The ascription system opens up all sorts of possibilities for concrete review patterns. Reviewers should keep in mind that for each message the full modification and review history is available, so that the team can think about how to make good use of it. Therefore, what follows are some examples to illustrate the review facilities that <tt>poascribe</tt> provides.

=== Basic Reviewing ===

At the very basic level (which is the only level in classical review by stages), messages can be classified into simply unreviewed and reviewed, without further qualifiers. Alice now wants to review all unreviewed messages in a group of PO files, say <tt>kdetoys</tt> module. She issues (<tt>di</tt> is short for <tt>diff</tt>):

<code bash>
$ cd $KDEREPO/trunk/l10n-support
$ poascribe di LANG/summit/messages/kdetoys/
LANG/summit/messages/kdegames/bovo.po (2)
LANG/summit/messages/kdegames/kdiamond.po (7)
LANG/summit/messages/kdegames/palapeli.po (12)
===== Diffed for review: 21
</code>

Unreviewed messages have now been marked and diffed, inside the listed PO files. What is this about "diffing"? If the files had already been reviewed before, some of the messages modified since then (those marked for review) may have changed very little (e.g. a few words in a paragraph-length message, or even just punctuation). Therefore, for each message marked for review, Alice also wants to see the diff since last review to current version. Here are two messages in typical review states added by <tt>poascribe di</tt>:

<code po>
#. +> trunk stable
#. ascto: charlie:m
#: gui/mainwindow.cc:372
#, ediff
msgid "GAME OVER. {-You won-}{+Tie+}!"
msgstr "KRAJ IGRE. {-Pobeda-}{+Nerešeno+}!"

#. +> trunk stable
#. ascto: bob:m charlie:m
#: game-state.cpp:117
#, ediff-total
msgid "Click the pause button again to resume the game."
msgstr "Kliknite ponovo na dugme pauze da nastavite igru."
</code>

and the first one in Kate:

[[Image:Kate_poascribe_diff_01.png|center|Message diffed for review by poascribe in Kate.]]

In the first message, the first to note is the <tt>#. ascto:</tt> comment. This comment succinctly lists who did what with the message since the last review; here <tt>charlie:m</tt> means that Charlie is the one who modified it. Then, there is the <tt>ediff</tt> flag, which alice can use it to jump through messages marked for review. Finally, the original and translation have been diffed; here they show that, since the last review, the message was fuzzied by changing "You won" to "Tie", and what Charlie did in translation to unfuzzy it. Even on a message as short as this, the diff tells something useful to Alice: the phrase "Game over" likely has a formulaic translation, and the fact that it is not part of the diff means that the earlier reviewer had made sure it is consistent, so Alice does not have to check that.

The <tt>#. ascto:</tt> comment of the second message reveals that both Charlie and Bob had been translating it. <tt>ediff-total</tt> flag instead of plain <tt>ediff</tt> means that this message had no review at all up to now, so there are no embedded diffs in text fields.

Alice can now go through marked files and messages, review translations, and possibly make modifications. When making changes in a message with embedded diffs, she can freely edit text outside of difference segments and within <tt>{+...+}</tt> segments (as these are the ones which belong to current version of the text). While reviewing, Alice does ''not'' remove any of the added message elements while reviewing (save for an occasional difference segment, when translation should be modified), as these elements are needed for later. If a message is particularly hard and Alice wants to defer its review for later, she can add the <tt>unreviewed</tt> (or <tt>nrev</tt> for short) flag to it.

Once the review is complete, Alice simply commits the reviewed files:

<code bash>
$ poascribe ci LANG/summit/messages/kdetoys/
LANG/summit/messages/kdegames/bovo.po (0/2)
LANG/summit/messages/kdegames/kdiamond.po (0/7)
LANG/summit/messages/kdegames/palapeli.po (3/12)
===== Ascription summary:
- modified reviewed
translated 3 21
>>>>> VCS is committing catalogs:
Sending LANG/summit/messages/kdegames/palapeli.po
Sending LANG/summit-ascript/messages/kdegames/bovo.po
Sending LANG/summit-ascript/messages/kdegames/kdiamond.po
Sending LANG/summit-ascript/messages/kdegames/palapeli.po
Transmitting file data ....
Committed revision 1284220.
$
</code>

Three things have happened here. First, all review states (flags, embedded diffs, etc.) have been removed, restoring the PO file to normal. Then, any modifications that Alice have made during review are ascribed to her (here 3 out of 21 messages). Finally, all marked messages are ascribed as reviewed by Alice (any with <tt>unreviewed</tt>/<tt>nrev</tt> flags would have been omitted here). When committing, the only summit catalog that got committed is the one with modifications made during review, and all the ascription catalogs were committed because of the reviews recorded in them.

When many files with few changes in each are to be reviewed, it becomes burdensome to manually open each and every diffed for review, and then to make sure that all are committed with <tt>poascribe ci</tt>. To make this easier, <tt>-w torevivew.out</tt> option can be added to <tt>poascribe di</tt>, which requests that paths of all diffed PO files are written into <tt>torevivew.out</tt> file. This file can then be used to batch open POs for review in the editor, as well as fed back on <tt>poascribe ci</tt> with <tt>-f torevivew.out</tt>. There is also the <tt>-o</tt> option which causes <tt>poascribe</tt> to directly open PO files in a PO editor (though this is currently applicable only to Lokalize). Putting it together, to efficiently review a whole bunch of small changes throughout many files, Alice can:

<code bash>
$ poascribe di PATHS... -w toreview.out -o lokalize
$ # ...only marked messages opened in Lokalize, review them...
$ poascribe ci -f toreview.out
</code>

=== Selecting Messages for Review ===

Invocations of <tt>poascribe di</tt> without any options, as in the previous section, were actually equivalent to this:

<code bash>
$ poascribe di -s modar PATHS...
</code>

Option <tt>-s</tt> is issuing the message ''selector''. <tt>modar</tt> is the default selector for <tt>diff</tt> mode, and stands for MODified-After-Review: it selects the earliest historical modification of the message after the last (or no) review of that message, if there is any such. By selecting a historical modification of the message, the diff from it to current version can be computed and embedded into the PO file, as in previous examples.

There are various specialized selectors, and fall into two groups: ''shallow selectors'' and ''history selectors''. Shallow selectors look only into the current version of the message, and cannot select historical versions, which means that they cannot provide embedded diffs. History selectors (<tt>modar</tt> is of this type) can select messages from history and provide diffs. Several selectors can be issued on the command line, and the message is selected only if all selectors select it. Shallow selectors are then normally used as a pre-filter for history selectors. For example, to select messages modified after last reviewed, but only those found in stable branch, <tt>branch</tt> and <tt>modar</tt> selectors are chained:

<code bash>
$ poascribe di -s branch:stable -s modar PATHS...
</code>

It is important that the history selector is given last, because the last selector determines which historical message is selected. If the ordering had been reversed here, same messages would get selected, but they would not have embedded diffs, because <tt>branch</tt> is a shallow selector.

Selectors can take parameters themselves, like <tt>branch:stable</tt> in the previous example. Parameters are separated from the selector name by any non-alphanumeric character; this is colon by convention, but if a parameter contains a colon, something like slash, tilde, etc. can be used. Number of parameters can be variable, and <tt>modar</tt> in particular can take from none to three. If Alice wants to review only those messages modified ''by Charlie'' since last review, she states this by first argument to <tt>modar</tt>:

<code bash>
$ poascribe di -s modar:charlie PATHS...
</code>

If Alice does not give too much credit to other reviewers, she can request selection of messages modified after last review ''by her'' with second parameter to <tt>modar</tt>:

<code bash>
$ poascribe di -s modar::alice PATHS...
</code>

Here the first parameter ("modified by..."), which is not needed, must be explicitly skipped, before going to the second parameter ("reviewed by..."). The third optional parameter of <tt>modar</tt> will be mentioned in the next section.

When a selector parameter is a user name, normally it can also be a comma-separated list of user names (<tt>modar:bob,charlie</tt>) or prefixed with tilde to negate, i.e. select all other users (<tt>modar:~alice</tt>).

Any selector can be negated by prepending <tt>n</tt> to its name. For example, the history selector <tt>modafter:DATE</tt> selects first modification after the given date; to select messages modified after last review, but only if modified during June 2010:

<code bash>
$ poascribe di -s modafter:2010-06 -s nmodafter:2010-07 -s modar PATHS...
</code>

Negating a history selector produces a shallow selector: while <tt>modafter</tt> is history selector, <tt>nmodafter</tt> is shallow. But the order of the two in the previous command line is not important, as the last selector is the usual <tt>modar</tt>.

Selectors can be issued in other modes too. If the PO file is big and Alice has reviewed messages up to and including entry 246 when she has to pause until another day, she can commit reviews only up to this entry by issuing the <tt>espan</tt> selector:

<code bash>
$ poascribe ci -s espan::246 PATHS...
</code>

(the first parameter to <tt>espan</tt> is the first entry number, given if messages are not to be selected from the first). There is also the counterpart <tt>lspan</tt> selector, which works with referent line numbers (those of <tt>msgid</tt> keywords) instead of entry numbers.

=== Fine-Grained Reviews ===

In the introduction, several distinct types of what can go wrong in translation were described. Not all reviewers may be able to check translation against all those problems. Here is a typical scenario of this kind:

Alice is very computer-savvy and knows the translation project inside and out, which means that she can review well for context, terminology, and technical style. But, her language style leaves something to be desired, which shows through longer sentences and passages. Dan, on the other hand, is a very literary person, but not that much into the technical aspects. Dan's style reviews would thus be a perfect complement to Alice's general reviews.

<tt>poascribe</tt> can support this scenario in the following way. A ''review type'' tag for language style is defined in the ascription configuration, using the <tt>review-tags</tt> field:

<code text>
[global]
...
review-tags = lstyle
</code>

(The value to <tt>review-tags</tt> is a space-separated list of identifiers, when more than one special review type is needed.) With this addition to configuration, Alice can continue to review as she did before, without any changes to her workflow.

Dan selects messages for review similarly to Alice, but aditionally giving the <tt>lstyle</tt> tag as ''third'' parameter of <tt>modar</tt>, and indicating that ascribed reviews should be tagged as <tt>lstyle</tt>:

<code bash>
$ poascribe di -s modar:::lstyle -t lstyle PATHS...
</code>

After finishing the review, Dan commits as usual:

<code bash>
$ poascribe ci PATHS...
</code>

If Dan is always going to review the language style, in order not to have to issue the selector and tag in the command line all the time, he can make them default per mode in <tt>~/.pologyrc</tt>:

<code text>
[poascribe]
user = dan
selectors/diff = modar:::lstyle
tags/diff = lstyle
</code>

With this Dan can use plain <tt>poascribe di</tt> just like Alice does.

The important point of review tags is that they make reviews by types independent. For example, Dan may come around to review the language style of the given message after several modifications and general reviews have been ascribed to it -- <tt>modar:::lstyle</tt> will simply ignore all reviews except for <tt>lstyle</tt> reviews. This is going to be reflected in the <tt>ascto:</tt> comment to marked messages:

<code po>
#...
#. ascto: charlie:m alice:r bob:m
#...
msgid "..."
msgstr "..."
</code>

Here Alice has made one review between Charlie's and Bob's modifications, and that review, being general instead of <tt>lstyle</tt>, did not cause <tt>modar</tt> to stop at it. After Dan reviews this message for language style, Alice runs selection for review and gets this:

<code po>
#...
#. ascto: bob:m dan:r(lstyle)
#...
msgid "..."
msgstr "..."
</code>

Again, since <tt>lstyle</tt> reviews do not mix with general reviews, Dan's review did not hide Bob's modification that Alice did not check so far.

(General review too has a tag assigned, the empty string, in case the reviewer needs to explicitly issue it in some context.)

== Daily Use for The Coordinator ==

After setting up the ascription system, the team coordinator should have to do very little to maintain it.

=== Ascribing Merges ===

Modifications made to summit catalogs by merging with templates must also be ascribed. Therefore, after merging the summit (<tt>posummit ... merge ...</tt>) the coordinator substitutes the VCS command:

<code bash>
$ svn commit LANG/summit/messages/ -m "Merged summit."
</code>

with the <tt>poascribe</tt> command in <tt>commit</tt> mode:

<code bash>
$ poascribe ci LANG/summit/messages/ -m "Merged summit."
</code>

Since the user is not explicitly given by <tt>-u</tt> option, this will ascribe merge modifications to the coordinator (more precisely, to the user set as default in <tt>~/.pologyrc</tt>), which is just fine. It is also possible to define a special user only for ascribing merge modifications, though there is no known advantage to that.

Since <tt>-C</tt> option is not issued, <tt>poascribe</tt> will automatically commit all modified summit and ascription catalogs when done.

=== Shuffling Ascription Catalogs ===

Sometimes summit catalogs are shuffled in the repository: moved to another module, renamed, one catalog split into two, two catalogs merged into one. Such shuffling should be exactly mirrored in the ascription tree, and this too is done on the repository side, at the same time. This relies on the ascription root being set exactly to <tt>summit-ascript</tt> in the ascription configuration. So the team coordinator has nothing special to do here.

If instead in the central KDE repository the translation team is working in an external repository, by consequence the ascription system must be set up in that repository. But so long as <tt>process_orphans.sh</tt> script from <tt>trunk/l10n-support/scripts/</tt> is used to shuffle catalogs in the external repository as well, the ascription catalogs will be properly handled.

== Filtering for Release ==

The last component of the ascription system is how to prevent insufficiently reviewed messages from leaking into a release. In context of Pology and summit workflow, <tt>poascribe</tt> itself is not used directly to this end. Instead, in the summit configuration (as opposed to ascription configuration), the team coordinator defines filters which pass messages by applying selectors.

Each top level PO tree has its own summit configuration file, named <tt>MSGTREE.extras.summit</tt>:

<code text>
...
LANG/
summit/
messages/
messages.extras.summit
docmessages/
docmessages.extras.summit
</code>

For the simple case of all reviews being general reviews, the filter is added to summit configuration like this (anywhere within <tt>*.extras.summit</tt> file):

<code python>
S.ascription_filters = [
("regular", ["nmodar"]),
]
</code>

Here the filter is named <tt>regular</tt>, and is defined as application of <tt>nmodar</tt> selector, the negation of <tt>modar</tt>. This simply means: pass all messages ''not'' modified after the last review.

When the team coordinator scatters to branches (executes <tt>posummit scatter</tt>), messages from summit POs which do not pass this filter will not be sent to branch POs. The count of stopped messages by branch PO will be reported in the output as scattering proceeds.

Why did we have to name the filter <tt>regular</tt>? (Those knowing some Python will also notice that it is defined as a list element.) Because it is possible to define more than one filter, and select which one is used on each scattering. For example, the coordinator may wish that, when the release is near and time is short to review everything, messages from a few experienced translators can be passed into release without review. If those translators are Alice and Bob, an "emergency" filter can be defined like this:

<code python>
S.ascription_filters = [
("regular", ["nmodar"]),
("emergency", ["nmodar:~alice,bob"]),
]
</code>

By default, <tt>posummit</tt> uses the first filter in the list. When the coordinator needs to do emergency scattering, he requests the emergency filter by the <tt>-a</tt> option:

<code bash>
$ cd $KDEREPO/trunk/l10n-support
$ posummit scripts/messages.summit LANG scatter -a emergency
</code>

What if several selectors are needed to pass the message? For example, the language style review (the earlier example with Alice and Dan) too may be requested for regular scattering, but omitted from emergency scattering. The filter setup for this scenario looks like this:

<code python>
S.ascription_filters = [
("regular", ["nmodar", "nmodar:::lstyle"]),
("emergency", ["nmodar:~alice,bob"]),
]
</code>

The regular filter now reads: pass the message if it has not been modified after the last (general) review ''and'' has not been modified after the last style review.

Simple combination of predefined selectors by AND-conditions may not be sufficient for more involved scenarios. When this is the case, the coordinator may write (or ask someone to write) a custom selector in Python, and plug it in as the second element in the filter tuple (instead of the list of predefined selectors).

=== Writing a Selector Function ===

((To be written.))

Localization/Tools/Pology

2010-08-01T16:46:49Z

Ilic: Adaptations to changes in Pology directory structure.

{{Template:I18n/Language Navigation Bar|Localization/Tools/Pology}}

{{LocalizationBrowser|
section=[[Localization#Tools|Tools]]|
name=Pology|
prereqs=[[Localization/Tools/Gettext_Tools|Gettext Tools]]|
}}

== About ==

Pology is a Python framework for custom processing of PO files. It aims to facilitate easy, fast and robust creation of scripts for tackling problems encountered in the "field", everyday translation work, and to collect all sorts of specific, narrow purpose tools written in this direction. It does not aim to be a collection of several feature-rounded, monolithic, and general purpose tools (though it may contain some which could qualify). In particular, it does not attempt to handle any other translation formats but PO. All of Pology's end-user tools and programming interfaces are geared towards the PO format and conventions. The name itself should be parsed as PO-logy, "the study of POs".

At the moment, Pology still has not reached release state. But it can already be used effectively for day-to-day work, especially through end-user scripts. Obtaining and preparing Pology for use is simple: fetch its code repository, set <tt>PATH</tt> to use the scripts that come with it, and possibly set <tt>PYTHONPATH</tt> to be able to write own code based on Pology. The following commands should suffice:

<code bash>
$ svn co svn://anonsvn.kde.org/home/kde/trunk/l10n-support/pology
$ export PATH=$PWD/pology/bin:$PATH
$ export PYTHONPATH=$PWD/pology:$PYTHONPATH
$ . $PWD/pology/completion/bash/pology
</code>

Of course, for continuous use, environment variables should rather be set in <tt>~/.bashrc</tt> (or another shell configuration file). The last line sources programmable shell completion for some of Pology's commands. After these steps are successfully performed, Pology is fully prepared for use and scripting.

== Ready-Made Tools ==

Pology provides a number of tools for end use, with varying degrees of specificity, embodied as several scripts within <tt>scripts/</tt> subfolder of Pology's source. Details of operation of each script are provided within Pology documentation (<tt>doc/html/index.html</tt>, node <tt>pology.scripts</tt>), and the following sections give overview and some examples of their functionality.

=== Sieving ===

((To be done.))

=== Diffing and Patching ===

Line-oriented diffing and patching, as conducted by <tt>diff(1)</tt> and <tt>patch(1)</tt> commands, is not quite appropriate for PO files. Due to PO content being composed of variably-formatted multiline entries, which combine translator, programmer, and automatically controlled elements, line-oriented diff may indicate difference where semantically there is none, or not show real difference in a useful form. One could even claim line diffing of PO files to be almost useless, especially for the purpose of sending patches for translation.

For this reason, Pology contains two scripts, <tt>poediff</tt> and <tt>poepatch</tt>, which create message-oriented, ''embedded'' diffs of PO files. These diffs can be used both for reviewing changes and applying patches to PO files. Concept of embedded diffing and details on operation of the mentioned scripts are described in a [[Localization/Tools/Pology/PO_Embedded_Diffing|separate article]].

=== Reformatting ===

((To be done.))

=== Heavy Artillery ===

((To be done.))

== Writing Own Tools ==

Pology comes with detailed API documentation, but for a quick start into writing custom tools based on Pology, the following sections will describe and illustrate some of its more salient elements.

=== Catalogs and Messages ===

For an obligatory hello-world demonstration, let us create a PO template named <tt>hello.pot</tt> with a single message of this greet:

<code python>
from pology.catalog import Catalog
from pology.message import Message

cat = Catalog("hello.pot", create=True, truncate=True)
msg = Message()
msg.msgid = u"Hello, world!"
cat.add(msg)
cat.sync()
</code>

Most of these few lines are self-explanatory, except the last one: modifications to catalogs in Pology are never automatically written to disk, instead the <tt>sync()</tt> method must be called to initiate writes. Catalog is not gone after this, but you can continue to use it normally, including further syncings. In this example, after syncing the file <tt>hello.pot</tt> will be created in current working directory.

Practically, however, it is usually Gettext tools that will be used to create templates and catalogs, while a much more common use of Pology is to iterate over existing catalogs. The following code will open a catalog with various greetings, look for all messages that contain "hello" in the original text but do ''not'' contain "zdravo" in the translation, and report their content to standard output:

<code python>
from pology.catalog import Catalog
from pology.msgreport import report_msg_content

cat = Catalog("greets.po")
for msg in cat:
if "hello" in msg.msgid.lower():
matched = False
for text in msg.msgstr:
if "zdravo" in text.lower():
matched = True
break
if not matched:
report_msg_content(msg, cat)
</code>

Note how <tt>msgstr</tt> is represented as a list regardless of whether the message is plural or not, the difference being only in the number of elements. This removes the special case of singular/plural translations, and makes programmer always think of plural messages (though plural of original text is accessed through <tt>msgid_plural</tt> instance variable). Function <tt>report_msg_content</tt> will output the message to standard output, nicely formatted and preceded with a line stating the originating catalog and message's referent line and entry number in it. But <tt>report_msg_content</tt> can do much more, e.g. highlight parts of the message in the shell, add notes and delimiters, and so on (its API documentation provides all the details). Since no changes were done to the catalog, it is perfectly fine, even appropriate, not to call <tt>sync()</tt> at the end.

Of course, the previous snippet is just an illustration of iterating through catalogs and examining messages, in practice superfluous next to the functionality already provided by <tt>find-messages</tt> sieve:

<code bash>
$ posieve find-messages -smsgid:'hello' -snmsgstr:'zdravo' greets.po
</code>

((To be continued...))

=== Sieves ===

((To be done.))

=== Hooks ===

((To be done.))

=== Language Support ===

((To be done.))

Localization/Concepts/Transcript

2010-07-23T21:53:57Z

Ilic: Example of dynamic contexts with dependent labels.

{{LocalizationBrowser|
section=[[Localization#Concepts|Concepts]]|
name=Translation Scripting with Transcript|
prereqs=[[Localization/Concepts/PO_Odyssey|The PO Format]]|
extread=[http://developer.mozilla.org/en/docs/JavaScript JavaScript Guide and Reference]|
}}

== Translation Scripting? ==

Current state of affairs in localization of user-visible strings (messages) in application interfaces is such that translator is sometimes forced to supply an inadequate translation. This problem typically occurs when a message contains a placeholder to be substituted at runtime, or when two unrelated strings become related by placement in the interface. In either case, the modest requirements of english language on congruence of words in a sentence allow original string to remain grammatically correct, while not so in many other languages.

Translators and programmers sometimes try to work out a change in the code which would provide a more workable alternative. However, this process is difficult, and worse yet, the outcome is still language-dependent: solving the problem for a few languages does not necessarily solve it for all other.

One way to overcome these problems in a more general and compartmentalized manner, is to provide translators with a way to modify translated strings at runtime, depending on the context (eg. particular placeholder substitutes). In other words, to script translation. Translator should be able to operate on any interface string he wishes, while the programmer should not bear any extra burden (or know about translation scripting at all).

== The Transcript Engine ==

KDE4 comes ready with a translation scripting system, the ''Transcript''. Several strategic choices were made in its design:

* programmers are unaware of scripting, which means that translator can script any message without outside coordination

* unless translator wants to script a message, he is faced with familiar, standard Gettext PO environment (i.e. translators too ''can'' be scripting-agnostic)

* scripting is a low-level bolt-on to Gettext environment, to keep existing PO tools in the game

* to have enough power for unforeseen needs, a general-purpose scripting language is provided: JavaScript (with extensions for interfacing with Transcript)

{{note|For comparison, another translation scripting approach taking some different decisions (new translation environment down to the file format, scripting facilities more specialized, etc.) is brewing at http://wiki.mozilla.org/L20n}}

To script a particular message, translator writes short scripting calls into msgstr in the PO file which expand into parts of msgstr (''interpolations''), and JavaScript code which defines these calls into accompanying Transcript module file (eg. {{path|foo.po}} can be augmented with {{path|foo.js}}).

In case the application draws translations from several PO files, scripting calls defined in one of the Transcript modules are available in all used PO files. Since every KDE app uses {{path|kdelibs.po}}, calls defined in {{path|kdelibs.js}} are available everywhere.

The scripting process is illustrated by several examples. More detailed explanations of the elements are given in following sections.

=== A Useless Example ===

In Nevernessian it is impolite to speak out a greet with the same tone of voice throughout; instead, the name of the person must be shouted out. Hence, the translator wants to capitalize the placeholder substitute in the following login greet in {{path|neverness.po}}:

<code po>
#: neverness_login.cpp:10
msgid "Hello, %1!"
msgstr "Heelyy, %1!"</code>

So the translator adds a scripted msgstr, with an interpolation:

<code po>
#: neverness_login.cpp:10
msgid "Hello, %1!"
msgstr "Heelyy, %1!"
"|/|"
"Heelyy, $[shout %1]!"</code>

The first thing to note is that, while a bit longer, msgstr is still a proper PO msgstr, which means that it can be edited and processed by the usual PO tools.

The first part of msgstr is same as before, and called the ''fallback'' in this context: if the scripted part happens to fail in some way, the fallback translation is used. The fallback is followed by the ''fence'' <tt>|/|</tt>, which separates the fallback and scripted translation (and, for that matter, indicates that this message is scripted).

Finally, there is the scripted translation after the fence. Compared to fallback, it contains the interpolation <tt>$[shout %1]</tt>, which is supposed to evaluate to a capitalized version of the placeholder substitute. It is composed of the call name, <tt>shout</tt>, and one argument to it, the <tt>%1</tt> placeholder which will be replaced by its substitute. The syntax and expansion rules for interpolations are similar to Unix shell.

The call <tt>shout</tt> itself is defined in the Transcript module {{path|neverness.js}}, which contains only these lines:

<code javascript>
function capitalize (str) {
return str.toUpperCase();
}

Ts.setcall("shout", capitalize);</code>

Here the function <tt>capitalize</tt> is an ordinary JavaScript function which takes a string argument and returns all-caps version of it.

The link with the PO file is established by the call to <tt>Ts.setcall()</tt> -- the Transcript interface is represented by the property functions of the <tt>Ts</tt> object. In this variant, the <tt>Ts.setcall()</tt> takes the name of the call for the interpolations in the PO messages (a string), and the JavaScript function which will actually be invoked (''bound'' to the call).

That's it, now the fair Nevernesse folks are greeted properly.

=== Basic Case Resolution ===

One problem frequently encountered is wrong noun case when placeholder is substituted in the msgstr. For example, in many languages every KDE app has such a problem in the Help menu, with one or both of "About %1..." and "%1 &Handbook". This can be scripted in {{path|kdelibs.po}} like this:

<code po>
msgid "&About %1"
msgstr "&O %1"
"|/|"
"&O $[get-case dative %1]"</code>

The <tt>get-case</tt> interpolation is supposed to get the <tt>dative</tt> case of whatever app name the <tt>%1</tt> happens to be. The Transcript module {{path|kdelibs.js}} contains the definition of <tt>get-case</tt>, as well as the dictionary of cases:

<code javascript>
function getProperty (prop, key) {
return _dict_[key][prop];
}
Ts.setcall("get-case", getProperty);

_dict_ = {};
function addDictCases (key, gen, dat, acc, ins) {
if (!_dict_[key])
_dict_[key] = {};
_dict_[key]["genitive"] = gen;
_dict_[key]["dative"] = dat;
_dict_[key]["accusative"] = acc;
_dict_[key]["instrumental"] = ins;
}

// dictionary entries follow:
addDictCases("KWrite", "KWritea", "KWriteu", "KWrite", "KWriteom");
addDictCases("Konsole", "Konsole", "Konsoli", "Konsolu", "Konsolom");
...</code>

Function <tt>getProperty</tt>, bound to <tt>get-case</tt> call, simply returns the entry from the dictionary of forms. Function <tt>addDictCases</tt> is responsible for adding the static entries (name and its cases) into the dictionary, which is done in the final few lines for all apps of interest.

This completes the example, but for better modularization, it is also possible split out the dictionary insertion in a separate file, eg. {{path|appdict.js}}:

<code javascript>
// appdict.js
addDictCases("KWrite", "KWritea", "KWriteu", "KWrite", "KWriteom");
addDictCases("Konsole", "Konsole", "Konsoli", "Konsolu", "Konsolom");
...</code>

and use Transcript interface to load this file in the {{path|kdelibs.js}}:

<code javascript>
// kdelibs.js
...
...
...
Ts.load("appdict");</code>

Note that <tt>Ts.load()</tt> takes filename without extension, and assumes its location is relative to the folder of the parent file (ie. in this case {{path|kdelibs.js}} and {{path|appdict.js}} should be in the same folder).

=== Dynamic Case Setting ===

The previous scripted example solves the original problem, but introduces the burden of maintaining the dictionary insertion file. There is no way around this when the placeholder substitutes are "dead" strings from outside (eg. from {{path|.desktop}} files), but when they are coming from KDE's PO files at runtime, this burden can be removed.

The app name in KDE's Help menu indeed comes from the app PO file, and it is of course encountered at runtime before the menu strings come into focus. This allows setting the cases of app name in the PO msgstr which contains it. For example, {{path|katepart.po}} contains the "KWrite" string, and the forms could be set at that point:

<code po>
msgid "KWrite"
msgstr "KWrite"
"|/|"
"$[set-cases KWritea KWriteu KWrite KWriteom]"</code>

The <tt>set-cases</tt> is a ''side-effect'' interpolation: it should set the dictionary entries, but this particular message should in any case use the ordinary translation. Assuming all the definitions from previous example are still in effect, here is how <tt>set-cases</tt> could be defined in {{path|kdelibs.js}}:

<code javascript>
function dynamicSetCases (gen, dat, acc, ins) {
addDictCases(Ts.msgstrf(), gen, dat, acc, ins);
Ts.fallback();
}
Ts.setcall("set-cases", dynamicSetCases);</code>

In other words, this is little more than a wrapper to "static" <tt>addDictCases</tt> from previous example, but two new elements of Transcript interface appear. First is the <tt>Ts.msgstrf()</tt> function, which returns the finalized ordinary translation (placeholders substituted), and which is needed in this case as the dictionary key. Second is the <tt>Ts.fallback()</tt> function, which signalizes the Transcript engine to disregard the result of the scripted part of msgstr and use the ordinary translation.

Admittedly, the use of <tt>Ts.fallback()</tt> in this case is not necessary, but given for introductory purpose; <tt>dynamicSetCases</tt> might as well ''return'' the ordinary translation via <tt>Ts.msgstrf()</tt>.

== The PO Shell ==

This section gives the details of how the interpolations in the PO msgstr are expanded before evaluation.

The interpolations are parts of msgstr between <tt>$[...]</tt>, and are parsed into a number of strings. The first string is the name of the call registered in the scripting module via <tt>Ts.setcall()</tt>, and the rest are the arguments to bound JavaScript function. The arguments are typically passed as JavaScript type <tt>String</tt>, except in a special case detailed below.

The special characters in the interpolation are whitespace, single quote (') and backslash (\). Whitespace separates arguments, whereas single quote can be used for text which contains whitespaces. The backslash is used as escape; it can escape whitespace in non-quoted text, or single quotes in quoted text. This is pretty much like a typical Unix shell.

Double quotes are not special. Single quotes are used instead of double quotes because it makes it easier to edit interpolations in PO files, where double quotes would have to be escaped. This also means that when escape is needed in the interpolation, it must be escaped once itself for the PO msgstr.

The biggest difference from the shell expansion is that unlike with shell variables, the placeholders are expanded such that no characters within them are treated as special. Otherwise, many tricky problems could arise with whitespace or single quotes contained inside.

The call name bound to a JavaScript function using <tt>Ts.setcall()</tt> does not have to be a proper JavaScript identifier, but any Unicode string not containing the interpolation-special characters. This means that more "natural" call names can be used inside the msgstr, like those with dashes or non-Latin1 characters. E.g. call names can be in the language of the PO file, for an aesthetic impression.

Sub-interpolations, <tt>$[... $[...] ...]</tt>, are also possible. If put inside inside single quotes, they will be treated as ordinary text. Thus, if a literal closing square bracket is needed as an argument inside the interpolation, it can be given within single quotes.

In case that the argument is given as <tt>^''number''</tt>, it will evaluate to the value supplied for the corresponding placeholder -- unlike the <tt>%''number''</tt> form, which evaluates to the placeholder substitution string. Also, the argument type as seen by JavaScript need no longer be <tt>String</tt>, but will correspond to the value substituted (e.g. <tt>Number</tt>). This distinction may be important in some situations. Even if the value is originally a string, the substitution string may have extra formatting (padding, tags...), which may not be desirable. In particular, the number substitutes may come tagged or locale-formatted, such that they cannot be easily parsed back into <tt>Number</tt>. In these cases, <tt>^''number''</tt> form can be used to get the raw values.

== The Transcript Interface ==

Transcript provides extensions to the JavaScript, which interface with the PO file and the Transcript environment. They are all function properties of the <tt>Ts</tt> object, accessible as <tt>Ts.''func''(''args'')</tt>.

; <tt>setcall (''name'', ''func'')</tt>
; <tt>setcall (''name'', ''func'', ''obj'')</tt>
: Binds the call name to the JavaScript function, for use in the interpolations inside the PO file.
:: <tt>''name''</tt> name of the call. Can be any Unicode string, for ease of use in the msgstr
:: <tt>''func''</tt> function object
:: <tt>''obj''</tt> object to act as <tt>this</tt> inside the function; if omitted, <tt>this</tt> refers to global object
: Returns <tt>Undefined</tt>.

; <tt>hascall (''name'')</tt>
: Check if the call for use in PO file has been set.
:: <tt>''name''</tt> name of the call
: Returns <tt>true</tt> if set, <tt>false</tt> otherwise.

; <tt>acall (''name'', ''arg''*)</tt>
: Applies previously set PO call to a list of arguments, within the scripting module itself. Indirect calls can be made this way, e.g. by passing a call name as a string within PO file interpolation.
:: <tt>''name''</tt> name of the call
:: <tt>''arg''*</tt> arguments to the call
: Returns what the indirect call would return.

; <tt>load (''file''*)</tt>
: Evaluates the code in the specified files, in the left to right order. File paths are expected to be relative to current module's folder.
:: <tt>''file''</tt> file name without extension
: Returns <tt>Undefined</tt>.

; <tt>fallback ()</tt>
: Forces Transcript to use ordinary translation, regardless of whether the interpolation evaluates successfully or fails. The evaluation of the script is not aborted, any other interpolations will evaluate too.
:  
: Returns <tt>Undefined</tt>.

; <tt>msgid ()</tt>
: Returns msgid of the last message, with placeholders intact.

; <tt>msgstrf ()</tt>
: Returns finalized ordinary translation of the last message, with placeholders substituted.

; <tt>msgctxt ()</tt>
: Returns msgctxt of the last message, with placeholders intact.

; <tt>msgkey ()</tt>
: Returns a <tt>String</tt> which is implementation-dependent combination of msgctxt and msgid with placeholders intact. Used to uniquely identify the message within the PO file, usefull as a dictionary key.

; <tt>nsubs ()</tt>
: Returns the number of substitutes provided for placeholders in the last message. It is equal to the highest-numbered placeholder for a proper i18n call in the application code, but i18n calls do not have to be quite proper.

; <tt>subs (''index'')</tt>
: Used to access values of placeholder substitutes provided to the last message. Numbering is zero-based.
:: <tt>''index''</tt> index of placeholder substitute
: Returns <tt>String</tt>, regardless of the value type substituted in the application code.

; <tt>vals (''index'')</tt>
: Used to access values of placeholder values provided to the last message. Numbering is zero-based.
:: <tt>''index''</tt> index of placeholder value
: Returns the matching JavaScript type to the value type used in the application code. E.g. strings will be <tt>String</tt>, integers and doubles <tt>Number</tt>. May also return <tt>Undefined</tt>, in case the value cannot be represented as a reasonable JavaScript type.

; <tt>dynctxt (''key'')</tt>
: Programmers may append ''dynamic'' context to the message, which is a pair of strings: a context key and its value. Use this function to retrieve such contexts.
:: <tt>''key''</tt> context key string
: Returns <tt>String</tt>, or <tt>Undefined</tt> if dynamic context with the given key does not exist.

; <tt>dbgputs (''msg'')</tt>
: Outputs a debug message in the shell when KDE has been compiled with debug option.
:: <tt>''msg''</tt> message string
: Returns <tt>Undefined</tt>.

; <tt>setcallForall (''name'', ''func'', ''obj'')</tt>
; <tt>setcallForall (''name'', ''func'')</tt>
: Similar to <tt>setcall()</tt>, but the function is executed on every message after it has been finalized (if the message was explicitly scripted, the function is executed after the script has been evaluated). The function receives no arguments, and its return value is ignored, i.e. it cannot change the finalized message; the call is used for side-effects. The <tt>''name''</tt> parameter is used only for reporting errors. When several <tt>setcallForall()</tt> have been issued, the functions are invoked in the order of issue.
:  
: More precisely, these calls are made on every message ''after'' the Transcript engine has been initialized, which happens on first explicitly scripted message. Thus, if earlier application of the calls is necessary, Transcript can be jump-started by scripting one early encountered message with only a single empty interpolation: <tt>msgstr "...|/|$[]"</tt>; this will do nothing for that message, but it will start the engine.
:  
: Returns <tt>Undefined</tt>.

; <tt>toUpperFirst (''text'')</tt>
: Converts the first letter in the string into upper case, even if the first character in the string is not a letter. Unicode specification is used to determine what a letter is (i.e. works for non-ASCII letters).
:: <tt>''text''</tt> string to process
: Returns <tt>String</tt>.

; <tt>toLowerFirst (''text'')</tt>
: Converts the first letter in the string into lower case, under same assumptions as for <tt>toUpperFirst</tt>.
:: <tt>''text''</tt> string to process
: Returns <tt>String</tt>.

; <tt>loadProps (''file''*)</tt>
: Loads [[#Property Maps|property maps]] from the specified files. File paths are expected to be relative to current module's folder.
:: <tt>''file''</tt> file name without extension
: Returns <tt>Undefined</tt>.

; <tt>getProp (''phrase'', ''prop'')</tt>
: Fetches the property value of the phrase, as defined by [[#Property Maps|property map]].
:: <tt>''phrase''</tt> text for which the property is requested
:: <tt>''prop''</tt> property key string
: Returns <tt>String</tt>, or <tt>Undefined</tt> if the given text has no such property.

; <tt>setProp (''phrase'', ''prop'', ''value'')</tt>
: Sets the property value of the phrase. The phrase and property key will be automatically normalized, as they would have been when loaded from a [[#Property Maps|property map]] file.
:: <tt>''phrase''</tt> text for which the property is set
:: <tt>''prop''</tt> property key string
:: <tt>''value''</tt> value of the property (string)
: Returns <tt>Undefined</tt>.

; <tt>normKey (''phrase'')</tt>
: Normalizes the text by removing all whitespace, removing the accelerator, and lowercasing it, e.g. for use in near-match lookups.
:: <tt>''phrase''</tt> text to be normalized
: Returns <tt>String</tt>.

; <tt>getConfString (''key'')</tt>
; <tt>getConfString (''key'', ''value'')</tt>
; <tt>getConfBool (''key'')</tt>
; <tt>getConfBool (''key'', ''value'')</tt>
; <tt>getConfNumber (''key'')</tt>
; <tt>getConfNumber (''key'', ''value'')</tt>
: Fetches a value of the requested type from the [[#User Configuration|user-configuration]] file. A default value of the same type can be stated too (or else it defaults to <tt>undefined</tt>).
:: <tt>''key''</tt> the name of the field in the configuration file
:: <tt>''value''</tt> the default value, in case the key is not found
: Returns the configuration value if the key is found, or the default value or <tt>undefined</tt> otherwise. If the configuration string is not a valid representation of the requested type, again the default or <tt>Undefined</tt> is returned instead.

=== Property Maps ===

There is a frequent need to have different properties attached to the particular pieces of text. For example, case, gender, etc. may be considered as properties of a noun, or phrase with a subject/object role in a sentence. Sometimes it may be convenient (or necessary) to define and maintain the phrases and their properties in an external file. ''Property map'' is Transcript's built-in (i.e. efficient) way to handle such definitions.

Property map files are text files in a simple dictionary format for writing down phrases with their properties (filenames must end in <tt>.pmap</tt>). The format is best shown on the example:
<code text>
# cities.pmap
=:Athens:Atina:nom=Atina:gen=Atine:dat=Atini:acc=Atinu::
=:Paris:Pariz:nom=Pariz:gen=Pariza:dat=Parizu:acc=Pariz::
</code>
Each entry starts with two characters, <tt>=:</tt> in this example, which may be chosen arbitrarily (but must be non-letters, and non-#), and can change from entry to entry. The first character defines the separator in the property key-value pair, and the second character is the separator of pairs. E.g. the <tt>gen=Atine</tt> segment above is defining the genitive case of the localized form of Athens. The "pairs" which actually lack the property key (first two in the entries above) are considered phrase identifiers. The completely empty pair (<tt>::</tt> in the example) terminates the entry.

The property map is loaded inside the scripting module using the <tt>Ts.loadProps()</tt> call. The map file should normally reside in the same folder as the scripting module that uses it; if so, the map from the above example is loaded simply with <tt>Ts.loadProps("cities")</tt>. The property values are obtained in the scripts using <tt>Ts.getProp()</tt> call. E.g. the genitive form of Athens would be obtained either by <tt>Ts.getProp("Athens", "gen")</tt> or <tt>Ts.getProp("Atina", "gen")</tt> -- because the map defined both as phrase identifiers for Athens. The phrase/key lookup is performed using normalized values; this normalization can be checked or used for own purposes by <tt>Ts.normKey()</tt> call.

The comments are a bit of an issue in property map files. Since the phrases and property values can be anything (hence the possibility to choose separators), fixing any single character as your typical to-the-end-of-line comment would introduce limitations. Instead, the #-comments are allowed only ''between'' the entries, span to the end of line, and # cannot be a separator character.

The whitespace in property values is mostly preserved (for the keys it does not matter, as it gets stripped for lookups). However, if the leading sequence of whitespace contains a newline, all whitespace to the first newline and including it is stripped; symmetrically, in a trailing whitespace sequence, the last newline (if any) and the whitespace following it are removed.

=== User Configuration ===

Using a configuration file located at {{path|$HOME/.transcriptrc}}, the user can be allowed to configure certain aspects of Transcript operation. The configuration file is in ini-style, where group names are language codes to which the selections apply. For example:
<code text>
# Settings for German.
[de]
serve-sausages = no
wine-instead-of-beer = yes

# Settings for Italian.
[it]
with-mozzarella = no
</code>
Within scripting modules, the configuration values can be fetched using <tt>Ts.getConf*()</tt> series of calls. These calls will automatically look for the configuration fields under the proper language group, as the scripting modules are language-dependent too.

For boolean-valued keys, <tt>0</tt>, <tt>no</tt>, and <tt>false</tt> can be used to indicate negative (false) values, and everything else is treated as positive (true). Also, the boolean values are not case sensitive.

== Repository Organization ==

For the PO file in the KDE repository {{path|LL/messages/kdemodule/foo.po}}, the corresponding Transcript module should be located at {{path|LL/scripts/kdemodule/foo/foo.js}}. Note the extra subfolder in the module path, named like the basename of the PO/JS file. This subfolder is introduced because the module's main JS file might need other supporting files, which can then be located conveniently in the same folder.

The {{path|autogen.sh}} script, used for some time already to generate build system support for PO files, will also generate build support for Transcript modules. In particular, it must be rerun whenever a new module subfolder is added (e.g. the {{path|LL/scripts/kdemodule/foo}}), but not when particular files within it are added or removed. Every file in the module subfolder will be installed automatically, so put there only what is needed at runtime.

== Real-Life Examples ==

=== CJK Accelerator Keys ===
In CJK environment, accelerator keys are wrapped in the parenthesis. For example, translation of "&File" in Korean is "파일(&F)", and in Japanese, "ファイル(&F)". When it comes to the names of action, it is shared among toolbar icon names, menu bar, and so on.

However, in toolbar icon name, "Configure shortcut" dialog, etc., those access keys are remaining in the name, makes the name little bit awkward. In this situation, we can use transcript to get rid of those accelerator keys. Currently, the code is in Japanese and Korean kdelibs4.js file.

=== Korean Postpositions ===
In Korean language, postpositions are widely used. Among those, 8 postpositions are in pair and they change forms according to the word. Because PO file itself doesn't know what the substituted word(like %1) is, proper postpositions couldn't be added to the string.

Fortunately, the rule is programmable, and proper postpositions could be added via transcript. Right now a little part of postposition rules is programmed into Korean version of kdelibs4.js.

=== Automatic Property Calls ===
Leveraging [[#Property Maps|property maps]] interface, we can make a generalization of the [[#Dynamic Case Setting|dynamic case setting]] example. We want to set named properties by key-value pairs for a given message, and then, to use calls named as property keys to retrieve the values when that message is used as placeholder replacement in another message.

For example, Nevernessians need to match the noun case of "baloon" when it is inserted into sentence "I see a %1", and additionally the adjective form of "red" to baloon in "I see a red %1", so they do:
<code po>
msgid "baloon"
msgstr ""
"beeluun"
"|/|"
"$[properties see-it beeluuno see-red raado]"

msgid "I see a %1"
msgstr ""
"O sii e %1"
"|/|"
"O sii e $[see-it %1]"

msgid "I see a red %1"
msgstr ""
"O sii e raad %1"
"|/|"
"O sii e $[see-red %1] $[see-it %1]"
</code>

The <tt>properties</tt> call, which does all the work -- setting properties, setting new calls by property keys -- looks like this:
<code javascript>
// Set properties of the phrase given by the finalized msgstr in the PO file.
// The arguments to the call are consecutive pairs of keys and values,
// as many as needed (i.e. total number of arguments must be even).
//
// The property keys are registered as PO calls taking single argument,
// which can be used to retrive the property values for this msgstr
// when it is later used as placeholder replacement in another message.
//
// Always signals fallback.
//
function setMsgstrProps (/*KEY1, VALUE1, ...*/)
{
if (arguments.length % 2 != 0)
throw Error("Property setter given odd number of arguments.");

// Collect finalized msgstr.
phrase = Ts.msgstrf()

// Go through all key-value pairs.
for (var i = 0; i < arguments.length; i += 2) {
var pkey = arguments[i];
var pval = arguments[i + 1];

// Set the value of the property for this phrase.
Ts.setProp(phrase, pkey, pval);

// Set the PO call for getting this property, if not already set.
if (!Ts.hascall(pkey)) {
Ts.setcall(pkey,
function (phr) { return Ts.getProp(phr, this.pkey) },
{"pkey" : pkey});
}
}

throw Ts.fallback();
}
Ts.setcall("properties", setMsgstrProps);
// NOTE: You can replace "properties" in the line above with any UTF-8 string,
// e.g. one in your language so that it blends nicely inside POs.
</code>

=== Labels Dependent on Dynamic Values ===
Sometimes the text of a standalone label should grammatically conform to the a changing value elsewhere in the interface. The typical example of this is when a sentence-like row of different widgets is used to specify an action:
<code text>
Search in files modified:
( ) ...
(*) during the last {day|week|month|year}
( ) ...
</code>
<tt>( )</tt> stand for radio buttons, and <tt>{...|...|...}</tt> for a list box with several choices. The label <tt>"during the last"</tt> contains the adjective "last", which in some languages needs to have different forms according to the noun it describes (one of the choices from list box). Whenever the list box value changes, the label should change to match it. But this label is a standalone message in the PO file, so based on what can its translation be scripted?

The solution is to ask the programmer to make the label refresh on every change of the list box value, and to add a ''dynamic context'' to the label's message (using <tt>inContext</tt> method of <tt>KLocalizedString</tt>). The dynamic context is a key-value string pair which indicates the current list box value; the key and possible values should be documented in the message's context. If the programmer did this properly, the translator should now see in the PO file:
<code po>
msgctxt ""
"Part of logical sentence 'Search in files modified >during the last< "
"day|week|...'; provides dynamic context 'last-what': 'd' for day, "
"'w' for week, 'm' for month, 'y' for year."
msgid "during the last"
</code>
The translation can now be scripted based on the <tt>last-what</tt> dynamic context:
<code po>
msgstr ""
"duureeng thee leestee"
"|/|"
"duureeng $[by-context last-what 'd|m' 'thee leestee' 'w|y' 'thaa leestaa']"
</code>
The <tt>by-context</tt> call takes the context key, followed by pairs of context value matches (regular expressions) and corresponding phrases; in this example, one form is selected for context values <tt>d</tt> and <tt>m</tt>, and another form for <tt>w</tt> and <tt>y</tt>. The definition of this call is:
<code javascript>
// Select a phrase according to dynamic context.
// The first argument is the context keyword,
// followed by arbitrary numer of pairs of context values and
// corresponding phrases, optionally followed by default phrase.
//
// Context values are actually regular experessions, so that if one phrase
// corresponds to several context values, it does not have to be repeated
// but its context value can be given e.g. as 'foo|bar|...'.
//
// If the context was not matched, default phrase is returned if
// it was given; otherwise fallback is signaled.
//
function selectByContext (/* ctxt_key,
valrx_1, phrase_1, ..., valrx_N, phrase_N
[, default_phrase] */)
{
if (arguments.length < 1) {
throw Error("Selector by context needs at least one argument.");
}

// Collect context value for the given key.
var ctxtkey = arguments[0];
var ctxtval = Ts.dynctxt(ctxtkey);

// Match the context and select the corresponding phrase.
var phrase;
for (var i = 1; i < arguments.length; i += 2) {
if (ctxtval.match(RegExp(arguments[i]))) {
phrase = arguments[i + 1];
break;
}
}

// If context was not matched, select default phrase or signal fallback.
if (phrase == undefined) {
if (arguments.length % 2 == 0) {
phrase = arguments[arguments.length - 1];
} else {
throw Ts.fallback();
}
}

return phrase;
}
Ts.setcall("by-context", selectByContext);
// NOTE: You can replace "by-context" in the line above with any UTF-8 string,
// e.g. one in your language so that it blends nicely inside POs.
</code>

Localization/Workflows/PO Ascription

2010-06-19T13:55:21Z

Ilic: ascription-config file moved one level up, with implied adjustments.

{{Template:I18n/Language Navigation Bar|Localization/Workflows/PO_Summit}}

{{LocalizationBrowser|
section=[[Localization#Workflows|Workflows]]|
name=Reviewing by Ascriptions|
prereqs=[[Localization/Workflows/PO_Summit|Translating in Summit]]|
related=[[Localization/Tools/Pology|Pology]], [[Localization/Workflows/Coordinator|Language Coordinator]]|
}}

== Why Review Translations? ==

Especially to new translators, it may not be obvious to which extent the translation needs to be reviewed. If the translator has exercised due diligence, how "wrong" can the translation be? Even if the translator has good command of the source language (English in context of this article), the answer is "very wrong", when all aspects are considered. Here are some of them.

With comparatively simple grammar of English, the meaning of a short English sentence -- as typically encountered in application user interfaces -- is very dependent on the surrounding context. This context may not be obvious when the translator is going through isolated messages in the translation file, so he may commit the worst of errors from the user's viewpoint, the senseless translation. An experienced reviewer will have developed sense for troublesome contexts, and will have several means to decisively determine the context (including, for example, running the development version of the application).

Even if the context is correctly established, the translator may use "wrong" terminology, which is the next worse thing for the user. A term used in translation does not need to be wrong by itself, in fact it may be exactly the correct term -- in another translation project. The reviewer will have more experience with terminology of the present project, and be able to bring the translation in line with it.

Style in the technical sense is a consistent choice between several perfectly valid constructs in target language when applied to text in the given technical context. For example, how to translate menu titles and items, button labels, or tooltips. The choices may include noun or verb forms, particular grammar categories, tone of address, and so on. There may be a style guide to the project which details such choices, and the reviewer will know it well.

Style in the linguistic sense is especially applicable to longer texts, such as tooltips in user interfaces, and passages in documentation. A typical error of a new translator is to closely adhere to English style and grammar. This may produce translation which is semantically and grammatically valid in the target language, but very out of style -- the "translationese". Reviewer is there to naturalize such passages.

Finally, the reviewer may be an experienced translator, but that does not mean that his own translations need no review. Immersion into the source language, distraction, fatigue, will lead the reviewer into any of the above errors in translation, only with less frequency. So reviewers should also review one anothers' translations.

== Classical Reviewing by Stages ==

Classical review workflow by stages seems simple enough. Translator translates a PO file (or updates existing translation), and declares it ready to review. A reviewer reviews it, and declares it ready to "commit". Committing here should be understood generally, as inclusion into the pool from which translations are periodically shipped to end users. A committer finally commits the file. The process is iterative: the reviewer may return the file to the translator, and translator later again declare it as ready for review. There may be several stages of review (such as ''proof-reading'', ''approving''), each of which may return the translation to a previous stage, or forward it to some special stage. The process can also be more finer grained, where each message in the file goes through stages separately.

Regardless of the particularities, workflows of this kind all have the following in common. Members of the translation team are assigned ''roles'' -- such as ''translator'', ''reviewer'', ''approver'', ''committer'' -- by which they enter into the workflow (single person can have more roles). The later review stages must wait for the earlier stages to complete, and the translation cannot be updated again before the current version clears the pipeline (or the pipeline is aborted). Most importantly, once the translation is committed, it becomes part of simply "admitted" translations, with no further qualifiers.

The system of prescribed roles requires that team members assign them between themselves, stick to them, and shuffle them along the way. The prescribed review pipeline requires a tool to enforce and keep track of the stages in which translations are. This makes the review workflow complex and rigid, most probably with choke points for efficiency. Distribution of roles may become disbalanced by people coming and going, or the workflow tool may be prohibitive to some scenarios (e.g. single translator making small adjustments in dozens of files across the project, but having to upload each manually through a web interface).

Of course, "rigid", "complex", "inefficient", are comparative qualifications, so what is it that the classical review by stages can be compared to in this way?

== Reviewing by Ascriptions ==

Reviewing by ascriptions is even simpler conceptually, and yet less rigid, less complex, and much more efficient than the review by stages. It works on the message-level, rather than file-level. Anyone can simply translate some messages and directly commit modified files, without any review, but with ''ascribing'' modifications to own name. Anyone can review any committed messages at any moment, commit the modifications-on-review and ascribe reviews to own name and (possibly) to certain class -- full review, review of context, of terminology, of style, etc. Only when the translation is to be shipped to end users, the ''insufficiently'' reviewed messages are automatically omitted from the package, by evaluating the ''ascription history'' of each message.

Most importantly, based on the ascription history, the reviewer can select only some particular messages, and review only the difference between their historical and current versions. For example, Alice can select to review only messages modified since she or Bob had last reviewed them for style; she could see the difference from that last review to current version, e.g. if in the whole paragraph only a single word has changed by Charlie when he reviewed the terminology. In terms of PO workflow, the ascription history propagates through merges, so the reviewer can compare the change in original and the change in translation since the last review, to judge if one fits the other.

Since everyone just commits, translations can be efficiently kept in a version control repository, with the ascription system added on top. After having done some translating, the team member simply substitutes commit command of the version control system (VCS) with ascribe-modifications command of the ascription system (AS, which calls the underlying VCS internally). After reviewing, the team member uses ascribe-reviews command of the AS to commit reviews to ascription history (as well as modifications made during the review). To select messages for review, the team member issues diff-for-review command of the AS (with suitable parameters to narrow the set) and selected messages are marked in-place in PO files and [[Localization/Tools/Pology/PO_Embedded_Diffing|embedded with differences]], and possibly popped open in a PO editor.

When the translations are to be released, the team coordinator issues filter-for-release command of the AS, which takes the working PO files and creates final PO files with insufficiently reviewed messages removed. "Release time" is used here only figuratively: this should be a fully automatic process, so it can be performed at any interval of convenience.

What constitutes "sufficient review" can be defined in fine detail. It could be specified that messages modified by Alice need to have only review for terminology, but not necessarily for style; Charlie may belong to the group which needs to be reviewed on style, but not necessarily on context; Bob's reviews for style may be nice to have, but never blocking if missing. These decisions do not preclude released messages to be reviewed later on missing points, after higher priority reviews have been completed. The definition of sufficiency may be changed at any point, e.g. as team members get more experienced and require less review, without interfering with direct translation and review work.



In summary, with reviewing by ascriptions the lean efficiency of raw VCS operation is preserved while providing for great flexibility of review. All team members can be given commit access, no web or email detours are needed. There are no prescribed roles, but an equivalent of role assignment happens automatically at last possible moment, and can take into account both translators' and reviewers' abilities. There is no staging between completing and committing the translation, which enables translator to keep on polishing the translation undisturbed until the reviewer comes around. There is no inefficiency in handling small changes throughout many files, since single AS command commits all changes just as single VCS command would. AS in effect abstracts VCS, so general team members do not have to know the particularities of the underlying VCS. On commit operations, AS can also apply checks (e.g. decline to commit syntactically invalid PO files) and modifications (e.g. update translator's data in the PO header).

== Ascription System in Pology ==

[[Localization/Tools/Pology|Pology]] is a collection of various modular tools for supporting translation based on PO files. Among them is the script <tt>poascribe</tt>, which implements an ascription system (AS); at present, it can use Subversion or Git as the underlying VCS. <tt>poascribe</tt> is still in experimental stage, so what follows is a brief description of how to use it in context of the KDE translation project. However, very little is truly specific to KDE; the only major assumption is that there exists a VCS repository with PO files of a given language grouped together, and that the translation team can use it without special restrictions.

Very important for the AS is how branches are handled (in KDE, the rolling trunk and stable branches). AS can in principle be deployed by branch, but then there is the added complexity of porting translations between branches, which ascriptions should follow. Therefore, the AS implemented by <tt>poascribe</tt> is currently limited to assumption that there is a single branch of translations at all times. The article [[Localization/Workflows/PO_Summit|"Translating in Summit"]] explains how a KDE translation team can set up and operate such a single branch, the summit, and this is the prerequisite for the following instructions. (Note that the summit system is useful on its own, and should be conductive to any kind of review workflow.)

== Setting Up ==

The summit branch for the language <tt>LANG</tt> is positioned like this in the KDE repository:

<code text>
$KDEREPO/
trunk/
l10n-support/
LANG/
summit/
messages/
docmessages/
</code>

The team coordinator already has this part of the repository tree locally due to regular summit operations. For the same reason Pology is already set up. Setting up the ascription system is now simple. The file <tt>ascription-config</tt> is created in the parent directory of the summit:

<code text>
...
LANG/
ascription-config
summit/
messages/
docmessages/
</code>

with the following contents:

<code text>
# ---------------------------
# Global ascription settings.

[global]

# Roots of the catalog and ascription trees.
catalog-root = summit
ascript-root = summit-ascript

# The underlying version control system.
version-control = svn

# Data for updating catalog headers.
# - language code
language = LANG
# - full language name
language-team = LANGUAGE
# - email address of the team
team-email = kde-i18n-LANG@kde.org

# Default commit message.
commit-message = Translation updates.

# -----------------------
# Registered translators.

[user-alice]
name = Alice Akmalryn
original-name = Алиса Акмалрин
email = alice.akmalryn@someplacenice.org

[user-bob]
name = Bob Byomkin
original-name = Бобан Бјомкин
email = bob.byomkin@otherplacenice.org

# ...and so on.
</code>

Some notes:

* The <tt>ascript-root</tt> setting should be exactly <tt>summit-ascript</tt>, for the reason mentioned later.

* <tt>commit-message</tt> field, if defined, allows team members to commit without providing a commit message. The value given by this field will be used by default, with translator's user name appended to the end in special syntax. For example: <tt>Translation updates. [>alice]</tt>. (Translator's user name is also appended to manually supplied commit messages.) Translators can still supply a commit message when they wish, as shown later. If this field is not set, the commit message is supplied as usual on committing.

* Team members are defined by <tt>[user-USERNAME]</tt> sections. Ascription user names can be any valid ASCII identifier: ASCII letters, digits and underscores only, digit cannot be the first character. Ascription user names have no technical relation to the underlying VCS accounts, though it is mnemonically convenient if they are the same (in case of SVN). This means that a translator who does not have a VCS account (yet) can and should be added here, with assigned user name (best one suitable as SVN account name later); why this should be done will be explained later.

* <tt>original-name</tt> field in user sections is there in case the preferred renderings of the name in English and in target language are not the same. When this is not the case, <tt>original-name</tt> can be omitted.

As soon as the <tt>ascription-config</tt> file is committed, the ascription system is ready for operation. Only regular modifications to this file are those of adding new team members. (On the other hand, team members should never be removed, because even after they no longer contribute, their ascription records remain in the system.)

=== Initial Ascription ===

The most common situation at start of ascription workflow is that there already exists a body of translations, contributed to by many different people over time. The coordinator should ascribe all existing translations as initial modifications, but to whom? It cannot be said precisely who translated what. The solution is to introduce a generic user in <tt>ascription-config</tt>, suitably known as "Unknown Hero" (or "Lost Translator", you can be inventive):

<code>
[user-uhero]
name = Unknown Hero
original-name = Незнани јунак
</code>

and ascribe all existing translations as modified and reviewed by this user. The coordinator does this with the following command:

<code bash>
$ cd $KDEREPO/trunk/l10n-support
$ poascribe commit -u uhero --all-reviewed -C LANG/summit/
</code>

The argument <tt>commit</tt> is the ascription mode, and the <tt>-u</tt> option provides the user name to which ascriptions are made. This is an important point: ascriptions are made to a user defined in ascription configuration, and have nothing to do with VCS accounts; someone who has the account can commit in the name of someone who does not. It is the <tt>--all-reviewed</tt> option that declares all messages to be reviewed as well (note that it is normally used only this once, and not for normal day to day reviewing). The <tt>-C</tt> option prevents automatic adding and committing to version control, which is useful for this initial step. Finally the paths which contain all summit catalogs are given.

When the <tt>poascribe</tt> command is issued, a progress bar will appear, and the following output will start to unfold:

<code text>
LANG/summit/messages/extragear-base/rellinks.po (50/50)
LANG/summit/messages/extragear-base/autorefresh.po (13/13)
LANG/summit/messages/extragear-base/babelfish.po (38/38)
...
LANG/summit/messages/qt/libphonon.po (13/13)
LANG/summit/messages/qt/phonon-xine.po (24/24)
LANG/summit/messages/qt/phonon_gstreamer.po (12/12)
===== Ascription summary:
- modified reviewed
translated 111775 111775
fuzzy 26943 26943
obsolete/t 2965 2965
obsolete/f 1626 1626
</code>

The number in parenthesis indicates how many messages have been ascribed in the given PO file (modified/reviewed), and at the end the totals are given. Ascribing the complete summit for the first time will take quite some time (on the order of 10-20 minutes).



After the initial ascription has been made, the ascription tree will appear next to the summit tree. This tree will contain one ascription PO file for each summit PO file, with the same name and relative location within the tree:

<code text>
...
LANG/
ascription-config
summit/
messages/
kdelibs/
kcertpart.po
kdelibs4.po
...
...
docmessages/
summit-ascript/
messages/
kdelibs/
kcertpart.po
kdelibs4.po
...
...
docmessages/
</code>

During the ascription some summit PO files may have been modified as well, in that any previous fields (<tt>#| ...</tt>) on translated messages have been removed. (These fields are sometimes erroneously left in by older PO editors.)

The newly created ascription tree (and any modifications to summit tree) can now be committed as usual:

<code bash>
$ svn add LANG/summit-ascript
$ svn commit LANG/summit LANG/summit-ascript -m "Initial ascription."
</code>

== Daily Use for Translators ==

Team members other than the coordinator, whether translators or reviewers, need to keep around only the <tt>trunk/l10n-support/LANG/</tt> directory. But they always need to update this directory fully (rather than just one particular module or file under <tt>.../*messages/</tt>), so that the summit tree and the ascription tree (and configuration) are kept in sync.

In order not to have to issue their own user name (<tt>-u</tt> option to <tt>poascribe</tt>) all the time, translators can set it in Pology user configuration <tt>~/.pologyrc</tt>, in <tt>[poascribe]</tt> section:

<code text>
[poascribe]
user = alice
</code>

Translators can then submit updated PO files simply by substituting <tt>svn commit</tt> (or whatever the VCS commit command is) with <tt>poascribe commit</tt> (<tt>co</tt> or <tt>ci</tt> for short):

<code bash>
$ poascribe ci LANG/summit/messages/kdefoo/*fooapp*.po
LANG/summit/messages/kdefoo/fooapp.po (144)
LANG/summit/messages/kdefoo/libfooapp.po (25)
===== Ascription summary:
- modified
translated 169
>>>>> VCS is committing catalogs:
Sending LANG/summit/messages/kdefoo/fooapp.po
Sending LANG/summit/messages/kdefoo/libfooapp.po
Sending LANG/summit-ascript/messages/kdefoo/fooapp.po
Sending LANG/summit-ascript/messages/kdefoo/libfooapp.po
Transmitting file data ....
Committed revision 1267069.
$
</code>

<tt>poascribe</tt> will add ascription records into ascription catalogs corresponding to summit catalogs to be committed, and commit them all. Like <tt>svn commit</tt>, <tt>poascribe ci</tt> can take any number of file or directory paths, and can be issued from any working directory (it will always find ascription catalogs). If default commit message has not been set in the ascription configuration, <tt>poascribe</tt> will ask for it; or it can be given in command line through <tt>-m</tt> option.

=== Translators Without Commit Access ===

With the ascription system in place, every regular team member should have commit access. But, there may be some period of time before new translators are given accounts, revision control may be too technical for some, and even those with the account may not be able to commit temporarily for some reason.

These translators may send in their work by email, to ''any'' team member with commit access (not necessarily the coordinator or a reviewer); this team member can commit received files without any review, as review can be conducted at any later time. If Bob sends some files to Alice, she can commit them immediately by stating Bob's user name:

<code bash>
$ poascribe ci -u bob ...
</code>

For this to work, the translator who sent in the files has to be defined in the ascription configuration. There are no hidden costs or security issues to this (as opposed to opening a VCS account), so every new translator should be defined there before any work of that person is committed.

== Daily Use for Reviewers ==

The ascription system opens up all sorts of possibilities for concrete review patterns. Reviewers should keep in mind that for each message the full modification and review history is available, so that the team can think about how to make good use of it. Therefore, what follows are some examples to illustrate the review facilities that <tt>poascribe</tt> provides.

=== Basic Reviewing ===

At the very basic level (which is the only level in classical review by stages), messages can be classified into simply unreviewed and reviewed, without further qualifiers. Alice now wants to review all unreviewed messages in a group of PO files, say <tt>kdetoys</tt> module. She issues (<tt>di</tt> is short for <tt>diff</tt>):

<code bash>
$ cd $KDEREPO/trunk/l10n-support
$ poascribe di LANG/summit/messages/kdetoys/
LANG/summit/messages/kdegames/bovo.po (2)
LANG/summit/messages/kdegames/kdiamond.po (7)
LANG/summit/messages/kdegames/palapeli.po (12)
===== Diffed for review: 21
</code>

Unreviewed messages have now been marked and diffed, inside the listed PO files. What is this about "diffing"? If the files had already been reviewed before, some of the messages modified since then (those marked for review) may have changed very little (e.g. a few words in a paragraph-length message, or even just punctuation). Therefore, for each message marked for review, Alice also wants to see the diff since last review to current version. Here are two messages in typical review states added by <tt>poascribe di</tt>:

<code po>
#. +> trunk stable
#. ascto: charlie:m
#: gui/mainwindow.cc:372
#, ediff
msgid "GAME OVER. {-You won-}{+Tie+}!"
msgstr "KRAJ IGRE. {-Pobeda-}{+Nerešeno+}!"

#. +> trunk stable
#. ascto: bob:m charlie:m
#: game-state.cpp:117
#, ediff-total
msgid "Click the pause button again to resume the game."
msgstr "Kliknite ponovo na dugme pauze da nastavite igru."
</code>

and the first one in Kate:

[[Image:Kate_poascribe_diff_01.png|center|Message diffed for review by poascribe in Kate.]]

In the first message, the first to note is the <tt>#. ascto:</tt> comment. This comment succinctly lists who did what with the message since the last review; here <tt>charlie:m</tt> means that Charlie is the one who modified it. Then, there is the <tt>ediff</tt> flag, which alice can use it to jump through messages marked for review. Finally, the original and translation have been diffed; here they show that, since the last review, the message was fuzzied by changing "You won" to "Tie", and what Charlie did in translation to unfuzzy it. Even on a message as short as this, the diff tells something useful to Alice: the phrase "Game over" likely has a formulaic translation, and the fact that it is not part of the diff means that the earlier reviewer had made sure it is consistent, so Alice does not have to check that.

The <tt>#. ascto:</tt> comment of the second message reveals that both Charlie and Bob had been translating it. <tt>ediff-total</tt> flag instead of plain <tt>ediff</tt> means that this message had no review at all up to now, so there are no embedded diffs in text fields.

Alice can now go through marked files and messages, review translations, and possibly make modifications. When making changes in a message with embedded diffs, she can freely edit text outside of difference segments and within <tt>{+...+}</tt> segments (as these are the ones which belong to current version of the text). While reviewing, Alice does ''not'' remove any of the added message elements while reviewing (save for an occasional difference segment, when translation should be modified), as these elements are needed for later. If a message is particularly hard and Alice wants to defer its review for later, she can add the <tt>unreviewed</tt> (or <tt>nrev</tt> for short) flag to it.

Once the review is complete, Alice simply commits the reviewed files:

<code bash>
$ poascribe ci LANG/summit/messages/kdetoys/
LANG/summit/messages/kdegames/bovo.po (0/2)
LANG/summit/messages/kdegames/kdiamond.po (0/7)
LANG/summit/messages/kdegames/palapeli.po (3/12)
===== Ascription summary:
- modified reviewed
translated 3 21
>>>>> VCS is committing catalogs:
Sending LANG/summit/messages/kdegames/palapeli.po
Sending LANG/summit-ascript/messages/kdegames/bovo.po
Sending LANG/summit-ascript/messages/kdegames/kdiamond.po
Sending LANG/summit-ascript/messages/kdegames/palapeli.po
Transmitting file data ....
Committed revision 1284220.
$
</code>

Three things have happened here. First, all review states (flags, embedded diffs, etc.) have been removed, restoring the PO file to normal. Then, any modifications that Alice have made during review are ascribed to her (here 3 out of 21 messages). Finally, all marked messages are ascribed as reviewed by Alice (any with <tt>unreviewed</tt>/<tt>nrev</tt> flags would have been omitted here). When committing, the only summit catalog that got committed is the one with modifications made during review, and all the ascription catalogs were committed because of the reviews recorded in them.

When many files with few changes in each are to be reviewed, it becomes burdensome to manually open each and every diffed for review, and then to make sure that all are committed with <tt>poascribe ci</tt>. To make this easier, <tt>-w torevivew.out</tt> option can be added to <tt>poascribe di</tt>, which requests that paths of all diffed PO files are written into <tt>torevivew.out</tt> file. This file can then be used to batch open POs for review in the editor, as well as fed back on <tt>poascribe re</tt> with <tt>-f torevivew.out</tt>. There is also the <tt>-o</tt> option which causes <tt>poascribe</tt> to directly open PO files in a PO editor (though this is currently applicable only to Lokalize). Putting it together, to efficiently review a whole bunch of small changes throughout many files, Alice can:

<code bash>
$ poascribe di PATHS... -w toreview.out -o lokalize
$ # ...only marked messages opened in Lokalize, review them...
$ poascribe ci -f toreview.out
</code>

=== Selecting Messages for Review ===

Invocations of <tt>poascribe di</tt> without any options, as in the previous section, were actually equivalent to this:

<code bash>
$ poascribe di -s modar PATHS...
</code>

Option <tt>-s</tt> is issuing the message ''selector''. <tt>modar</tt> is the default selector for <tt>diff</tt> mode, and stands for MODified-After-Review: it selects the earliest historical modification of the message after the last (or no) review of that message, if there is any such. By selecting a historical modification of the message, the diff from it to current version can be computed and embedded into the PO file, as in previous examples.

There are various specialized selectors, and fall into two groups: ''shallow selectors'' and ''history selectors''. Shallow selectors look only into the current version of the message, and cannot select historical versions, which means that they cannot provide embedded diffs. History selectors (<tt>modar</tt> is of this type) can select messages from history and provide diffs. Several selectors can be issued on the command line, and the message is selected only if all selectors select it. Shallow selectors are then normally used as a pre-filter for history selectors. For example, to select messages modified after last reviewed, but only those found in stable branch, <tt>branch</tt> and <tt>modar</tt> selectors are chained:

<code bash>
$ poascribe di -s branch:stable -s modar PATHS...
</code>

It is important that the history selector is given last, because the last selector determines which historical message is selected. If the ordering had been reversed here, same messages would get selected, but they would not have embedded diffs, because <tt>branch</tt> is a shallow selector.

Selectors can take parameters themselves, like <tt>branch:stable</tt> in the previous example. Parameters are separated from the selector name by any non-alphanumeric character; this is colon by convention, but if a parameter contains a colon, something like slash, tilde, etc. can be used. Number of parameters can be variable, and <tt>modar</tt> in particular can take from none to three. If Alice wants to review only those messages modified ''by Charlie'' since last review, she states this by first argument to <tt>modar</tt>:

<code bash>
$ poascribe di -s modar:charlie PATHS...
</code>

If Alice does not give too much credit to other reviewers, she can request selection of messages modified after last review ''by her'' with second parameter to <tt>modar</tt>:

<code bash>
$ poascribe di -s modar::alice PATHS...
</code>

Here the first parameter ("modified by..."), which is not needed, must be explicitly skipped, before going to the second parameter ("reviewed by..."). The third optional parameter of <tt>modar</tt> will be mentioned in the next section.

When a selector parameter is a user name, normally it can also be a comma-separated list of user names (<tt>modar:bob,charlie</tt>) or prefixed with tilde to negate, i.e. select all other users (<tt>modar:~alice</tt>).

Any selector can be negated by prepending <tt>n</tt> to its name. For example, the history selector <tt>modafter:DATE</tt> selects first modification after the given date; to select messages modified after last review, but only if modified during June 2010:

<code bash>
$ poascribe di -s modafter:2010-06 -s nmodafter:2010-07 -s modar PATHS...
</code>

Negating a history selector produces a shallow selector: while <tt>modafter</tt> is history selector, <tt>nmodafter</tt> is shallow. But the order of the two in the previous command line is not important, as the last selector is the usual <tt>modar</tt>.

Selectors can be issued in other modes too. If the PO file is big and Alice has reviewed messages up to and including entry 246 when she has to pause until another day, she can commit reviews only up to this entry by issuing the <tt>espan</tt> selector:

<code bash>
$ poascribe ci -s espan::246 PATHS...
</code>

(the first parameter to <tt>espan</tt> is the first entry number, given if messages are not to be selected from the first). There is also the counterpart <tt>lspan</tt> selector, which works with referent line numbers (those of <tt>msgid</tt> keywords) instead of entry numbers.

=== Fine-Grained Reviews ===

In the introduction, several distinct types of what can go wrong in translation were described. Not all reviewers may be able to check translation against all those problems. Here is a typical scenario of this kind:

Alice is very computer-savvy and knows the translation project inside and out, which means that she can review well for context, terminology, and technical style. But, her language style leaves something to be desired, which shows through longer sentences and passages. Dan, on the other hand, is a very literary person, but not that much into the technical aspects. Dan's style reviews would thus be a perfect complement to Alice's general reviews.

<tt>poascribe</tt> can support this scenario in the following way. A ''review type'' tag for language style is defined in the ascription configuration, using the <tt>review-tags</tt> field:

<code text>
[global]
...
review-tags = lstyle
</code>

(The value to <tt>review-tags</tt> is a space-separated list of identifiers, when more than one special review type is needed.) With this addition to configuration, Alice can continue to review as she did before, without any changes to her workflow.

Dan selects messages for review similarly to Alice, but aditionally giving the <tt>lstyle</tt> tag as ''third'' parameter of <tt>modar</tt>, and indicating that ascribed reviews should be tagged as <tt>lstyle</tt>:

<code bash>
$ poascribe di -s modar:::lstyle -t lstyle PATHS...
</code>

After finishing the review, Dan commits as usual:

<code bash>
$ poascribe ci PATHS...
</code>

If Dan is always going to review the language style, in order not to have to issue the selector and tag in the command line all the time, he can make them default per mode in <tt>~/.pologyrc</tt>:

<code text>
[poascribe]
user = dan
selectors/diff = modar:::lstyle
tags/diff = lstyle
</code>

With this Dan can use plain <tt>poascribe di</tt> just like Alice does.

The important point of review tags is that they make reviews by types independent. For example, Dan may come around to review the language style of the given message after several modifications and general reviews have been ascribed to it -- <tt>modar:::lstyle</tt> will simply ignore all reviews except for <tt>lstyle</tt> reviews. This is going to be reflected in the <tt>ascto:</tt> comment to marked messages:

<code po>
#...
#. ascto: charlie:m alice:r bob:m
#...
msgid "..."
msgstr "..."
</code>

Here Alice has made one review between Charlie's and Bob's modifications, and that review, being general instead of <tt>lstyle</tt>, did not cause <tt>modar</tt> to stop at it. After Dan reviews this message for language style, Alice runs selection for review and gets this:

<code po>
#...
#. ascto: bob:m dan:r(lstyle)
#...
msgid "..."
msgstr "..."
</code>

Again, since <tt>lstyle</tt> reviews do not mix with general reviews, Dan's review did not hide Bob's modification that Alice did not check so far.

(General review too has a tag assigned, the empty string, in case the reviewer needs to explicitly issue it in some context.)

== Daily Use for The Coordinator ==

After setting up the ascription system, the team coordinator should have to do very little to maintain it.

=== Ascribing Merges ===

Modifications made to summit catalogs by merging with templates must also be ascribed. Therefore, after merging the summit (<tt>posummit ... merge ...</tt>) the coordinator substitutes the VCS command:

<code bash>
$ svn commit LANG/summit/messages/ -m "Merged summit."
</code>

with the <tt>poascribe</tt> command in <tt>commit</tt> mode:

<code bash>
$ poascribe ci LANG/summit/messages/ -m "Merged summit."
</code>

Since the user is not explicitly given by <tt>-u</tt> option, this will ascribe merge modifications to the coordinator (more precisely, to the user set as default in <tt>~/.pologyrc</tt>), which is just fine. It is also possible to define a special user only for ascribing merge modifications, though there is no known advantage to that.

Since <tt>-C</tt> option is not issued, <tt>poascribe</tt> will automatically commit all modified summit and ascription catalogs when done.

=== Shuffling Ascription Catalogs ===

Sometimes summit catalogs are shuffled in the repository: moved to another module, renamed, one catalog split into two, two catalogs merged into one. Such shuffling should be exactly mirrored in the ascription tree, and this too is done on the repository side, at the same time. This relies on the ascription root being set exactly to <tt>summit-ascript</tt> in the ascription configuration. So the team coordinator has nothing special to do here.

If instead in the central KDE repository the translation team is working in an external repository, by consequence the ascription system must be set up in that repository. But so long as <tt>process_orphans.sh</tt> script from <tt>trunk/l10n-support/scripts/</tt> is used to shuffle catalogs in the external repository as well, the ascription catalogs will be properly handled.

== Filtering for Release ==

The last component of the ascription system is how to prevent insufficiently reviewed messages from leaking into a release. In context of Pology and summit workflow, <tt>poascribe</tt> itself is used directly to this end. Instead, in the summit configuration (as opposed to ascription configuration), the team coordinator defines filters which pass messages by applying selectors.

Each top level PO tree has its own summit configuration file, named <tt>MSGTREE.extras.summit</tt>:

<code text>
...
LANG/
summit/
messages/
messages.extras.summit
docmessages/
docmessages.extras.summit
</code>

For the simple case of all reviews being general reviews, the filter is added to summit configuration like this (anywhere within <tt>*.extras.summit</tt> file):

<code python>
S.ascription_filters = [
("regular", ["nmodar"]),
]
</code>

Here the filter is named <tt>regular</tt>, and is defined as application of <tt>nmodar</tt> selector, the negation of <tt>modar</tt>. This simply means: pass all messages ''not'' modified after the last review.

When the team coordinator scatters to branches (executes <tt>posummit scatter</tt>), messages from summit POs which do not pass this filter will not be sent to branch POs. The count of stopped messages by branch PO will be reported in the output as scattering proceeds.

Why did we have to name the filter <tt>regular</tt>? (Those knowing some Python will also notice that it is defined as a list element.) Because it is possible to define more than one filter, and select which one is used on each scattering. For example, the coordinator may wish that, when the release is near and time is short to review everything, messages from a few experienced translators can be passed into release without review. If those translators are Alice and Bob, an "emergency" filter can be defined like this:

<code python>
S.ascription_filters = [
("regular", ["nmodar"]),
("emergency", ["nmodar:~alice,bob"]),
]
</code>

By default, <tt>posummit</tt> uses the first filter in the list. When the coordinator needs to do emergency scattering, he requests the emergency filter by the <tt>-a</tt> option:

<code bash>
$ cd $KDEREPO/trunk/l10n-support
$ posummit scripts/messages.summit LANG scatter -a emergency
</code>

What if several selectors are needed to pass the message? For example, the language style review (the earlier example with Alice and Dan) too may be requested for regular scattering, but omitted from emergency scattering. The filter setup for this scenario looks like this:

<code python>
S.ascription_filters = [
("regular", ["nmodar", "nmodar:::lstyle"]),
("emergency", ["nmodar:~alice,bob"]),
]
</code>

The regular filter now reads: pass the message if it has not been modified after the last (general) review ''and'' has not been modified after the last style review.

Simple combination of predefined selectors by AND-conditions may not be sufficient for more involved scenarios. When this is the case, the coordinator may write (or ask someone to write) a custom selector in Python, and plug it in as the second element in the filter tuple (instead of the list of predefined selectors).

=== Writing a Selector Function ===

((To be written.))

Localization/Tools/Pology/PO Embedded Diffing

2010-06-13T08:27:44Z

Ilic: Wording.

{{Template:I18n/Language Navigation Bar|Localization/Tools/Pology/PO_Embedded_Diffing}}

{{LocalizationBrowser|
section=[[Localization#Tools|Tools]]|
name=Embedded Diffing with Pology|
prereqs=[[Localization/Tools/Pology|Pology]]|
}}

== Diffing of PO So Far ==

'''Line-level''' diffing assumes that the maximal generaly recognizable unit of text file is one line, that each line carries a meaning of significant standalone value, and that the ordering of lines has been deliberately chosen and cannot be arbitrary. For example, this is typical of program code.

Superficially, PO files could also be considered "a programming language of translations", and amenable to same treatment on diffing. However, some of the above assumptions which make line-level diffing viable are violated with PO format. Firstly, minimal unit of PO file is one message, whereas one line is of negligible semantic value. Secondly, ordering of messages can in principle be arbitrary (e.g. dependent on the order of extraction from source code), such that two line-wise very different PO files are actually equivalent from translator's viewpoint. And thirdly, good number of lines in the PO file are auxiliary, neither original text nor translation, generated either automatically or by the programmer (e.g. source references, extracted comments), all of which are out of translator's scope for modifications.

Thus, a common way to use line-level diffing with PO files so far was only for review, and with some preparations. Due to myriad of equivalent but line-wise different representations of PO content, it is quite useless to send line diffs as patches; translators are instructed to always send full PO files to the reviewer/commiter, no matter the amount of modifications. Then, the reviewer merges the received PO file (new version), and possibly the original (old version), with current template, without wrapping of long lines in text fields. This "normalizes" old and new files with respect to all semantically non-significant elements, and only then can line diff be performed. Additionally, since a long non-wrapped line of translated text may differ only in some segments, a dedicated diff viewer which can highlight '''word-level''' differences has to be used; ordinary diff syntax highlighting (e.g. in shell, or in general text editor) won't cut it.

Even with such preparations and dedicated diff viewer at hand, there is at least one significant case which is still not reasonably covered: when a fuzzy message, which had previous fields (PO was merged with <tt>--previous</tt> option to <tt>msgmerge</tt>), has been updated and unfuzzied. For example:

{|width="100%"
|-
|style="padding-right: 1em; width: 1%"|'''old'''
|style="padding-bottom: 0.5em"|
<code po>
#: main.c:110
#, fuzzy
#| msgid "The Record of The Witch River"
msgid "Records of The Witch River"
msgstr "Beleška o Veštičjoj reci"
</code>
|-
|style="padding-right: 1em; width: 1%"|'''new'''
|style="padding-bottom: 0.5em"|
<code po>
#: main.c:110
msgid "Records of The Witch River"
msgstr "Beleške o Veštičjoj reci"
</code>
|-
|style="padding-right: 0.5em; width: 1%"|'''diff'''
|style="padding-bottom: 0.5em"|
<code diff>
⁠ #: main.c:110
- #, fuzzy
- #| msgid "The Record of The Witch River"
msgid "Records of The Witch River"
- msgstr "Beleška o Veštičjoj reci"
+ msgstr "Beleške o Veštičjoj reci"
</code>
|}

Here, the diff viewer will know to show word-level diff for modified translation, but it cannot know that it should also show word-level diff between the removed previous and current <tt>msgid</tt> fields, so that reviewer can see what has changed ''in the original'' text (i.e. why the message became fuzzy), and based on that judge the change in translation.

Finally, a dedicated PO editor may be able to show the truly proper, '''message-level''' difference (such as [[Localization/Tools/Lokalize|Lokalize]] can, operating in [http://docs.kde.org/stable/en/kdesdk/lokalize/sync.html merge mode]). Even then, however, there is still the need to send around full PO files, and, to some extent, to normalize them before comparing in the editor. The diff format becomes tied to and defined in terms of the given PO editor (which you may not want to use in general), instead of being intrinsically defined and modularly processable (such as line diffs are).

The rest of this article will therefore propose a format and semantics for self-contained, message-level diffing of PO files -- the '''embedded diff''' -- and present two [[Localization/Tools/Pology|Pology]] scripts which embody it as proof of concept (but are also quite practically applicable).

== The Embedded Diff ==

Difference between two PO messages should be primarily, though not exclusively, composed of differences between its text fields (<tt>msgid</tt>, <tt>msgctxt</tt>, etc.) Then, to be easily judged, differences in text should be presented as succinctly and locally as possible -- think of a long paragraph where only some spelling or punctuation has changed. Finally, the format of the complete message diff should be easily comprehendable to translators used to PO format itself, and as far as possible, to existing PO tools too.

Previous considerations lead to the following decision: ''PO message diff will also be a PO message''. Or, in other words, the diff will be ''embedded'' into the regular parts of a PO message. An embedded diff (''ediff'' for short) message should be at least syntactically a valid PO entry, if not always semantically (i.e. if part of PO file, it should pass <tt>msgfmt</tt>, though not necessarily <tt>msgfmt --check</tt>). To enable ediffs to be passed around as patches for PO files, the embedding should be automatically resolvable (up to significant message parts) to the old and new messages from which the diff was created.

In this way, if ediff messages are packed into a PO file (an ediff PO), existing PO tools can be used to review and modify the diff. For example, highlighting in a text editor will need only minimal upgrades to show the embedded differences (more on that below), and otherwise it will already highlight ediff message parts as usual.

To complete the definition of ediffs in detail, the following questions should be answered:

* How to represent embedded differences in text?

* Which parts of the message should be diffed?

* How to pair for diffing messages from two files?

* How to present collection of diffed messages?

=== Embedding Differences into Text ===

Once the difference between the old and new strings, the word-level difference, has been determined, it should be decided how to embed it into the new (or, equivalently, old) string. One possibility is that of <tt>wdiff(1)</tt>, where removed and added text segments are by default wrapped with <tt><nowiki>[-...-]</nowiki></tt> and <tt><nowiki>{+...+}</nowiki></tt>, respectively:

{|width="100%"
|-
|style="padding-right: 1em; width: 1%"|'''old'''
|style="padding-bottom: 0.5em"|
<code text>
"The Record of The Witch River"
</code>
|-
|style="padding-right: 1em; width: 1%"|'''new'''
|style="padding-bottom: 0.5em"|
<code text>
"Records of The Witch River"
</code>
|-
|style="padding-right: 0.5em; width: 1%"|'''diff'''
|style="padding-bottom: 1em"|
<code text>
"[-The Record-]{+Records+} of The Witch River"
</code>
|}

However, search through PO files of several translation projects (KDE, Gnome, OpenOffice, Fedora, Mozilla) reveals the <tt><nowiki>[-</nowiki></tt> character combination to be frequently encountered, e.g. in synopsis of command usage. While there must exist an escaping mechanism for cases when diff wrappers are encountered in the original text, to enable unambiguous resolving of ediffs, it is nevertheless prudent to pick wrappers which reduce frequency of escaping (e.g. syntax highlighting may not be able to discern non-diff wrapper-like segments). Searching through same collection of PO files produces no hits for <tt>{-</tt>, so this combination is picked instead as wrapper of removed text segments:

{|width="100%"
|-
|style="padding-right: 1em; width: 1%"|'''old'''
|style="padding-bottom: 0.5em"|
<code text>
"The Record of The Witch River"
</code>
|-
|style="padding-right: 1em; width: 1%"|'''new'''
|style="padding-bottom: 0.5em"|
<code text>
"Records of The Witch River"
</code>
|-
|style="padding-right: 0.5em; width: 1%"|'''ediff'''
|style="padding-bottom: 1em"|
<code text>
"{-The Record-}{+Records+} of The Witch River"
</code>
|}

If the text itself contains a wrapper-like combination, it is escaped by inserting tilde (<tt>~</tt>) between the brace and plus/minus sign:

{|width="100%"
|-
|style="padding-right: 1em; width: 1%"|'''old'''
|style="padding-bottom: 0.5em"|
<code text>
"Foo {+ bar"
</code>
|-
|style="padding-right: 1em; width: 1%"|'''new'''
|style="padding-bottom: 0.5em"|
<code text>
"Foo {+ qwyx"
</code>
|-
|style="padding-right: 0.5em; width: 1%"|'''ediff'''
|style="padding-bottom: 1em"|
<code text>
"Foo {~+ {-bar-}{+qwyx+}"
</code>
|}

If the text already contains a substring with brace-tilde-plus/minus, then another tilde is inserted, and so on. Thus the ediff can be unambiguously resolved to old and new versions of the string. Inserting the tilde between the two characters of wrapper combinations also makes it easier on the syntax higlighting, as the difference highlighting trigger is automatically removed.

It may happen that a given string is not merely empty in the old or new PO message, but that it does not exist at all (e.g. <tt>msgctxt</tt> field). Thus an ediff can be made between existing and non-existing strings too, in which case a tilde is appended to the very end of the ediff:

{|width="100%"
|-
|style="padding-right: 1em; width: 1%"|'''old'''
|style="padding-bottom: 0.5em"|
<code text>
</code>
|-
|style="padding-right: 1em; width: 1%"|'''new'''
|style="padding-bottom: 0.5em"|
<code text>
"a-context-note"
</code>
|-
|style="padding-right: 0.5em; width: 1%"|'''ediff'''
|style="padding-bottom: 1em"|
<code text>
"{+a-context-note+}~"
</code>
|}

Again, escaping is provided for by inserting further tildes if the ediff between two existing strings would result in trailing tilde (old: <tt>"~"</tt>, new: <tt>"foo~"</tt>, ediff: <tt>"{+foo+}~~"</tt>).

How exactly the difference between two strings is formed, may be left to implementation. In fact, an implementation may allow translator to select between several diffing algorithms, depending on personal taste and situation. For example, the default algorithm of <tt>poediff</tt> does the following: words are diffed as whole, all non-word segments (interpunction, markup tags, etc.) character by character, and equal non-word segments between different words are taken into the difference segment. Hence the above ediff

<code text>
"{-The Record-}{+Records+} of The Witch River"
</code>

instead of smaller

<code text>
"{-The -}Record{+s+} of The Witch River"
</code>

as the former is (tentatively) easier to comprehend.

Every difference segment in an ediff PO message will be represented like this, thus it is sufficient to upgrade PO syntax highlighting of an editor to indiscriminately highlight <tt>{-...-}</tt> and <tt>{+...+}</tt> segments everywhere in the message.

=== Message Parts Included in Diffing ===

A PO message consists of several types of parts: text fields, comments, flags, source references, etc. It would not be very constructive to diff all of them; for example, while <tt>msgstr</tt> fields should clearly be included into diffing, source references most probably should not. In order not to consider pros and cons of inclusion for each and every message part, there already exists a clear split of message parts into two groups, one of which will be taken into diffing, and the other ignored. These two groups are:

* ''extraction-invariant'' parts -- those which do not depend on placement (or even presence) of message in the sources, such as <tt>msgid</tt> field, <tt>msgstr</tt> fields, manual comments, etc.

* ''extraction-prescribed'' parts -- those which cannot exist independently of the source from which the message is extracted, such as format flags or extracted comments.

Extraction-invariant parts are the ones which will be diffed. Working definition of exactly which parts belong into this group is provided by obsolete messages in PO files. Thus, these parts are:

* current original: <tt>msgctxt</tt>, <tt>msgid</tt>, and <tt>msgid_plural</tt> fields

* previous original: commented <tt>#| msgctxt</tt>, <tt>#| msgid</tt>, and <tt>#| msgid_plural</tt> fields

* translation: <tt>msgstr</tt> fields

* translator (manual) comments

* fuzzy state (whether the <tt>fuzzy</tt> flag is present)

* obsolete state (whether the message is obsolete)

All the listed fields and manual comments are presented in ediff message as wrapped word-level differences, as described earlier. Change in states, fuzzy and obsolete, is represented slightly differently. A special "extracted" (automatic) comment is added to the ediff message, starting with <tt>ediff:</tt> and listing any extra info needed to describe the ediff, including the state changes. Here is an example of two messages and the ediff they would produce (whether two messages such as these would get paired for diffing in the first place, will be discussed later on):

{|width="100%"
|-
|style="padding-right: 1em; width: 1%"|'''old'''
|style="padding-bottom: 0.5em"|
<code po>
#, fuzzy
#~| msgid "Accurate subpolar weather cycles"
#~ msgid "Accurate subpolar climate cycles"
#~ msgstr "Tačni ciklusi subpolarnog vremena"
</code>
|-
|style="padding-right: 1em; width: 1%"|'''new'''
|style="padding-bottom: 0.5em"|
<code po>
#. ui: property (text), widget (QCheckBox, accCyclesTrop)
#: config.ui:180
#, fuzzy
#| msgid "Accurate tropical weather cycles"
msgctxt "some-superfluous-context"
msgid "Accurate tropical climate cycles"
msgstr "Tačni ciklusi tropskog vremena"
</code>
|-
|style="padding-right: 0.5em; width: 1%"|'''ediff'''
|style="padding-bottom: 1em"|
<code po>
#. ediff: state {-obsolete-}
#. ui: property (text), widget (QCheckBox, accCyclesTrop)
#: config.ui:180
#, fuzzy
#| msgid "Accurate {-subpolar-}{+tropical+} weather cycles"
msgctxt "{+some-superfluous-context+}~"
msgid "Accurate {-subpolar-}{+tropical+} climate cycles"
msgstr "Tačni ciklusi {-subpolarnog-}{+tropskog+} vremena"
</code>
|}

Here is how this ediff message will look like in KWrite and Kate, starting with KDE 4.2:

[[Image:Kate_PO_Ediff_Highlighting.png|center|Ediff syntax highlighting in Kate]]

(In fact, ediff-aware [http://websvn.kde.org/*checkout*/trunk/KDE/kdelibs/kate/syntax/data/gettext.xml?content-type=text%2Fplain PO syntax highlighting definition] for Kate can be used with Kate versions as early as KDE 3.4. The definition file should simply be placed as <tt>~/.kde/share/apps/katepart/syntax/gettext.xml</tt>, and it will override the system definition.)

First thing to note is that the ediff message contains not only extraction-invariant parts, but also copies verbatim extraction-prescribed parts from the new message. Effectively, the ediff is embedded into the copy of new message. Extraction-prescribed parts are not simply discarded in order to provide more context when reviewing the diff. Here, for example, the extracted comment states that the text is a checkbox label, which may be important for the style of translation.

The other important element is the <tt>#. ediff:</tt> dummy extracted comment, which here indicates that the obsolete state has been "removed", i.e. the message was unobsoleted going from old to new version. Aside from state changes, few other indicators may rarely be present, as described in API documentation of <tt>pology.misc.diff</tt> module, function <tt>msg_ediff</tt>. Some of these indicators will be mentioned later on. The ediff comment will be present only when necessary, if there are any indicators to show.

If diffing of two messages would always be conducted part for part, for all parts which are taken into diffing, then in some cases the resulting ediff would not be very useful. Consider how the first example in the article, the line-level diff of a fuzzy and translated message, would look like as ediff if performed part for part:

{|width="100%"
|-
|style="padding-right: 1em; width: 1%"|'''old'''
|style="padding-bottom: 0.5em"|
<code po>
#: main.c:110
#, fuzzy
#| msgid "The Record of The Witch River"
msgid "Records of The Witch River"
msgstr "Beleška o Veštičjoj reci"
</code>
|-
|style="padding-right: 1em; width: 1%"|'''new'''
|style="padding-bottom: 0.5em"|
<code po>
#: main.c:110
msgid "Records of The Witch River"
msgstr "Beleške o Veštičjoj reci"
</code>
|-
|style="padding-right: 0.5em; width: 1%"|'''ediff'''
|style="padding-bottom: 0.5em"|
<code po>
#. ediff: state {-fuzzy-}
#: main.c:110
#| msgid "{-The Record of The Witch River-}~"
msgid "Records of The Witch River"
msgstr "{-Beleška-}{+Beleške+} o Veštičjoj reci"
</code>
|}

The same problem as with the line-level diff persists: instead of showing the difference from previous to current <tt>msgid</tt> field, the current <tt>msgid</tt> is left untouched, and previous <tt>msgid</tt> is simply shown to have been entirely removed.

Therefore, instead of straightforward, part for part diffing, a special transformation will take place when ''exactly one'' of the two diffed messages is fuzzy and equipped with previous original text fields. This splits into two directions: from fuzzy to non-fuzzy, and from non-fuzzy to fuzzy.

Diffing from fuzzy to non-fuzzy message, such as above, is the more usual of the two directions. It is typical of translation updated after merging with template, where some fuzzied messages will have been resolved. In this case, the original old and new messages are transformed thusly prior to diffing (<tt>*-rest</tt> denotes all diffed parts that are neither original text nor fuzzy state):

{|width="100%"
|-
|style="padding-right: 1em; width: 1%"|'''old'''
|style="padding-bottom: 0.5em"|
<code text>
fuzzy --> fuzzy
old-previous-fields --> old-previous-fields
old-current-fields --> old-previous-fields
old-rest --> old-rest
</code>
|-
|style="padding-right: 1em; width: 1%"|'''new'''
|style="padding-bottom: 0.5em"|
<code text>
- --> -
- --> old-current-fields
new-current-fields --> new-current-fields
new-rest --> new-rest
</code>
|}

In this way, ediff message's current fields will show the important difference, that of previous fields of old fuzzy message and current fields of new non-fuzzy message. Ediff message's previous fields will show the less important difference of old fuzzy messages previous and current fields, but ''only'' if it is not equal to the difference in current fields; otherwise it is eliminated. This may sound confusing, but the final ediff produced in this way is quite intuitive:

{|width="100%"
|-
|style="padding-right: 1em; width: 1%"|'''old'''
|style="padding-bottom: 0.5em"|
<code po>
#: main.c:110
#, fuzzy
#| msgid "The Record of The Witch River"
msgid "Records of The Witch River"
msgstr "Beleška o Veštičjoj reci"
</code>
|-
|style="padding-right: 1em; width: 1%"|'''new'''
|style="padding-bottom: 0.5em"|
<code po>
#: main.c:110
msgid "Records of The Witch River"
msgstr "Beleške o Veštičjoj reci"
</code>
|-
|style="padding-right: 0.5em; width: 1%"|'''ediff'''
|style="padding-bottom: 0.5em"|
<code po>
#. ediff: state {-fuzzy-}
#: main.c:110
msgid "{-The Record-}{+Records+} of The Witch River"
msgstr "{-Beleška-}{+Beleške+} o Veštičjoj reci"
</code>
|}

The reviewer here sees that the message was unfuzzied, the change in the original text that caused the message to become fuzzy, and the change made in translation to unfuzzy it. Old version (in removed and equal segments) of original and translation is that of the message before it got fuzzied, and new version (in added and equal segments) is that of the message after it was unfuzzied.

From non-fuzzy to fuzzy message should be the less frequent direction of diffing. It corresponds e.g. to case where the diff is taken from older fully complete translation to the one just after merging with newest template. In this case, the transformation is as follows:

{|width="100%"
|-
|style="padding-right: 1em; width: 1%"|'''old'''
|style="padding-bottom: 0.5em"|
<code text>
- --> -
- --> new-previous-fields
old-current-fields --> old-current-fields
old-rest --> old-rest
</code>
|-
|style="padding-right: 1em; width: 1%"|'''new'''
|style="padding-bottom: 0.5em"|
<code text>
fuzzy --> fuzzy
new-previous-fields --> new-current-fields
new-current-fields --> new-current-fields
new-rest --> new-rest
</code>
|}

Again, the difference presented by ediff messages's current fields will be the most important one, the difference in previous fields the less important one, and eliminated if equal to the other. Here is what this will do if applied to one step earlier (just after merging) of the running example:

{|width="100%"
|-
|style="padding-right: 1em; width: 1%"|'''old'''
|style="padding-bottom: 0.5em"|
<code po>
#: main.c:89
msgid "The Record of The Witch River"
msgstr "Beleška o Veštičjoj reci"
</code>
|-
|style="padding-right: 1em; width: 1%"|'''new'''
|style="padding-bottom: 0.5em"|
<code po>
#: main.c:110
#, fuzzy
#| msgid "The Record of The Witch River"
msgid "Records of The Witch River"
msgstr "Beleška o Veštičjoj reci"
</code>
|-
|style="padding-right: 0.5em; width: 1%"|'''ediff'''
|style="padding-bottom: 0.5em"|
<code po>
#. ediff: state {+fuzzy+}
#: main.c:110
#, fuzzy
msgid "{-The Record-}{+Records+} of The Witch River"
msgstr "Beleška o Veštičjoj reci"
</code>
|}

The reviewer here sees that the message became fuzzy from new to old, and the change in original text that caused this. (On a side note, remember that ediff message is constructed by embedding differences into a copy of new message, so the source reference and the fuzzy flag from the new message appear here in the ediff message.)

To be able to use the ediff as patch, it is necessary to reconstruct original old and new messages after resolving ediff into transformed old and new messages. This step is fortunatelly unambiguous; one just needs to check whether the non-fuzzy of the two resolved messages has previous fields, or, if not (due to elimination of equal difference in previous fields, or because template was merged without the <tt>--previous</tt> option), whether the current fields are equal. Then the back-transformation to original old and new messages can be performed.

=== Pairing Messages From Two Catalogs ===

So far the article has described how to make an embedded diff out of two messages, once it has been decided that those messages should be diffed. However, on the lowest level, the user decides not which messages to diff, but which two PO files to diff. The implementation should then automatically ''pair'' for diffing messages from the two catalogs, and this section described several possible ways to do this.

In the first instance, messages should obviously be '''paired by key''' (primary pairing), the unique combination of <tt>msgctxt</tt> and <tt>msgid</tt> fields. In the most usual case -- reviewing an ediff from incomplete catalog with some fuzzy and untranslated messages, to an updated catalog with some or all of those messages translated -- pairing by key will be fully sufficient, as both catalogs contain exactly the same set of messages by keys. These two messages will be plainly paired by key:

{|width="100%"
|-
|style="padding-right: 1em; width: 1%"|'''old'''
|style="padding-bottom: 0.5em"|
<code po>
#: main.c:110
#, fuzzy
#| msgid "The Record of The Witch River"
msgid "Records of The Witch River"
msgstr "Beleška o Veštičjoj reci"
</code>
|-
|style="padding-right: 1em; width: 1%"|'''new'''
|style="padding-bottom: 0.5em"|
<code po>
#: main.c:110
msgid "Records of The Witch River"
msgstr "Beleške o Veštičjoj reci"
</code>
|}

But what should happen if some messages cannot be paired by key? Consider the earlier example where diff was taken from older fully translated, to newer merged catalog:

{|width="100%"
|-
|style="padding-right: 1em; width: 1%"|'''old'''
|style="padding-bottom: 0.5em"|
<code po>
#: main.c:89
msgid "The Record of The Witch River"
msgstr "Beleška o Veštičjoj reci"
</code>
|-
|style="padding-right: 1em; width: 1%"|'''new'''
|style="padding-bottom: 0.5em"|
<code po>
#: main.c:110
#, fuzzy
#| msgid "The Record of The Witch River"
msgid "Records of The Witch River"
msgstr "Beleška o Veštičjoj reci"
</code>
|}

The keys, here just current <tt>msgid</tt> fields, of the two message do not match, so they cannot be paired by key. Yet it would be rather ungainly to represent the old message as fully removed in the ediff, and the new message as fully added:

{|width="100%"
|-
|style="padding-right: 1em; width: 1%"|'''ediff'''
|style="padding-bottom: 0.5em"|
<code po>
#: main.c:89
msgid "{-The Record of The Witch River-}~"
msgstr "{-Beleška o Veštičjoj reci-}~"
⁠
#. ediff: state {+fuzzy+}
#: main.c:110
#, fuzzy
#| msgid "{+The Record of The Witch River+}~"
msgid "{+Records of The Witch River+}~"
msgstr "{+Beleška o Veštičjoj reci+}~"
</code>
|}

(That the message has been fully added or removed can be seen by trailing tilde in the <tt>msgid</tt> field, which indicates that the old or new <tt>msgid</tt> does not exist at all, and so neither the message with it.)

Instead, messages left unpaired by key should be tested for '''pairing by pivoting''' around previous fields (secondary pairing). The two messages above will thus be paired due to the fact that the current <tt>msgid</tt> of the old message is equal to the previous <tt>msgid</tt> of the new message, and will produce a single ediff message as shown earlier.

Finally, consider the third related combination, when the old catalog has not yet been merged with the template, while the new catalog has both been merged and its translation updated:

{|width="100%"
|-
|style="padding-right: 1em; width: 1%"|'''old'''
|style="padding-bottom: 0.5em"|
<code po>
#: main.c:89
msgid "The Record of The Witch River"
msgstr "Beleška o Veštičjoj reci"
</code>
|-
|style="padding-right: 1em; width: 1%"|'''new'''
|style="padding-bottom: 0.5em"|
<code po>
#: main.c:110
msgid "Records of The Witch River"
msgstr "Beleške o Veštičjoj reci"
</code>
|}

Once again, it would be a waste to present the old message as fully removed and the new message as fully added in the ediff. When a message is left unpaired after both pairing by key and pairing by pivoting, then the two catalogs should be merged in the background -- as if the new is the template for the old, and vice versa -- and then tested for chained pairing by pivoting and by key with merged catalog as intermediary. This '''pairing by merging''', or tertiary pairing, will then produce another natural ediff:

{|width="100%"
|-
|style="padding-right: 0.5em; width: 1%"|'''ediff'''
|style="padding-bottom: 0.5em"|
<code po>
#: main.c:110
msgid "{-The Record-}{+Records+} of The Witch River"
msgstr "{-Beleška-}{+Beleške+} o Veštičjoj reci"
</code>
|}

Implementations can decide which pairing modes beyond the primary, by key, to use. There should not be much reason not to perform secondary pairing, by pivoting, too. If tertiary pairing, by merging, is implemented, it should be provided that the user can disable it (sometimes this pairing may produce strange results, and needs <tt>msgmerge</tt> to be available on the system).

=== Collecting Diffed Messages ===

For ediff of two PO files to also be a syntactically valid PO file, constructed ediff messages should be preceded by a PO header on output. At first glance, this PO header could be itself the ediff of PO headers of the catalogs which were diffed. However, there are several issues with this approach:

* Reviewer of the ediff PO file would not be informed at once whereas there was any difference between the headers. Headers tend to be long, and a point change in one of the fields may go visually unnoticed.

* Depending on the amount of changes between the two headers, the resulting ediff message of the header could be too badly formed to represent the header as such (e.g. if header fields in <tt>msgstr</tt> were added or removed, embedded difference wrappers would invalidate MIME-header format of <tt>msgstr</tt>), and thus confuse the PO tools.

* How would the diff of two ''collections'' of PO files (e.g. directories) be packed into a single ediff PO, such as can normally be done with line-level diffing?

To avert these difficulties, the following is done instead. First, the header of ediff PO is constructed as a minimal valid header (i.e. one that <tt>msgfmt</tt> would not complain about) independent of the content of original headers. <tt>poediff</tt> will produce something like:

<code po>
# +- ediff -+
msgid ""
msgstr ""
"Project-Id-Version: ediff\n"
"PO-Revision-Date: 2009-02-08 01:20+0100\n"
"Last-Translator: J. Random Translator\n"
"Language-Team: Differs\n"
"MIME-Version: 1.0\n"
"Content-Type: text/plain; charset=UTF-8\n"
"Content-Transfer-Encoding: 8bit\n"
"X-Ediff-Header-Context: ~\n"
</code>

<tt>PO-Revision-Date</tt> header field is naturally set to the date when the ediff is made. <tt>Last-Translator</tt> and <tt>Language-Team</tt> can be somehow pulled from environment (<tt>poediff</tt> will source them from <tt>~/.pologyrc</tt>, or set some dummy values as above if not present). Encoding of the ediff PO can be chosen at will by the implementation, so long as all following ediff messages can be encoded with it (<tt>poediff</tt> will always use UTF-8). The purpose of the <tt>X-Ediff-Header-Context</tt> field will be explained shortly.

It is the first next message in the ediff PO that will actually be the ediff of headers of the two diffed PO files. Headers are diffed just like any other message, but the resulting ediff is equiped with few extra decorations:

<code po>
# =========================================================
# Translation of The Witch River into Serbian.
# Koja Kojic <koja.kojic@nedohodnik.net>, 2008.
# {+Era Eric <era.eric@ledopad.net>, 2008.+}~
msgctxt "~"
msgid ""
"- l10n-wr/sr/wriver-main.po\n"
"+ l10n-wr/sr-mod/wriver-main.po\n"
msgstr ""
"Project-Id-Version: wriver 0.1\n"
"POT-Creation-Date: 2008-09-22 09:17+0200\n"
"PO-Revision-Date: 2008-09-{-25 20:44-}{+28 21:49+}+0100\n"
"Last-Translator: {-Koja Kojic <koja.kojic@nedohodnik-}"
"{+Era Eric <era.eric@ledopad+}.net>\n"
"Language-Team: Serbian\n"
"MIME-Version: 1.0\n"
"Content-Type: text/plain; charset=UTF-8\n"
"Content-Transfer-Encoding: 8bit\n"
</code>

First observe the normal ediff segments: translator comment with a new translator who updated the PO file has been added, and <tt>PO-Revision-Date</tt> and <tt>Last-Translator</tt> header fields contain ediffs reflecting the update. These are the only actual differences between the two headers.

More interesting are the extra decorations:

* The very first translator comment (here a long line of equality signs) can be anything, and serves to give good visual distinction to the header ediff. This is more so convenient when the ediff PO contains diffs of several pairs of PO files.

* That a particular message in the ediff PO is a header ediff, is indicated by the <tt>msgctxt</tt> set to a special value, here a single tilde. This value is given up front by the <tt>X-Ediff-Header-Context</tt> of the ediff PO header. It should be computed during diffing such that it does not conflict with <tt>msgctxt</tt> of one of the normal message ediffs (e.g. it may simply be a sufficiently long sequence tildes).

* The <tt>msgid</tt> field of the header ediff contains newline-separated paths of the diffed PO files. More precisely, the two lines of the <tt>msgid</tt> field are in form <code text>[+-] file-path[ <<< comment]\n</code> (The trailing newline of the second file path is elided if the <tt>msgstr</tt> does not end in newline, to prevent <tt>msgfmt</tt> from complaining.) The optional, <<<-separated comment to the file path can be used for any purpose, one which will be demonstrated when describing functionality of <tt>poediff</tt>.

Although when a PO file is properly updated there should always be some difference in its header, it may happen that there is none. In such case, the header ediff message is still added, but such that it only contains the extra decorations -- the visual separator comment, special <tt>msgctxt</tt>, and <tt>msgid</tt> with file paths. All other comments and <tt>msgstr</tt> should be empty, and the empty <tt>msgstr</tt> indicates that there was no difference in headers. Presence of this "empty" header ediff is necessary to provide diffed file paths and, if several POs were diffed, to separate them in the ediff PO.

[[Image:Kate_Dir_Ediff_PO.png|right|150x150px|Embedded diff of directory of PO files in Kate.]]

After the header ediff message, ordinary ediff messages follow. As already obvious by now, when several POs were diffed to construct a single ediff PO, each next PO in the ediff simply opens with a new header ediff message. Click on the thumbnail on the right to see how an ediff of two PO files looks like in entirety, with syntax highlighting in Kate.

Especially when diffing several PO files, it may happen that two ediff messages have equal keys (<tt>msgid</tt> and <tt>msgctxt</tt> fields) and thus cannot be both added as such to the ediff PO. In such case, the ediff message which was added after the first with the same key, will have its <tt>msgctxt</tt> field ''padded'' by few random alphanumerics, such as to make its key unique. This padding sequence will be recorded in the ediff comment. For example:

<code po>
# =========================================================
msgctxt "~"
msgid "...(first PO header ediff)..."
msgstr "..."
⁠
#. ediff: state {-fuzzy-}
msgid "White{+ horizon+}"
msgstr "Belo{+ obzorje+}"
⁠
# =========================================================
msgctxt "~"
msgid "...(second PO header ediff)..."
msgstr "..."
⁠
#. ediff: state {-fuzzy-}, ctxtpad q9ac3
msgctxt "|q9ac3~"
msgid "White{+ horizon+}"
msgstr "Belo{+ obzorje+}"
</code>

The padding sequence is appended to the original <tt>msgctxt</tt>, separated by the pipe character. If there was no original <tt>msgctxt</tt>, the padding sequence is further extended by a tilde.

== Producing Ediffs: <tt>poediff</tt> ==

Pology's <tt>poediff</tt> script implements embedded diffing as outlined in the previous section. To diff two PO files, executing the usual:

<code bash>
$ poediff orig/foo.po mod/foo.po
</code>

will write the ediff PO (if there is any difference) to standard output, with some basic shell highlighting of difference segments. Option <tt>-o</tt> can be used to output the ediff PO to file instead. Other options include <tt>--no-merge</tt> (<tt>-n</tt>) to not perform pairing by merging, and <tt>--strip-headers</tt> (<tt>-s</tt>) to take a quick look of differences without headers in the way (output with stripped headers is not a valid PO, and cannot be used as patch).

It is equally simple to make ediff of directories:

<code bash>
$ poediff orig/ mod/
</code>

Diffing of directories is by default recursive and takes into ediff added and removed catalogs. When a catalog has been added or removed, the <tt>msgid</tt> of corresponding header ediff will have one of the file paths, new or old, empty.

To have non-dummy translator name and email address added to the header of ediff PO, set them in Pology's configuration file, <tt>~/.pologyrc</tt>:

<code ini>
[user]
name = Koja Kojic
original-name = Која Којић
email = koja.kojic@nedohodnik.net
</code>

(These entries cover all contexts in Pology where such information is of use; <tt>original-name</tt> field is used when the name is to be written differently in English and translator's native language.)

This is pretty much where the story about <tt>poediff</tt> would end, if not for its ability to take into account the underlying VCS, if one is used to control PO files.

=== Diffing With Underlying VCS ===

When PO files are under version control, <tt>poediff</tt> can operate in VCS mode using the option <tt>-c VCSKEY</tt> (<tt>--vcs</tt>), where <tt>VCSKEY</tt> is the keyword of one of VCS known to <tt>poediff</tt>. Then, instead of giving two paths to diff, any number of version-controlled paths (files or directories) is given. Without other options, all modified PO files in these paths are diffed against the last commit known to local repository. For example, if an application uses Subversion repository, PO files in its <tt>po/</tt> directory can be diffed with:

<code bash>
$ poediff -c svn app/po/
</code>

Some other set of revisions to diff can be given by the option <tt>-r REV1<nowiki>[:REV2]</nowiki></tt> (<tt>--revision</tt>). <tt>REV1</tt> and <tt>REV2</tt> are not necessarily proper revision IDs, but any strings that the underlying VCS will be able to convert into revision IDs. If <tt>REV2</tt> is omitted, diffing is preformed from <tt>REV1</tt> to current working copy.

When ediff is made in VCS mode, <tt>msgid</tt> fields of header ediffs will use <<<-separated comments to file paths to indicate revision IDs:

<code po>
# =========================================================
# ...
msgctxt "~"
msgid ""
"- app/po/lang.po <<< 20537\n"
"+ app/po/lang.po"
msgstr "..."
</code>

As for supported VCS, Pology currently knows about Subversion (<tt>svn</tt>) and Git (<tt>git</tt>). To add a new VCS, its functionality should be wrapped for Pology's use in <tt>pology.misc.vcs</tt>, using the interface defined by <tt>VcsBase</tt> class.

== Ediffs as Patches: <tt>poepatch</tt> ==

For basic functionality, applying an ediff patch is implementationally even easier than applying a line-level patch. Each ediff message in turn is resolved into originating old and new message, and if either the old or new message exists in target PO and is equal by extraction-invariant parts, then the message patch is applied, otherwise rejected.

Applying the patch means overwriting extraction-invariant parts of the target message with those of the new message from the ediff, and leaving other parts untouched. If the target message is already equal to new message by extraction-invariant parts, then the patch is silently ignored. This means that if the same patch is applied twice, second application makes no modifications to the target catalog. Likewise if, by chance, the changes given by the patch were already independently introduced in the target catalog.

Command-line interface of <tt>poepatch</tt> is much like <tt>patch(1)</tt>, sans the myriad of its more obscure options. There is the <tt>-p</tt> option to strip leading elements of file paths in the ediff, and <tt>-d</tt> option to append to them a directory path where target POs are to be found. If the files were diffed with underlying VCS as in the previous example, then the ediff could be applied in any of the following ways:

<code bash>
$ cd repos/app/po && poepatch <ediff.po
$ cd repos/ && poepatch -p0 <ediff.po
$ poepatch -d repos/app/po <ediff.po
</code>

Header patch (coming from the header ediff message) is applied in a slightly more relaxed fashion: some of the header fields are ignored when checking whether the patch is applicable. These are the fields which are known to be volatile as the PO file goes through different translators, and do not influence the processing of the catalog directed by the header (such as encoding or plural forms). Currently, these fields are: <tt>POT-Creation-Date</tt>, <tt>PO-Revision-Date</tt>, <tt>Last-Translator</tt>, <tt>X-Generator</tt>. When a header patch is accepted, these fields in the target header are overwritten with those from the patch (including being added or removed).

=== Handling Rejected Ediffs ===

Any rejected ediff messages will be written out to <tt>stdin.rej.po</tt> if the patch was read from standard input, or to <tt><catname>.rej.po</tt> if it was given as file through <tt>-i</tt> option (e.g. <tt>ediff.rej.po</tt> for input <tt>ediff.po</tt>).

File with rejected ediff messages will again be an ediff PO file. It will have the header as before, except that its title comment will mention, in free prose, that this particular ediff PO contains rejects of some patching operation. Afterwards, ediff messages rejected as patch will follow. Header ediff messages will be present whether rejected or not, for the same purpose of separation and provision of file paths, but they will be stripped of comments and <tt>msgstr</tt> when the header patch itself was not rejected.

Furthermore, to every straigh-out rejected ediff message an <tt>ediff-no-match</tt> flag will be added. This is done, naturally, because some ediff messages may not be rejected straight-out, but go through some post-processing instead. Consider the following scenario. A catalog has been merged to produce the fuzzy message:

{|width="100%"
|-
|style="padding-right: 1em; width: 1%"|'''old'''
|style="padding-bottom: 0.5em"|
<code po>
#: tools/power.c:348
msgid "Active sonar low frequency"
msgstr "Niska frekvencija aktivnog sonara"
</code>
|-
|style="padding-right: 1em; width: 1%"|'''new'''
|style="padding-bottom: 0.5em"|
<code po>
#: tools/power.c:361
#, fuzzy
#| msgid "Active sonar low frequency"
msgid "Active sonar high frequency"
msgstr "Niska frekvencija aktivnog sonara"
</code>
|}

Translator updates the translation, which produces the usual ediff message on update from fuzzy to translated:

{|width="100%"
|-
|style="padding-right: 0.5em; width: 1%"|'''ediff'''
|style="padding-bottom: 0.5em"|
<code po>
#. ediff: state {-fuzzy-}
#: tools/power.c:361
msgid "Active sonar {-low-}{+high+} frequency"
msgstr "{-Niska-}{+Visoka+} frekvencija aktivnog sonara"
</code>
|}

However, ''before'' this patch could have been applied, the programmer adds a trailing colon to the same message, and the catalog is merged again to produce:

{|width="100%"
|-
|style="padding-right: 1em; width: 1%"|'''new2'''
|style="padding-bottom: 0.5em"|
<code po>
#: tools/power.c:361
#, fuzzy
#| msgid "Active sonar low frequency"
msgid "Active sonar high frequency:"
msgstr "Niska frekvencija aktivnog sonara"
</code>
|}

The patch can no longer be cleanly applied, due to the extra colon added in the meantime to the <tt>msgid</tt>, so it has to be rejected. If nothing else is done, it would appear in the file of rejects as:

<code po>
#. ediff: state {-fuzzy-}
#: tools/power.c:361
#, ediff-no-match
msgid "Active sonar {-low-}{+high+} frequency"
msgstr "{-Niska-}{+Visoka+} frekvencija aktivnog sonara"
</code>

It seems a bit wastefull to reject such a near-match patch without any indication that it could be easily adapted to suit the latest message in the target PO. Therefore, when an ediff message is rejected, the following analysis is performed: by trying out message pairings as described for diffing, could the old message from the patch be paired with a current message from the target PO, and that current message with the new message from the patch? Or, in other words, can an existing message in the target PO be "fitted in between" the old and new messages defined by the patch? If this is the case, instead of the original, two special ediff messages -- ''split rejects'' -- are constructed and written out: one from the old to the current message, and another from the current to the new message. They are flagged as <tt>ediff-to-cur</tt> and <tt>ediff-to-new</tt>, respectively:

<code po>
#: tools/power.c:361
#, fuzzy, ediff-to-cur
#| msgid "Active sonar low frequency"
msgid "Active sonar high frequency{+:+}"
msgstr "Niska frekvencija aktivnog sonara"
⁠
#. ediff: state {-fuzzy-}
#: tools/power.c:361
#, ediff-to-new
#| msgid "Active sonar {-low-}{+high+} frequency{+:+}"
msgid "Active sonar {-low-}{+high+} frequency"
msgstr "{-Niska-}{+Visoka+} frekvencija aktivnog sonara"
</code>

There are more ways to interpret split rejects, depending on the circumstances. In this example, from <tt>ediff-to-cur</tt> message reviewer can see what had changed in the target message after the translator made the ediff. This can also be seen by comparing difference embedded into previous and current <tt>msgid</tt> fields in the <tt>ediff-to-new</tt> message. With slightly more extensive editing, the reviewer can fold these two messages into an applicable patch:

<code po>
#. ediff: state {-fuzzy-}
#: tools/power.c:361
#, ediff
msgid "Active sonar {-low-}{+high+} frequency:"
msgstr "{-Niska-}{+Visoka+} frekvencija aktivnog sonara"
</code>

Given that the file of rejected ediffs is also an ediff PO, after edits to make some of the rejected patches applicable, it can be ''reapplied'' as patch. If that is done, <tt>poepatch</tt> will silently ignore all ediff messages having <tt>ediff-no-match</tt> and <tt>ediff-to-new</tt> flag, as these have already been determined inapplicable. That is why in the example above the reviewer has replaced the <tt>ediff-to-new</tt> with the plain <tt>ediff</tt> flag on the folded ediff.

=== Embedding Patches ===

Depending on the type of text which is being translated, and distance of translation language's grammar, ortography and style from English, it may be difficult to review an ediff in isolation. In general, messages in ediff PO will lack ''positional context'', which is in the full PO provided by messages immediately preceding and following the target message. For example, a long passage from documentation probably needs no positional context. But a short newly added message such as "Crimson" could very well need one, if it has neither <tt>msgctxt</tt> nor extracted comment describing it. Is it really a color? What grammatical ending should it have, in a language which matches adjective to noun gender? Several messages around it in the full PO could easily show whether it is just another color in a row, and their grammatical endings (determined by a translator earlier) would show the needed ending for the new color.

Then, if an ediff message needs some editing before being applied, it may not be easy to do this directly in the ediff PO. Everything is fine so long as only added text segments (<tt>{+...+}</tt>) are edited, but if the sentence needs to be restructured more thoroughly, reviewer would have to make sure to put all additions into existing or new <tt>{+...+}</tt> segments, and to wrap all removals as <tt>{-...-}</tt> segments. If this is not carefully performed, the patch will not be applicable any more, as old message resolved from it will no longer exactly match a message in the target PO.

For these reasons, <tt>poepatch</tt> can apply the patch such as ''not to resolve the ediff'', but to set all its extraction-invariant fields to the message in the patched PO. In effect, the patched PO becomes an ediff PO by itself, but only in the messages which were actually patched. To mark these messages for lookup, <tt>ediff</tt> flag is added to them. If the message in the ediff PO was:

<code po>
#: title.c:274
msgid "Tutorial"
msgstr "{-Tutorijal-}{+Podučavanje+}"
</code>

then when the patch is successfully applied with embedding, the patched message in target PO will look like this:

<code po>
#: main.c:110
msgid "Records of The Witch River"
msgstr "Beleške o Veštičjoj reci"
⁠
#: title.c:292
#, ediff
msgid "Tutorial"
msgstr "{-Tutorijal-}{+Podučavanje+}"
⁠
#: title.c:328
msgid "Start the Expedition"
msgstr "Pođi u ekspediciju"
</code>

Other than the addition <tt>ediff</tt> flag, note that the patched message also kept its source reference, rather than it being overwritten by that from ediff PO. Same holds for all extraction-prescribed parts.

Reviewer can now jump from <tt>ediff</tt> to <tt>edif</tt> flag, always having the full positional context for each patched message, and being able to edit it to heart's content, with only minimal care not to invalidate the ediff format. Wrapped difference segments can be entirely removed, non-wrapped segments can be freely edited; it should only not happen that a wrapped segment looses its opening or closing sequence. But this does not mean that the reviewer ''needs'' to remove or touch difference segments at all, that is, to unembed patched messages by hand -- <tt>poepatch</tt> will do that automatically, when run on embedded-patched POs with a particular option.

A patch is applied with embedding by issuing the <tt>-e</tt> (<tt>--embed</tt>) option (<tt>E</tt> in parenthesis in progress output indicates that embedding is engaged):

<code bash>
$ poepatch -e <ediff.po
patched (E): lang.po
$
</code>

When the patched PO has been reviewed and patched messages possibly edited, all remaining embedded differences are removed, i.e. resolved to new versions, by running:

<code bash>
$ poepatch -u lang.po
</code>

Only those messages having the <tt>ediff</tt> flag are resolved, therefore the reviewer should ''never'' remove them (unless manually unembedding the whole patched message).

What happens with rejected patches when embedding is engaged? They are also added into the target PO, with heuristic positioning, and no separate ediff PO with rejects is created. Same as with plain patching, straight-out rejects will have <tt>ediff-no-match</tt> flag, and split rejects <tt>ediff-to-cur</tt> and <tt>ediff-to-new</tt>. If these are not manually resolved during the review (<tt>ediff-no-match</tt> messages removed, <tt>ediff-to-*</tt> messages removed or folded), when <tt>poepatch</tt> is run to unembed the patch, it will remove all <tt>ediff-no-match</tt> and <tt>ediff-to-new</tt> messages, and resolve <tt>ediff-to-cur</tt> messages to new version (effectively rejecting patches from which split rejects had originated).

== Lightweight Diffing when Updating Translation ==

Rather than for reviewing changes and sending patches, the translator may want to have only a convenient diff between the previous and new original text when updating a catalog merged with <tt>--previous</tt> option. Repeating the (by now) usual example message:

<code po>
#: main.c:110
#, fuzzy
#| msgid "The Record of The Witch River"
msgid "Records of The Witch River"
msgstr "Beleška o Veštičjoj reci"
</code>

Obviously, a dedicated PO editor could automatically present appropriate diff for messages like these (e.g. [[Localization/Tools/Lokalize|Lokalize]] can do it since inception), so for translators using such an editor, diffing of fuzzies is either already available or a strong candidate for a feature request. The rest of this section therefore applies only to translators who stick to general text editors (with or without some power-assists for editing PO files).

One ''could'' use <tt>poediff</tt> and <tt>poepatch</tt> for the purpose of diffing of fuzzies, by first diffing from previous complete to new merged catalog, embedding the patch into the merged catalog, updating translation, and then unembedding ediffs. One drawback of this, however, is that two version of the catalog are necessary, when only the merged one contains all the information. More drawbacks are that the catalog has to be cleared of ediffs after updating, or that each diffed message contains more information than necessary (<tt>ediff</tt> flag, <tt>#. ediff:</tt> comment) since all ediffs are of one and the same type.

In short, full embedded diffing is too heavyweight as an assist to updating fuzzy messages. Instead, there is a Pology sieve intended for this particular purpose, <tt>diff-previous</tt>, which embeds the difference from previous to current original text into previous fields. Executing:

<code bash>
$ posieve diff-previous lang.po
</code>

will modify fuzzy messages in-place in the processed catalog, to the following partial ediff:

<code po>
#: main.c:110
#, fuzzy
#| msgid "{-The Record-}{+Records+} of The Witch River"
msgid "Records of The Witch River"
msgstr "Beleška o Veštičjoj reci"
</code>

The format of embedding is the same as before, so the ediff-aware syntax highlighting works here too. When the message is updated and properly unfuzzied (both <tt>fuzzy</tt> flag and previous fields removed), none of the ediff remains, so there is no need for post-unembedding. In other words, translator updates translation just as used to before, with the free benefit of having the ediff of original text.

Note that no embedding should remain in previous fields by the time the catalog is merged again, as it would throw off <tt>msgmerge</tt> when matching to create new fuzzies. This will normally not be an issue, as the translator will have unfuzzied all messages when updating the catalog after last merging. However, if there were a lot of fuzzy messages, and translator didn't have the time to update all of them, <tt>diff-previous</tt> sieve can also be used to unembed any remaining ediffs, restoring the original values of previous fields. It just needs to be run on the catalog with extra parameter <tt>strip</tt>:

<code bash>
$ posieve diff-previous -sstrip lang.po
</code>

Localization/Tools/Pology/PO Embedded Diffing

2010-06-13T08:26:31Z

Ilic: Example corrected: do not remove, but change ediff-to-new to ediff.

{{Template:I18n/Language Navigation Bar|Localization/Tools/Pology/PO_Embedded_Diffing}}

{{LocalizationBrowser|
section=[[Localization#Tools|Tools]]|
name=Embedded Diffing with Pology|
prereqs=[[Localization/Tools/Pology|Pology]]|
}}

== Diffing of PO So Far ==

'''Line-level''' diffing assumes that the maximal generaly recognizable unit of text file is one line, that each line carries a meaning of significant standalone value, and that the ordering of lines has been deliberately chosen and cannot be arbitrary. For example, this is typical of program code.

Superficially, PO files could also be considered "a programming language of translations", and amenable to same treatment on diffing. However, some of the above assumptions which make line-level diffing viable are violated with PO format. Firstly, minimal unit of PO file is one message, whereas one line is of negligible semantic value. Secondly, ordering of messages can in principle be arbitrary (e.g. dependent on the order of extraction from source code), such that two line-wise very different PO files are actually equivalent from translator's viewpoint. And thirdly, good number of lines in the PO file are auxiliary, neither original text nor translation, generated either automatically or by the programmer (e.g. source references, extracted comments), all of which are out of translator's scope for modifications.

Thus, a common way to use line-level diffing with PO files so far was only for review, and with some preparations. Due to myriad of equivalent but line-wise different representations of PO content, it is quite useless to send line diffs as patches; translators are instructed to always send full PO files to the reviewer/commiter, no matter the amount of modifications. Then, the reviewer merges the received PO file (new version), and possibly the original (old version), with current template, without wrapping of long lines in text fields. This "normalizes" old and new files with respect to all semantically non-significant elements, and only then can line diff be performed. Additionally, since a long non-wrapped line of translated text may differ only in some segments, a dedicated diff viewer which can highlight '''word-level''' differences has to be used; ordinary diff syntax highlighting (e.g. in shell, or in general text editor) won't cut it.

Even with such preparations and dedicated diff viewer at hand, there is at least one significant case which is still not reasonably covered: when a fuzzy message, which had previous fields (PO was merged with <tt>--previous</tt> option to <tt>msgmerge</tt>), has been updated and unfuzzied. For example:

{|width="100%"
|-
|style="padding-right: 1em; width: 1%"|'''old'''
|style="padding-bottom: 0.5em"|
<code po>
#: main.c:110
#, fuzzy
#| msgid "The Record of The Witch River"
msgid "Records of The Witch River"
msgstr "Beleška o Veštičjoj reci"
</code>
|-
|style="padding-right: 1em; width: 1%"|'''new'''
|style="padding-bottom: 0.5em"|
<code po>
#: main.c:110
msgid "Records of The Witch River"
msgstr "Beleške o Veštičjoj reci"
</code>
|-
|style="padding-right: 0.5em; width: 1%"|'''diff'''
|style="padding-bottom: 0.5em"|
<code diff>
⁠ #: main.c:110
- #, fuzzy
- #| msgid "The Record of The Witch River"
msgid "Records of The Witch River"
- msgstr "Beleška o Veštičjoj reci"
+ msgstr "Beleške o Veštičjoj reci"
</code>
|}

Here, the diff viewer will know to show word-level diff for modified translation, but it cannot know that it should also show word-level diff between the removed previous and current <tt>msgid</tt> fields, so that reviewer can see what has changed ''in the original'' text (i.e. why the message became fuzzy), and based on that judge the change in translation.

Finally, a dedicated PO editor may be able to show the truly proper, '''message-level''' difference (such as [[Localization/Tools/Lokalize|Lokalize]] can, operating in [http://docs.kde.org/stable/en/kdesdk/lokalize/sync.html merge mode]). Even then, however, there is still the need to send around full PO files, and, to some extent, to normalize them before comparing in the editor. The diff format becomes tied to and defined in terms of the given PO editor (which you may not want to use in general), instead of being intrinsically defined and modularly processable (such as line diffs are).

The rest of this article will therefore propose a format and semantics for self-contained, message-level diffing of PO files -- the '''embedded diff''' -- and present two [[Localization/Tools/Pology|Pology]] scripts which embody it as proof of concept (but are also quite practically applicable).

== The Embedded Diff ==

Difference between two PO messages should be primarily, though not exclusively, composed of differences between its text fields (<tt>msgid</tt>, <tt>msgctxt</tt>, etc.) Then, to be easily judged, differences in text should be presented as succinctly and locally as possible -- think of a long paragraph where only some spelling or punctuation has changed. Finally, the format of the complete message diff should be easily comprehendable to translators used to PO format itself, and as far as possible, to existing PO tools too.

Previous considerations lead to the following decision: ''PO message diff will also be a PO message''. Or, in other words, the diff will be ''embedded'' into the regular parts of a PO message. An embedded diff (''ediff'' for short) message should be at least syntactically a valid PO entry, if not always semantically (i.e. if part of PO file, it should pass <tt>msgfmt</tt>, though not necessarily <tt>msgfmt --check</tt>). To enable ediffs to be passed around as patches for PO files, the embedding should be automatically resolvable (up to significant message parts) to the old and new messages from which the diff was created.

In this way, if ediff messages are packed into a PO file (an ediff PO), existing PO tools can be used to review and modify the diff. For example, highlighting in a text editor will need only minimal upgrades to show the embedded differences (more on that below), and otherwise it will already highlight ediff message parts as usual.

To complete the definition of ediffs in detail, the following questions should be answered:

* How to represent embedded differences in text?

* Which parts of the message should be diffed?

* How to pair for diffing messages from two files?

* How to present collection of diffed messages?

=== Embedding Differences into Text ===

Once the difference between the old and new strings, the word-level difference, has been determined, it should be decided how to embed it into the new (or, equivalently, old) string. One possibility is that of <tt>wdiff(1)</tt>, where removed and added text segments are by default wrapped with <tt><nowiki>[-...-]</nowiki></tt> and <tt><nowiki>{+...+}</nowiki></tt>, respectively:

{|width="100%"
|-
|style="padding-right: 1em; width: 1%"|'''old'''
|style="padding-bottom: 0.5em"|
<code text>
"The Record of The Witch River"
</code>
|-
|style="padding-right: 1em; width: 1%"|'''new'''
|style="padding-bottom: 0.5em"|
<code text>
"Records of The Witch River"
</code>
|-
|style="padding-right: 0.5em; width: 1%"|'''diff'''
|style="padding-bottom: 1em"|
<code text>
"[-The Record-]{+Records+} of The Witch River"
</code>
|}

However, search through PO files of several translation projects (KDE, Gnome, OpenOffice, Fedora, Mozilla) reveals the <tt><nowiki>[-</nowiki></tt> character combination to be frequently encountered, e.g. in synopsis of command usage. While there must exist an escaping mechanism for cases when diff wrappers are encountered in the original text, to enable unambiguous resolving of ediffs, it is nevertheless prudent to pick wrappers which reduce frequency of escaping (e.g. syntax highlighting may not be able to discern non-diff wrapper-like segments). Searching through same collection of PO files produces no hits for <tt>{-</tt>, so this combination is picked instead as wrapper of removed text segments:

{|width="100%"
|-
|style="padding-right: 1em; width: 1%"|'''old'''
|style="padding-bottom: 0.5em"|
<code text>
"The Record of The Witch River"
</code>
|-
|style="padding-right: 1em; width: 1%"|'''new'''
|style="padding-bottom: 0.5em"|
<code text>
"Records of The Witch River"
</code>
|-
|style="padding-right: 0.5em; width: 1%"|'''ediff'''
|style="padding-bottom: 1em"|
<code text>
"{-The Record-}{+Records+} of The Witch River"
</code>
|}

If the text itself contains a wrapper-like combination, it is escaped by inserting tilde (<tt>~</tt>) between the brace and plus/minus sign:

{|width="100%"
|-
|style="padding-right: 1em; width: 1%"|'''old'''
|style="padding-bottom: 0.5em"|
<code text>
"Foo {+ bar"
</code>
|-
|style="padding-right: 1em; width: 1%"|'''new'''
|style="padding-bottom: 0.5em"|
<code text>
"Foo {+ qwyx"
</code>
|-
|style="padding-right: 0.5em; width: 1%"|'''ediff'''
|style="padding-bottom: 1em"|
<code text>
"Foo {~+ {-bar-}{+qwyx+}"
</code>
|}

If the text already contains a substring with brace-tilde-plus/minus, then another tilde is inserted, and so on. Thus the ediff can be unambiguously resolved to old and new versions of the string. Inserting the tilde between the two characters of wrapper combinations also makes it easier on the syntax higlighting, as the difference highlighting trigger is automatically removed.

It may happen that a given string is not merely empty in the old or new PO message, but that it does not exist at all (e.g. <tt>msgctxt</tt> field). Thus an ediff can be made between existing and non-existing strings too, in which case a tilde is appended to the very end of the ediff:

{|width="100%"
|-
|style="padding-right: 1em; width: 1%"|'''old'''
|style="padding-bottom: 0.5em"|
<code text>
</code>
|-
|style="padding-right: 1em; width: 1%"|'''new'''
|style="padding-bottom: 0.5em"|
<code text>
"a-context-note"
</code>
|-
|style="padding-right: 0.5em; width: 1%"|'''ediff'''
|style="padding-bottom: 1em"|
<code text>
"{+a-context-note+}~"
</code>
|}

Again, escaping is provided for by inserting further tildes if the ediff between two existing strings would result in trailing tilde (old: <tt>"~"</tt>, new: <tt>"foo~"</tt>, ediff: <tt>"{+foo+}~~"</tt>).

How exactly the difference between two strings is formed, may be left to implementation. In fact, an implementation may allow translator to select between several diffing algorithms, depending on personal taste and situation. For example, the default algorithm of <tt>poediff</tt> does the following: words are diffed as whole, all non-word segments (interpunction, markup tags, etc.) character by character, and equal non-word segments between different words are taken into the difference segment. Hence the above ediff

<code text>
"{-The Record-}{+Records+} of The Witch River"
</code>

instead of smaller

<code text>
"{-The -}Record{+s+} of The Witch River"
</code>

as the former is (tentatively) easier to comprehend.

Every difference segment in an ediff PO message will be represented like this, thus it is sufficient to upgrade PO syntax highlighting of an editor to indiscriminately highlight <tt>{-...-}</tt> and <tt>{+...+}</tt> segments everywhere in the message.

=== Message Parts Included in Diffing ===

A PO message consists of several types of parts: text fields, comments, flags, source references, etc. It would not be very constructive to diff all of them; for example, while <tt>msgstr</tt> fields should clearly be included into diffing, source references most probably should not. In order not to consider pros and cons of inclusion for each and every message part, there already exists a clear split of message parts into two groups, one of which will be taken into diffing, and the other ignored. These two groups are:

* ''extraction-invariant'' parts -- those which do not depend on placement (or even presence) of message in the sources, such as <tt>msgid</tt> field, <tt>msgstr</tt> fields, manual comments, etc.

* ''extraction-prescribed'' parts -- those which cannot exist independently of the source from which the message is extracted, such as format flags or extracted comments.

Extraction-invariant parts are the ones which will be diffed. Working definition of exactly which parts belong into this group is provided by obsolete messages in PO files. Thus, these parts are:

* current original: <tt>msgctxt</tt>, <tt>msgid</tt>, and <tt>msgid_plural</tt> fields

* previous original: commented <tt>#| msgctxt</tt>, <tt>#| msgid</tt>, and <tt>#| msgid_plural</tt> fields

* translation: <tt>msgstr</tt> fields

* translator (manual) comments

* fuzzy state (whether the <tt>fuzzy</tt> flag is present)

* obsolete state (whether the message is obsolete)

All the listed fields and manual comments are presented in ediff message as wrapped word-level differences, as described earlier. Change in states, fuzzy and obsolete, is represented slightly differently. A special "extracted" (automatic) comment is added to the ediff message, starting with <tt>ediff:</tt> and listing any extra info needed to describe the ediff, including the state changes. Here is an example of two messages and the ediff they would produce (whether two messages such as these would get paired for diffing in the first place, will be discussed later on):

{|width="100%"
|-
|style="padding-right: 1em; width: 1%"|'''old'''
|style="padding-bottom: 0.5em"|
<code po>
#, fuzzy
#~| msgid "Accurate subpolar weather cycles"
#~ msgid "Accurate subpolar climate cycles"
#~ msgstr "Tačni ciklusi subpolarnog vremena"
</code>
|-
|style="padding-right: 1em; width: 1%"|'''new'''
|style="padding-bottom: 0.5em"|
<code po>
#. ui: property (text), widget (QCheckBox, accCyclesTrop)
#: config.ui:180
#, fuzzy
#| msgid "Accurate tropical weather cycles"
msgctxt "some-superfluous-context"
msgid "Accurate tropical climate cycles"
msgstr "Tačni ciklusi tropskog vremena"
</code>
|-
|style="padding-right: 0.5em; width: 1%"|'''ediff'''
|style="padding-bottom: 1em"|
<code po>
#. ediff: state {-obsolete-}
#. ui: property (text), widget (QCheckBox, accCyclesTrop)
#: config.ui:180
#, fuzzy
#| msgid "Accurate {-subpolar-}{+tropical+} weather cycles"
msgctxt "{+some-superfluous-context+}~"
msgid "Accurate {-subpolar-}{+tropical+} climate cycles"
msgstr "Tačni ciklusi {-subpolarnog-}{+tropskog+} vremena"
</code>
|}

Here is how this ediff message will look like in KWrite and Kate, starting with KDE 4.2:

[[Image:Kate_PO_Ediff_Highlighting.png|center|Ediff syntax highlighting in Kate]]

(In fact, ediff-aware [http://websvn.kde.org/*checkout*/trunk/KDE/kdelibs/kate/syntax/data/gettext.xml?content-type=text%2Fplain PO syntax highlighting definition] for Kate can be used with Kate versions as early as KDE 3.4. The definition file should simply be placed as <tt>~/.kde/share/apps/katepart/syntax/gettext.xml</tt>, and it will override the system definition.)

First thing to note is that the ediff message contains not only extraction-invariant parts, but also copies verbatim extraction-prescribed parts from the new message. Effectively, the ediff is embedded into the copy of new message. Extraction-prescribed parts are not simply discarded in order to provide more context when reviewing the diff. Here, for example, the extracted comment states that the text is a checkbox label, which may be important for the style of translation.

The other important element is the <tt>#. ediff:</tt> dummy extracted comment, which here indicates that the obsolete state has been "removed", i.e. the message was unobsoleted going from old to new version. Aside from state changes, few other indicators may rarely be present, as described in API documentation of <tt>pology.misc.diff</tt> module, function <tt>msg_ediff</tt>. Some of these indicators will be mentioned later on. The ediff comment will be present only when necessary, if there are any indicators to show.

If diffing of two messages would always be conducted part for part, for all parts which are taken into diffing, then in some cases the resulting ediff would not be very useful. Consider how the first example in the article, the line-level diff of a fuzzy and translated message, would look like as ediff if performed part for part:

{|width="100%"
|-
|style="padding-right: 1em; width: 1%"|'''old'''
|style="padding-bottom: 0.5em"|
<code po>
#: main.c:110
#, fuzzy
#| msgid "The Record of The Witch River"
msgid "Records of The Witch River"
msgstr "Beleška o Veštičjoj reci"
</code>
|-
|style="padding-right: 1em; width: 1%"|'''new'''
|style="padding-bottom: 0.5em"|
<code po>
#: main.c:110
msgid "Records of The Witch River"
msgstr "Beleške o Veštičjoj reci"
</code>
|-
|style="padding-right: 0.5em; width: 1%"|'''ediff'''
|style="padding-bottom: 0.5em"|
<code po>
#. ediff: state {-fuzzy-}
#: main.c:110
#| msgid "{-The Record of The Witch River-}~"
msgid "Records of The Witch River"
msgstr "{-Beleška-}{+Beleške+} o Veštičjoj reci"
</code>
|}

The same problem as with the line-level diff persists: instead of showing the difference from previous to current <tt>msgid</tt> field, the current <tt>msgid</tt> is left untouched, and previous <tt>msgid</tt> is simply shown to have been entirely removed.

Therefore, instead of straightforward, part for part diffing, a special transformation will take place when ''exactly one'' of the two diffed messages is fuzzy and equipped with previous original text fields. This splits into two directions: from fuzzy to non-fuzzy, and from non-fuzzy to fuzzy.

Diffing from fuzzy to non-fuzzy message, such as above, is the more usual of the two directions. It is typical of translation updated after merging with template, where some fuzzied messages will have been resolved. In this case, the original old and new messages are transformed thusly prior to diffing (<tt>*-rest</tt> denotes all diffed parts that are neither original text nor fuzzy state):

{|width="100%"
|-
|style="padding-right: 1em; width: 1%"|'''old'''
|style="padding-bottom: 0.5em"|
<code text>
fuzzy --> fuzzy
old-previous-fields --> old-previous-fields
old-current-fields --> old-previous-fields
old-rest --> old-rest
</code>
|-
|style="padding-right: 1em; width: 1%"|'''new'''
|style="padding-bottom: 0.5em"|
<code text>
- --> -
- --> old-current-fields
new-current-fields --> new-current-fields
new-rest --> new-rest
</code>
|}

In this way, ediff message's current fields will show the important difference, that of previous fields of old fuzzy message and current fields of new non-fuzzy message. Ediff message's previous fields will show the less important difference of old fuzzy messages previous and current fields, but ''only'' if it is not equal to the difference in current fields; otherwise it is eliminated. This may sound confusing, but the final ediff produced in this way is quite intuitive:

{|width="100%"
|-
|style="padding-right: 1em; width: 1%"|'''old'''
|style="padding-bottom: 0.5em"|
<code po>
#: main.c:110
#, fuzzy
#| msgid "The Record of The Witch River"
msgid "Records of The Witch River"
msgstr "Beleška o Veštičjoj reci"
</code>
|-
|style="padding-right: 1em; width: 1%"|'''new'''
|style="padding-bottom: 0.5em"|
<code po>
#: main.c:110
msgid "Records of The Witch River"
msgstr "Beleške o Veštičjoj reci"
</code>
|-
|style="padding-right: 0.5em; width: 1%"|'''ediff'''
|style="padding-bottom: 0.5em"|
<code po>
#. ediff: state {-fuzzy-}
#: main.c:110
msgid "{-The Record-}{+Records+} of The Witch River"
msgstr "{-Beleška-}{+Beleške+} o Veštičjoj reci"
</code>
|}

The reviewer here sees that the message was unfuzzied, the change in the original text that caused the message to become fuzzy, and the change made in translation to unfuzzy it. Old version (in removed and equal segments) of original and translation is that of the message before it got fuzzied, and new version (in added and equal segments) is that of the message after it was unfuzzied.

From non-fuzzy to fuzzy message should be the less frequent direction of diffing. It corresponds e.g. to case where the diff is taken from older fully complete translation to the one just after merging with newest template. In this case, the transformation is as follows:

{|width="100%"
|-
|style="padding-right: 1em; width: 1%"|'''old'''
|style="padding-bottom: 0.5em"|
<code text>
- --> -
- --> new-previous-fields
old-current-fields --> old-current-fields
old-rest --> old-rest
</code>
|-
|style="padding-right: 1em; width: 1%"|'''new'''
|style="padding-bottom: 0.5em"|
<code text>
fuzzy --> fuzzy
new-previous-fields --> new-current-fields
new-current-fields --> new-current-fields
new-rest --> new-rest
</code>
|}

Again, the difference presented by ediff messages's current fields will be the most important one, the difference in previous fields the less important one, and eliminated if equal to the other. Here is what this will do if applied to one step earlier (just after merging) of the running example:

{|width="100%"
|-
|style="padding-right: 1em; width: 1%"|'''old'''
|style="padding-bottom: 0.5em"|
<code po>
#: main.c:89
msgid "The Record of The Witch River"
msgstr "Beleška o Veštičjoj reci"
</code>
|-
|style="padding-right: 1em; width: 1%"|'''new'''
|style="padding-bottom: 0.5em"|
<code po>
#: main.c:110
#, fuzzy
#| msgid "The Record of The Witch River"
msgid "Records of The Witch River"
msgstr "Beleška o Veštičjoj reci"
</code>
|-
|style="padding-right: 0.5em; width: 1%"|'''ediff'''
|style="padding-bottom: 0.5em"|
<code po>
#. ediff: state {+fuzzy+}
#: main.c:110
#, fuzzy
msgid "{-The Record-}{+Records+} of The Witch River"
msgstr "Beleška o Veštičjoj reci"
</code>
|}

The reviewer here sees that the message became fuzzy from new to old, and the change in original text that caused this. (On a side note, remember that ediff message is constructed by embedding differences into a copy of new message, so the source reference and the fuzzy flag from the new message appear here in the ediff message.)

To be able to use the ediff as patch, it is necessary to reconstruct original old and new messages after resolving ediff into transformed old and new messages. This step is fortunatelly unambiguous; one just needs to check whether the non-fuzzy of the two resolved messages has previous fields, or, if not (due to elimination of equal difference in previous fields, or because template was merged without the <tt>--previous</tt> option), whether the current fields are equal. Then the back-transformation to original old and new messages can be performed.

=== Pairing Messages From Two Catalogs ===

So far the article has described how to make an embedded diff out of two messages, once it has been decided that those messages should be diffed. However, on the lowest level, the user decides not which messages to diff, but which two PO files to diff. The implementation should then automatically ''pair'' for diffing messages from the two catalogs, and this section described several possible ways to do this.

In the first instance, messages should obviously be '''paired by key''' (primary pairing), the unique combination of <tt>msgctxt</tt> and <tt>msgid</tt> fields. In the most usual case -- reviewing an ediff from incomplete catalog with some fuzzy and untranslated messages, to an updated catalog with some or all of those messages translated -- pairing by key will be fully sufficient, as both catalogs contain exactly the same set of messages by keys. These two messages will be plainly paired by key:

{|width="100%"
|-
|style="padding-right: 1em; width: 1%"|'''old'''
|style="padding-bottom: 0.5em"|
<code po>
#: main.c:110
#, fuzzy
#| msgid "The Record of The Witch River"
msgid "Records of The Witch River"
msgstr "Beleška o Veštičjoj reci"
</code>
|-
|style="padding-right: 1em; width: 1%"|'''new'''
|style="padding-bottom: 0.5em"|
<code po>
#: main.c:110
msgid "Records of The Witch River"
msgstr "Beleške o Veštičjoj reci"
</code>
|}

But what should happen if some messages cannot be paired by key? Consider the earlier example where diff was taken from older fully translated, to newer merged catalog:

{|width="100%"
|-
|style="padding-right: 1em; width: 1%"|'''old'''
|style="padding-bottom: 0.5em"|
<code po>
#: main.c:89
msgid "The Record of The Witch River"
msgstr "Beleška o Veštičjoj reci"
</code>
|-
|style="padding-right: 1em; width: 1%"|'''new'''
|style="padding-bottom: 0.5em"|
<code po>
#: main.c:110
#, fuzzy
#| msgid "The Record of The Witch River"
msgid "Records of The Witch River"
msgstr "Beleška o Veštičjoj reci"
</code>
|}

The keys, here just current <tt>msgid</tt> fields, of the two message do not match, so they cannot be paired by key. Yet it would be rather ungainly to represent the old message as fully removed in the ediff, and the new message as fully added:

{|width="100%"
|-
|style="padding-right: 1em; width: 1%"|'''ediff'''
|style="padding-bottom: 0.5em"|
<code po>
#: main.c:89
msgid "{-The Record of The Witch River-}~"
msgstr "{-Beleška o Veštičjoj reci-}~"
⁠
#. ediff: state {+fuzzy+}
#: main.c:110
#, fuzzy
#| msgid "{+The Record of The Witch River+}~"
msgid "{+Records of The Witch River+}~"
msgstr "{+Beleška o Veštičjoj reci+}~"
</code>
|}

(That the message has been fully added or removed can be seen by trailing tilde in the <tt>msgid</tt> field, which indicates that the old or new <tt>msgid</tt> does not exist at all, and so neither the message with it.)

Instead, messages left unpaired by key should be tested for '''pairing by pivoting''' around previous fields (secondary pairing). The two messages above will thus be paired due to the fact that the current <tt>msgid</tt> of the old message is equal to the previous <tt>msgid</tt> of the new message, and will produce a single ediff message as shown earlier.

Finally, consider the third related combination, when the old catalog has not yet been merged with the template, while the new catalog has both been merged and its translation updated:

{|width="100%"
|-
|style="padding-right: 1em; width: 1%"|'''old'''
|style="padding-bottom: 0.5em"|
<code po>
#: main.c:89
msgid "The Record of The Witch River"
msgstr "Beleška o Veštičjoj reci"
</code>
|-
|style="padding-right: 1em; width: 1%"|'''new'''
|style="padding-bottom: 0.5em"|
<code po>
#: main.c:110
msgid "Records of The Witch River"
msgstr "Beleške o Veštičjoj reci"
</code>
|}

Once again, it would be a waste to present the old message as fully removed and the new message as fully added in the ediff. When a message is left unpaired after both pairing by key and pairing by pivoting, then the two catalogs should be merged in the background -- as if the new is the template for the old, and vice versa -- and then tested for chained pairing by pivoting and by key with merged catalog as intermediary. This '''pairing by merging''', or tertiary pairing, will then produce another natural ediff:

{|width="100%"
|-
|style="padding-right: 0.5em; width: 1%"|'''ediff'''
|style="padding-bottom: 0.5em"|
<code po>
#: main.c:110
msgid "{-The Record-}{+Records+} of The Witch River"
msgstr "{-Beleška-}{+Beleške+} o Veštičjoj reci"
</code>
|}

Implementations can decide which pairing modes beyond the primary, by key, to use. There should not be much reason not to perform secondary pairing, by pivoting, too. If tertiary pairing, by merging, is implemented, it should be provided that the user can disable it (sometimes this pairing may produce strange results, and needs <tt>msgmerge</tt> to be available on the system).

=== Collecting Diffed Messages ===

For ediff of two PO files to also be a syntactically valid PO file, constructed ediff messages should be preceded by a PO header on output. At first glance, this PO header could be itself the ediff of PO headers of the catalogs which were diffed. However, there are several issues with this approach:

* Reviewer of the ediff PO file would not be informed at once whereas there was any difference between the headers. Headers tend to be long, and a point change in one of the fields may go visually unnoticed.

* Depending on the amount of changes between the two headers, the resulting ediff message of the header could be too badly formed to represent the header as such (e.g. if header fields in <tt>msgstr</tt> were added or removed, embedded difference wrappers would invalidate MIME-header format of <tt>msgstr</tt>), and thus confuse the PO tools.

* How would the diff of two ''collections'' of PO files (e.g. directories) be packed into a single ediff PO, such as can normally be done with line-level diffing?

To avert these difficulties, the following is done instead. First, the header of ediff PO is constructed as a minimal valid header (i.e. one that <tt>msgfmt</tt> would not complain about) independent of the content of original headers. <tt>poediff</tt> will produce something like:

<code po>
# +- ediff -+
msgid ""
msgstr ""
"Project-Id-Version: ediff\n"
"PO-Revision-Date: 2009-02-08 01:20+0100\n"
"Last-Translator: J. Random Translator\n"
"Language-Team: Differs\n"
"MIME-Version: 1.0\n"
"Content-Type: text/plain; charset=UTF-8\n"
"Content-Transfer-Encoding: 8bit\n"
"X-Ediff-Header-Context: ~\n"
</code>

<tt>PO-Revision-Date</tt> header field is naturally set to the date when the ediff is made. <tt>Last-Translator</tt> and <tt>Language-Team</tt> can be somehow pulled from environment (<tt>poediff</tt> will source them from <tt>~/.pologyrc</tt>, or set some dummy values as above if not present). Encoding of the ediff PO can be chosen at will by the implementation, so long as all following ediff messages can be encoded with it (<tt>poediff</tt> will always use UTF-8). The purpose of the <tt>X-Ediff-Header-Context</tt> field will be explained shortly.

It is the first next message in the ediff PO that will actually be the ediff of headers of the two diffed PO files. Headers are diffed just like any other message, but the resulting ediff is equiped with few extra decorations:

<code po>
# =========================================================
# Translation of The Witch River into Serbian.
# Koja Kojic <koja.kojic@nedohodnik.net>, 2008.
# {+Era Eric <era.eric@ledopad.net>, 2008.+}~
msgctxt "~"
msgid ""
"- l10n-wr/sr/wriver-main.po\n"
"+ l10n-wr/sr-mod/wriver-main.po\n"
msgstr ""
"Project-Id-Version: wriver 0.1\n"
"POT-Creation-Date: 2008-09-22 09:17+0200\n"
"PO-Revision-Date: 2008-09-{-25 20:44-}{+28 21:49+}+0100\n"
"Last-Translator: {-Koja Kojic <koja.kojic@nedohodnik-}"
"{+Era Eric <era.eric@ledopad+}.net>\n"
"Language-Team: Serbian\n"
"MIME-Version: 1.0\n"
"Content-Type: text/plain; charset=UTF-8\n"
"Content-Transfer-Encoding: 8bit\n"
</code>

First observe the normal ediff segments: translator comment with a new translator who updated the PO file has been added, and <tt>PO-Revision-Date</tt> and <tt>Last-Translator</tt> header fields contain ediffs reflecting the update. These are the only actual differences between the two headers.

More interesting are the extra decorations:

* The very first translator comment (here a long line of equality signs) can be anything, and serves to give good visual distinction to the header ediff. This is more so convenient when the ediff PO contains diffs of several pairs of PO files.

* That a particular message in the ediff PO is a header ediff, is indicated by the <tt>msgctxt</tt> set to a special value, here a single tilde. This value is given up front by the <tt>X-Ediff-Header-Context</tt> of the ediff PO header. It should be computed during diffing such that it does not conflict with <tt>msgctxt</tt> of one of the normal message ediffs (e.g. it may simply be a sufficiently long sequence tildes).

* The <tt>msgid</tt> field of the header ediff contains newline-separated paths of the diffed PO files. More precisely, the two lines of the <tt>msgid</tt> field are in form <code text>[+-] file-path[ <<< comment]\n</code> (The trailing newline of the second file path is elided if the <tt>msgstr</tt> does not end in newline, to prevent <tt>msgfmt</tt> from complaining.) The optional, <<<-separated comment to the file path can be used for any purpose, one which will be demonstrated when describing functionality of <tt>poediff</tt>.

Although when a PO file is properly updated there should always be some difference in its header, it may happen that there is none. In such case, the header ediff message is still added, but such that it only contains the extra decorations -- the visual separator comment, special <tt>msgctxt</tt>, and <tt>msgid</tt> with file paths. All other comments and <tt>msgstr</tt> should be empty, and the empty <tt>msgstr</tt> indicates that there was no difference in headers. Presence of this "empty" header ediff is necessary to provide diffed file paths and, if several POs were diffed, to separate them in the ediff PO.

[[Image:Kate_Dir_Ediff_PO.png|right|150x150px|Embedded diff of directory of PO files in Kate.]]

After the header ediff message, ordinary ediff messages follow. As already obvious by now, when several POs were diffed to construct a single ediff PO, each next PO in the ediff simply opens with a new header ediff message. Click on the thumbnail on the right to see how an ediff of two PO files looks like in entirety, with syntax highlighting in Kate.

Especially when diffing several PO files, it may happen that two ediff messages have equal keys (<tt>msgid</tt> and <tt>msgctxt</tt> fields) and thus cannot be both added as such to the ediff PO. In such case, the ediff message which was added after the first with the same key, will have its <tt>msgctxt</tt> field ''padded'' by few random alphanumerics, such as to make its key unique. This padding sequence will be recorded in the ediff comment. For example:

<code po>
# =========================================================
msgctxt "~"
msgid "...(first PO header ediff)..."
msgstr "..."
⁠
#. ediff: state {-fuzzy-}
msgid "White{+ horizon+}"
msgstr "Belo{+ obzorje+}"
⁠
# =========================================================
msgctxt "~"
msgid "...(second PO header ediff)..."
msgstr "..."
⁠
#. ediff: state {-fuzzy-}, ctxtpad q9ac3
msgctxt "|q9ac3~"
msgid "White{+ horizon+}"
msgstr "Belo{+ obzorje+}"
</code>

The padding sequence is appended to the original <tt>msgctxt</tt>, separated by the pipe character. If there was no original <tt>msgctxt</tt>, the padding sequence is further extended by a tilde.

== Producing Ediffs: <tt>poediff</tt> ==

Pology's <tt>poediff</tt> script implements embedded diffing as outlined in the previous section. To diff two PO files, executing the usual:

<code bash>
$ poediff orig/foo.po mod/foo.po
</code>

will write the ediff PO (if there is any difference) to standard output, with some basic shell highlighting of difference segments. Option <tt>-o</tt> can be used to output the ediff PO to file instead. Other options include <tt>--no-merge</tt> (<tt>-n</tt>) to not perform pairing by merging, and <tt>--strip-headers</tt> (<tt>-s</tt>) to take a quick look of differences without headers in the way (output with stripped headers is not a valid PO, and cannot be used as patch).

It is equally simple to make ediff of directories:

<code bash>
$ poediff orig/ mod/
</code>

Diffing of directories is by default recursive and takes into ediff added and removed catalogs. When a catalog has been added or removed, the <tt>msgid</tt> of corresponding header ediff will have one of the file paths, new or old, empty.

To have non-dummy translator name and email address added to the header of ediff PO, set them in Pology's configuration file, <tt>~/.pologyrc</tt>:

<code ini>
[user]
name = Koja Kojic
original-name = Која Којић
email = koja.kojic@nedohodnik.net
</code>

(These entries cover all contexts in Pology where such information is of use; <tt>original-name</tt> field is used when the name is to be written differently in English and translator's native language.)

This is pretty much where the story about <tt>poediff</tt> would end, if not for its ability to take into account the underlying VCS, if one is used to control PO files.

=== Diffing With Underlying VCS ===

When PO files are under version control, <tt>poediff</tt> can operate in VCS mode using the option <tt>-c VCSKEY</tt> (<tt>--vcs</tt>), where <tt>VCSKEY</tt> is the keyword of one of VCS known to <tt>poediff</tt>. Then, instead of giving two paths to diff, any number of version-controlled paths (files or directories) is given. Without other options, all modified PO files in these paths are diffed against the last commit known to local repository. For example, if an application uses Subversion repository, PO files in its <tt>po/</tt> directory can be diffed with:

<code bash>
$ poediff -c svn app/po/
</code>

Some other set of revisions to diff can be given by the option <tt>-r REV1<nowiki>[:REV2]</nowiki></tt> (<tt>--revision</tt>). <tt>REV1</tt> and <tt>REV2</tt> are not necessarily proper revision IDs, but any strings that the underlying VCS will be able to convert into revision IDs. If <tt>REV2</tt> is omitted, diffing is preformed from <tt>REV1</tt> to current working copy.

When ediff is made in VCS mode, <tt>msgid</tt> fields of header ediffs will use <<<-separated comments to file paths to indicate revision IDs:

<code po>
# =========================================================
# ...
msgctxt "~"
msgid ""
"- app/po/lang.po <<< 20537\n"
"+ app/po/lang.po"
msgstr "..."
</code>

As for supported VCS, Pology currently knows about Subversion (<tt>svn</tt>) and Git (<tt>git</tt>). To add a new VCS, its functionality should be wrapped for Pology's use in <tt>pology.misc.vcs</tt>, using the interface defined by <tt>VcsBase</tt> class.

== Ediffs as Patches: <tt>poepatch</tt> ==

For basic functionality, applying an ediff patch is implementationally even easier than applying a line-level patch. Each ediff message in turn is resolved into originating old and new message, and if either the old or new message exists in target PO and is equal by extraction-invariant parts, then the message patch is applied, otherwise rejected.

Applying the patch means overwriting extraction-invariant parts of the target message with those of the new message from the ediff, and leaving other parts untouched. If the target message is already equal to new message by extraction-invariant parts, then the patch is silently ignored. This means that if the same patch is applied twice, second application makes no modifications to the target catalog. Likewise if, by chance, the changes given by the patch were already independently introduced in the target catalog.

Command-line interface of <tt>poepatch</tt> is much like <tt>patch(1)</tt>, sans the myriad of its more obscure options. There is the <tt>-p</tt> option to strip leading elements of file paths in the ediff, and <tt>-d</tt> option to append to them a directory path where target POs are to be found. If the files were diffed with underlying VCS as in the previous example, then the ediff could be applied in any of the following ways:

<code bash>
$ cd repos/app/po && poepatch <ediff.po
$ cd repos/ && poepatch -p0 <ediff.po
$ poepatch -d repos/app/po <ediff.po
</code>

Header patch (coming from the header ediff message) is applied in a slightly more relaxed fashion: some of the header fields are ignored when checking whether the patch is applicable. These are the fields which are known to be volatile as the PO file goes through different translators, and do not influence the processing of the catalog directed by the header (such as encoding or plural forms). Currently, these fields are: <tt>POT-Creation-Date</tt>, <tt>PO-Revision-Date</tt>, <tt>Last-Translator</tt>, <tt>X-Generator</tt>. When a header patch is accepted, these fields in the target header are overwritten with those from the patch (including being added or removed).

=== Handling Rejected Ediffs ===

Any rejected ediff messages will be written out to <tt>stdin.rej.po</tt> if the patch was read from standard input, or to <tt><catname>.rej.po</tt> if it was given as file through <tt>-i</tt> option (e.g. <tt>ediff.rej.po</tt> for input <tt>ediff.po</tt>).

File with rejected ediff messages will again be an ediff PO file. It will have the header as before, except that its title comment will mention, in free prose, that this particular ediff PO contains rejects of some patching operation. Afterwards, ediff messages rejected as patch will follow. Header ediff messages will be present whether rejected or not, for the same purpose of separation and provision of file paths, but they will be stripped of comments and <tt>msgstr</tt> when the header patch itself was not rejected.

Furthermore, to every straigh-out rejected ediff message an <tt>ediff-no-match</tt> flag will be added. This is done, naturally, because some ediff messages may not be rejected straight-out, but go through some post-processing instead. Consider the following scenario. A catalog has been merged to produce the fuzzy message:

{|width="100%"
|-
|style="padding-right: 1em; width: 1%"|'''old'''
|style="padding-bottom: 0.5em"|
<code po>
#: tools/power.c:348
msgid "Active sonar low frequency"
msgstr "Niska frekvencija aktivnog sonara"
</code>
|-
|style="padding-right: 1em; width: 1%"|'''new'''
|style="padding-bottom: 0.5em"|
<code po>
#: tools/power.c:361
#, fuzzy
#| msgid "Active sonar low frequency"
msgid "Active sonar high frequency"
msgstr "Niska frekvencija aktivnog sonara"
</code>
|}

Translator updates the translation, which produces the usual ediff message on update from fuzzy to translated:

{|width="100%"
|-
|style="padding-right: 0.5em; width: 1%"|'''ediff'''
|style="padding-bottom: 0.5em"|
<code po>
#. ediff: state {-fuzzy-}
#: tools/power.c:361
msgid "Active sonar {-low-}{+high+} frequency"
msgstr "{-Niska-}{+Visoka+} frekvencija aktivnog sonara"
</code>
|}

However, ''before'' this patch could have been applied, the programmer adds a trailing colon to the same message, and the catalog is merged again to produce:

{|width="100%"
|-
|style="padding-right: 1em; width: 1%"|'''new2'''
|style="padding-bottom: 0.5em"|
<code po>
#: tools/power.c:361
#, fuzzy
#| msgid "Active sonar low frequency"
msgid "Active sonar high frequency:"
msgstr "Niska frekvencija aktivnog sonara"
</code>
|}

The patch can no longer be cleanly applied, due to the extra colon added in the meantime to the <tt>msgid</tt>, so it has to be rejected. If nothing else is done, it would appear in the file of rejects as:

<code po>
#. ediff: state {-fuzzy-}
#: tools/power.c:361
#, ediff-no-match
msgid "Active sonar {-low-}{+high+} frequency"
msgstr "{-Niska-}{+Visoka+} frekvencija aktivnog sonara"
</code>

It seems a bit wastefull to reject such a near-match patch without any indication that it could be easily adapted to suit the latest message in the target PO. Therefore, when an ediff message is rejected, the following analysis is performed: by trying out message pairings as described for diffing, could the old message from the patch be paired with a current message from the target PO, and that current message with the new message from the patch? Or, in other words, can an existing message in the target PO be "fitted in between" the old and new messages defined by the patch? If this is the case, instead of the original, two special ediff messages -- ''split rejects'' -- are constructed and written out: one from the old to the current message, and another from the current to the new message. They are flagged as <tt>ediff-to-cur</tt> and <tt>ediff-to-new</tt>, respectively:

<code po>
#: tools/power.c:361
#, fuzzy, ediff-to-cur
#| msgid "Active sonar low frequency"
msgid "Active sonar high frequency{+:+}"
msgstr "Niska frekvencija aktivnog sonara"
⁠
#. ediff: state {-fuzzy-}
#: tools/power.c:361
#, ediff-to-new
#| msgid "Active sonar {-low-}{+high+} frequency{+:+}"
msgid "Active sonar {-low-}{+high+} frequency"
msgstr "{-Niska-}{+Visoka+} frekvencija aktivnog sonara"
</code>

There are more ways to interpret split rejects, depending on the circumstances. In this example, from <tt>ediff-to-cur</tt> message reviewer can see what had changed in the target message after the translator made the ediff. This can also be seen by comparing difference embedded into previous and current <tt>msgid</tt> fields in the <tt>ediff-to-new</tt> message. With slightly more extensive editing, the reviewer can fold these two messages into an applicable patch:

<code po>
#. ediff: state {-fuzzy-}
#: tools/power.c:361
#, ediff
msgid "Active sonar {-low-}{+high+} frequency:"
msgstr "{-Niska-}{+Visoka+} frekvencija aktivnog sonara"
</code>

Given that the file of rejected ediffs is also an ediff PO, after edits to make some of the rejected patches applicable, it can be ''reapplied'' as patch. If that is done, <tt>poepatch</tt> will silently ignore all ediff messages having <tt>ediff-no-match</tt> and <tt>ediff-to-new</tt> flag, as these have already been determined inapplicable. That is why in the example above the reviewer has replaced the <tt>ediff-to-new</tt> with the plain <tt>ediff</tt> flag on the folded ediff.

=== Embedding Patches ===

Depending on the type of text which is being translated, and distance of translation language's grammar, ortography and style from English, it may be difficult to review an ediff in isolation. In general, messages in ediff PO will lack ''positional context'', which is in the full PO provided by messages immediately preceding and following the target message. For example, a long passage from documentation probably needs no positional context. But a short newly added message such as "Crimson" could very well need one, if it has neither <tt>msgctxt</tt> nor extracted comment describing it. Is it really a color? What grammatical ending should it have, in a language which matches adjective to noun gender? Several messages around it in the full PO could easily show whether it is just another color in a row, and their grammatical endings (determined by a translator earlier) would show the needed ending for the new color.

Then, if an ediff message needs some editing before being applied, it may not be easy to do this directly in the ediff PO. Everything is fine so long as only added text segments (<tt>{+...+}</tt>) are edited, but if the sentence needs to be restructured more thoroughly, reviewer would have to make sure to put all additions into existing or new <tt>{+...+}</tt> segments, and to wrap all removals as <tt>{-...-}</tt> segments. If this is not carefully performed, the patch will not be applicable any more, as old message resolved from it will no longer exactly match a message in the target PO.

For these reasons, <tt>poepatch</tt> can apply the patch such as ''not to resolve the ediff'', but to set all its extraction-invariant fields to the message in the patched PO. In effect, the patched PO becomes an ediff PO by itself, but only in the messages which were actually patched. To mark these messages for lookup, <tt>ediff</tt> flag is added to them. If the message in the ediff PO was:

<code po>
#: title.c:274
msgid "Tutorial"
msgstr "{-Tutorijal-}{+Podučavanje+}"
</code>

then when the patch is successfully applied with embedding, the patched message in target PO will look like this:

<code po>
#: main.c:110
msgid "Records of The Witch River"
msgstr "Beleške o Veštičjoj reci"
⁠
#: title.c:292
#, ediff
msgid "Tutorial"
msgstr "{-Tutorijal-}{+Podučavanje+}"
⁠
#: title.c:328
msgid "Start the Expedition"
msgstr "Pođi u ekspediciju"
</code>

Other than the addition <tt>ediff</tt> flag, note that the patched message also kept its source reference, rather than it being overwritten by that from ediff PO. Same holds for all extraction-prescribed parts.

Reviewer can now jump from <tt>ediff</tt> to <tt>edif</tt> flag, always having the full positional context for each patched message, and being able to edit it to heart's content, with only minimal care not to invalidate the ediff format. Wrapped difference segments can be entirely removed, non-wrapped segments can be freely edited; it should only not happen that a wrapped segment looses its opening or closing sequence. But this does not mean that the reviewer ''needs'' to remove or touch difference segments at all, that is, to unembed patched messages by hand -- <tt>poepatch</tt> will do that automatically, when run on embedded-patched POs with a particular option.

A patch is applied with embedding by issuing the <tt>-e</tt> (<tt>--embed</tt>) option (<tt>E</tt> in parenthesis in progress output indicates that embedding is engaged):

<code bash>
$ poepatch -e <ediff.po
patched (E): lang.po
$
</code>

When the patched PO has been reviewed and patched messages possibly edited, all remaining embedded differences are removed, i.e. resolved to new versions, by running:

<code bash>
$ poepatch -u lang.po
</code>

Only those messages having the <tt>ediff</tt> flag are resolved, therefore the reviewer should ''never'' remove them (unless manually unembedding the whole patched message).

What happens with rejected patches when embedding is engaged? They are also added into the target PO, with heuristic positioning, and no separate ediff PO with rejects is created. Same as with plain patching, straight-out rejects will have <tt>ediff-no-match</tt> flag, and split rejects <tt>ediff-to-cur</tt> and <tt>ediff-to-new</tt>. If these are not manually resolved during the review (<tt>ediff-no-match</tt> messages removed, <tt>ediff-to-*</tt> messages removed or folded), when <tt>poepatch</tt> is run to unembed the patch, it will remove all <tt>ediff-no-match</tt> and <tt>ediff-to-new</tt> messages, and resolve <tt>ediff-to-cur</tt> messages to new version (effectively rejecting patches from which split rejects had originated).

== Lightweight Diffing when Updating Translation ==

Rather than for reviewing changes and sending patches, the translator may want to have only a convenient diff between the previous and new original text when updating a catalog merged with <tt>--previous</tt> option. Repeating the (by now) usual example message:

<code po>
#: main.c:110
#, fuzzy
#| msgid "The Record of The Witch River"
msgid "Records of The Witch River"
msgstr "Beleška o Veštičjoj reci"
</code>

Obviously, a dedicated PO editor could automatically present appropriate diff for messages like these (e.g. [[Localization/Tools/Lokalize|Lokalize]] can do it since introduction), so for translators using such an editor, diffing of fuzzies is either already available or a strong candidate for a feature request. The rest of this section therefore applies only to translators who stick to general text editors (with or without some power-assists for editing PO files).

One ''could'' use <tt>poediff</tt> and <tt>poepatch</tt> for the purpose of diffing of fuzzies, by first diffing from previous complete to new merged catalog, embedding the patch into the merged catalog, updating translation, and then unembedding ediffs. One drawback of this, however, is that two version of the catalog are necessary, when only the merged one contains all the information. More drawbacks are that the catalog has to be cleared of ediffs after updating, or that each diffed message contains more information than necessary (<tt>ediff</tt> flag, <tt>#. ediff:</tt> comment) since all ediffs are of one and the same type.

In short, full embedded diffing is too heavyweight as an assist to updating fuzzy messages. Instead, there is a Pology sieve intended for this particular purpose, <tt>diff-previous</tt>, which embeds the difference from previous to current original text into previous fields. Executing:

<code bash>
$ posieve diff-previous lang.po
</code>

will modify fuzzy messages in-place in the processed catalog, to the following partial ediff:

<code po>
#: main.c:110
#, fuzzy
#| msgid "{-The Record-}{+Records+} of The Witch River"
msgid "Records of The Witch River"
msgstr "Beleška o Veštičjoj reci"
</code>

The format of embedding is the same as before, so the ediff-aware syntax highlighting works here too. When the message is updated and properly unfuzzied (both <tt>fuzzy</tt> flag and previous fields removed), none of the ediff remains, so there is no need for post-unembedding. In other words, translator updates translation just as used to before, with the free benefit of having the ediff of original text.

Note that no embedding should remain in previous fields by the time the catalog is merged again, as it would throw off <tt>msgmerge</tt> when matching to create new fuzzies. This will normally not be an issue, as the translator will have unfuzzied all messages when updating the catalog after last merging. However, if there were a lot of fuzzy messages, and translator didn't have the time to update all of them, <tt>diff-previous</tt> sieve can also be used to unembed any remaining ediffs, restoring the original values of previous fields. It just needs to be run on the catalog with extra parameter <tt>strip</tt>:

<code bash>
$ posieve diff-previous -sstrip lang.po
</code>

Development/Tutorials/Localization/i18n Semantics

2010-06-12T16:24:21Z

Ilic: Mention new 'row' subcue.

== Abstract ==

Typical way of formatting user visible strings in application interfaces, for a long time has been that of plain text or at most visual markup like HTML tags. In most textual content environments, shift to ''semantic'' markup has been recognized as superior to visual (for example, the Docbook XML for documentation). Why not go down the same road for UI strings?

== Semantic Markup by Examples ==

In the semantic model, user interface strings are marked for their ''context'', and text elements within for their ''meaning'' rather than visual appearance. Consider few i18n examples of usual, non-semantic format:
<code cpp>
i18n("Move");

i18n("Descending");

i18n("<qt><b>%1</b> does not exist</qt>", fname);

i18n("<h1>History Sidebar</h1> You can configure the history sidebar here.");
</code>

Using KDE UI Text (KUIT for short) semantic markup, these strings would be formated like this:
<code cpp>
i18nc("@action:button", "Move");

i18nc("@item:inmenu", "Descending");

i18nc("@info", "<filename>%1</filename> does not exist", fname);

i18nc("@info:whatsthis",
"<title>History Sidebar</title>"
"<para>You can configure the history sidebar here.</para>");
</code>

Two distinct differences between ordinary and KUIT markup can be observed.

The first is the use of context i18n calls, the <tt>i18nc()</tt>, to convey the usage context of the string by means of the ''context marker''. The first message above, "Move", has been assigned the <tt>@action:button</tt> marker, where <tt>@action</tt> is the ''semantic role'' which describes the text as an action to be taken (e.g. operation on data or opening of a new dialog), and <tt>:button</tt> is the ''interface subcue'' saying that this text is displayed on a pushbutton widget. The second message, "Descending", has been marked as semantically a list item (<tt>@item</tt>), displayed in a menu (<tt>:inmenu</tt>). The interface subcue can be left out if none is appropriate, as has been done in the third message.

The other difference is the use of the semantic tags, which convey the meaning of a word or phrase within the text. The <tt><filename>%1</filename></tt> part of the third message tells that the substituted text is the name of a file. The <tt><title></tt> and <tt><para></tt> tags in the last message clearly lay out structure of a longer informational text.

{{note|The context marker can be added when within some of the standard XML sources too. In Qt Designer forms (<tt>.ui</tt> files), each text label to a widget has <tt>comment</tt> attribute (presented as "disambiguation" property within Designer, or "comment" prior to Qt 4.5), which can be used in the same manner as context argument of <tt>i18nc()</tt> call. Similarly, in the KXmlGui (<tt>.rc</tt>) and KConfigXT (<tt>.kcfg</tt>) files, tags <tt><text></tt>, <tt>label</tt>, and <tt>whatsthis</tt> can have a <tt>context</tt> attribute. For example:
<code>
<label context="@label">Hide trivial details</label>
<whatsthis context="@info:whatsthis">Option to hide drivel</whatsthis>
<text context="@item:inmenu">&New...</text>
</code>}}

Even when context marker is present, sometimes the programmer may want to provide an additional "free-form" description to translators, in order shed more light on particularly ambiguous strings. The free-form description is just separated by a whitespace from the context marker proper, like this:
<code cpp>
i18nc("@item:inmenu Sorting order", "Descending");
</code>

One particularly useful and yet general piece of free-form description, is the title under which the message is grouped -- the menu title for actions in menu, the group title of set of radio-buttons, the listbox label for items in list, etc. For example:
<code cpp>
iconSizeBox = new QGroupBox(i18nc("@title:group", "Icon Size"), this);
//...
iconSize1 = new QRadioButton(i18nc("@option:radio Icon Size", "Small"), this);
iconSize2 = new QRadioButton(i18nc("@option:radio Icon Size", "Medium"), this);
iconSize3 = new QRadioButton(i18nc("@option:radio Icon Size", "Large"), this);
//...
</code>

== Advantages of Semantic Markup ==

KUIT markup has advantages both to users and to translators of applications that make use of it.

For the users, the use of semantic tags means consistent formatting of same kinds of text. A notorious example of inconsistent visual formatting are filenames and paths, which are sometimes put in as is, sometimes in quotes (and ordinary quotes at that, rather than proper English fancy quotes), and sometimes in bold tags. Furthermore, the text withing the tag may be modified when semantically marked; for example, the standard "/" path delimiters in a <tt><filename></tt> text will be substituted for platform specific ones.

Translators will benefit from both context markers and tags. For the <tt>@action</tt> role of the "Move" string in the example above, the translator may use command form of the verb, while gerund form (like "Moving") may be more appropriate for the <tt>@title</tt> role, which would be used if the string was title of the menu, window, etc. The interface subcue, like <tt>:button</tt> above, if present, additionally enables the translator to mentally picture the actual runtime GUI. Tags within the text will also benefit translators, as they may clarify the structure of the sentence, especially in presence of placeholder substitutions.

Context markers also serve a technical purpose: they decide whether what form of visual formatting is used. For example, any <tt>@title</tt> role will use plain text, whereas <tt>@info</tt> will frequently produce rich text, depending on the subcue.

None the least, semantic markup removes the burden from programmers of thinking about the visual formatting to apply, like "''Should I put the path in quotes or <b>?''", or "''Should the title be <h2> or <h3>?''", and so on.

== Context Markers ==

Context marker consist of the semantic role and the interface subcue, in the form of <tt>@role:subcue</tt>. Each message should be given a role, but the subcue may be left out. The subcue should be given only when the string clearly maps to the user interface element that it describes.

Every role/subcue combination determines the default visual formatting of resulting string, whether it comes out as plain or rich text. See the section on [[#Limitations to Semantic Markup|limitations]] for a way to override the default formatting when necessary.

{{warning|Roles and subcues in the context marker cannot be specified at will, but must be drawn from the sets defined below. This is important for several reasons, one being that translators have deliberated about and agreed upon the meanings of presented sets, rather than having to second-guess arbitrary combinations.}}

KUIT defines the following roles and subcues (with notes on default visual formatting):

; <tt>@action</tt>
: Text to all clickable widgets that cause some action to be performed, like an operation on the data, view restructuring, or opening a dialog. The button texts and menu entries (except submenu titles) all fall into this category.
:
:: <tt>:button</tt> - pushbuttons in windows and dialogs
:: <tt>:inmenu</tt> - menu entries that perform an action
:: <tt>:intoolbar</tt> - toolbar buttons
:
: All <tt>@action</tt> markers are formatted as '''plain text''' by default.

; <tt>@title</tt>
: Text that is semantically a title in the interface. These would include window titles, menu titles, tab names, option group names in configuration dialogs, and column and row names in list views.
:
:: <tt>:window</tt> - window title (also dock name)
:: <tt>:menu</tt> - menu name
:: <tt>:tab</tt> - tab name
:: <tt>:group</tt> - option group
:: <tt>:column</tt> - column name
:: <tt>:row</tt> - row name
:
: All <tt>@title</tt> markers are formatted as '''plain text''' by default.

; <tt>@option</tt>
: Text to options which user can turn on and off, or choose between. These are the labels to checkboxes (either in dialogs or in menus) and radio buttons.
:
:: <tt>:check</tt> - checkbox label
:: <tt>:radio</tt> - radio-button label
:
: All <tt>@option</tt> markers are formatted as '''plain text''' by default.

; <tt>@label</tt>
: Text labels to various widgets in the interface, which are none of <tt>@action</tt>, <tt>@title</tt>, <tt>@option</tt>. These include labels to sliders, spinboxes, combo, list and text boxes, font and color choosers.
:
:: <tt>:slider</tt> - slider labels (but end-ranges are @item:inrange!)
:: <tt>:spinbox</tt> - spinbox labels
:: <tt>:listbox</tt> - list and combo boxes
:: <tt>:textbox</tt> - text and edit boxes
:: <tt>:chooser</tt> - chooser widgets (fonts, colors, etc.)
:
: All <tt>@label</tt> markers are formatted as '''plain text''' by default.

; <tt>@item</tt>
: Strings that can be considered one from a range of possibilities or properties. Entries in listings, dropdown and combo boxes are obvious, but also some menu items (e.g. encoding selection, sort orderings), end-labels to ranges (e.g. high/low, more/less), inserts into longer texts, and properites (e.g. file types, permissions) frequently displayed in tabular form (e.g. column views, property dialogs).
:
:: <tt>:inmenu</tt> - items presented as menu entries
:: <tt>:inlistbox</tt> - items in list and combo boxes
:: <tt>:intable</tt> - items presented in table-like forms
:: <tt>:inrange</tt> - range labels to sliders, etc.
:: <tt>:intext</tt> - words and phrases inserted into other messages
:
: All <tt>@item</tt> markers are formatted as '''plain text''' by default.

; <tt>@info</tt>
: General texts for user's information, that do not fall under any of the previous roles. These are for example tooltip and "What's This?" texts, text in message boxes, fields in status bar, and strings in progress dialogs.
:
:: <tt>:tooltip</tt> - hovering tooltips
:: <tt>:whatsthis</tt> - "What's This?" explanations of widgets
:: <tt>:status</tt> - texts in status displays (e.g. in status bar)
:: <tt>:progress</tt> - the current state of ongoing process
:: <tt>:tipoftheday</tt> - introductory tips on application startup
:: <tt>:credit</tt> - contributor names and their contributions
:: <tt>:shell</tt> - info output to the terminal, rather than to GUI
:
: Standalone <tt>@info</tt>, without a subcue, is formatted as '''rich text''' by default; the same holds for <tt>:tooltip</tt>, <tt>:whatsthis</tt>, and <tt>:tipoftheday</tt> subcues. With <tt>:status</tt>, <tt>:progress</tt>, and <tt>:credit</tt>, '''plain text''' is produced. <tt>:shell</tt> produces '''terminal text''' (like plain, but with possible shell escape sequences).

== Semantic Tags ==

KUIT semantic tags come in several logical groups:
* ''phrase tags'' - those that ascribe meaning to certain phrases and inserts
* ''sentence tags'' - which describe the purpose of a complete sentence in text
* ''structure tags'' - used to order longer text into paragraphs, titles, etc.

=== Phrase tags ===

Phrase tags will by default not admit any subtags; where some subtags can be used, it is so indicated. KUIT defines the folowing phrase tags:

; <tt><application></tt>
: Name of an application.
<code cpp>
i18nc("@action:inmenu",
"Open with <application>%1</application>", appName);
</code>

; <tt><bcode></tt>
: Line-breaking body of code, for short listings.
<code cpp>
i18nc("@info:whatsthis",
"You can try the following snippet:<bcode>"
"\\begin{equation}"
" C_{x_i} = \\frac{C_z^2}{e \\pi \\lambda}"
"\\end{equation}"
"</bcode>");
</code>

; <tt><command></tt>
: Name of shell command or system call. Its man section can be provided via <tt>section</tt> attribute.
<code cpp>
i18nc("@info",
"This will call <command>%1</command> internally.", cmdName);

i18nc("@info",
"Consult man entry for <command section='1'>%1</command>", cmdName);
</code>

; <tt><email></tt>
: Email address. Without attributes, the tag text is the address. Address can also be given with <tt>address</tt> attribute, in which case the tag text is the name or description attached to the address.
<code cpp>
i18nc("@info",
"Send bug reports to <email>%1</email>.", emailNull);

i18nc("@info",
"Send praises to <email address='%1'>the author</email>.", emailMy);
</code>
: The construct will be hyperlinked in rich text format.

; <tt><emphasis></tt>
: Emphasize a word or phrase in the text.
<code cpp>
i18nc("@info:progress",
"Checking <emphasis>feedback</emphasis> circuits...");
</code>
: For strong emphasis, attribute <tt>strong</tt> (since KDE 4.3) may be used, with value <tt>1</tt>, <tt>yes</tt>, or <tt>true</tt>.

; <tt><envar></tt>
: Environment variable. The $ sign will be prepended automatically in formatted text.
<code cpp>
i18nc("@info",
"Assure that your <envar>PATH</envar> is properly set.");
</code>

; <tt><filename></tt>
: File or folder name or path. The path separators will be transformed into what is native to the platform.
<code cpp>
i18nc("@info", "Cannot read <filename>%1</filename>.", filename);

i18nc("@info",
"<filename><envar>HOME</envar>/.foorc</filename> does not exist.");
</code>
: The <tt><envar></tt> can be used as subtag.

; <tt><icode></tt>
: Inline code, like shell command lines.
<code cpp>
i18nc("@info:tooltip",
"Execute <icode>svn merge</icode> on selected revisions.");
</code>
: The <tt><placeholder></tt> can be used as subtag.

; <tt><interface></tt>
: Path to GUI interface element. If there is more than one element in the path, use "|" or "->" to delimit elements, which will be converted into canonical delimiter.
<code cpp>
i18nc("@info:whatsthis",
"If you make a mistake, click "
"<interface>Reset</interface> to start again.");
i18nc("@info:whatsthis",
"The line colors can be changed under "
"<interface>Settings->Visuals</interface>.");
</code>

; <tt><link></tt>
: Link to a URL-addressable resource. Without attributes, the tag text is the URL; alternatively, the URL can be given by <tt>url</tt> attribute, and then the tag text serves as description.
<code cpp>
i18nc("@info:tooltip",
"Go to <link>%1</link> website.", urlKDE);

i18nc("@info:tooltip",
"Go to <link url='%1'>the KDE website</link>.", urlKDE);
</code>
: The variant with URL/description separation is preferred when applicable. The construct will be hyperlinked in rich text format.

; <tt><message></tt>
: An external message to be reported to the user.
<code cpp>
i18nc("@info",
"The fortune cookie says: <message>%1</message>", trouble);
</code>

; <tt><nl></tt>
: Line break, counterpart to HTML's <tt><br/></tt>. In plain text it will format as a newline character, and in rich text as <tt><br/></tt>.
<code cpp>
i18nc("@info",
"Do you really want to delete:<nl/>"
"<filename>%1</filename>", fileName);
</code>

; <tt><numid></tt>
: By default, numbers supplied as arguments to i18n calls are formatted into localized form. If the number is supposed to be a numeric identifier instead, like a port number, use this tag to signal numeric-id environment.
<code cpp>
i18nc("@info:progress",
"Connecting to <numid>%1</numid>...", portNo);
</code>

; <tt><placeholder></tt>
: A placeholder text, either something to be replaced by the user, or a generic item in a list.
<code cpp>
i18nc("@info",
"Replace <placeholder>name</placeholder> with your name.");

i18nc("@item:inlistbox",
"<placeholder>All images</placeholder>");
</code>

; <tt><resource></tt>
: General named resource. Names of documents, sessions, projects, toolbars, plugins, schemes and themes, accounts, etc.
<code cpp>
i18nc("@info", "Apply color scheme <resource>%1</resource>?", colScheme);
</code>

; <tt><shortcut></tt>
: Combination of keys to press. Separate the keys by "+" or "-", and the shortcut will be converted into canonical form.
<code cpp>
i18nc("@info:whatsthis",
"Cycle through layouts using <shortcut>Alt+Space</shortcut>.");
</code>

=== Sentence tags ===

Sentence tags mark complete sentences in text, and will admit any phrase tags as subtags. The following are defined:

; <tt><note></tt>
: The sentence is a side note of significance to the topic.
<code cpp>
i18nc("@info",
"Probably the best known of all duck species is the Mallard. "
"It breeds throughout the temperate areas around the world. "
"<note>Most domestic ducks are derived from Mallard.</note>");
</code>
: Do not explicitly add "Note:", it will be added automatically. If you really need other label than "Note", use attribute <tt>label</tt>, e.g. <tt>"<note label='Trivia'>...</note>"</tt>.

; <tt><warning></tt>
: The sentence is a warning.
<code cpp>
i18nc("@info",
"Really delete this key?"
"<warning>This cannot be undone.</warning>");
</code>
: Do not explicitly add "Warning:", it will be added automatically. If you really need other label than "Warning", use attribute <tt>label</tt>, e.g. <tt>"<warning label='Danger'>...</warning>"</tt>.

=== Structure tags ===

Structure tags are used to split longer texts into titles, paragraphs, and lists. By default they can contain any phrase or sentence tags, unless indicated otherwise.

; <tt><para></tt>
: Text paragraph.

; <tt><title></tt>
: The title of the text. Must be the first tag if present, but can be omitted.

; <tt><subtitle></tt>
: Subtitle in the text. Must be followed by at least one <tt><para></tt>.

; <tt><list></tt>
: List of items. Can contain only <tt><item></tt> as subtags. List is considered an element of the paragraph, so the <tt><list></tt> must be found inside <tt><para></tt>.

; <tt><item></tt>
: List item.

If any of the structure tags is present, then there must be no text outside of structure tags. The following is not valid KUIT markup:
<code cpp>
// invalid markup
i18nc("@info",
"<title>History Sidebar</title>"
"You can configure the history sidebar here."); // <para> missing
</code>

== Limitations to Semantic Markup ==

Semantic markup cannot be used in "dumb" strings, which do not pass through KDE's i18n subsystem. These would be, for example, strings in <tt>.desktop</tt> format files. But ''not'' the strings in UI files, as in Qt Designer they can be equipped with both context markers (via comment field to text properties) and semantic tags.

Qt's rich text HTML tags can be used concurrently with KUIT tags, but this is not advised unless necessary. They may be needed, for example, to create tables or insert images, as KUIT does not implement this functionality at the moment.

Sometimes, the visual formatting may not be quite appropriate for the output device; every role/subcue combination has a preset default formatting. For example, if the <tt>@info</tt> role is applied to a string which is used in a widget that does not handle rich text, it will come out with HTML tags displayed verbatim. To handle this, visual formatting can be explicitly signaled by <tt>/''format''</tt> modifier appended to context marker:

<code cpp>
i18nc("@info/plain",
"<filename>%1</filename> does not exist", fname);
</code>

Presently, the possible format modifiers are <tt>/plain</tt>, <tt>/rich</tt> and <tt>/term</tt> (for terminal format, possible use of escape sequences).

To specify context in kpartgui rc file use context attribute (<text context="@title:menu">File</text>). A context-aware version of I18N_NOOP is I18N_NOOP2.

== Should I Go For Semantic Markup? ==

Admittedly, KUIT markup is an additional thing to be learned and applied throughout the course of development. By now you may be wondering if it is worthwile to invest time into that, particularly in view of two cases:
* starting work on a new application, and
* porting messages in existing applications.

You are strongly advised to use KUIT for new code. Compared to the total time spent on code, writing UI messages is only a small fraction. Context markers will help translators a lot, and message tags will provide consistent visual text formatting to your application.

When modifying existing code, there are two issues. First, obviously it is a daunting task to go through hundreds (or worse) of messages and equip them with semantic markup. Second, by changing the messages, the translators too will have to review their existing translations; however, it is not expected that the porting will take so "epic" proportions that the translators cannot keep up. Sumarily, feel free to do as you see fit.

Additionally, for porting, keep in mind that it is not all-or-nothing proposal. Any amount of semantic messages are useful to translators, and users can only see the difference for the better. Thus, for example, deciding to make all ''new'' messages semantic and slowly over time fix old messages, is a perfectly fine strategy.

To make your job easier, there is an i18n-checker script that will report the problems in KUIT markup, as well as check some other i18n nuances. It is run daily on KDE code repository-wide as part Krazy-framework, but you can also run it locally on your code. It can be found in {{path|trunk/quality/krazy2/plugins/general/i18ncheckarg}}. Furthermore, as of this moment Krazy checks only C++ sources, while when you run the script locally, some of the XML sources containing translatable strings will be checked as well. <tt>i18ncheckarg</tt> requires Perl libxml bindings, which are probably already packaged for your distribution (Debian package is libxml-libxml-perl).

{{note|By default <tt>i18ncheckarg</tt> takes a single filename to check, but using <tt>--allsources</tt> option you can provide as many file or directory paths as you wish; typically you would give top directory path to your sources. Also, to report all missing context markers use <tt>--ctxmark</tt> option, or otherwise missing markers will be reported only if some threshold of marked-to-total number of messages in a file is reached.}}

Last but not the least, there is also a chic-effect to the KUIT. Its wide use, together with some under-the-hood elements on translators' disposal, will make KDE4's i18n layer without peer in free or proprietary software world. Insofar as you consider localization excellence an important part of the overall KDE excellence, this is something that may also tip your decision :) -- Your Friendly Translator.

Localization/Workflows/PO Ascription

2010-05-29T10:51:18Z

Ilic: Corrected note on -C option. Mentioned removal of previous fields.

{{Template:I18n/Language Navigation Bar|Localization/Workflows/PO_Summit}}

{{LocalizationBrowser|
section=[[Localization#Workflows|Workflows]]|
name=Reviewing by Ascriptions|
prereqs=[[Localization/Workflows/PO_Summit|Translating in Summit]]|
related=[[Localization/Tools/Pology|Pology]], [[Localization/Workflows/Coordinator|Language Coordinator]]|
}}

== Why Review Translations? ==

Especially to new translators, it may not be obvious to which extent the translation needs to be reviewed. If the translator has exercised due diligence, how "wrong" can the translation be? Even if the translator has good command of the source language (English in context of this article), the answer is "very wrong", when all aspects are considered. Here are some of them.

With comparatively simple grammar of English, the meaning of a short English sentence -- as typically encountered in application user interfaces -- is very dependent on the surrounding context. This context may not be obvious when the translator is going through isolated messages in the translation file, so he may commit the worst of errors from the user's viewpoint, the senseless translation. An experienced reviewer will have developed sense for troublesome contexts, and will have several means to decisively determine the context (including, for example, running the development version of the application).

Even if the context is correctly established, the translator may use "wrong" terminology, which is the next worse thing for the user. A term used in translation does not need to be wrong by itself, in fact it may be exactly the correct term -- in another translation project. The reviewer will have more experience with terminology of the present project, and be able to bring the translation in line with it.

Style in the technical sense is a consistent choice between several perfectly valid constructs in target language when applied to text in the given technical context. For example, how to translate menu titles and items, button labels, or tooltips. The choices may include noun or verb forms, particular grammar categories, tone of address, and so on. There may be a style guide to the project which details such choices, and the reviewer will know it well.

Style in the linguistic sense is especially applicable to longer texts, such as tooltips in user interfaces, and passages in documentation. A typical error of a new translator is to closely adhere to English style and grammar. This may produce translation which is semantically and grammatically valid in the target language, but very out of style -- the "translationese". Reviewer is there to naturalize such passages.

Finally, the reviewer may be an experienced translator, but that does not mean that his own translations need no review. Immersion into the source language, distraction, fatigue, will lead the reviewer into any of the above errors in translation, only with less frequency. So reviewers should also review one anothers' translations.

== Classical Reviewing by Stages ==

Classical review workflow by stages seems simple enough. Translator translates a PO file (or updates existing translation), and declares it ready to review. A reviewer reviews it, and declares it ready to "commit". Committing here should be understood generally, as inclusion into the pool from which translations are periodically shipped to end users. A committer finally commits the file. The process is iterative: the reviewer may return the file to the translator, and translator later again declare it as ready for review. There may be several stages of review (such as ''proof-reading'', ''approving''), each of which may return the translation to a previous stage, or forward it to some special stage. The process can also be more finer grained, where each message in the file goes through stages separately.

Regardless of the particularities, workflows of this kind all have the following in common. Members of the translation team are assigned ''roles'' -- such as ''translator'', ''reviewer'', ''approver'', ''committer'' -- by which they enter into the workflow (single person can have more roles). The later review stages must wait for the earlier stages to complete, and the translation cannot be updated again before the current version clears the pipeline (or the pipeline is aborted). Most importantly, once the translation is committed, it becomes part of simply "admitted" translations, with no further qualifiers.

The system of prescribed roles requires that team members assign them between themselves, stick to them, and shuffle them along the way. The prescribed review pipeline requires a tool to enforce and keep track of the stages in which translations are. This makes the review workflow complex and rigid, most probably with choke points for efficiency. Distribution of roles may become disbalanced by people coming and going, or the workflow tool may be prohibitive to some scenarios (e.g. single translator making small adjustments in dozens of files across the project, but having to upload each manually through a web interface).

Of course, "rigid", "complex", "inefficient", are comparative qualifications, so what is it that the classical review by stages can be compared to in this way?

== Reviewing by Ascriptions ==

Reviewing by ascriptions is even simpler conceptually, and yet less rigid, less complex, and much more efficient than the review by stages. It works on the message-level, rather than file-level. Anyone can simply translate some messages and directly commit modified files, without any review, but with ''ascribing'' modifications to own name. Anyone can review any committed messages at any moment, commit the modifications-on-review and ascribe reviews to own name and (possibly) to certain class -- full review, review of context, of terminology, of style, etc. Only when the translation is to be shipped to end users, the ''insufficiently'' reviewed messages are automatically omitted from the package, by evaluating the ''ascription history'' of each message.

Most importantly, based on the ascription history, the reviewer can select only some particular messages, and review only the difference between their historical and current versions. For example, Alice can select to review only messages modified since she or Bob had last reviewed them for style; she could see the difference from that last review to current version, e.g. if in the whole paragraph only a single word has changed by Charlie when he reviewed the terminology. In terms of PO workflow, the ascription history propagates through merges, so the reviewer can compare the change in original and the change in translation since the last review, to judge if one fits the other.

Since everyone just commits, translations can be efficiently kept in a version control repository, with the ascription system added on top. After having done some translating, the team member simply substitutes commit command of the version control system (VCS) with ascribe-modifications command of the ascription system (AS, which calls the underlying VCS internally). After reviewing, the team member uses ascribe-reviews command of the AS to commit reviews to ascription history (as well as modifications made during the review). To select messages for review, the team member issues diff-for-review command of the AS (with suitable parameters to narrow the set) and selected messages are marked in-place in PO files and [[Localization/Tools/Pology/PO_Embedded_Diffing|embedded with differences]], and possibly popped open in a PO editor.

When the translations are to be released, the team coordinator issues filter-for-release command of the AS, which takes the working PO files and creates final PO files with insufficiently reviewed messages removed. "Release time" is used here only figuratively: this should be a fully automatic process, so it can be performed at any interval of convenience.

What constitutes "sufficient review" can be defined in fine detail. It could be specified that messages modified by Alice need to have only review for terminology, but not necessarily for style; Charlie may belong to the group which needs to be reviewed on style, but not necessarily on context; Bob's reviews for style may be nice to have, but never blocking if missing. These decisions do not preclude released messages to be reviewed later on missing points, after higher priority reviews have been completed. The definition of sufficiency may be changed at any point, e.g. as team members get more experienced and require less review, without interfering with direct translation and review work.



In summary, with reviewing by ascriptions the lean efficiency of raw VCS operation is preserved while providing for great flexibility of review. All team members can be given commit access, no web or email detours are needed. There are no prescribed roles, but an equivalent of role assignment happens automatically at last possible moment, and can take into account both translators' and reviewers' abilities. There is no staging between completing and committing the translation, which enables translator to keep on polishing the translation undisturbed until the reviewer comes around. There is no inefficiency in handling small changes throughout many files, since single AS command commits all changes just as single VCS command would. AS in effect abstracts VCS, so general team members do not have to know the particularities of the underlying VCS. On commit operations, AS can also apply checks (e.g. decline to commit syntactically invalid PO files) and modifications (e.g. update translator's data in the PO header).

== Ascription System in Pology ==

[[Localization/Tools/Pology|Pology]] is a collection of various modular tools for supporting translation based on PO files. Among them is the script <tt>poascribe</tt>, which implements an ascription system (AS); at present, it can use Subversion or Git as the underlying VCS. <tt>poascribe</tt> is still in experimental stage, so what follows is a brief description of how to use it in context of the KDE translation project. However, very little is truly specific to KDE; the only major assumption is that there exists a VCS repository with PO files of a given language grouped together, and that the translation team can use it without special restrictions.

Very important for the AS is how branches are handled (in KDE, the rolling trunk and stable branches). AS can in principle be deployed by branch, but then there is the added complexity of porting translations between branches, which ascriptions should follow. Therefore, the AS implemented by <tt>poascribe</tt> is currently limited to assumption that there is a single branch of translations at all times. The article [[Localization/Workflows/PO_Summit|"Translating in Summit"]] explains how a KDE translation team can set up and operate such a single branch, the summit, and this is the prerequisite for the following instructions. (Note that the summit system is useful on its own, and should be conductive to any kind of review workflow.)

== Setting Up ==

The summit branch for the language <tt>LANG</tt> is positioned like this in the KDE repository:

<code text>
$KDEREPO/
trunk/
l10n-support/
LANG/
summit/
messages/
docmessages/
</code>

The team coordinator already has this part of the repository tree locally due to regular summit operations. For the same reason Pology is already set up. Setting up the ascription system is now simple. The file <tt>ascription-config</tt> is created in the root of the summit:

<code text>
...
LANG/
summit/
messages/
docmessages/
ascription-config
</code>

with the following contents:

<code text>
# ---------------------------
# Global ascription settings.

[global]

# The root of the ascription catalog tree.
ascript-root = ../summit-ascript

# The underlying version control system.
version-control = svn

# Data for updating catalog headers.
# - language code
language = LANG
# - full language name
language-team = LANGUAGE
# - email address of the team
team-email = kde-i18n-LANG@kde.org

# Default commit message.
commit-message = Translation updates.

# -----------------------
# Registered translators.

[user-alice]
name = Alice Akmalryn
original-name = Алиса Акмалрин
email = alice.akmalryn@someplacenice.org

[user-bob]
name = Bob Byomkin
original-name = Бобан Бјомкин
email = bob.byomkin@otherplacenice.org

# ...and so on.
</code>

Some notes:

* The <tt>ascript-root</tt> setting should be exactly <tt>../summit-ascript</tt>, for the reason mentioned later.

* <tt>commit-message</tt> field, if defined, allows team members to commit without providing a commit message. The value given by this field will be used by default, with translator's user name appended to the end in special syntax. For example: <tt>Translation updates. [>alice]</tt>. (Translator's user name is also appended to manually supplied commit messages.) Translators can still supply a commit message when they wish, as shown later. If this field is not set, the commit message is supplied as usual on committing.

* Team members are defined by <tt>[user-USERNAME]</tt> sections. Ascription user names can be any valid ASCII identifier: ASCII letters, digits and underscores only, digit cannot be the first character. Ascription user names have no technical relation to the underlying VCS accounts, though it is mnemonically convenient if they are the same (in case of SVN). This means that a translator who does not have a VCS account (yet) can and should be added here, with assigned user name (best one suitable as SVN account name later); why this should be done will be explained later.

* <tt>original-name</tt> field in user sections is there in case the preferred renderings of the name in English and in target language are not the same. When this is not the case, <tt>original-name</tt> can be omitted.

As soon as the <tt>ascription-config</tt> file is committed, the ascription system is ready for operation. Only regular modifications to this file are those of adding new team members. (On the other hand, team members should never be removed, because even after they no longer contribute, their ascription records remain in the system.)

=== Initial Ascription ===

The most common situation at start of ascription workflow is that there already exists a body of translations, contributed to by many different people over time. The coordinator should ascribe all existing translations as initial modifications, but to whom? It cannot be said precisely who translated what. The solution is to introduce a generic user in <tt>ascription-config</tt>, suitably known as "Unknown Hero" (or "Lost Translator", you can be inventive):

<code>
[user-uhero]
name = Unknown Hero
original-name = Незнани јунак
</code>

and ascribe all existing translations as modified and reviewed by this user. The coordinator does this with the following command:

<code bash>
$ cd $KDEREPO/trunk/l10n-support
$ poascribe commit -u uhero --all-reviewed -C LANG/summit/
</code>

The argument <tt>commit</tt> is the ascription mode, and the <tt>-u</tt> option provides the user name to which ascriptions are made. This is an important point: ascriptions are made to a user defined in ascription configuration, and have nothing to do with VCS accounts; someone who has the account can commit in the name of someone who does not. It is the <tt>--all-reviewed</tt> option that declares all messages to be reviewed as well (note that it is normally used only this once, and not for normal day to day reviewing). The <tt>-C</tt> option prevents automatic adding and committing to version control, which is useful for this initial step. Finally the paths which contain all summit catalogs are given.

When the <tt>poascribe</tt> command is issued, a progress bar will appear, and the following output will start to unfold:

<code text>
LANG/summit/messages/extragear-base/rellinks.po (50/50)
LANG/summit/messages/extragear-base/autorefresh.po (13/13)
LANG/summit/messages/extragear-base/babelfish.po (38/38)
...
LANG/summit/messages/qt/libphonon.po (13/13)
LANG/summit/messages/qt/phonon-xine.po (24/24)
LANG/summit/messages/qt/phonon_gstreamer.po (12/12)
===== Ascription summary:
- modified reviewed
translated 111775 111775
fuzzy 26943 26943
obsolete/t 2965 2965
obsolete/f 1626 1626
</code>

The number in parenthesis indicates how many messages have been ascribed in the given PO file (modified/reviewed), and at the end the totals are given. Ascribing the complete summit for the first time will take quite some time (on the order of 10-20 minutes).



After the initial ascription has been made, the ascription tree will appear next to the summit tree. This tree will contain one ascription PO file for each summit PO file, with the same name and relative location within the tree:

<code text>
...
LANG/
summit/
messages/
kdelibs/
kcertpart.po
kdelibs4.po
...
...
docmessages/
ascription-config
summit-ascript/
messages/
kdelibs/
kcertpart.po
kdelibs4.po
...
...
docmessages/
</code>

During the ascription some summit PO files may have been modified as well, in that any previous fields (<tt>#| ...</tt>) on translated messages have been removed. (These fields are sometimes erroneously left in by older PO editors.)

The newly created ascription tree (and any modifications to summit tree) can now be committed as usual:

<code bash>
$ svn add LANG/summit-ascript
$ svn commit LANG/summit LANG/summit-ascript -m "Initial ascription."
</code>

== Daily Use for Translators ==

Team members other than the coordinator, whether translators or reviewers, need to keep around only the <tt>trunk/l10n-support/LANG/</tt> directory. But they always need to update this directory fully (rather than just one particular module or file under <tt>.../*messages/</tt>), so that the summit tree and the ascription tree (and configuration) are kept in sync.

In order not to have to issue their own user name (<tt>-u</tt> option to <tt>poascribe</tt>) all the time, translators can set it in Pology user configuration <tt>~/.pologyrc</tt>, in <tt>[poascribe]</tt> section:

<code text>
[poascribe]
user = alice
</code>

Translators can then submit updated PO files simply by substituting <tt>svn commit</tt> (or whatever the VCS commit command is) with <tt>poascribe commit</tt> (<tt>co</tt> or <tt>ci</tt> for short):

<code bash>
$ poascribe ci LANG/summit/messages/kdefoo/*fooapp*.po
LANG/summit/messages/kdefoo/fooapp.po (144)
LANG/summit/messages/kdefoo/libfooapp.po (25)
===== Ascription summary:
- modified
translated 169
>>>>> VCS is committing catalogs:
Sending LANG/summit/messages/kdefoo/fooapp.po
Sending LANG/summit/messages/kdefoo/libfooapp.po
Sending LANG/summit-ascript/messages/kdefoo/fooapp.po
Sending LANG/summit-ascript/messages/kdefoo/libfooapp.po
Transmitting file data ....
Committed revision 1267069.
$
</code>

<tt>poascribe</tt> will add ascription records into ascription catalogs corresponding to summit catalogs to be committed, and commit them all. Like <tt>svn commit</tt>, <tt>poascribe ci</tt> can take any number of file or directory paths, and can be issued from any working directory (it will always find ascription catalogs). If default commit message has not been set in the ascription configuration, <tt>poascribe</tt> will ask for it; or it can be given in command line through <tt>-m</tt> option.

=== Translators Without Commit Access ===

With the ascription system in place, every regular team member should have commit access. But, there may be some period of time before new translators are given accounts, revision control may be too technical for some, and even those with the account may not be able to commit temporarily for some reason.

These translators may send in their work by email, to ''any'' team member with commit access (not necessarily the coordinator or a reviewer); this team member can commit received files without any review, as review can be conducted at any later time. If Bob sends some files to Alice, she can commit them immediately by stating Bob's user name:

<code bash>
$ poascribe ci -u bob ...
</code>

For this to work, the translator who sent in the files has to be defined in the ascription configuration. There are no hidden costs or security issues to this (as opposed to opening a VCS account), so every new translator should be defined there before any work of that person is committed.

== Daily Use for Reviewers ==

The ascription system opens up all sorts of possibilities for concrete review patterns. Reviewers should keep in mind that for each message the full modification and review history is available, so that the team can think about how to make good use of it. Therefore, what follows are some examples to illustrate the review facilities that <tt>poascribe</tt> provides.

=== Basic Reviewing ===

At the very basic level (which is the only level in classical review by stages), messages can be classified into simply unreviewed and reviewed, without further qualifiers. Alice now wants to review all unreviewed messages in a group of PO files, say <tt>kdetoys</tt> module. She issues (<tt>di</tt> is short for <tt>diff</tt>):

<code bash>
$ cd $KDEREPO/trunk/l10n-support
$ poascribe di LANG/summit/messages/kdetoys/
LANG/summit/messages/kdegames/bovo.po (2)
LANG/summit/messages/kdegames/kdiamond.po (7)
LANG/summit/messages/kdegames/palapeli.po (12)
===== Diffed for review: 21
</code>

Unreviewed messages have now been marked and diffed, inside the listed PO files. What is this about "diffing"? If the files had already been reviewed before, some of the messages modified since then (those marked for review) may have changed very little (e.g. a few words in a paragraph-length message, or even just punctuation). Therefore, for each message marked for review, Alice also wants to see the diff since last review to current version. Here are two messages in typical review states added by <tt>poascribe di</tt>:

<code po>
#. +> trunk stable
#. ascto: charlie:m
#: gui/mainwindow.cc:372
#, ediff
msgid "GAME OVER. {-You won-}{+Tie+}!"
msgstr "KRAJ IGRE. {-Pobeda-}{+Nerešeno+}!"

#. +> trunk stable
#. ascto: bob:m charlie:m
#: game-state.cpp:117
#, ediff-total
msgid "Click the pause button again to resume the game."
msgstr "Kliknite ponovo na dugme pauze da nastavite igru."
</code>

and the first one in Kate:

[[Image:Kate_poascribe_diff_01.png|center|Message diffed for review by poascribe in Kate.]]

In the first message, the first to note is the <tt>#. ascto:</tt> comment. This comment succinctly lists who did what with the message since the last review; here <tt>charlie:m</tt> means that Charlie is the one who modified it. Then, there is the <tt>ediff</tt> flag, which alice can use it to jump through messages marked for review. Finally, the original and translation have been diffed; here they show that, since the last review, the message was fuzzied by changing "You won" to "Tie", and what Charlie did in translation to unfuzzy it. Even on a message as short as this, the diff tells something useful to Alice: the phrase "Game over" likely has a formulaic translation, and the fact that it is not part of the diff means that the earlier reviewer had made sure it is consistent, so Alice does not have to check that.

The <tt>#. ascto:</tt> comment of the second message reveals that both Charlie and Bob had been translating it. <tt>ediff-total</tt> flag instead of plain <tt>ediff</tt> means that this message had no review at all up to now, so there are no embedded diffs in text fields.

Alice can now go through marked files and messages, review translations, and possibly make modifications. When making changes in a message with embedded diffs, she can freely edit text outside of difference segments and within <tt>{+...+}</tt> segments (as these are the ones which belong to current version of the text). While reviewing, Alice does ''not'' remove any of the added message elements while reviewing (save for an occasional difference segment, when translation should be modified), as these elements are needed for later. If a message is particularly hard and Alice wants to defer its review for later, she can add the <tt>unreviewed</tt> (or <tt>nrev</tt> for short) flag to it.

Once the review is complete, Alice simply commits the reviewed files:

<code bash>
$ poascribe ci LANG/summit/messages/kdetoys/
LANG/summit/messages/kdegames/bovo.po (0/2)
LANG/summit/messages/kdegames/kdiamond.po (0/7)
LANG/summit/messages/kdegames/palapeli.po (3/12)
===== Ascription summary:
- modified reviewed
translated 3 21
>>>>> VCS is committing catalogs:
Sending LANG/summit/messages/kdegames/palapeli.po
Sending LANG/summit-ascript/messages/kdegames/bovo.po
Sending LANG/summit-ascript/messages/kdegames/kdiamond.po
Sending LANG/summit-ascript/messages/kdegames/palapeli.po
Transmitting file data ....
Committed revision 1284220.
$
</code>

Three things have happened here. First, all review states (flags, embedded diffs, etc.) have been removed, restoring the PO file to normal. Then, any modifications that Alice have made during review are ascribed to her (here 3 out of 21 messages). Finally, all marked messages are ascribed as reviewed by Alice (any with <tt>unreviewed</tt>/<tt>nrev</tt> flags would have been omitted here). When committing, the only summit catalog that got committed is the one with modifications made during review, and all the ascription catalogs were committed because of the reviews recorded in them.

When many files with few changes in each are to be reviewed, it becomes burdensome to manually open each and every diffed for review, and then to make sure that all are committed with <tt>poascribe ci</tt>. To make this easier, <tt>-w torevivew.out</tt> option can be added to <tt>poascribe di</tt>, which requests that paths of all diffed PO files are written into <tt>torevivew.out</tt> file. This file can then be used to batch open POs for review in the editor, as well as fed back on <tt>poascribe re</tt> with <tt>-f torevivew.out</tt>. There is also the <tt>-o</tt> option which causes <tt>poascribe</tt> to directly open PO files in a PO editor (though this is currently applicable only to Lokalize). Putting it together, to efficiently review a whole bunch of small changes throughout many files, Alice can:

<code bash>
$ poascribe di PATHS... -w toreview.out -o lokalize
$ # ...only marked messages opened in Lokalize, review them...
$ poascribe ci -f toreview.out
</code>

=== Selecting Messages for Review ===

Invocations of <tt>poascribe di</tt> without any options, as in the previous section, were actually equivalent to this:

<code bash>
$ poascribe di -s modar PATHS...
</code>

Option <tt>-s</tt> is issuing the message ''selector''. <tt>modar</tt> is the default selector for <tt>diff</tt> mode, and stands for MODified-After-Review: it selects the earliest historical modification of the message after the last (or no) review of that message, if there is any such. By selecting a historical modification of the message, the diff from it to current version can be computed and embedded into the PO file, as in previous examples.

There are various specialized selectors, and fall into two groups: ''shallow selectors'' and ''history selectors''. Shallow selectors look only into the current version of the message, and cannot select historical versions, which means that they cannot provide embedded diffs. History selectors (<tt>modar</tt> is of this type) can select messages from history and provide diffs. Several selectors can be issued on the command line, and the message is selected only if all selectors select it. Shallow selectors are then normally used as a pre-filter for history selectors. For example, to select messages modified after last reviewed, but only those found in stable branch, <tt>branch</tt> and <tt>modar</tt> selectors are chained:

<code bash>
$ poascribe di -s branch:stable -s modar PATHS...
</code>

It is important that the history selector is given last, because the last selector determines which historical message is selected. If the ordering had been reversed here, same messages would get selected, but they would not have embedded diffs, because <tt>branch</tt> is a shallow selector.

Selectors can take parameters themselves, like <tt>branch:stable</tt> in the previous example. Parameters are separated from the selector name by any non-alphanumeric character; this is colon by convention, but if a parameter contains a colon, something like slash, tilde, etc. can be used. Number of parameters can be variable, and <tt>modar</tt> in particular can take from none to three. If Alice wants to review only those messages modified ''by Charlie'' since last review, she states this by first argument to <tt>modar</tt>:

<code bash>
$ poascribe di -s modar:charlie PATHS...
</code>

If Alice does not give too much credit to other reviewers, she can request selection of messages modified after last review ''by her'' with second parameter to <tt>modar</tt>:

<code bash>
$ poascribe di -s modar::alice PATHS...
</code>

Here the first parameter ("modified by..."), which is not needed, must be explicitly skipped, before going to the second parameter ("reviewed by..."). The third optional parameter of <tt>modar</tt> will be mentioned in the next section.

When a selector parameter is a user name, normally it can also be a comma-separated list of user names (<tt>modar:bob,charlie</tt>) or prefixed with tilde to negate, i.e. select all other users (<tt>modar:~alice</tt>).

Any selector can be negated by prepending <tt>n</tt> to its name. For example, the history selector <tt>modafter:DATE</tt> selects first modification after the given date; to select messages modified after last review, but only if modified during June 2010:

<code bash>
$ poascribe di -s modafter:2010-06 -s nmodafter:2010-07 -s modar PATHS...
</code>

Negating a history selector produces a shallow selector: while <tt>modafter</tt> is history selector, <tt>nmodafter</tt> is shallow. But the order of the two in the previous command line is not important, as the last selector is the usual <tt>modar</tt>.

Selectors can be issued in other modes too. If the PO file is big and Alice has reviewed messages up to and including entry 246 when she has to pause until another day, she can commit reviews only up to this entry by issuing the <tt>espan</tt> selector:

<code bash>
$ poascribe ci -s espan::246 PATHS...
</code>

(the first parameter to <tt>espan</tt> is the first entry number, given if messages are not to be selected from the first). There is also the counterpart <tt>lspan</tt> selector, which works with referent line numbers (those of <tt>msgid</tt> keywords) instead of entry numbers.

=== Fine-Grained Reviews ===

In the introduction, several distinct types of what can go wrong in translation were described. Not all reviewers may be able to check translation against all those problems. Here is a typical scenario of this kind:

Alice is very computer-savvy and knows the translation project inside and out, which means that she can review well for context, terminology, and technical style. But, her language style leaves something to be desired, which shows through longer sentences and passages. Dan, on the other hand, is a very literary person, but not that much into the technical aspects. Dan's style reviews would thus be a perfect complement to Alice's general reviews.

<tt>poascribe</tt> can support this scenario in the following way. A ''review type'' tag for language style is defined in the ascription configuration, using the <tt>review-tags</tt> field:

<code text>
[global]
...
review-tags = lstyle
</code>

(The value to <tt>review-tags</tt> is a space-separated list of identifiers, when more than one special review type is needed.) With this addition to configuration, Alice can continue to review as she did before, without any changes to her workflow.

Dan selects messages for review similarly to Alice, but aditionally giving the <tt>lstyle</tt> tag as ''third'' parameter of <tt>modar</tt>, and indicating that ascribed reviews should be tagged as <tt>lstyle</tt>:

<code bash>
$ poascribe di -s modar:::lstyle -t lstyle PATHS...
</code>

After finishing the review, Dan commits as usual:

<code bash>
$ poascribe ci PATHS...
</code>

If Dan is always going to review the language style, in order not to have to issue the selector and tag in the command line all the time, he can make them default per mode in <tt>~/.pologyrc</tt>:

<code text>
[poascribe]
user = dan
selectors/diff = modar:::lstyle
tags/diff = lstyle
</code>

With this Dan can use plain <tt>poascribe di</tt> just like Alice does.

The important point of review tags is that they make reviews by types independent. For example, Dan may come around to review the language style of the given message after several modifications and general reviews have been ascribed to it -- <tt>modar:::lstyle</tt> will simply ignore all reviews except for <tt>lstyle</tt> reviews. This is going to be reflected in the <tt>ascto:</tt> comment to marked messages:

<code po>
#...
#. ascto: charlie:m alice:r bob:m
#...
msgid "..."
msgstr "..."
</code>

Here Alice has made one review between Charlie's and Bob's modifications, and that review, being general instead of <tt>lstyle</tt>, did not cause <tt>modar</tt> to stop at it. After Dan reviews this message for language style, Alice runs selection for review and gets this:

<code po>
#...
#. ascto: bob:m dan:r(lstyle)
#...
msgid "..."
msgstr "..."
</code>

Again, since <tt>lstyle</tt> reviews do not mix with general reviews, Dan's review did not hide Bob's modification that Alice did not check so far.

(General review too has a tag assigned, the empty string, in case the reviewer needs to explicitly issue it in some context.)

== Daily Use for The Coordinator ==

After setting up the ascription system, the team coordinator should have to do very little to maintain it.

=== Ascribing Merges ===

Modifications made to summit catalogs by merging with templates must also be ascribed. Therefore, after merging the summit (<tt>posummit ... merge ...</tt>) the coordinator substitutes the VCS command:

<code bash>
$ svn commit LANG/summit/messages/ -m "Merged summit."
</code>

with the <tt>poascribe</tt> command in <tt>commit</tt> mode:

<code bash>
$ poascribe ci LANG/summit/messages/ -m "Merged summit."
</code>

Since the user is not explicitly given by <tt>-u</tt> option, this will ascribe merge modifications to the coordinator (more precisely, to the user set as default in <tt>~/.pologyrc</tt>), which is just fine. It is also possible to define a special user only for ascribing merge modifications, though there is no known advantage to that.

Since <tt>-C</tt> option is not issued, <tt>poascribe</tt> will automatically commit all modified summit and ascription catalogs when done.

=== Shuffling Ascription Catalogs ===

Sometimes summit catalogs are shuffled in the repository: moved to another module, renamed, one catalog split into two, two catalogs merged into one. Such shuffling should be exactly mirrored in the ascription tree, and this too is done on the repository side, at the same time. This relies on the ascription root being set exactly to <tt>../summit-ascript</tt> in the ascription configuration. So the team coordinator has nothing special to do here.

If instead in the central KDE repository the translation team is working in an external repository, by consequence the ascription system must be set up in that repository. But so long as <tt>process_orphans.sh</tt> script from <tt>trunk/l10n-support/scripts/</tt> is used to shuffle catalogs in the external repository as well, the ascription catalogs will be properly handled.

== Filtering for Release ==

The last component of the ascription system is how to prevent insufficiently reviewed messages from leaking into a release. In context of Pology and summit workflow, <tt>poascribe</tt> itself is used directly to this end. Instead, in the summit configuration (as opposed to ascription configuration), the team coordinator defines filters which pass messages by applying selectors.

Each top level PO tree has its own summit configuration file, named <tt>MSGTREE.extras.summit</tt>:

<code text>
...
LANG/
summit/
messages/
messages.extras.summit
docmessages/
docmessages.extras.summit
</code>

For the simple case of all reviews being general reviews, the filter is added to summit configuration like this (anywhere within <tt>*.extras.summit</tt> file):

<code python>
S.ascription_filters = [
("regular", ["nmodar"]),
]
</code>

Here the filter is named <tt>regular</tt>, and is defined as application of <tt>nmodar</tt> selector, the negation of <tt>modar</tt>. This simply means: pass all messages ''not'' modified after the last review.

When the team coordinator scatters to branches (executes <tt>posummit scatter</tt>), messages from summit POs which do not pass this filter will not be sent to branch POs. The count of stopped messages by branch PO will be reported in the output as scattering proceeds.

Why did we have to name the filter <tt>regular</tt>? (Those knowing some Python will also notice that it is defined as a list element.) Because it is possible to define more than one filter, and select which one is used on each scattering. For example, the coordinator may wish that, when the release is near and time is short to review everything, messages from a few experienced translators can be passed into release without review. If those translators are Alice and Bob, an "emergency" filter can be defined like this:

<code python>
S.ascription_filters = [
("regular", ["nmodar"]),
("emergency", ["nmodar:~alice,bob"]),
]
</code>

By default, <tt>posummit</tt> uses the first filter in the list. When the coordinator needs to do emergency scattering, he requests the emergency filter by the <tt>-a</tt> option:

<code bash>
$ cd $KDEREPO/trunk/l10n-support
$ posummit scripts/messages.summit LANG scatter -a emergency
</code>

What if several selectors are needed to pass the message? For example, the language style review (the earlier example with Alice and Dan) too may be requested for regular scattering, but omitted from emergency scattering. The filter setup for this scenario looks like this:

<code python>
S.ascription_filters = [
("regular", ["nmodar", "nmodar:::lstyle"]),
("emergency", ["nmodar:~alice,bob"]),
]
</code>

The regular filter now reads: pass the message if it has not been modified after the last (general) review ''and'' has not been modified after the last style review.

Simple combination of predefined selectors by AND-conditions may not be sufficient for more involved scenarios. When this is the case, the coordinator may write (or ask someone to write) a custom selector in Python, and plug it in as the second element in the filter tuple (instead of the list of predefined selectors).

=== Writing a Selector Function ===

((To be written.))

Localization/Workflows/PO Ascription

2010-05-29T09:55:04Z

Ilic: Updated output in examples.

{{Template:I18n/Language Navigation Bar|Localization/Workflows/PO_Summit}}

{{LocalizationBrowser|
section=[[Localization#Workflows|Workflows]]|
name=Reviewing by Ascriptions|
prereqs=[[Localization/Workflows/PO_Summit|Translating in Summit]]|
related=[[Localization/Tools/Pology|Pology]], [[Localization/Workflows/Coordinator|Language Coordinator]]|
}}

== Why Review Translations? ==

Especially to new translators, it may not be obvious to which extent the translation needs to be reviewed. If the translator has exercised due diligence, how "wrong" can the translation be? Even if the translator has good command of the source language (English in context of this article), the answer is "very wrong", when all aspects are considered. Here are some of them.

With comparatively simple grammar of English, the meaning of a short English sentence -- as typically encountered in application user interfaces -- is very dependent on the surrounding context. This context may not be obvious when the translator is going through isolated messages in the translation file, so he may commit the worst of errors from the user's viewpoint, the senseless translation. An experienced reviewer will have developed sense for troublesome contexts, and will have several means to decisively determine the context (including, for example, running the development version of the application).

Even if the context is correctly established, the translator may use "wrong" terminology, which is the next worse thing for the user. A term used in translation does not need to be wrong by itself, in fact it may be exactly the correct term -- in another translation project. The reviewer will have more experience with terminology of the present project, and be able to bring the translation in line with it.

Style in the technical sense is a consistent choice between several perfectly valid constructs in target language when applied to text in the given technical context. For example, how to translate menu titles and items, button labels, or tooltips. The choices may include noun or verb forms, particular grammar categories, tone of address, and so on. There may be a style guide to the project which details such choices, and the reviewer will know it well.

Style in the linguistic sense is especially applicable to longer texts, such as tooltips in user interfaces, and passages in documentation. A typical error of a new translator is to closely adhere to English style and grammar. This may produce translation which is semantically and grammatically valid in the target language, but very out of style -- the "translationese". Reviewer is there to naturalize such passages.

Finally, the reviewer may be an experienced translator, but that does not mean that his own translations need no review. Immersion into the source language, distraction, fatigue, will lead the reviewer into any of the above errors in translation, only with less frequency. So reviewers should also review one anothers' translations.

== Classical Reviewing by Stages ==

Classical review workflow by stages seems simple enough. Translator translates a PO file (or updates existing translation), and declares it ready to review. A reviewer reviews it, and declares it ready to "commit". Committing here should be understood generally, as inclusion into the pool from which translations are periodically shipped to end users. A committer finally commits the file. The process is iterative: the reviewer may return the file to the translator, and translator later again declare it as ready for review. There may be several stages of review (such as ''proof-reading'', ''approving''), each of which may return the translation to a previous stage, or forward it to some special stage. The process can also be more finer grained, where each message in the file goes through stages separately.

Regardless of the particularities, workflows of this kind all have the following in common. Members of the translation team are assigned ''roles'' -- such as ''translator'', ''reviewer'', ''approver'', ''committer'' -- by which they enter into the workflow (single person can have more roles). The later review stages must wait for the earlier stages to complete, and the translation cannot be updated again before the current version clears the pipeline (or the pipeline is aborted). Most importantly, once the translation is committed, it becomes part of simply "admitted" translations, with no further qualifiers.

The system of prescribed roles requires that team members assign them between themselves, stick to them, and shuffle them along the way. The prescribed review pipeline requires a tool to enforce and keep track of the stages in which translations are. This makes the review workflow complex and rigid, most probably with choke points for efficiency. Distribution of roles may become disbalanced by people coming and going, or the workflow tool may be prohibitive to some scenarios (e.g. single translator making small adjustments in dozens of files across the project, but having to upload each manually through a web interface).

Of course, "rigid", "complex", "inefficient", are comparative qualifications, so what is it that the classical review by stages can be compared to in this way?

== Reviewing by Ascriptions ==

Reviewing by ascriptions is even simpler conceptually, and yet less rigid, less complex, and much more efficient than the review by stages. It works on the message-level, rather than file-level. Anyone can simply translate some messages and directly commit modified files, without any review, but with ''ascribing'' modifications to own name. Anyone can review any committed messages at any moment, commit the modifications-on-review and ascribe reviews to own name and (possibly) to certain class -- full review, review of context, of terminology, of style, etc. Only when the translation is to be shipped to end users, the ''insufficiently'' reviewed messages are automatically omitted from the package, by evaluating the ''ascription history'' of each message.

Most importantly, based on the ascription history, the reviewer can select only some particular messages, and review only the difference between their historical and current versions. For example, Alice can select to review only messages modified since she or Bob had last reviewed them for style; she could see the difference from that last review to current version, e.g. if in the whole paragraph only a single word has changed by Charlie when he reviewed the terminology. In terms of PO workflow, the ascription history propagates through merges, so the reviewer can compare the change in original and the change in translation since the last review, to judge if one fits the other.

Since everyone just commits, translations can be efficiently kept in a version control repository, with the ascription system added on top. After having done some translating, the team member simply substitutes commit command of the version control system (VCS) with ascribe-modifications command of the ascription system (AS, which calls the underlying VCS internally). After reviewing, the team member uses ascribe-reviews command of the AS to commit reviews to ascription history (as well as modifications made during the review). To select messages for review, the team member issues diff-for-review command of the AS (with suitable parameters to narrow the set) and selected messages are marked in-place in PO files and [[Localization/Tools/Pology/PO_Embedded_Diffing|embedded with differences]], and possibly popped open in a PO editor.

When the translations are to be released, the team coordinator issues filter-for-release command of the AS, which takes the working PO files and creates final PO files with insufficiently reviewed messages removed. "Release time" is used here only figuratively: this should be a fully automatic process, so it can be performed at any interval of convenience.

What constitutes "sufficient review" can be defined in fine detail. It could be specified that messages modified by Alice need to have only review for terminology, but not necessarily for style; Charlie may belong to the group which needs to be reviewed on style, but not necessarily on context; Bob's reviews for style may be nice to have, but never blocking if missing. These decisions do not preclude released messages to be reviewed later on missing points, after higher priority reviews have been completed. The definition of sufficiency may be changed at any point, e.g. as team members get more experienced and require less review, without interfering with direct translation and review work.



In summary, with reviewing by ascriptions the lean efficiency of raw VCS operation is preserved while providing for great flexibility of review. All team members can be given commit access, no web or email detours are needed. There are no prescribed roles, but an equivalent of role assignment happens automatically at last possible moment, and can take into account both translators' and reviewers' abilities. There is no staging between completing and committing the translation, which enables translator to keep on polishing the translation undisturbed until the reviewer comes around. There is no inefficiency in handling small changes throughout many files, since single AS command commits all changes just as single VCS command would. AS in effect abstracts VCS, so general team members do not have to know the particularities of the underlying VCS. On commit operations, AS can also apply checks (e.g. decline to commit syntactically invalid PO files) and modifications (e.g. update translator's data in the PO header).

== Ascription System in Pology ==

[[Localization/Tools/Pology|Pology]] is a collection of various modular tools for supporting translation based on PO files. Among them is the script <tt>poascribe</tt>, which implements an ascription system (AS); at present, it can use Subversion or Git as the underlying VCS. <tt>poascribe</tt> is still in experimental stage, so what follows is a brief description of how to use it in context of the KDE translation project. However, very little is truly specific to KDE; the only major assumption is that there exists a VCS repository with PO files of a given language grouped together, and that the translation team can use it without special restrictions.

Very important for the AS is how branches are handled (in KDE, the rolling trunk and stable branches). AS can in principle be deployed by branch, but then there is the added complexity of porting translations between branches, which ascriptions should follow. Therefore, the AS implemented by <tt>poascribe</tt> is currently limited to assumption that there is a single branch of translations at all times. The article [[Localization/Workflows/PO_Summit|"Translating in Summit"]] explains how a KDE translation team can set up and operate such a single branch, the summit, and this is the prerequisite for the following instructions. (Note that the summit system is useful on its own, and should be conductive to any kind of review workflow.)

== Setting Up ==

The summit branch for the language <tt>LANG</tt> is positioned like this in the KDE repository:

<code text>
$KDEREPO/
trunk/
l10n-support/
LANG/
summit/
messages/
docmessages/
</code>

The team coordinator already has this part of the repository tree locally due to regular summit operations. For the same reason Pology is already set up. Setting up the ascription system is now simple. The file <tt>ascription-config</tt> is created in the root of the summit:

<code text>
...
LANG/
summit/
messages/
docmessages/
ascription-config
</code>

with the following contents:

<code text>
# ---------------------------
# Global ascription settings.

[global]

# The root of the ascription catalog tree.
ascript-root = ../summit-ascript

# The underlying version control system.
version-control = svn

# Data for updating catalog headers.
# - language code
language = LANG
# - full language name
language-team = LANGUAGE
# - email address of the team
team-email = kde-i18n-LANG@kde.org

# Default commit message.
commit-message = Translation updates.

# -----------------------
# Registered translators.

[user-alice]
name = Alice Akmalryn
original-name = Алиса Акмалрин
email = alice.akmalryn@someplacenice.org

[user-bob]
name = Bob Byomkin
original-name = Бобан Бјомкин
email = bob.byomkin@otherplacenice.org

# ...and so on.
</code>

Some notes:

* The <tt>ascript-root</tt> setting should be exactly <tt>../summit-ascript</tt>, for the reason mentioned later.

* <tt>commit-message</tt> field, if defined, allows team members to commit without providing a commit message. The value given by this field will be used by default, with translator's user name appended to the end in special syntax. For example: <tt>Translation updates. [>alice]</tt>. (Translator's user name is also appended to manually supplied commit messages.) Translators can still supply a commit message when they wish, as shown later. If this field is not set, the commit message is supplied as usual on committing.

* Team members are defined by <tt>[user-USERNAME]</tt> sections. Ascription user names can be any valid ASCII identifier: ASCII letters, digits and underscores only, digit cannot be the first character. Ascription user names have no technical relation to the underlying VCS accounts, though it is mnemonically convenient if they are the same (in case of SVN). This means that a translator who does not have a VCS account (yet) can and should be added here, with assigned user name (best one suitable as SVN account name later); why this should be done will be explained later.

* <tt>original-name</tt> field in user sections is there in case the preferred renderings of the name in English and in target language are not the same. When this is not the case, <tt>original-name</tt> can be omitted.

As soon as the <tt>ascription-config</tt> file is committed, the ascription system is ready for operation. Only regular modifications to this file are those of adding new team members. (On the other hand, team members should never be removed, because even after they no longer contribute, their ascription records remain in the system.)

=== Initial Ascription ===

The most common situation at start of ascription workflow is that there already exists a body of translations, contributed to by many different people over time. The coordinator should ascribe all existing translations as initial modifications, but to whom? It cannot be said precisely who translated what. The solution is to introduce a generic user in <tt>ascription-config</tt>, suitably known as "Unknown Hero" (or "Lost Translator", you can be inventive):

<code>
[user-uhero]
name = Unknown Hero
original-name = Незнани јунак
</code>

and ascribe all existing translations as modified and reviewed by this user. The coordinator does this with the following command:

<code bash>
$ cd $KDEREPO/trunk/l10n-support
$ poascribe commit -u uhero --all-reviewed -C LANG/summit/
</code>

The argument <tt>commit</tt> is the ascription mode, and the <tt>-u</tt> option provides the user name to which ascriptions are made. This is an important point: ascriptions are made to a user defined in ascription configuration, and have nothing to do with VCS accounts; someone who has the account can commit in the name of someone who does not. It is the <tt>--all-reviewed</tt> option that declares all messages to be reviewed as well (note that it is normally used only this once, and not for normal day to day reviewing). The <tt>-C</tt> option prevent automatic committing by <tt>poascribe</tt>, which is useful for this initial step. Finally the paths which contain all summit catalogs are given.

When the <tt>poascribe</tt> command is issued, a progress bar will appear, and the following output will start to unfold:

<code text>
LANG/summit/messages/extragear-base/rellinks.po (50/50)
LANG/summit/messages/extragear-base/autorefresh.po (13/13)
LANG/summit/messages/extragear-base/babelfish.po (38/38)
...
LANG/summit/messages/qt/libphonon.po (13/13)
LANG/summit/messages/qt/phonon-xine.po (24/24)
LANG/summit/messages/qt/phonon_gstreamer.po (12/12)
===== Ascription summary:
- modified reviewed
translated 111775 111775
fuzzy 26943 26943
obsolete/t 2965 2965
obsolete/f 1626 1626
</code>

The number in parenthesis indicates how many messages have been ascribed in the given PO file (modified/reviewed), and at the end the totals are given. Ascribing the complete summit for the first time will take quite some time (on the order of 10-20 minutes).



After the initial ascription has been made, the ascription tree will appear next to the summit tree. This tree will contain one ascription PO file for each summit PO file, with the same name and relative location within the tree:

<code text>
...
LANG/
summit/
messages/
kdelibs/
kcertpart.po
kdelibs4.po
...
...
docmessages/
ascription-config
summit-ascript/
messages/
kdelibs/
kcertpart.po
kdelibs4.po
...
...
docmessages/
</code>

The ascription tree can now be committed as usual (<tt>poascribe</tt> will have already added it):

<code bash>
$ svn commit LANG/summit-ascript/ -m "Initial ascription."
</code>

== Daily Use for Translators ==

Team members other than the coordinator, whether translators or reviewers, need to keep around only the <tt>trunk/l10n-support/LANG/</tt> directory. But they always need to update this directory fully (rather than just one particular module or file under <tt>.../*messages/</tt>), so that the summit tree and the ascription tree (and configuration) are kept in sync.

In order not to have to issue their own user name (<tt>-u</tt> option to <tt>poascribe</tt>) all the time, translators can set it in Pology user configuration <tt>~/.pologyrc</tt>, in <tt>[poascribe]</tt> section:

<code text>
[poascribe]
user = alice
</code>

Translators can then submit updated PO files simply by substituting <tt>svn commit</tt> (or whatever the VCS commit command is) with <tt>poascribe commit</tt> (<tt>co</tt> or <tt>ci</tt> for short):

<code bash>
$ poascribe ci LANG/summit/messages/kdefoo/*fooapp*.po
LANG/summit/messages/kdefoo/fooapp.po (144)
LANG/summit/messages/kdefoo/libfooapp.po (25)
===== Ascription summary:
- modified
translated 169
>>>>> VCS is committing catalogs:
Sending LANG/summit/messages/kdefoo/fooapp.po
Sending LANG/summit/messages/kdefoo/libfooapp.po
Sending LANG/summit-ascript/messages/kdefoo/fooapp.po
Sending LANG/summit-ascript/messages/kdefoo/libfooapp.po
Transmitting file data ....
Committed revision 1267069.
$
</code>

<tt>poascribe</tt> will add ascription records into ascription catalogs corresponding to summit catalogs to be committed, and commit them all. Like <tt>svn commit</tt>, <tt>poascribe ci</tt> can take any number of file or directory paths, and can be issued from any working directory (it will always find ascription catalogs). If default commit message has not been set in the ascription configuration, <tt>poascribe</tt> will ask for it; or it can be given in command line through <tt>-m</tt> option.

=== Translators Without Commit Access ===

With the ascription system in place, every regular team member should have commit access. But, there may be some period of time before new translators are given accounts, revision control may be too technical for some, and even those with the account may not be able to commit temporarily for some reason.

These translators may send in their work by email, to ''any'' team member with commit access (not necessarily the coordinator or a reviewer); this team member can commit received files without any review, as review can be conducted at any later time. If Bob sends some files to Alice, she can commit them immediately by stating Bob's user name:

<code bash>
$ poascribe ci -u bob ...
</code>

For this to work, the translator who sent in the files has to be defined in the ascription configuration. There are no hidden costs or security issues to this (as opposed to opening a VCS account), so every new translator should be defined there before any work of that person is committed.

== Daily Use for Reviewers ==

The ascription system opens up all sorts of possibilities for concrete review patterns. Reviewers should keep in mind that for each message the full modification and review history is available, so that the team can think about how to make good use of it. Therefore, what follows are some examples to illustrate the review facilities that <tt>poascribe</tt> provides.

=== Basic Reviewing ===

At the very basic level (which is the only level in classical review by stages), messages can be classified into simply unreviewed and reviewed, without further qualifiers. Alice now wants to review all unreviewed messages in a group of PO files, say <tt>kdetoys</tt> module. She issues (<tt>di</tt> is short for <tt>diff</tt>):

<code bash>
$ cd $KDEREPO/trunk/l10n-support
$ poascribe di LANG/summit/messages/kdetoys/
LANG/summit/messages/kdegames/bovo.po (2)
LANG/summit/messages/kdegames/kdiamond.po (7)
LANG/summit/messages/kdegames/palapeli.po (12)
===== Diffed for review: 21
</code>

Unreviewed messages have now been marked and diffed, inside the listed PO files. What is this about "diffing"? If the files had already been reviewed before, some of the messages modified since then (those marked for review) may have changed very little (e.g. a few words in a paragraph-length message, or even just punctuation). Therefore, for each message marked for review, Alice also wants to see the diff since last review to current version. Here are two messages in typical review states added by <tt>poascribe di</tt>:

<code po>
#. +> trunk stable
#. ascto: charlie:m
#: gui/mainwindow.cc:372
#, ediff
msgid "GAME OVER. {-You won-}{+Tie+}!"
msgstr "KRAJ IGRE. {-Pobeda-}{+Nerešeno+}!"

#. +> trunk stable
#. ascto: bob:m charlie:m
#: game-state.cpp:117
#, ediff-total
msgid "Click the pause button again to resume the game."
msgstr "Kliknite ponovo na dugme pauze da nastavite igru."
</code>

and the first one in Kate:

[[Image:Kate_poascribe_diff_01.png|center|Message diffed for review by poascribe in Kate.]]

In the first message, the first to note is the <tt>#. ascto:</tt> comment. This comment succinctly lists who did what with the message since the last review; here <tt>charlie:m</tt> means that Charlie is the one who modified it. Then, there is the <tt>ediff</tt> flag, which alice can use it to jump through messages marked for review. Finally, the original and translation have been diffed; here they show that, since the last review, the message was fuzzied by changing "You won" to "Tie", and what Charlie did in translation to unfuzzy it. Even on a message as short as this, the diff tells something useful to Alice: the phrase "Game over" likely has a formulaic translation, and the fact that it is not part of the diff means that the earlier reviewer had made sure it is consistent, so Alice does not have to check that.

The <tt>#. ascto:</tt> comment of the second message reveals that both Charlie and Bob had been translating it. <tt>ediff-total</tt> flag instead of plain <tt>ediff</tt> means that this message had no review at all up to now, so there are no embedded diffs in text fields.

Alice can now go through marked files and messages, review translations, and possibly make modifications. When making changes in a message with embedded diffs, she can freely edit text outside of difference segments and within <tt>{+...+}</tt> segments (as these are the ones which belong to current version of the text). While reviewing, Alice does ''not'' remove any of the added message elements while reviewing (save for an occasional difference segment, when translation should be modified), as these elements are needed for later. If a message is particularly hard and Alice wants to defer its review for later, she can add the <tt>unreviewed</tt> (or <tt>nrev</tt> for short) flag to it.

Once the review is complete, Alice simply commits the reviewed files:

<code bash>
$ poascribe ci LANG/summit/messages/kdetoys/
LANG/summit/messages/kdegames/bovo.po (0/2)
LANG/summit/messages/kdegames/kdiamond.po (0/7)
LANG/summit/messages/kdegames/palapeli.po (3/12)
===== Ascription summary:
- modified reviewed
translated 3 21
>>>>> VCS is committing catalogs:
Sending LANG/summit/messages/kdegames/palapeli.po
Sending LANG/summit-ascript/messages/kdegames/bovo.po
Sending LANG/summit-ascript/messages/kdegames/kdiamond.po
Sending LANG/summit-ascript/messages/kdegames/palapeli.po
Transmitting file data ....
Committed revision 1284220.
$
</code>

Three things have happened here. First, all review states (flags, embedded diffs, etc.) have been removed, restoring the PO file to normal. Then, any modifications that Alice have made during review are ascribed to her (here 3 out of 21 messages). Finally, all marked messages are ascribed as reviewed by Alice (any with <tt>unreviewed</tt>/<tt>nrev</tt> flags would have been omitted here). When committing, the only summit catalog that got committed is the one with modifications made during review, and all the ascription catalogs were committed because of the reviews recorded in them.

When many files with few changes in each are to be reviewed, it becomes burdensome to manually open each and every diffed for review, and then to make sure that all are committed with <tt>poascribe ci</tt>. To make this easier, <tt>-w torevivew.out</tt> option can be added to <tt>poascribe di</tt>, which requests that paths of all diffed PO files are written into <tt>torevivew.out</tt> file. This file can then be used to batch open POs for review in the editor, as well as fed back on <tt>poascribe re</tt> with <tt>-f torevivew.out</tt>. There is also the <tt>-o</tt> option which causes <tt>poascribe</tt> to directly open PO files in a PO editor (though this is currently applicable only to Lokalize). Putting it together, to efficiently review a whole bunch of small changes throughout many files, Alice can:

<code bash>
$ poascribe di PATHS... -w toreview.out -o lokalize
$ # ...only marked messages opened in Lokalize, review them...
$ poascribe ci -f toreview.out
</code>

=== Selecting Messages for Review ===

Invocations of <tt>poascribe di</tt> without any options, as in the previous section, were actually equivalent to this:

<code bash>
$ poascribe di -s modar PATHS...
</code>

Option <tt>-s</tt> is issuing the message ''selector''. <tt>modar</tt> is the default selector for <tt>diff</tt> mode, and stands for MODified-After-Review: it selects the earliest historical modification of the message after the last (or no) review of that message, if there is any such. By selecting a historical modification of the message, the diff from it to current version can be computed and embedded into the PO file, as in previous examples.

There are various specialized selectors, and fall into two groups: ''shallow selectors'' and ''history selectors''. Shallow selectors look only into the current version of the message, and cannot select historical versions, which means that they cannot provide embedded diffs. History selectors (<tt>modar</tt> is of this type) can select messages from history and provide diffs. Several selectors can be issued on the command line, and the message is selected only if all selectors select it. Shallow selectors are then normally used as a pre-filter for history selectors. For example, to select messages modified after last reviewed, but only those found in stable branch, <tt>branch</tt> and <tt>modar</tt> selectors are chained:

<code bash>
$ poascribe di -s branch:stable -s modar PATHS...
</code>

It is important that the history selector is given last, because the last selector determines which historical message is selected. If the ordering had been reversed here, same messages would get selected, but they would not have embedded diffs, because <tt>branch</tt> is a shallow selector.

Selectors can take parameters themselves, like <tt>branch:stable</tt> in the previous example. Parameters are separated from the selector name by any non-alphanumeric character; this is colon by convention, but if a parameter contains a colon, something like slash, tilde, etc. can be used. Number of parameters can be variable, and <tt>modar</tt> in particular can take from none to three. If Alice wants to review only those messages modified ''by Charlie'' since last review, she states this by first argument to <tt>modar</tt>:

<code bash>
$ poascribe di -s modar:charlie PATHS...
</code>

If Alice does not give too much credit to other reviewers, she can request selection of messages modified after last review ''by her'' with second parameter to <tt>modar</tt>:

<code bash>
$ poascribe di -s modar::alice PATHS...
</code>

Here the first parameter ("modified by..."), which is not needed, must be explicitly skipped, before going to the second parameter ("reviewed by..."). The third optional parameter of <tt>modar</tt> will be mentioned in the next section.

When a selector parameter is a user name, normally it can also be a comma-separated list of user names (<tt>modar:bob,charlie</tt>) or prefixed with tilde to negate, i.e. select all other users (<tt>modar:~alice</tt>).

Any selector can be negated by prepending <tt>n</tt> to its name. For example, the history selector <tt>modafter:DATE</tt> selects first modification after the given date; to select messages modified after last review, but only if modified during June 2010:

<code bash>
$ poascribe di -s modafter:2010-06 -s nmodafter:2010-07 -s modar PATHS...
</code>

Negating a history selector produces a shallow selector: while <tt>modafter</tt> is history selector, <tt>nmodafter</tt> is shallow. But the order of the two in the previous command line is not important, as the last selector is the usual <tt>modar</tt>.

Selectors can be issued in other modes too. If the PO file is big and Alice has reviewed messages up to and including entry 246 when she has to pause until another day, she can commit reviews only up to this entry by issuing the <tt>espan</tt> selector:

<code bash>
$ poascribe ci -s espan::246 PATHS...
</code>

(the first parameter to <tt>espan</tt> is the first entry number, given if messages are not to be selected from the first). There is also the counterpart <tt>lspan</tt> selector, which works with referent line numbers (those of <tt>msgid</tt> keywords) instead of entry numbers.

=== Fine-Grained Reviews ===

In the introduction, several distinct types of what can go wrong in translation were described. Not all reviewers may be able to check translation against all those problems. Here is a typical scenario of this kind:

Alice is very computer-savvy and knows the translation project inside and out, which means that she can review well for context, terminology, and technical style. But, her language style leaves something to be desired, which shows through longer sentences and passages. Dan, on the other hand, is a very literary person, but not that much into the technical aspects. Dan's style reviews would thus be a perfect complement to Alice's general reviews.

<tt>poascribe</tt> can support this scenario in the following way. A ''review type'' tag for language style is defined in the ascription configuration, using the <tt>review-tags</tt> field:

<code text>
[global]
...
review-tags = lstyle
</code>

(The value to <tt>review-tags</tt> is a space-separated list of identifiers, when more than one special review type is needed.) With this addition to configuration, Alice can continue to review as she did before, without any changes to her workflow.

Dan selects messages for review similarly to Alice, but aditionally giving the <tt>lstyle</tt> tag as ''third'' parameter of <tt>modar</tt>, and indicating that ascribed reviews should be tagged as <tt>lstyle</tt>:

<code bash>
$ poascribe di -s modar:::lstyle -t lstyle PATHS...
</code>

After finishing the review, Dan commits as usual:

<code bash>
$ poascribe ci PATHS...
</code>

If Dan is always going to review the language style, in order not to have to issue the selector and tag in the command line all the time, he can make them default per mode in <tt>~/.pologyrc</tt>:

<code text>
[poascribe]
user = dan
selectors/diff = modar:::lstyle
tags/diff = lstyle
</code>

With this Dan can use plain <tt>poascribe di</tt> just like Alice does.

The important point of review tags is that they make reviews by types independent. For example, Dan may come around to review the language style of the given message after several modifications and general reviews have been ascribed to it -- <tt>modar:::lstyle</tt> will simply ignore all reviews except for <tt>lstyle</tt> reviews. This is going to be reflected in the <tt>ascto:</tt> comment to marked messages:

<code po>
#...
#. ascto: charlie:m alice:r bob:m
#...
msgid "..."
msgstr "..."
</code>

Here Alice has made one review between Charlie's and Bob's modifications, and that review, being general instead of <tt>lstyle</tt>, did not cause <tt>modar</tt> to stop at it. After Dan reviews this message for language style, Alice runs selection for review and gets this:

<code po>
#...
#. ascto: bob:m dan:r(lstyle)
#...
msgid "..."
msgstr "..."
</code>

Again, since <tt>lstyle</tt> reviews do not mix with general reviews, Dan's review did not hide Bob's modification that Alice did not check so far.

(General review too has a tag assigned, the empty string, in case the reviewer needs to explicitly issue it in some context.)

== Daily Use for The Coordinator ==

After setting up the ascription system, the team coordinator should have to do very little to maintain it.

=== Ascribing Merges ===

Modifications made to summit catalogs by merging with templates must also be ascribed. Therefore, after merging the summit (<tt>posummit ... merge ...</tt>) the coordinator substitutes the VCS command:

<code bash>
$ svn commit LANG/summit/messages/ -m "Merged summit."
</code>

with the <tt>poascribe</tt> command in <tt>commit</tt> mode:

<code bash>
$ poascribe ci LANG/summit/messages/ -m "Merged summit."
</code>

Since the user is not explicitly given by <tt>-u</tt> option, this will ascribe merge modifications to the coordinator (more precisely, to the user set as default in <tt>~/.pologyrc</tt>), which is just fine. It is also possible to define a special user only for ascribing merge modifications, though there is no known advantage to that.

Since <tt>-C</tt> option is not issued, <tt>poascribe</tt> will automatically commit all modified summit and ascription catalogs when done.

=== Shuffling Ascription Catalogs ===

Sometimes summit catalogs are shuffled in the repository: moved to another module, renamed, one catalog split into two, two catalogs merged into one. Such shuffling should be exactly mirrored in the ascription tree, and this too is done on the repository side, at the same time. This relies on the ascription root being set exactly to <tt>../summit-ascript</tt> in the ascription configuration. So the team coordinator has nothing special to do here.

If instead in the central KDE repository the translation team is working in an external repository, by consequence the ascription system must be set up in that repository. But so long as <tt>process_orphans.sh</tt> script from <tt>trunk/l10n-support/scripts/</tt> is used to shuffle catalogs in the external repository as well, the ascription catalogs will be properly handled.

== Filtering for Release ==

The last component of the ascription system is how to prevent insufficiently reviewed messages from leaking into a release. In context of Pology and summit workflow, <tt>poascribe</tt> itself is used directly to this end. Instead, in the summit configuration (as opposed to ascription configuration), the team coordinator defines filters which pass messages by applying selectors.

Each top level PO tree has its own summit configuration file, named <tt>MSGTREE.extras.summit</tt>:

<code text>
...
LANG/
summit/
messages/
messages.extras.summit
docmessages/
docmessages.extras.summit
</code>

For the simple case of all reviews being general reviews, the filter is added to summit configuration like this (anywhere within <tt>*.extras.summit</tt> file):

<code python>
S.ascription_filters = [
("regular", ["nmodar"]),
]
</code>

Here the filter is named <tt>regular</tt>, and is defined as application of <tt>nmodar</tt> selector, the negation of <tt>modar</tt>. This simply means: pass all messages ''not'' modified after the last review.

When the team coordinator scatters to branches (executes <tt>posummit scatter</tt>), messages from summit POs which do not pass this filter will not be sent to branch POs. The count of stopped messages by branch PO will be reported in the output as scattering proceeds.

Why did we have to name the filter <tt>regular</tt>? (Those knowing some Python will also notice that it is defined as a list element.) Because it is possible to define more than one filter, and select which one is used on each scattering. For example, the coordinator may wish that, when the release is near and time is short to review everything, messages from a few experienced translators can be passed into release without review. If those translators are Alice and Bob, an "emergency" filter can be defined like this:

<code python>
S.ascription_filters = [
("regular", ["nmodar"]),
("emergency", ["nmodar:~alice,bob"]),
]
</code>

By default, <tt>posummit</tt> uses the first filter in the list. When the coordinator needs to do emergency scattering, he requests the emergency filter by the <tt>-a</tt> option:

<code bash>
$ cd $KDEREPO/trunk/l10n-support
$ posummit scripts/messages.summit LANG scatter -a emergency
</code>

What if several selectors are needed to pass the message? For example, the language style review (the earlier example with Alice and Dan) too may be requested for regular scattering, but omitted from emergency scattering. The filter setup for this scenario looks like this:

<code python>
S.ascription_filters = [
("regular", ["nmodar", "nmodar:::lstyle"]),
("emergency", ["nmodar:~alice,bob"]),
]
</code>

The regular filter now reads: pass the message if it has not been modified after the last (general) review ''and'' has not been modified after the last style review.

Simple combination of predefined selectors by AND-conditions may not be sufficient for more involved scenarios. When this is the case, the coordinator may write (or ask someone to write) a custom selector in Python, and plug it in as the second element in the filter tuple (instead of the list of predefined selectors).

=== Writing a Selector Function ===

((To be written.))

Localization/Workflows/PO Ascription

2010-05-29T09:49:54Z

Ilic: Updated to new poascribe mode semantics.

{{Template:I18n/Language Navigation Bar|Localization/Workflows/PO_Summit}}

{{LocalizationBrowser|
section=[[Localization#Workflows|Workflows]]|
name=Reviewing by Ascriptions|
prereqs=[[Localization/Workflows/PO_Summit|Translating in Summit]]|
related=[[Localization/Tools/Pology|Pology]], [[Localization/Workflows/Coordinator|Language Coordinator]]|
}}

== Why Review Translations? ==

Especially to new translators, it may not be obvious to which extent the translation needs to be reviewed. If the translator has exercised due diligence, how "wrong" can the translation be? Even if the translator has good command of the source language (English in context of this article), the answer is "very wrong", when all aspects are considered. Here are some of them.

With comparatively simple grammar of English, the meaning of a short English sentence -- as typically encountered in application user interfaces -- is very dependent on the surrounding context. This context may not be obvious when the translator is going through isolated messages in the translation file, so he may commit the worst of errors from the user's viewpoint, the senseless translation. An experienced reviewer will have developed sense for troublesome contexts, and will have several means to decisively determine the context (including, for example, running the development version of the application).

Even if the context is correctly established, the translator may use "wrong" terminology, which is the next worse thing for the user. A term used in translation does not need to be wrong by itself, in fact it may be exactly the correct term -- in another translation project. The reviewer will have more experience with terminology of the present project, and be able to bring the translation in line with it.

Style in the technical sense is a consistent choice between several perfectly valid constructs in target language when applied to text in the given technical context. For example, how to translate menu titles and items, button labels, or tooltips. The choices may include noun or verb forms, particular grammar categories, tone of address, and so on. There may be a style guide to the project which details such choices, and the reviewer will know it well.

Style in the linguistic sense is especially applicable to longer texts, such as tooltips in user interfaces, and passages in documentation. A typical error of a new translator is to closely adhere to English style and grammar. This may produce translation which is semantically and grammatically valid in the target language, but very out of style -- the "translationese". Reviewer is there to naturalize such passages.

Finally, the reviewer may be an experienced translator, but that does not mean that his own translations need no review. Immersion into the source language, distraction, fatigue, will lead the reviewer into any of the above errors in translation, only with less frequency. So reviewers should also review one anothers' translations.

== Classical Reviewing by Stages ==

Classical review workflow by stages seems simple enough. Translator translates a PO file (or updates existing translation), and declares it ready to review. A reviewer reviews it, and declares it ready to "commit". Committing here should be understood generally, as inclusion into the pool from which translations are periodically shipped to end users. A committer finally commits the file. The process is iterative: the reviewer may return the file to the translator, and translator later again declare it as ready for review. There may be several stages of review (such as ''proof-reading'', ''approving''), each of which may return the translation to a previous stage, or forward it to some special stage. The process can also be more finer grained, where each message in the file goes through stages separately.

Regardless of the particularities, workflows of this kind all have the following in common. Members of the translation team are assigned ''roles'' -- such as ''translator'', ''reviewer'', ''approver'', ''committer'' -- by which they enter into the workflow (single person can have more roles). The later review stages must wait for the earlier stages to complete, and the translation cannot be updated again before the current version clears the pipeline (or the pipeline is aborted). Most importantly, once the translation is committed, it becomes part of simply "admitted" translations, with no further qualifiers.

The system of prescribed roles requires that team members assign them between themselves, stick to them, and shuffle them along the way. The prescribed review pipeline requires a tool to enforce and keep track of the stages in which translations are. This makes the review workflow complex and rigid, most probably with choke points for efficiency. Distribution of roles may become disbalanced by people coming and going, or the workflow tool may be prohibitive to some scenarios (e.g. single translator making small adjustments in dozens of files across the project, but having to upload each manually through a web interface).

Of course, "rigid", "complex", "inefficient", are comparative qualifications, so what is it that the classical review by stages can be compared to in this way?

== Reviewing by Ascriptions ==

Reviewing by ascriptions is even simpler conceptually, and yet less rigid, less complex, and much more efficient than the review by stages. It works on the message-level, rather than file-level. Anyone can simply translate some messages and directly commit modified files, without any review, but with ''ascribing'' modifications to own name. Anyone can review any committed messages at any moment, commit the modifications-on-review and ascribe reviews to own name and (possibly) to certain class -- full review, review of context, of terminology, of style, etc. Only when the translation is to be shipped to end users, the ''insufficiently'' reviewed messages are automatically omitted from the package, by evaluating the ''ascription history'' of each message.

Most importantly, based on the ascription history, the reviewer can select only some particular messages, and review only the difference between their historical and current versions. For example, Alice can select to review only messages modified since she or Bob had last reviewed them for style; she could see the difference from that last review to current version, e.g. if in the whole paragraph only a single word has changed by Charlie when he reviewed the terminology. In terms of PO workflow, the ascription history propagates through merges, so the reviewer can compare the change in original and the change in translation since the last review, to judge if one fits the other.

Since everyone just commits, translations can be efficiently kept in a version control repository, with the ascription system added on top. After having done some translating, the team member simply substitutes commit command of the version control system (VCS) with ascribe-modifications command of the ascription system (AS, which calls the underlying VCS internally). After reviewing, the team member uses ascribe-reviews command of the AS to commit reviews to ascription history (as well as modifications made during the review). To select messages for review, the team member issues diff-for-review command of the AS (with suitable parameters to narrow the set) and selected messages are marked in-place in PO files and [[Localization/Tools/Pology/PO_Embedded_Diffing|embedded with differences]], and possibly popped open in a PO editor.

When the translations are to be released, the team coordinator issues filter-for-release command of the AS, which takes the working PO files and creates final PO files with insufficiently reviewed messages removed. "Release time" is used here only figuratively: this should be a fully automatic process, so it can be performed at any interval of convenience.

What constitutes "sufficient review" can be defined in fine detail. It could be specified that messages modified by Alice need to have only review for terminology, but not necessarily for style; Charlie may belong to the group which needs to be reviewed on style, but not necessarily on context; Bob's reviews for style may be nice to have, but never blocking if missing. These decisions do not preclude released messages to be reviewed later on missing points, after higher priority reviews have been completed. The definition of sufficiency may be changed at any point, e.g. as team members get more experienced and require less review, without interfering with direct translation and review work.



In summary, with reviewing by ascriptions the lean efficiency of raw VCS operation is preserved while providing for great flexibility of review. All team members can be given commit access, no web or email detours are needed. There are no prescribed roles, but an equivalent of role assignment happens automatically at last possible moment, and can take into account both translators' and reviewers' abilities. There is no staging between completing and committing the translation, which enables translator to keep on polishing the translation undisturbed until the reviewer comes around. There is no inefficiency in handling small changes throughout many files, since single AS command commits all changes just as single VCS command would. AS in effect abstracts VCS, so general team members do not have to know the particularities of the underlying VCS. On commit operations, AS can also apply checks (e.g. decline to commit syntactically invalid PO files) and modifications (e.g. update translator's data in the PO header).

== Ascription System in Pology ==

[[Localization/Tools/Pology|Pology]] is a collection of various modular tools for supporting translation based on PO files. Among them is the script <tt>poascribe</tt>, which implements an ascription system (AS); at present, it can use Subversion or Git as the underlying VCS. <tt>poascribe</tt> is still in experimental stage, so what follows is a brief description of how to use it in context of the KDE translation project. However, very little is truly specific to KDE; the only major assumption is that there exists a VCS repository with PO files of a given language grouped together, and that the translation team can use it without special restrictions.

Very important for the AS is how branches are handled (in KDE, the rolling trunk and stable branches). AS can in principle be deployed by branch, but then there is the added complexity of porting translations between branches, which ascriptions should follow. Therefore, the AS implemented by <tt>poascribe</tt> is currently limited to assumption that there is a single branch of translations at all times. The article [[Localization/Workflows/PO_Summit|"Translating in Summit"]] explains how a KDE translation team can set up and operate such a single branch, the summit, and this is the prerequisite for the following instructions. (Note that the summit system is useful on its own, and should be conductive to any kind of review workflow.)

== Setting Up ==

The summit branch for the language <tt>LANG</tt> is positioned like this in the KDE repository:

<code text>
$KDEREPO/
trunk/
l10n-support/
LANG/
summit/
messages/
docmessages/
</code>

The team coordinator already has this part of the repository tree locally due to regular summit operations. For the same reason Pology is already set up. Setting up the ascription system is now simple. The file <tt>ascription-config</tt> is created in the root of the summit:

<code text>
...
LANG/
summit/
messages/
docmessages/
ascription-config
</code>

with the following contents:

<code text>
# ---------------------------
# Global ascription settings.

[global]

# The root of the ascription catalog tree.
ascript-root = ../summit-ascript

# The underlying version control system.
version-control = svn

# Data for updating catalog headers.
# - language code
language = LANG
# - full language name
language-team = LANGUAGE
# - email address of the team
team-email = kde-i18n-LANG@kde.org

# Default commit message.
commit-message = Translation updates.

# -----------------------
# Registered translators.

[user-alice]
name = Alice Akmalryn
original-name = Алиса Акмалрин
email = alice.akmalryn@someplacenice.org

[user-bob]
name = Bob Byomkin
original-name = Бобан Бјомкин
email = bob.byomkin@otherplacenice.org

# ...and so on.
</code>

Some notes:

* The <tt>ascript-root</tt> setting should be exactly <tt>../summit-ascript</tt>, for the reason mentioned later.

* <tt>commit-message</tt> field, if defined, allows team members to commit without providing a commit message. The value given by this field will be used by default, with translator's user name appended to the end in special syntax. For example: <tt>Translation updates. [>alice]</tt>. (Translator's user name is also appended to manually supplied commit messages.) Translators can still supply a commit message when they wish, as shown later. If this field is not set, the commit message is supplied as usual on committing.

* Team members are defined by <tt>[user-USERNAME]</tt> sections. Ascription user names can be any valid ASCII identifier: ASCII letters, digits and underscores only, digit cannot be the first character. Ascription user names have no technical relation to the underlying VCS accounts, though it is mnemonically convenient if they are the same (in case of SVN). This means that a translator who does not have a VCS account (yet) can and should be added here, with assigned user name (best one suitable as SVN account name later); why this should be done will be explained later.

* <tt>original-name</tt> field in user sections is there in case the preferred renderings of the name in English and in target language are not the same. When this is not the case, <tt>original-name</tt> can be omitted.

As soon as the <tt>ascription-config</tt> file is committed, the ascription system is ready for operation. Only regular modifications to this file are those of adding new team members. (On the other hand, team members should never be removed, because even after they no longer contribute, their ascription records remain in the system.)

=== Initial Ascription ===

The most common situation at start of ascription workflow is that there already exists a body of translations, contributed to by many different people over time. The coordinator should ascribe all existing translations as initial modifications, but to whom? It cannot be said precisely who translated what. The solution is to introduce a generic user in <tt>ascription-config</tt>, suitably known as "Unknown Hero" (or "Lost Translator", you can be inventive):

<code>
[user-uhero]
name = Unknown Hero
original-name = Незнани јунак
</code>

and ascribe all existing translations as modified and reviewed by this user. The coordinator does this with the following command:

<code bash>
$ cd $KDEREPO/trunk/l10n-support
$ poascribe commit -u uhero --all-reviewed -C LANG/summit/
</code>

The argument <tt>commit</tt> is the ascription mode, and the <tt>-u</tt> option provides the user name to which ascriptions are made. This is an important point: ascriptions are made to a user defined in ascription configuration, and have nothing to do with VCS accounts; someone who has the account can commit in the name of someone who does not. It is the <tt>--all-reviewed</tt> option that declares all messages to be reviewed as well (note that it is normally used only this once, and not for normal day to day reviewing). The <tt>-C</tt> option prevent automatic committing by <tt>poascribe</tt>, which is useful for this initial step. Finally the paths which contain all summit catalogs are given.

When the <tt>poascribe</tt> command is issued, a progress bar will appear, and the following output will start to unfold:

<code text>
LANG/summit/messages/extragear-base/rellinks.po (50/50)
LANG/summit/messages/extragear-base/autorefresh.po (13/13)
LANG/summit/messages/extragear-base/babelfish.po (38/38)
...
LANG/summit/messages/qt/libphonon.po (13/13)
LANG/summit/messages/qt/phonon-xine.po (24/24)
LANG/summit/messages/qt/phonon_gstreamer.po (12/12)
===== Ascription summary:
- modified reviewed
translated 111775 111775
fuzzy 26943 26943
obsolete/t 2965 2965
obsolete/f 1626 1626
</code>

The number in parenthesis indicates how many messages have been ascribed in the given PO file (modified/reviewed), and at the end the totals are given. Ascribing the complete summit for the first time will take quite some time (on the order of 10-20 minutes).



After the initial ascription has been made, the ascription tree will appear next to the summit tree. This tree will contain one ascription PO file for each summit PO file, with the same name and relative location within the tree:

<code text>
...
LANG/
summit/
messages/
kdelibs/
kcertpart.po
kdelibs4.po
...
...
docmessages/
ascription-config
summit-ascript/
messages/
kdelibs/
kcertpart.po
kdelibs4.po
...
...
docmessages/
</code>

The ascription tree can now be committed as usual (<tt>poascribe</tt> will have already added it):

<code bash>
$ svn commit LANG/summit-ascript/ -m "Initial ascription."
</code>

== Daily Use for Translators ==

Team members other than the coordinator, whether translators or reviewers, need to keep around only the <tt>trunk/l10n-support/LANG/</tt> directory. But they always need to update this directory fully (rather than just one particular module or file under <tt>.../*messages/</tt>), so that the summit tree and the ascription tree (and configuration) are kept in sync.

In order not to have to issue their own user name (<tt>-u</tt> option to <tt>poascribe</tt>) all the time, translators can set it in Pology user configuration <tt>~/.pologyrc</tt>, in <tt>[poascribe]</tt> section:

<code text>
[poascribe]
user = alice
</code>

Translators can then submit updated PO files simply by substituting <tt>svn commit</tt> (or whatever the VCS commit command is) with <tt>poascribe commit</tt> (<tt>co</tt> or <tt>ci</tt> for short):

<code bash>
$ poascribe ci LANG/summit/messages/kdefoo/*fooapp*.po
LANG/summit/messages/kdefoo/fooapp.po (144)
LANG/summit/messages/kdefoo/libfooapp.po (25)
===== Ascription summary:
- modified
translated 169
Sending LANG/summit/messages/kdefoo/fooapp.po
Sending LANG/summit/messages/kdefoo/libfooapp.po
Sending LANG/summit-ascript/messages/kdefoo/fooapp.po
Sending LANG/summit-ascript/messages/kdefoo/libfooapp.po
Transmitting file data ....
Committed revision 1267069.
$
</code>

<tt>poascribe</tt> will add ascription records into ascription catalogs corresponding to summit catalogs to be committed, and commit them all. Like <tt>svn commit</tt>, <tt>poascribe ci</tt> can take any number of file or directory paths, and can be issued from any working directory (it will always find ascription catalogs). If default commit message has not been set in the ascription configuration, <tt>poascribe</tt> will ask for it; or it can be given in command line through <tt>-m</tt> option.

=== Translators Without Commit Access ===

With the ascription system in place, every regular team member should have commit access. But, there may be some period of time before new translators are given accounts, revision control may be too technical for some, and even those with the account may not be able to commit temporarily for some reason.

These translators may send in their work by email, to ''any'' team member with commit access (not necessarily the coordinator or a reviewer); this team member can commit received files without any review, as review can be conducted at any later time. If Bob sends some files to Alice, she can commit them immediately by stating Bob's user name:

<code bash>
$ poascribe ci -u bob ...
</code>

For this to work, the translator who sent in the files has to be defined in the ascription configuration. There are no hidden costs or security issues to this (as opposed to opening a VCS account), so every new translator should be defined there before any work of that person is committed.

== Daily Use for Reviewers ==

The ascription system opens up all sorts of possibilities for concrete review patterns. Reviewers should keep in mind that for each message the full modification and review history is available, so that the team can think about how to make good use of it. Therefore, what follows are some examples to illustrate the review facilities that <tt>poascribe</tt> provides.

=== Basic Reviewing ===

At the very basic level (which is the only level in classical review by stages), messages can be classified into simply unreviewed and reviewed, without further qualifiers. Alice now wants to review all unreviewed messages in a group of PO files, say <tt>kdetoys</tt> module. She issues (<tt>di</tt> is short for <tt>diff</tt>):

<code bash>
$ cd $KDEREPO/trunk/l10n-support
$ poascribe di LANG/summit/messages/kdetoys/
LANG/summit/messages/kdegames/bovo.po (2)
LANG/summit/messages/kdegames/kdiamond.po (7)
LANG/summit/messages/kdegames/palapeli.po (12)
===== Diffed for review: 21
</code>

Unreviewed messages have now been marked and diffed, inside the listed PO files. What is this about "diffing"? If the files had already been reviewed before, some of the messages modified since then (those marked for review) may have changed very little (e.g. a few words in a paragraph-length message, or even just punctuation). Therefore, for each message marked for review, Alice also wants to see the diff since last review to current version. Here are two messages in typical review states added by <tt>poascribe di</tt>:

<code po>
#. +> trunk stable
#. ascto: charlie:m
#: gui/mainwindow.cc:372
#, ediff
msgid "GAME OVER. {-You won-}{+Tie+}!"
msgstr "KRAJ IGRE. {-Pobeda-}{+Nerešeno+}!"

#. +> trunk stable
#. ascto: bob:m charlie:m
#: game-state.cpp:117
#, ediff-total
msgid "Click the pause button again to resume the game."
msgstr "Kliknite ponovo na dugme pauze da nastavite igru."
</code>

and the first one in Kate:

[[Image:Kate_poascribe_diff_01.png|center|Message diffed for review by poascribe in Kate.]]

In the first message, the first to note is the <tt>#. ascto:</tt> comment. This comment succinctly lists who did what with the message since the last review; here <tt>charlie:m</tt> means that Charlie is the one who modified it. Then, there is the <tt>ediff</tt> flag, which alice can use it to jump through messages marked for review. Finally, the original and translation have been diffed; here they show that, since the last review, the message was fuzzied by changing "You won" to "Tie", and what Charlie did in translation to unfuzzy it. Even on a message as short as this, the diff tells something useful to Alice: the phrase "Game over" likely has a formulaic translation, and the fact that it is not part of the diff means that the earlier reviewer had made sure it is consistent, so Alice does not have to check that.

The <tt>#. ascto:</tt> comment of the second message reveals that both Charlie and Bob had been translating it. <tt>ediff-total</tt> flag instead of plain <tt>ediff</tt> means that this message had no review at all up to now, so there are no embedded diffs in text fields.

Alice can now go through marked files and messages, review translations, and possibly make modifications. When making changes in a message with embedded diffs, she can freely edit text outside of difference segments and within <tt>{+...+}</tt> segments (as these are the ones which belong to current version of the text). While reviewing, Alice does ''not'' remove any of the added message elements while reviewing (save for an occasional difference segment, when translation should be modified), as these elements are needed for later. If a message is particularly hard and Alice wants to defer its review for later, she can add the <tt>unreviewed</tt> (or <tt>nrev</tt> for short) flag to it.

Once the review is complete, Alice simply commits the reviewed files:

<code bash>
$ poascribe ci LANG/summit/messages/kdetoys/
LANG/summit/messages/kdegames/bovo.po (0/2)
LANG/summit/messages/kdegames/kdiamond.po (0/7)
LANG/summit/messages/kdegames/palapeli.po (3/12)
===== Ascription summary:
- modified reviewed
translated 3 21
Sending LANG/summit/messages/kdegames/palapeli.po
Sending LANG/summit-ascript/messages/kdegames/bovo.po
Sending LANG/summit-ascript/messages/kdegames/kdiamond.po
Sending LANG/summit-ascript/messages/kdegames/palapeli.po
Transmitting file data ....
Committed revision 1284220.
$
</code>

Three things have happened here. First, all review states (flags, embedded diffs, etc.) have been removed, restoring the PO file to normal. Then, any modifications that Alice have made during review are ascribed to her (here 3 out of 21 messages). Finally, all marked messages are ascribed as reviewed by Alice (any with <tt>unreviewed</tt>/<tt>nrev</tt> flags would have been omitted here). When committing, the only summit catalog that got committed is the one with modifications made during review, and all the ascription catalogs were committed because of the reviews recorded in them.

When many files with few changes in each are to be reviewed, it becomes burdensome to manually open each and every diffed for review, and then to make sure that all are committed with <tt>poascribe ci</tt>. To make this easier, <tt>-w torevivew.out</tt> option can be added to <tt>poascribe di</tt>, which requests that paths of all diffed PO files are written into <tt>torevivew.out</tt> file. This file can then be used to batch open POs for review in the editor, as well as fed back on <tt>poascribe re</tt> with <tt>-f torevivew.out</tt>. There is also the <tt>-o</tt> option which causes <tt>poascribe</tt> to directly open PO files in a PO editor (though this is currently applicable only to Lokalize). Putting it together, to efficiently review a whole bunch of small changes throughout many files, Alice can:

<code bash>
$ poascribe di PATHS... -w toreview.out -o lokalize
$ # ...only marked messages opened in Lokalize, review them...
$ poascribe ci -f toreview.out
</code>

=== Selecting Messages for Review ===

Invocations of <tt>poascribe di</tt> without any options, as in the previous section, were actually equivalent to this:

<code bash>
$ poascribe di -s modar PATHS...
</code>

Option <tt>-s</tt> is issuing the message ''selector''. <tt>modar</tt> is the default selector for <tt>diff</tt> mode, and stands for MODified-After-Review: it selects the earliest historical modification of the message after the last (or no) review of that message, if there is any such. By selecting a historical modification of the message, the diff from it to current version can be computed and embedded into the PO file, as in previous examples.

There are various specialized selectors, and fall into two groups: ''shallow selectors'' and ''history selectors''. Shallow selectors look only into the current version of the message, and cannot select historical versions, which means that they cannot provide embedded diffs. History selectors (<tt>modar</tt> is of this type) can select messages from history and provide diffs. Several selectors can be issued on the command line, and the message is selected only if all selectors select it. Shallow selectors are then normally used as a pre-filter for history selectors. For example, to select messages modified after last reviewed, but only those found in stable branch, <tt>branch</tt> and <tt>modar</tt> selectors are chained:

<code bash>
$ poascribe di -s branch:stable -s modar PATHS...
</code>

It is important that the history selector is given last, because the last selector determines which historical message is selected. If the ordering had been reversed here, same messages would get selected, but they would not have embedded diffs, because <tt>branch</tt> is a shallow selector.

Selectors can take parameters themselves, like <tt>branch:stable</tt> in the previous example. Parameters are separated from the selector name by any non-alphanumeric character; this is colon by convention, but if a parameter contains a colon, something like slash, tilde, etc. can be used. Number of parameters can be variable, and <tt>modar</tt> in particular can take from none to three. If Alice wants to review only those messages modified ''by Charlie'' since last review, she states this by first argument to <tt>modar</tt>:

<code bash>
$ poascribe di -s modar:charlie PATHS...
</code>

If Alice does not give too much credit to other reviewers, she can request selection of messages modified after last review ''by her'' with second parameter to <tt>modar</tt>:

<code bash>
$ poascribe di -s modar::alice PATHS...
</code>

Here the first parameter ("modified by..."), which is not needed, must be explicitly skipped, before going to the second parameter ("reviewed by..."). The third optional parameter of <tt>modar</tt> will be mentioned in the next section.

When a selector parameter is a user name, normally it can also be a comma-separated list of user names (<tt>modar:bob,charlie</tt>) or prefixed with tilde to negate, i.e. select all other users (<tt>modar:~alice</tt>).

Any selector can be negated by prepending <tt>n</tt> to its name. For example, the history selector <tt>modafter:DATE</tt> selects first modification after the given date; to select messages modified after last review, but only if modified during June 2010:

<code bash>
$ poascribe di -s modafter:2010-06 -s nmodafter:2010-07 -s modar PATHS...
</code>

Negating a history selector produces a shallow selector: while <tt>modafter</tt> is history selector, <tt>nmodafter</tt> is shallow. But the order of the two in the previous command line is not important, as the last selector is the usual <tt>modar</tt>.

Selectors can be issued in other modes too. If the PO file is big and Alice has reviewed messages up to and including entry 246 when she has to pause until another day, she can commit reviews only up to this entry by issuing the <tt>espan</tt> selector:

<code bash>
$ poascribe ci -s espan::246 PATHS...
</code>

(the first parameter to <tt>espan</tt> is the first entry number, given if messages are not to be selected from the first). There is also the counterpart <tt>lspan</tt> selector, which works with referent line numbers (those of <tt>msgid</tt> keywords) instead of entry numbers.

=== Fine-Grained Reviews ===

In the introduction, several distinct types of what can go wrong in translation were described. Not all reviewers may be able to check translation against all those problems. Here is a typical scenario of this kind:

Alice is very computer-savvy and knows the translation project inside and out, which means that she can review well for context, terminology, and technical style. But, her language style leaves something to be desired, which shows through longer sentences and passages. Dan, on the other hand, is a very literary person, but not that much into the technical aspects. Dan's style reviews would thus be a perfect complement to Alice's general reviews.

<tt>poascribe</tt> can support this scenario in the following way. A ''review type'' tag for language style is defined in the ascription configuration, using the <tt>review-tags</tt> field:

<code text>
[global]
...
review-tags = lstyle
</code>

(The value to <tt>review-tags</tt> is a space-separated list of identifiers, when more than one special review type is needed.) With this addition to configuration, Alice can continue to review as she did before, without any changes to her workflow.

Dan selects messages for review similarly to Alice, but aditionally giving the <tt>lstyle</tt> tag as ''third'' parameter of <tt>modar</tt>, and indicating that ascribed reviews should be tagged as <tt>lstyle</tt>:

<code bash>
$ poascribe di -s modar:::lstyle -t lstyle PATHS...
</code>

After finishing the review, Dan commits as usual:

<code bash>
$ poascribe ci PATHS...
</code>

If Dan is always going to review the language style, in order not to have to issue the selector and tag in the command line all the time, he can make them default per mode in <tt>~/.pologyrc</tt>:

<code text>
[poascribe]
user = dan
selectors/diff = modar:::lstyle
tags/diff = lstyle
</code>

With this Dan can use plain <tt>poascribe di</tt> just like Alice does.

The important point of review tags is that they make reviews by types independent. For example, Dan may come around to review the language style of the given message after several modifications and general reviews have been ascribed to it -- <tt>modar:::lstyle</tt> will simply ignore all reviews except for <tt>lstyle</tt> reviews. This is going to be reflected in the <tt>ascto:</tt> comment to marked messages:

<code po>
#...
#. ascto: charlie:m alice:r bob:m
#...
msgid "..."
msgstr "..."
</code>

Here Alice has made one review between Charlie's and Bob's modifications, and that review, being general instead of <tt>lstyle</tt>, did not cause <tt>modar</tt> to stop at it. After Dan reviews this message for language style, Alice runs selection for review and gets this:

<code po>
#...
#. ascto: bob:m dan:r(lstyle)
#...
msgid "..."
msgstr "..."
</code>

Again, since <tt>lstyle</tt> reviews do not mix with general reviews, Dan's review did not hide Bob's modification that Alice did not check so far.

(General review too has a tag assigned, the empty string, in case the reviewer needs to explicitly issue it in some context.)

== Daily Use for The Coordinator ==

After setting up the ascription system, the team coordinator should have to do very little to maintain it.

=== Ascribing Merges ===

Modifications made to summit catalogs by merging with templates must also be ascribed. Therefore, after merging the summit (<tt>posummit ... merge ...</tt>) the coordinator substitutes the VCS command:

<code bash>
$ svn commit LANG/summit/messages/ -m "Merged summit."
</code>

with the <tt>poascribe</tt> command in <tt>commit</tt> mode:

<code bash>
$ poascribe ci LANG/summit/messages/ -m "Merged summit."
</code>

Since the user is not explicitly given by <tt>-u</tt> option, this will ascribe merge modifications to the coordinator (more precisely, to the user set as default in <tt>~/.pologyrc</tt>), which is just fine. It is also possible to define a special user only for ascribing merge modifications, though there is no known advantage to that.

Since <tt>-C</tt> option is not issued, <tt>poascribe</tt> will automatically commit all modified summit and ascription catalogs when done.

=== Shuffling Ascription Catalogs ===

Sometimes summit catalogs are shuffled in the repository: moved to another module, renamed, one catalog split into two, two catalogs merged into one. Such shuffling should be exactly mirrored in the ascription tree, and this too is done on the repository side, at the same time. This relies on the ascription root being set exactly to <tt>../summit-ascript</tt> in the ascription configuration. So the team coordinator has nothing special to do here.

If instead in the central KDE repository the translation team is working in an external repository, by consequence the ascription system must be set up in that repository. But so long as <tt>process_orphans.sh</tt> script from <tt>trunk/l10n-support/scripts/</tt> is used to shuffle catalogs in the external repository as well, the ascription catalogs will be properly handled.

== Filtering for Release ==

The last component of the ascription system is how to prevent insufficiently reviewed messages from leaking into a release. In context of Pology and summit workflow, <tt>poascribe</tt> itself is used directly to this end. Instead, in the summit configuration (as opposed to ascription configuration), the team coordinator defines filters which pass messages by applying selectors.

Each top level PO tree has its own summit configuration file, named <tt>MSGTREE.extras.summit</tt>:

<code text>
...
LANG/
summit/
messages/
messages.extras.summit
docmessages/
docmessages.extras.summit
</code>

For the simple case of all reviews being general reviews, the filter is added to summit configuration like this (anywhere within <tt>*.extras.summit</tt> file):

<code python>
S.ascription_filters = [
("regular", ["nmodar"]),
]
</code>

Here the filter is named <tt>regular</tt>, and is defined as application of <tt>nmodar</tt> selector, the negation of <tt>modar</tt>. This simply means: pass all messages ''not'' modified after the last review.

When the team coordinator scatters to branches (executes <tt>posummit scatter</tt>), messages from summit POs which do not pass this filter will not be sent to branch POs. The count of stopped messages by branch PO will be reported in the output as scattering proceeds.

Why did we have to name the filter <tt>regular</tt>? (Those knowing some Python will also notice that it is defined as a list element.) Because it is possible to define more than one filter, and select which one is used on each scattering. For example, the coordinator may wish that, when the release is near and time is short to review everything, messages from a few experienced translators can be passed into release without review. If those translators are Alice and Bob, an "emergency" filter can be defined like this:

<code python>
S.ascription_filters = [
("regular", ["nmodar"]),
("emergency", ["nmodar:~alice,bob"]),
]
</code>

By default, <tt>posummit</tt> uses the first filter in the list. When the coordinator needs to do emergency scattering, he requests the emergency filter by the <tt>-a</tt> option:

<code bash>
$ cd $KDEREPO/trunk/l10n-support
$ posummit scripts/messages.summit LANG scatter -a emergency
</code>

What if several selectors are needed to pass the message? For example, the language style review (the earlier example with Alice and Dan) too may be requested for regular scattering, but omitted from emergency scattering. The filter setup for this scenario looks like this:

<code python>
S.ascription_filters = [
("regular", ["nmodar", "nmodar:::lstyle"]),
("emergency", ["nmodar:~alice,bob"]),
]
</code>

The regular filter now reads: pass the message if it has not been modified after the last (general) review ''and'' has not been modified after the last style review.

Simple combination of predefined selectors by AND-conditions may not be sufficient for more involved scenarios. When this is the case, the coordinator may write (or ask someone to write) a custom selector in Python, and plug it in as the second element in the filter tuple (instead of the list of predefined selectors).

=== Writing a Selector Function ===

((To be written.))

Localization/Workflows/PO Ascription

2010-05-13T10:15:18Z

Ilic: Stray Cyrillic text in the example.

{{Template:I18n/Language Navigation Bar|Localization/Workflows/PO_Summit}}

{{LocalizationBrowser|
section=[[Localization#Workflows|Workflows]]|
name=Reviewing by Ascriptions|
prereqs=[[Localization/Workflows/PO_Summit|Translating in Summit]]|
related=[[Localization/Tools/Pology|Pology]], [[Localization/Workflows/Coordinator|Language Coordinator]]|
}}

== Why Review Translations? ==

Especially to new translators, it may not be obvious to which extent the translation needs to be reviewed. If the translator has exercised due diligence, how "wrong" can the translation be? Even if the translator has good command of the source language (English in context of this article), the answer is "very wrong", when all aspects are considered. Here are some of them.

With comparatively simple grammar of English, the meaning of a short English sentence -- as typically encountered in application user interfaces -- is very dependent on the surrounding context. This context may not be obvious when the translator is going through isolated messages in the translation file, so he may commit the worst of errors from the user's viewpoint, the senseless translation. An experienced reviewer will have developed sense for troublesome contexts, and will have several means to decisively determine the context (including, for example, running the development version of the application).

Even if the context is correctly established, the translator may use "wrong" terminology, which is the next worse thing for the user. A term used in translation does not need to be wrong by itself, in fact it may be exactly the correct term -- in another translation project. The reviewer will have more experience with terminology of the present project, and be able to bring the translation in line with it.

Style in the technical sense is a consistent choice between several perfectly valid constructs in target language when applied to text in the given technical context. For example, how to translate menu titles and items, button labels, or tooltips. The choices may include noun or verb forms, particular grammar categories, tone of address, and so on. There may be a style guide to the project which details such choices, and the reviewer will know it well.

Style in the linguistic sense is especially applicable to longer texts, such as tooltips in user interfaces, and passages in documentation. A typical error of a new translator is to closely adhere to English style and grammar. This may produce translation which is semantically and grammatically valid in the target language, but very out of style -- the "translationese". Reviewer is there to naturalize such passages.

Finally, the reviewer may be an experienced translator, but that does not mean that his own translations need no review. Immersion into the source language, distraction, fatigue, will lead the reviewer into any of the above errors in translation, only with less frequency. So reviewers should also review one anothers' translations.

== Classical Reviewing by Stages ==

Classical review workflow by stages seems simple enough. Translator translates a PO file (or updates existing translation), and declares it ready to review. A reviewer reviews it, and declares it ready to "commit". Committing here should be understood generally, as inclusion into the pool from which translations are periodically shipped to end users. A committer finally commits the file. The process is iterative: the reviewer may return the file to the translator, and translator later again declare it as ready for review. There may be several stages of review (such as ''proof-reading'', ''approving''), each of which may return the translation to a previous stage, or forward it to some special stage. The process can also be more finer grained, where each message in the file goes through stages separately.

Regardless of the particularities, workflows of this kind all have the following in common. Members of the translation team are assigned ''roles'' -- such as ''translator'', ''reviewer'', ''approver'', ''committer'' -- by which they enter into the workflow (single person can have more roles). The later review stages must wait for the earlier stages to complete, and the translation cannot be updated again before the current version clears the pipeline (or the pipeline is aborted). Most importantly, once the translation is committed, it becomes part of simply "admitted" translations, with no further qualifiers.

The system of prescribed roles requires that team members assign them between themselves, stick to them, and shuffle them along the way. The prescribed review pipeline requires a tool to enforce and keep track of the stages in which translations are. This makes the review workflow complex and rigid, most probably with choke points for efficiency. Distribution of roles may become disbalanced by people coming and going, or the workflow tool may be prohibitive to some scenarios (e.g. single translator making small adjustments in dozens of files across the project, but having to upload each manually through a web interface).

Of course, "rigid", "complex", "inefficient", are comparative qualifications, so what is it that the classical review by stages can be compared to in this way?

== Reviewing by Ascriptions ==

Reviewing by ascriptions is even simpler conceptually, and yet less rigid, less complex, and much more efficient than the review by stages. It works on the message-level, rather than file-level. Anyone can simply translate some messages and directly commit modified files, without any review, but with ''ascribing'' modifications to own name. Anyone can review any committed messages at any moment, commit the modifications-on-review and ascribe reviews to own name and (possibly) to certain class -- full review, review of context, of terminology, of style, etc. Only when the translation is to be shipped to end users, the ''insufficiently'' reviewed messages are automatically omitted from the package, by evaluating the ''ascription history'' of each message.

Most importantly, based on the ascription history, the reviewer can select only some particular messages, and review only the difference between their historical and current versions. For example, Alice can select to review only messages modified since she or Bob had last reviewed them for style; she could see the difference from that last review to current version, e.g. if in the whole paragraph only a single word has changed by Charlie when he reviewed the terminology. In terms of PO workflow, the ascription history propagates through merges, so the reviewer can compare the change in original and the change in translation since the last review, to judge if one fits the other.

Since everyone just commits, translations can be efficiently kept in a version control repository, with the ascription system added on top. After having done some translating, the team member simply substitutes commit command of the version control system (VCS) with ascribe-modifications command of the ascription system (AS, which calls the underlying VCS internally). After reviewing, the team member uses ascribe-reviews command of the AS to commit reviews to ascription history (as well as modifications made during the review). To select messages for review, the team member issues diff-for-review command of the AS (with suitable parameters to narrow the set) and selected messages are marked in-place in PO files and [[Localization/Tools/Pology/PO_Embedded_Diffing|embedded with differences]], and possibly popped open in a PO editor.

When the translations are to be released, the team coordinator issues filter-for-release command of the AS, which takes the working PO files and creates final PO files with insufficiently reviewed messages removed. "Release time" is used here only figuratively: this should be a fully automatic process, so it can be performed at any interval of convenience.

What constitutes "sufficient review" can be defined in fine detail. It could be specified that messages modified by Alice need to have only review for terminology, but not necessarily for style; Charlie may belong to the group which needs to be reviewed on style, but not necessarily on context; Bob's reviews for style may be nice to have, but never blocking if missing. These decisions do not preclude released messages to be reviewed later on missing points, after higher priority reviews have been completed. The definition of sufficiency may be changed at any point, e.g. as team members get more experienced and require less review, without interfering with direct translation and review work.



In summary, with reviewing by ascriptions the lean efficiency of raw VCS operation is preserved while providing for great flexibility of review. All team members can be given commit access, no web or email detours are needed. There are no prescribed roles, but an equivalent of role assignment happens automatically at last possible moment, and can take into account both translators' and reviewers' abilities. There is no staging between completing and committing the translation, which enables translator to keep on polishing the translation undisturbed until the reviewer comes around. There is no inefficiency in handling small changes throughout many files, since single AS command commits all changes just as single VCS command would. AS in effect abstracts VCS, so general team members do not have to know the particularities of the underlying VCS. On commit operations, AS can also apply checks (e.g. decline to commit syntactically invalid PO files) and modifications (e.g. update translator's data in the PO header).

== Ascription System in Pology ==

[[Localization/Tools/Pology|Pology]] is a collection of various modular tools for supporting translation based on PO files. Among them is the script <tt>poascribe</tt>, which implements an ascription system (AS); at present, it can use Subversion or Git as the underlying VCS. <tt>poascribe</tt> is still in experimental stage, so what follows is a brief description of how to use it in context of the KDE translation project. However, very little is truly specific to KDE; the only major assumption is that there exists a VCS repository with PO files of a given language grouped together, and that the translation team can use it without special restrictions.

Very important for the AS is how branches are handled (in KDE, the rolling trunk and stable branches). AS can in principle be deployed by branch, but then there is the added complexity of porting translations between branches, which ascriptions should follow. Therefore, the AS implemented by <tt>poascribe</tt> is currently limited to assumption that there is a single branch of translations at all times. The article [[Localization/Workflows/PO_Summit|"Translating in Summit"]] explains how a KDE translation team can set up and operate such a single branch, the summit, and this is the prerequisite for the following instructions. (Note that the summit system is useful on its own, and should be conductive to any kind of review workflow.)

== Setting Up ==

The summit branch for the language <tt>LANG</tt> is positioned like this in the KDE repository:

<code text>
$KDEREPO/
trunk/
l10n-support/
LANG/
summit/
messages/
docmessages/
</code>

The team coordinator already has this part of the repository tree locally due to regular summit operations. For the same reason Pology is already set up. Setting up the ascription system is now simple. The file <tt>ascription-config</tt> is created in the root of the summit:

<code text>
...
LANG/
summit/
messages/
docmessages/
ascription-config
</code>

with the following contents:

<code text>
# ---------------------------
# Global ascription settings.

[global]

# The root of the ascription catalog tree.
ascript-root = ../summit-ascript

# The underlying version control system.
version-control = svn

# Data for updating catalog headers.
# - language code
language = LANG
# - full language name
language-team = LANGUAGE
# - email address of the team
team-email = kde-i18n-LANG@kde.org

# Default commit message.
commit-message = Translation updates.

# -----------------------
# Registered translators.

[user-alice]
name = Alice Akmalryn
original-name = Алиса Акмалрин
email = alice.akmalryn@someplacenice.org

[user-bob]
name = Bob Byomkin
original-name = Бобан Бјомкин
email = bob.byomkin@otherplacenice.org

# ...and so on.
</code>

Some notes:

* The <tt>ascript-root</tt> setting should be exactly <tt>../summit-ascript</tt>, for the reason mentioned later.

* <tt>commit-message</tt> field, if defined, allows team members to commit without providing a commit message. The value given by this field will be used by default, with translator's user name appended to the end in special syntax. For example: <tt>Translation updates. [>alice]</tt>. (Translator's user name is also appended to manually supplied commit messages.) Translators can still supply a commit message when they wish, as shown later. If this field is not set, the commit message is supplied as usual on committing.

* Team members are defined by <tt>[user-USERNAME]</tt> sections. Ascription user names can be any valid ASCII identifier: ASCII letters, digits and underscores only, digit cannot be the first character. Ascription user names have no technical relation to the underlying VCS accounts, though it is mnemonically convenient if they are the same (in case of SVN). This means that a translator who does not have a VCS account (yet) can and should be added here, with assigned user name (best one suitable as SVN account name later); why this should be done will be explained later.

* <tt>original-name</tt> field in user sections is there in case the preferred renderings of the name in English and in target language are not the same. When this is not the case, <tt>original-name</tt> can be omitted.

As soon as the <tt>ascription-config</tt> file is committed, the ascription system is ready for operation. Only regular modifications to this file are those of adding new team members. (On the other hand, team members should never be removed, because even after they no longer contribute, their ascription records remain in the system.)

=== Initial Ascription ===

The most common situation at start of ascription workflow is that there already exists a body of translations, contributed to by many different people over time. The coordinator should ascribe all existing translations as initial modifications, but to whom? It cannot be said precisely who translated what. The solution is to introduce a generic user in <tt>ascription-config</tt>, suitably known as "Unknown Hero" (or "Lost Translator", you can be inventive):

<code>
[user-uhero]
name = Unknown Hero
original-name = Незнани јунак
</code>

and ascribe all existing translations as modified and reviewed by this user. The coordinator does this with the following command:

<code bash>
$ cd $KDEREPO/trunk/l10n-support
$ poascribe reviewed -u uhero -C LANG/summit/
</code>

The argument <tt>reviewed</tt> is the ascription mode, and the <tt>-u</tt> option provides the user name to which ascriptions are made. This is an important point: ascriptions are made to a user defined in ascription configuration, and have nothing to do with VCS accounts; someone who has the account can commit in the name of someone who does not. The <tt>-C</tt> option prevent automatic committing by <tt>poascribe</tt>, which is useful for this initial step. Finally the paths which contain all summit catalogs are given.

When the <tt>poascribe</tt> command is issued, a progress bar will appear, and the following output will start to unfold:

<code text>
! LANG/summit-ascript/messages/extragear-base/rellinks.po (50)
! LANG/summit-ascript/messages/extragear-base/autorefresh.po (13)
! LANG/summit-ascript/messages/extragear-base/babelfish.po (38)
...
! LANG/summit-ascript/messages/qt/libphonon.po (13)
! LANG/summit-ascript/messages/qt/phonon-xine.po (24)
! LANG/summit-ascript/messages/qt/phonon_gstreamer.po (12)
===! Translated: 111775
===! Fuzzy: 26943
===! Obsolete translated: 2965
===! Obsolete fuzzy: 1626
! LANG/summit-ascript/messages/extragear-base/rellinks.po (50)
! LANG/summit-ascript/messages/extragear-base/autorefresh.po (13)
! LANG/summit-ascript/messages/extragear-base/babelfish.po (38)
...
! LANG/summit-ascript/messages/qt/libphonon.po (13)
! LANG/summit-ascript/messages/qt/phonon-xine.po (24)
! LANG/summit-ascript/messages/qt/phonon_gstreamer.po (12)
===! Reviewed: 143309
</code>

The number in parenthesis indicates how many messages have been ascribed in the given PO file, and at the end the totals are given. Catalogs are processed twice, first to ascribe modifications and then reviews, because a review cannot be ascribed before the modification has been ascribed. Ascribing the complete summit for the first time will take quite some time (say 15-30 minutes).



After the initial ascription has been made, the ascription tree will appear next to the summit tree. This tree will contain one ascription PO file for each summit PO file, with the same name and relative location within the tree:

<code text>
...
LANG/
summit/
messages/
kdelibs/
kcertpart.po
kdelibs4.po
...
...
docmessages/
ascription-config
summit-ascript/
messages/
kdelibs/
kcertpart.po
kdelibs4.po
...
...
docmessages/
</code>

The ascription tree can now be committed as usual (<tt>poascribe</tt> will have already added it):

<code bash>
$ svn commit LANG/summit-ascript/ -m "Initial ascription."
</code>

== Daily Use for Translators ==

Team members other than the coordinator, whether translators or reviewers, need to keep around only the <tt>trunk/l10n-support/LANG/</tt> directory. But they always need to update this directory fully (rather than just one particular module or file under <tt>.../*messages/</tt>), so that the summit tree and the ascription tree (and configuration) are kept in sync.

In order not to have to issue their own user name (<tt>-u</tt> option to <tt>poascribe</tt>) all the time, translators can set it in Pology user configuration <tt>~/.pologyrc</tt>, in <tt>[poascribe]</tt> section:

<code text>
[poascribe]
user = alice
</code>

Translators can then submit updated PO files simply by substituting <tt>svn commit</tt> (or whatever the VCS commit command is) with <tt>poascribe modified</tt> (<tt>mo</tt> for short):

<code bash>
$ poascribe mo LANG/summit/messages/kdefoo/*fooapp*.po
! LANG/summit-ascript/messages/kdefoo/fooapp.po (144)
! LANG/summit-ascript/messages/kdefoo/libfooapp.po (25)
===! Translated: 169
Sending LANG/summit/messages/kdefoo/fooapp.po
Sending LANG/summit/messages/kdefoo/libfooapp.po
Sending LANG/summit-ascript/messages/kdefoo/fooapp.po
Sending LANG/summit-ascript/messages/kdefoo/libfooapp.po
Transmitting file data ....
Committed revision 1267069.
$
</code>

<tt>poascribe</tt> will add ascription records into ascription catalogs corresponding to summit catalogs to be committed, and commit them all. Like <tt>svn commit</tt>, <tt>poascribe mo</tt> can take any number of file or directory paths, and can be issued from any working directory (it will always find ascription catalogs). If default commit message has not been set in the ascription configuration, <tt>poascribe</tt> will ask for it; or it can be given in command line through <tt>-m</tt> option.

=== Translators Without Commit Access ===

With the ascription system in place, every regular team member should have commit access. But, there may be some period of time before new translators are given accounts, revision control may be too technical for some, and even those with the account may not be able to commit temporarily for some reason.

These translators may send in their work by email, to ''any'' team member with commit access (not necessarily the coordinator or a reviewer); this team member can commit received files without any review, as review can be conducted at any later time. If Bob sends some files to Alice, she can commit them immediately by stating Bob's user name:

<code bash>
$ poascribe mo -u bob ...
</code>

For this to work, the translator who sent in the files has to be defined in the ascription configuration. There are no hidden costs or security issues to this (as opposed to opening a VCS account), so every new translator should be defined there before any work of that person is committed.

== Daily Use for Reviewers ==

The ascription system opens up all sorts of possibilities for concrete review patterns. Reviewers should keep in mind that for each message the full modification and review history is available, so that the team can think about how to make good use of it. Therefore, what follows are some examples to illustrate the review facilities that <tt>poascribe</tt> provides.

=== Basic Reviewing ===

At the very basic level (which is the only level in classical review by stages), messages can be classified into simply unreviewed and reviewed, without further qualifiers. Alice now wants to review all unreviewed messages in a group of PO files, say <tt>kdetoys</tt> module. She issues (<tt>di</tt> is short for <tt>diff</tt>):

<code bash>
$ cd $KDEREPO/trunk/l10n-support
$ poascribe di LANG/summit/messages/kdetoys/
! LANG/summit/messages/kdegames/bovo.po (2)
! LANG/summit/messages/kdegames/kdiamond.po (7)
! LANG/summit/messages/kdegames/palapeli.po (12)
===! Diffed for review: 21
</code>

Unreviewed messages have now been marked and diffed, inside the listed PO files. What is this about "diffing"? If the files had already been reviewed before, some of the messages modified since then (those marked for review) may have changed very little (e.g. a few words in a paragraph-length message, or even just punctuation). Therefore, for each message marked for review, Alice also wants to see the diff since last review to current version. Here are two messages in typical review states added by <tt>poascribe di</tt>:

<code po>
#. +> trunk stable
#. ascto: charlie:m
#: gui/mainwindow.cc:372
#, ediff
msgid "GAME OVER. {-You won-}{+Tie+}!"
msgstr "KRAJ IGRE. {-Pobeda-}{+Nerešeno+}!"

#. +> trunk stable
#. ascto: bob:m charlie:m
#: game-state.cpp:117
#, ediff-total
msgid "Click the pause button again to resume the game."
msgstr "Kliknite ponovo na dugme pauze da nastavite igru."
</code>

and the first one in Kate:

[[Image:Kate_poascribe_diff_01.png|center|Message diffed for review by poascribe in Kate.]]

In the first message, the first to note is the <tt>#. ascto:</tt> comment. This comment succinctly lists who did what with the message since the last review; here <tt>charlie:m</tt> means that Charlie is the one who modified it. Then, there is the <tt>ediff</tt> flag, which alice can use it to jump through messages marked for review. Finally, the original and translation have been diffed; here they show that, since the last review, the message was fuzzied by changing "You won" to "Tie", and what Charlie did in translation to unfuzzy it. Even on a message as short as this, the diff tells something useful to Alice: the phrase "Game over" likely has a formulaic translation, and the fact that it is not part of the diff means that the earlier reviewer had made sure it is consistent, so Alice does not have to check that.

The <tt>#. ascto:</tt> comment of the second message reveals that both Charlie and Bob had been translating it. <tt>ediff-total</tt> flag instead of plain <tt>ediff</tt> means that this message had no review at all up to now, so there are no embedded diffs in text fields.

Alice can now go through marked files and messages, review translations, and possibly make modifications. When making changes in a message with embedded diffs, she can freely edit text outside of difference segments and within <tt>{+...+}</tt> segments (as these are the ones which belong to current version of the text). While reviewing, Alice does ''not'' remove any of the added message elements while reviewing (save for an occasional difference segment, when translation should be modified), as these elements are needed for later. If a message is particularly hard and Alice wants to defer its review for later, she can add the <tt>unreviewed</tt> (or <tt>urev</tt> or <tt>nrev</tt> for short) flag to it.

Once the review is complete, Alice commits the reviewed files in <tt>reviewed</tt> mode (short <tt>re</tt>):

<code bash>
$ poascribe re LANG/summit/messages/kdetoys/{bovo,kdiamond,palapeli}.po
! LANG/summit/messages/kdegames/bovo.po (2)
! LANG/summit/messages/kdegames/kdiamond.po (7)
! LANG/summit/messages/kdegames/palapeli.po (12)
===! Cleared reviews: 21
! LANG/summit-ascript/messages/kdegames/palapeli.po (3)
===! Translated: 3
! LANG/summit-ascript/messages/kdegames/bovo.po (2)
! LANG/summit-ascript/messages/kdegames/kdiamond.po (7)
! LANG/summit-ascript/messages/kdegames/palapeli.po (12)
===! Reviewed: 21
Sending LANG/summit/messages/kdegames/palapeli.po
Sending LANG/summit-ascript/messages/kdegames/bovo.po
Sending LANG/summit-ascript/messages/kdegames/kdiamond.po
Sending LANG/summit-ascript/messages/kdegames/palapeli.po
Transmitting file data ....
Committed revision 1284220.
$
</code>

Three things have happened here. First, all review states (flags, embedded diffs, etc.) have been removed, restoring the PO file to normal. Then, any modifications that Alice have made during review are ascribed to her (here 3 out of 21 messages). Finally, all marked messages are ascribed as reviewed by Alice (any with <tt>unreviewed</tt>/<tt>urev</tt>/<tt>nrev</tt> flags would have been omitted here). When committing, the only summit catalog that got committed is the one with modifications made during review, and all the ascription catalogs were committed because of the reviews recorded in them.

When many files with few changes in each are to be reviewed, it becomes burdensome to manually open each and every diffed for review, and then to make sure that all are committed with <tt>poascribe re</tt>. To make this easier, <tt>-w torevivew.out</tt> option can be added to <tt>poascribe di</tt>, which requests that paths of all diffed PO files are written into <tt>torevivew.out</tt> file. This file can then be used to batch open POs for review in the editor, as well as fed back on <tt>poascribe re</tt> with <tt>-f torevivew.out</tt>. There is also the <tt>-o</tt> option which causes <tt>poascribe</tt> to directly open PO files in a PO editor, though this is currently applicable only to Lokalize. Putting it together, to efficiently review a whole bunch of small changes throughout many files, Alice can:

<code bash>
$ poascribe di PATHS... -w toreview.out -o lokalize
$ # ...only marked messages opened in Lokalize, review them...
$ poascribe re -f toreview.out
</code>

=== Selecting Messages for Review ===

Invocations of <tt>poascribe di</tt> without any options, as in the previous section, were actually equivalent to this:

<code bash>
$ poascribe di -s modar PATHS...
</code>

Option <tt>-s</tt> is issuing the message ''selector''. <tt>modar</tt> is the default selector for <tt>diff</tt> mode, and stands for MODified-After-Review: it selects the earliest historical modification of the message after the last (or no) review of that message, if there is any such. By selecting a historical modification of the message, the diff from it to current version can be computed and embedded into the PO file, as in previous examples.

There are various specialized selectors, and fall into two groups: ''shallow selectors'' and ''history selectors''. Shallow selectors look only into the current version of the message, and cannot select historical versions, which means that they cannot provide embedded diffs. History selectors (<tt>modar</tt> is of this type) can select messages from history and provide diffs. Several selectors can be issued on the command line, and the message is selected only if all selectors select it. Shallow selectors are then normally used as a pre-filter for history selectors. For example, to select messages modified after last reviewed, but only those found in stable branch, <tt>branch</tt> and <tt>modar</tt> selectors are chained:

<code bash>
$ poascribe di -s branch:stable -s modar PATHS...
</code>

It is important that the history selector is given last, because the last selector determines which historical message is selected. If the ordering had been reversed here, same messages would get selected, but they would not have embedded diffs, because <tt>branch</tt> is a shallow selector.

Selectors can take parameters themselves, like <tt>branch:stable</tt> in the previous example. Parameters are separated from the selector name by any non-alphanumeric character; this is colon by convention, but if a parameter contains a colon, something like slash, tilde, etc. can be used. Number of parameters can be variable, and <tt>modar</tt> in particular can take from none to three. If Alice wants to review only those messages modified ''by Charlie'' since last review, she states this by first argument to <tt>modar</tt>:

<code bash>
$ poascribe di -s modar:charlie PATHS...
</code>

If Alice does not give too much credit to other reviewers, she can request selection of messages modified after last review ''by her'' with second parameter to <tt>modar</tt>:

<code bash>
$ poascribe di -s modar::alice PATHS...
</code>

Here the first parameter ("modified by..."), which is not needed, must be explicitly skipped, before going to the second parameter ("reviewed by..."). The third optional parameter of <tt>modar</tt> will be mentioned in the next section.

When a selector parameter is a user name, normally it can also be a comma-separated list of user names (<tt>modar:bob,charlie</tt>) or prefixed with tilde to negate, i.e. select all other users (<tt>modar:~alice</tt>).

Any selector can be negated by prepending <tt>n</tt> to its name. For example, the history selector <tt>modafter:DATE</tt> selects first modification after the given date; to select messages modified after last review, but only if modified during June 2010:

<code bash>
$ poascribe di -s modafter:2010-06 -s nmodafter:2010-07 -s modar PATHS...
</code>

Negating a history selector produces a shallow selector: while <tt>modafter</tt> is history selector, <tt>nmodafter</tt> is shallow. But the order of the two in the previous command line is not important, as the last selector is the usual <tt>modar</tt>.

Selectors can be issued in other modes too. If the PO file is big and Alice has reviewed messages up to and including entry 246 when she has to pause until another day, she can commit reviews only up to this entry by issuing the <tt>espan</tt> selector:

<code bash>
$ poascribe re -s espan::246 PATHS...
</code>

(the first parameter to <tt>espan</tt> is the first entry number, given if messages are not to be selected from the first). There is also the counterpart <tt>lspan</tt> selector, which works with referent line numbers (those of <tt>msgid</tt> keywords) instead of entry numbers.

=== Fine-Grained Reviews ===

In the introduction, several distinct types of what can go wrong in translation were described. Not all reviewers may be able to check translation against all those problems. Here is a typical scenario of this kind:

Alice is very computer-savvy and knows the translation project inside and out, which means that she can review well for context, terminology, and technical style. But, her language style leaves something to be desired, which shows through longer sentences and passages. Dan, on the other hand, is a very literary person, but not that much into the technical aspects. Dan's style reviews would thus be a perfect complement to Alice's general reviews.

<tt>poascribe</tt> can support this scenario in the following way. A ''review type'' tag for language style is defined in the ascription configuration, using the <tt>review-tags</tt> field:

<code text>
[global]
...
review-tags = lstyle
</code>

(The value to <tt>review-tags</tt> is a space-separated list of identifiers, when more than one special review type is needed.) With this addition to configuration, Alice can continue to review as she did before, without any changes to her workflow.

Dan selects messages for review quite similarly to Alice, with the exception of giving the <tt>lstyle</tt> tag as ''third'' parameter of <tt>modar</tt>:

<code bash>
$ poascribe di -s modar:::lstyle PATHS...
</code>

When committing reviews, Dan must also state this tag, in order to ascribe reviews as of language style type:

<code bash>
$ poascribe re -t lstyle PATHS...
</code>

If Dan is always going to review the language style, in order not to have to issue the selector and tag in the command line all the time, he can make them default per mode in <tt>~/.pologyrc</tt>:

<code text>
[poascribe]
user = dan
selector/diff = modar:::lstyle
tag/reviewed = lstyle
</code>

With this Dan can use plain <tt>poascribe di</tt> and <tt>poascribe re</tt>, just like Alice does.

The important point of review tags is that they make reviews by types independent. For example, Dan may come around to review the language style of the given message after several modifications and general reviews have been ascribed to it -- <tt>modar:::lstyle</tt> will simply ignore all reviews except for <tt>lstyle</tt> reviews. This is going to be reflected in the <tt>ascto:</tt> comment to marked messages:

<code po>
#...
#. ascto: charlie:m alice:r bob:m
#...
msgid "..."
msgstr "..."
</code>

Here Alice has made one review between Charlie's and Bob's modifications, and that review, being general instead of <tt>lstyle</tt>, did not cause <tt>modar</tt> to stop at it. After Dan reviews this message for language style, Alice runs selection for review and gets this:

<code po>
#...
#. ascto: bob:m dan:r(lstyle)
#...
msgid "..."
msgstr "..."
</code>

Again, since <tt>lstyle</tt> reviews do not mix with general reviews, Dan's review did not hide Bob's modification that Alice did not check so far.

(General review too has a tag assigned, the empty string, in case the reviewer needs to explicitly issue it in some context.)

== Daily Use for The Coordinator ==

After setting up the ascription system, the team coordinator should have to do very little to maintain it.

=== Ascribing Merges ===

Modifications made to summit catalogs by merging with templates must also be ascribed. Therefore, after merging the summit (<tt>posummit ... merge ...</tt>) the coordinator substitutes the VCS command:

<code bash>
$ svn commit LANG/summit/messages/ -m "Merged summit."
</code>

with the <tt>poascribe</tt> command in <tt>modified</tt> mode:

<code bash>
$ poascribe mo LANG/summit/messages/ -m "Merged summit."
</code>

Since the user is not explicitly given by <tt>-u</tt> option, this will ascribe merge modifications to the coordinator (more precisely, to the user set as default in <tt>~/.pologyrc</tt>), which is just fine. It is also possible to define a special user only for ascribing merge modifications, though there is no known advantage to that.

Since <tt>-C</tt> option is not issued, <tt>poascribe</tt> will automatically commit all modified summit and ascription catalogs when done.

=== Shuffling Ascription Catalogs ===

Sometimes summit catalogs are shuffled in the repository: moved to another module, renamed, one catalog split into two, two catalogs merged into one. Such shuffling should be exactly mirrored in the ascription tree, and this too is done on the repository side, at the same time. This relies on the ascription root being set exactly to <tt>../summit-ascript</tt> in the ascription configuration. So the team coordinator has nothing special to do here.

If instead in the central KDE repository the translation team is working in an external repository, by consequence the ascription system must be set up in that repository. But so long as <tt>process_orphans.sh</tt> script from <tt>trunk/l10n-support/scripts/</tt> is used to shuffle catalogs in the external repository as well, the ascription catalogs will be properly handled.

== Filtering for Release ==

The last component of the ascription system is how to prevent insufficiently reviewed messages from leaking into a release. In context of Pology and summit workflow, <tt>poascribe</tt> itself is used directly to this end. Instead, in the summit configuration (as opposed to ascription configuration), the team coordinator defines filters which pass messages by applying selectors.

Each top level PO tree has its own summit configuration file, named <tt>MSGTREE.extras.summit</tt>:

<code text>
...
LANG/
summit/
messages/
messages.extras.summit
docmessages/
docmessages.extras.summit
</code>

For the simple case of all reviews being general reviews, the filter is added to summit configuration like this (anywhere within <tt>*.extras.summit</tt> file):

<code python>
S.ascription_filters = [
("regular", ["nmodar"]),
]
</code>

Here the filter is named <tt>regular</tt>, and is defined as application of <tt>nmodar</tt> selector, the negation of <tt>modar</tt>. This simply means: pass all messages ''not'' modified after the last review.

When the team coordinator scatters to branches (executes <tt>posummit scatter</tt>), messages from summit POs which do not pass this filter will not be sent to branch POs. The count of stopped messages by branch PO will be reported in the output as scattering proceeds.

Why did we have to name the filter <tt>regular</tt>? (Those knowing some Python will also notice that it is defined as a list element.) Because it is possible to define more than one filter, and select which one is used on each scattering. For example, the coordinator may wish that, when the release is near and time is short to review everything, messages from a few experienced translators can be passed into release without review. If those translators are Alice and Bob, an "emergency" filter can be defined like this:

<code python>
S.ascription_filters = [
("regular", ["nmodar"]),
("emergency", ["nmodar:~alice,bob"]),
]
</code>

By default, <tt>posummit</tt> uses the first filter in the list. When the coordinator needs to do emergency scattering, he requests the emergency filter by the <tt>-a</tt> option:

<code bash>
$ cd $KDEREPO/trunk/l10n-support
$ posummit scripts/messages.summit LANG scatter -a emergency
</code>

What if several selectors are needed to pass the message? For example, the language style review (the earlier example with Alice and Dan) too may be requested for regular scattering, but omitted from emergency scattering. The filter setup for this scenario looks like this:

<code python>
S.ascription_filters = [
("regular", ["nmodar", "nmodar:::lstyle"]),
("emergency", ["nmodar:~alice,bob"]),
]
</code>

The regular filter now reads: pass the message if it has not been modified after the last (general) review ''and'' has not been modified after the last style review.

Simple combination of predefined selectors by AND-conditions may not be sufficient for more involved scenarios. When this is the case, the coordinator may write (or ask someone to write) a custom selector in Python, and plug it in as the second element in the filter tuple (instead of the list of predefined selectors).

=== Writing a Selector Function ===

((To be written.))

Localization/Workflows/PO Ascription

2010-01-28T15:18:07Z

Ilic: Eliminated mention of 'fuzzy' reserved user.

{{Template:I18n/Language Navigation Bar|Localization/Workflows/PO_Summit}}

{{LocalizationBrowser|
section=[[Localization#Workflows|Workflows]]|
name=Reviewing by Ascriptions|
prereqs=[[Localization/Workflows/PO_Summit|Translating in Summit]]|
related=[[Localization/Tools/Pology|Pology]], [[Localization/Workflows/Coordinator|Language Coordinator]]|
}}

== Why Review Translations? ==

Especially to new translators, it may not be obvious to which extent the translation needs to be reviewed. If the translator has exercised due diligence, how "wrong" can the translation be? Even if the translator has good command of the source language (English in context of this article), the answer is "very wrong", when all aspects are considered. Here are some of them.

With comparatively simple grammar of English, the meaning of a short English sentence -- as typically encountered in application user interfaces -- is very dependent on the surrounding context. This context may not be obvious when the translator is going through isolated messages in the translation file, so he may commit the worst of errors from the user's viewpoint, the senseless translation. An experienced reviewer will have developed sense for troublesome contexts, and will have several means to decisively determine the context (including, for example, running the development version of the application).

Even if the context is correctly established, the translator may use "wrong" terminology, which is the next worse thing for the user. A term used in translation does not need to be wrong by itself, in fact it may be exactly the correct term -- in another translation project. The reviewer will have more experience with terminology of the present project, and be able to bring the translation in line with it.

Style in the technical sense is a consistent choice between several perfectly valid constructs in target language when applied to text in the given technical context. For example, how to translate menu titles and items, button labels, or tooltips. The choices may include noun or verb forms, particular grammar categories, tone of address, and so on. There may be a style guide to the project which details such choices, and the reviewer will know it well.

Style in the linguistic sense is especially applicable to longer texts, such as tooltips in user interfaces, and passages in documentation. A typical error of a new translator is to closely adhere to English style and grammar. This may produce translation which is semantically and grammatically valid in the target language, but very out of style -- the "translationese". Reviewer is there to naturalize such passages.

Finally, the reviewer may be an experienced translator, but that does not mean that his own translations need no review. Immersion into the source language, distraction, fatigue, will lead the reviewer into any of the above errors in translation, only with less frequency. So reviewers should also review one anothers' translations.

== Classical Reviewing by Stages ==

Classical review workflow by stages seems simple enough. Translator translates a PO file (or updates existing translation), and declares it ready to review. A reviewer reviews it, and declares it ready to "commit". Committing here should be understood generally, as inclusion into the pool from which translations are periodically shipped to end users. A committer finally commits the file. The process is iterative: the reviewer may return the file to the translator, and translator later again declare it as ready for review. There may be several stages of review (such as ''proof-reading'', ''approving''), each of which may return the translation to a previous stage, or forward it to some special stage. The process can also be more finer grained, where each message in the file goes through stages separately.

Regardless of the particularities, workflows of this kind all have the following in common. Members of the translation team are assigned ''roles'' -- such as ''translator'', ''reviewer'', ''approver'', ''committer'' -- by which they enter into the workflow (single person can have more roles). The later review stages must wait for the earlier stages to complete, and the translation cannot be updated again before the current version clears the pipeline (or the pipeline is aborted). Most importantly, once the translation is committed, it becomes part of simply "admitted" translations, with no further qualifiers.

The system of prescribed roles requires that team members assign them between themselves, stick to them, and shuffle them along the way. The prescribed review pipeline requires a tool to enforce and keep track of the stages in which translations are. This makes the review workflow complex and rigid, most probably with choke points for efficiency. Distribution of roles may become disbalanced by people coming and going, or the workflow tool may be prohibitive to some scenarios (e.g. single translator making small adjustments in dozens of files across the project, but having to upload each manually through a web interface).

Of course, "rigid", "complex", "inefficient", are comparative qualifications, so what is it that the classical review by stages can be compared to in this way?

== Reviewing by Ascriptions ==

Reviewing by ascriptions is even simpler conceptually, and yet less rigid, less complex, and much more efficient than the review by stages. It works on the message-level, rather than file-level. Anyone can simply translate some messages and directly commit modified files, without any review, but with ''ascribing'' modifications to own name. Anyone can review any committed messages at any moment, commit the modifications-on-review and ascribe reviews to own name and (possibly) to certain class -- full review, review of context, of terminology, of style, etc. Only when the translation is to be shipped to end users, the ''insufficiently'' reviewed messages are automatically omitted from the package, by evaluating the ''ascription history'' of each message.

Most importantly, based on the ascription history, the reviewer can select only some particular messages, and review only the difference between their historical and current versions. For example, Alice can select to review only messages modified since she or Bob had last reviewed them for style; she could see the difference from that last review to current version, e.g. if in the whole paragraph only a single word has changed by Charlie when he reviewed the terminology. In terms of PO workflow, the ascription history propagates through merges, so the reviewer can compare the change in original and the change in translation since the last review, to judge if one fits the other.

Since everyone just commits, translations can be efficiently kept in a version control repository, with the ascription system added on top. After having done some translating, the team member simply substitutes commit command of the version control system (VCS) with ascribe-modifications command of the ascription system (AS, which calls the underlying VCS internally). After reviewing, the team member uses ascribe-reviews command of the AS to commit reviews to ascription history (as well as modifications made during the review). To select messages for review, the team member issues diff-for-review command of the AS (with suitable parameters to narrow the set) and selected messages are marked in-place in PO files and [[Localization/Tools/Pology/PO_Embedded_Diffing|embedded with differences]], and possibly popped open in a PO editor.

When the translations are to be released, the team coordinator issues filter-for-release command of the AS, which takes the working PO files and creates final PO files with insufficiently reviewed messages removed. "Release time" is used here only figuratively: this should be a fully automatic process, so it can be performed at any interval of convenience.

What constitutes "sufficient review" can be defined in fine detail. It could be specified that messages modified by Alice need to have only review for terminology, but not necessarily for style; Charlie may belong to the group which needs to be reviewed on style, but not necessarily on context; Bob's reviews for style may be nice to have, but never blocking if missing. These decisions do not preclude released messages to be reviewed later on missing points, after higher priority reviews have been completed. The definition of sufficiency may be changed at any point, e.g. as team members get more experienced and require less review, without interfering with direct translation and review work.



In summary, with reviewing by ascriptions the lean efficiency of raw VCS operation is preserved while providing for great flexibility of review. All team members can be given commit access, no web or email detours are needed. There are no prescribed roles, but an equivalent of role assignment happens automatically at last possible moment, and can take into account both translators' and reviewers' abilities. There is no staging between completing and committing the translation, which enables translator to keep on polishing the translation undisturbed until the reviewer comes around. There is no inefficiency in handling small changes throughout many files, since single AS command commits all changes just as single VCS command would. AS in effect abstracts VCS, so general team members do not have to know the particularities of the underlying VCS. On commit operations, AS can also apply checks (e.g. decline to commit syntactically invalid PO files) and modifications (e.g. update translator's data in the PO header).

== Ascription System in Pology ==

[[Localization/Tools/Pology|Pology]] is a collection of various modular tools for supporting translation based on PO files. Among them is the script <tt>poascribe</tt>, which implements an ascription system (AS); at present, it can use Subversion or Git as the underlying VCS. <tt>poascribe</tt> is still in experimental stage, so what follows is a brief description of how to use it in context of the KDE translation project. However, very little is truly specific to KDE; the only major assumption is that there exists a VCS repository with PO files of a given language grouped together, and that the translation team can use it without special restrictions.

Very important for the AS is how branches are handled (in KDE, the rolling trunk and stable branches). AS can in principle be deployed by branch, but then there is the added complexity of porting translations between branches, which ascriptions should follow. Therefore, the AS implemented by <tt>poascribe</tt> is currently limited to assumption that there is a single branch of translations at all times. The article [[Localization/Workflows/PO_Summit|"Translating in Summit"]] explains how a KDE translation team can set up and operate such a single branch, the summit, and this is the prerequisite for the following instructions. (Note that the summit system is useful on its own, and should be conductive to any kind of review workflow.)

== Setting Up ==

The summit branch for the language <tt>LANG</tt> is positioned like this in the KDE repository:

<code text>
$KDEREPO/
trunk/
l10n-support/
LANG/
summit/
messages/
docmessages/
</code>

The team coordinator already has this part of the repository tree locally due to regular summit operations. For the same reason Pology is already set up. Setting up the ascription system is now simple. The file <tt>ascription-config</tt> is created in the root of the summit:

<code text>
...
LANG/
summit/
messages/
docmessages/
ascription-config
</code>

with the following contents:

<code text>
# ---------------------------
# Global ascription settings.

[global]

# The root of the ascription catalog tree.
ascript-root = ../summit-ascript

# The underlying version control system.
version-control = svn

# Data for updating catalog headers.
# - language code
language = LANG
# - full language name
language-team = LANGUAGE
# - email address of the team
team-email = kde-i18n-LANG@kde.org

# Default commit message.
commit-message = Translation updates.

# -----------------------
# Registered translators.

[user-alice]
name = Alice Akmalryn
original-name = Алиса Акмалрин
email = alice.akmalryn@someplacenice.org

[user-bob]
name = Bob Byomkin
original-name = Бобан Бјомкин
email = bob.byomkin@otherplacenice.org

# ...and so on.
</code>

Some notes:

* The <tt>ascript-root</tt> setting should be exactly <tt>../summit-ascript</tt>, for the reason mentioned later.

* <tt>commit-message</tt> field, if defined, allows team members to commit without providing a commit message. The value given by this field will be used by default, with translator's user name appended to the end in special syntax. For example: <tt>Translation updates. [>alice]</tt>. (Translator's user name is also appended to manually supplied commit messages.) Translators can still supply a commit message when they wish, as shown later. If this field is not set, the commit message is supplied as usual on committing.

* Team members are defined by <tt>[user-USERNAME]</tt> sections. Ascription user names can be any valid ASCII identifier: ASCII letters, digits and underscores only, digit cannot be the first character. Ascription user names have no technical relation to the underlying VCS accounts, though it is mnemonically convenient if they are the same (in case of SVN). This means that a translator who does not have a VCS account (yet) can and should be added here, with assigned user name (best one suitable as SVN account name later); why this should be done will be explained later.

* <tt>original-name</tt> field in user sections is there in case the preferred renderings of the name in English and in target language are not the same. When this is not the case, <tt>original-name</tt> can be omitted.

As soon as the <tt>ascription-config</tt> file is committed, the ascription system is ready for operation. Only regular modifications to this file are those of adding new team members. (On the other hand, team members should never be removed, because even after they no longer contribute, their ascription records remain in the system.)

=== Initial Ascription ===

The most common situation at start of ascription workflow is that there already exists a body of translations, contributed to by many different people over time. The coordinator should ascribe all existing translations as initial modifications, but to whom? It cannot be said precisely who translated what. The solution is to introduce a generic user in <tt>ascription-config</tt>, suitably known as "Unknown Hero" (or "Lost Translator", you can be inventive):

<code>
[user-uhero]
name = Unknown Hero
original-name = Незнани јунак
</code>

and ascribe all existing translations as modified and reviewed by this user. The coordinator does this with the following command:

<code bash>
$ cd $KDEREPO/trunk/l10n-support
$ poascribe reviewed -u uhero -C LANG/summit/
</code>

The argument <tt>reviewed</tt> is the ascription mode, and the <tt>-u</tt> option provides the user name to which ascriptions are made. This is an important point: ascriptions are made to a user defined in ascription configuration, and have nothing to do with VCS accounts; someone who has the account can commit in the name of someone who does not. The <tt>-C</tt> option prevent automatic committing by <tt>poascribe</tt>, which is useful for this initial step. Finally the paths which contain all summit catalogs are given.

When the <tt>poascribe</tt> command is issued, a progress bar will appear, and the following output will start to unfold:

<code text>
! LANG/summit-ascript/messages/extragear-base/rellinks.po (50)
! LANG/summit-ascript/messages/extragear-base/autorefresh.po (13)
! LANG/summit-ascript/messages/extragear-base/babelfish.po (38)
...
! LANG/summit-ascript/messages/qt/libphonon.po (13)
! LANG/summit-ascript/messages/qt/phonon-xine.po (24)
! LANG/summit-ascript/messages/qt/phonon_gstreamer.po (12)
===! Translated: 111775
===! Fuzzy: 26943
===! Obsolete translated: 2965
===! Obsolete fuzzy: 1626
! LANG/summit-ascript/messages/extragear-base/rellinks.po (50)
! LANG/summit-ascript/messages/extragear-base/autorefresh.po (13)
! LANG/summit-ascript/messages/extragear-base/babelfish.po (38)
...
! LANG/summit-ascript/messages/qt/libphonon.po (13)
! LANG/summit-ascript/messages/qt/phonon-xine.po (24)
! LANG/summit-ascript/messages/qt/phonon_gstreamer.po (12)
===! Reviewed: 143309
</code>

The number in parenthesis indicates how many messages have been ascribed in the given PO file, and at the end the totals are given. Catalogs are processed twice, first to ascribe modifications and then reviews, because a review cannot be ascribed before the modification has been ascribed. Ascribing the complete summit for the first time will take quite some time (say 15-30 minutes).



After the initial ascription has been made, the ascription tree will appear next to the summit tree. This tree will contain one ascription PO file for each summit PO file, with the same name and relative location within the tree:

<code text>
...
LANG/
summit/
messages/
kdelibs/
kcertpart.po
kdelibs4.po
...
...
docmessages/
ascription-config
summit-ascript/
messages/
kdelibs/
kcertpart.po
kdelibs4.po
...
...
docmessages/
</code>

The ascription tree can now be committed as usual (<tt>poascribe</tt> will have already added it):

<code bash>
$ svn commit LANG/summit-ascript/ -m "Initial ascription."
</code>

== Daily Use for Translators ==

Team members other than the coordinator, whether translators or reviewers, need to keep around only the <tt>trunk/l10n-support/LANG/</tt> directory. But they always need to update this directory fully (rather than just one particular module or file under <tt>.../*messages/</tt>), so that the summit tree and the ascription tree (and configuration) are kept in sync.

In order not to have to issue their own user name (<tt>-u</tt> option to <tt>poascribe</tt>) all the time, translators can set it in Pology user configuration <tt>~/.pologyrc</tt>, in <tt>[poascribe]</tt> section:

<code text>
[poascribe]
user = alice
</code>

Translators can then submit updated PO files simply by substituting <tt>svn commit</tt> (or whatever the VCS commit command is) with <tt>poascribe modified</tt> (<tt>mo</tt> for short):

<code bash>
$ poascribe mo LANG/summit/messages/kdefoo/*fooapp*.po
! LANG/summit-ascript/messages/kdefoo/fooapp.po (144)
! LANG/summit-ascript/messages/kdefoo/libfooapp.po (25)
===! Translated: 169
Sending LANG/summit/messages/kdefoo/fooapp.po
Sending LANG/summit/messages/kdefoo/libfooapp.po
Sending LANG/summit-ascript/messages/kdefoo/fooapp.po
Sending LANG/summit-ascript/messages/kdefoo/libfooapp.po
Transmitting file data ....
Committed revision 1267069.
$
</code>

<tt>poascribe</tt> will add ascription records into ascription catalogs corresponding to summit catalogs to be committed, and commit them all. Like <tt>svn commit</tt>, <tt>poascribe mo</tt> can take any number of file or directory paths, and can be issued from any working directory (it will always find ascription catalogs). If default commit message has not been set in the ascription configuration, <tt>poascribe</tt> will ask for it; or it can be given in command line through <tt>-m</tt> option.

=== Translators Without Commit Access ===

With the ascription system in place, every regular team member should have commit access. But, there may be some period of time before new translators are given accounts, revision control may be too technical for some, and even those with the account may not be able to commit temporarily for some reason.

These translators may send in their work by email, to ''any'' team member with commit access (not necessarily the coordinator or a reviewer); this team member can commit received files without any review, as review can be conducted at any later time. If Bob sends some files to Alice, she can commit them immediately by stating Bob's user name:

<code bash>
$ poascribe mo -u bob ...
</code>

For this to work, the translator who sent in the files has to be defined in the ascription configuration. There are no hidden costs or security issues to this (as opposed to opening a VCS account), so every new translator should be defined there before any work of that person is committed.

== Daily Use for Reviewers ==

The ascription system opens up all sorts of possibilities for concrete review patterns. Reviewers should keep in mind that for each message the full modification and review history is available, so that the team can think about how to make good use of it. Therefore, what follows are some examples to illustrate the review facilities that <tt>poascribe</tt> provides.

=== Basic Reviewing ===

At the very basic level (which is the only level in classical review by stages), messages can be classified into simply unreviewed and reviewed, without further qualifiers. Alice now wants to review all unreviewed messages in a group of PO files, say <tt>kdetoys</tt> module. She issues (<tt>di</tt> is short for <tt>diff</tt>):

<code bash>
$ cd $KDEREPO/trunk/l10n-support
$ poascribe di LANG/summit/messages/kdetoys/
! LANG/summit/messages/kdegames/bovo.po (2)
! LANG/summit/messages/kdegames/kdiamond.po (7)
! LANG/summit/messages/kdegames/palapeli.po (12)
===! Diffed for review: 21
</code>

Unreviewed messages have now been marked and diffed, inside the listed PO files. What is this about "diffing"? If the files had already been reviewed before, some of the messages modified since then (those marked for review) may have changed very little (e.g. a few words in a paragraph-length message, or even just punctuation). Therefore, for each message marked for review, Alice also wants to see the diff since last review to current version. Here are two messages in typical review states added by <tt>poascribe di</tt>:

<code po>
#. +> trunk stable
#. ascto: charlie:m
#: gui/mainwindow.cc:372
#, ediff
msgid "GAME OVER. {-You won-}{+Tie+}!"
msgstr "KRAJ IGRE. {-Pobeda-}{+Nerešeno+}!"

#. +> trunk stable
#. ascto: bob:m charlie:m
#: game-state.cpp:117
#, ediff-total
msgid "Click the pause button again to resume the game."
msgstr "Кликните поново на дугме паузе за наставак игре."
</code>

and the first one in Kate:

[[Image:Kate_poascribe_diff_01.png|center|Message diffed for review by poascribe in Kate.]]

In the first message, the first to note is the <tt>#. ascto:</tt> comment. This comment succinctly lists who did what with the message since the last review; here <tt>charlie:m</tt> means that Charlie is the one who modified it. Then, there is the <tt>ediff</tt> flag, which alice can use it to jump through messages marked for review. Finally, the original and translation have been diffed; here they show that, since the last review, the message was fuzzied by changing "You won" to "Tie", and what Charlie did in translation to unfuzzy it. Even on a message as short as this, the diff tells something useful to Alice: the phrase "Game over" likely has a formulaic translation, and the fact that it is not part of the diff means that the earlier reviewer had made sure it is consistent, so Alice does not have to check that.

The <tt>#. ascto:</tt> comment of the second message reveals that both Charlie and Bob had been translating it. <tt>ediff-total</tt> flag instead of plain <tt>ediff</tt> means that this message had no review at all up to now, so there are no embedded diffs in text fields.

Alice can now go through marked files and messages, review translations, and possibly make modifications. When making changes in a message with embedded diffs, she can freely edit text outside of difference segments and within <tt>{+...+}</tt> segments (as these are the ones which belong to current version of the text). While reviewing, Alice does ''not'' remove any of the added message elements while reviewing (save for an occasional difference segment, when translation should be modified), as these elements are needed for later. If a message is particularly hard and Alice wants to defer its review for later, she can add the <tt>unreviewed</tt> (or <tt>urev</tt> or <tt>nrev</tt> for short) flag to it.

Once the review is complete, Alice commits the reviewed files in <tt>reviewed</tt> mode (short <tt>re</tt>):

<code bash>
$ poascribe re LANG/summit/messages/kdetoys/{bovo,kdiamond,palapeli}.po
! LANG/summit/messages/kdegames/bovo.po (2)
! LANG/summit/messages/kdegames/kdiamond.po (7)
! LANG/summit/messages/kdegames/palapeli.po (12)
===! Cleared reviews: 21
! LANG/summit-ascript/messages/kdegames/palapeli.po (3)
===! Translated: 3
! LANG/summit-ascript/messages/kdegames/bovo.po (2)
! LANG/summit-ascript/messages/kdegames/kdiamond.po (7)
! LANG/summit-ascript/messages/kdegames/palapeli.po (12)
===! Reviewed: 21
Sending LANG/summit/messages/kdegames/palapeli.po
Sending LANG/summit-ascript/messages/kdegames/bovo.po
Sending LANG/summit-ascript/messages/kdegames/kdiamond.po
Sending LANG/summit-ascript/messages/kdegames/palapeli.po
Transmitting file data ....
Committed revision 1284220.
$
</code>

Three things have happened here. First, all review states (flags, embedded diffs, etc.) have been removed, restoring the PO file to normal. Then, any modifications that Alice have made during review are ascribed to her (here 3 out of 21 messages). Finally, all marked messages are ascribed as reviewed by Alice (any with <tt>unreviewed</tt>/<tt>urev</tt>/<tt>nrev</tt> flags would have been omitted here). When committing, the only summit catalog that got committed is the one with modifications made during review, and all the ascription catalogs were committed because of the reviews recorded in them.

When many files with few changes in each are to be reviewed, it becomes burdensome to manually open each and every diffed for review, and then to make sure that all are committed with <tt>poascribe re</tt>. To make this easier, <tt>-w torevivew.out</tt> option can be added to <tt>poascribe di</tt>, which requests that paths of all diffed PO files are written into <tt>torevivew.out</tt> file. This file can then be used to batch open POs for review in the editor, as well as fed back on <tt>poascribe re</tt> with <tt>-f torevivew.out</tt>. There is also the <tt>-o</tt> option which causes <tt>poascribe</tt> to directly open PO files in a PO editor, though this is currently applicable only to Lokalize. Putting it together, to efficiently review a whole bunch of small changes throughout many files, Alice can:

<code bash>
$ poascribe di PATHS... -w toreview.out -o lokalize
$ # ...only marked messages opened in Lokalize, review them...
$ poascribe re -f toreview.out
</code>

=== Selecting Messages for Review ===

Invocations of <tt>poascribe di</tt> without any options, as in the previous section, were actually equivalent to this:

<code bash>
$ poascribe di -s modar PATHS...
</code>

Option <tt>-s</tt> is issuing the message ''selector''. <tt>modar</tt> is the default selector for <tt>diff</tt> mode, and stands for MODified-After-Review: it selects the earliest historical modification of the message after the last (or no) review of that message, if there is any such. By selecting a historical modification of the message, the diff from it to current version can be computed and embedded into the PO file, as in previous examples.

There are various specialized selectors, and fall into two groups: ''shallow selectors'' and ''history selectors''. Shallow selectors look only into the current version of the message, and cannot select historical versions, which means that they cannot provide embedded diffs. History selectors (<tt>modar</tt> is of this type) can select messages from history and provide diffs. Several selectors can be issued on the command line, and the message is selected only if all selectors select it. Shallow selectors are then normally used as a pre-filter for history selectors. For example, to select messages modified after last reviewed, but only those found in stable branch, <tt>branch</tt> and <tt>modar</tt> selectors are chained:

<code bash>
$ poascribe di -s branch:stable -s modar PATHS...
</code>

It is important that the history selector is given last, because the last selector determines which historical message is selected. If the ordering had been reversed here, same messages would get selected, but they would not have embedded diffs, because <tt>branch</tt> is a shallow selector.

Selectors can take parameters themselves, like <tt>branch:stable</tt> in the previous example. Parameters are separated from the selector name by any non-alphanumeric character; this is colon by convention, but if a parameter contains a colon, something like slash, tilde, etc. can be used. Number of parameters can be variable, and <tt>modar</tt> in particular can take from none to three. If Alice wants to review only those messages modified ''by Charlie'' since last review, she states this by first argument to <tt>modar</tt>:

<code bash>
$ poascribe di -s modar:charlie PATHS...
</code>

If Alice does not give too much credit to other reviewers, she can request selection of messages modified after last review ''by her'' with second parameter to <tt>modar</tt>:

<code bash>
$ poascribe di -s modar::alice PATHS...
</code>

Here the first parameter ("modified by..."), which is not needed, must be explicitly skipped, before going to the second parameter ("reviewed by..."). The third optional parameter of <tt>modar</tt> will be mentioned in the next section.

When a selector parameter is a user name, normally it can also be a comma-separated list of user names (<tt>modar:bob,charlie</tt>) or prefixed with tilde to negate, i.e. select all other users (<tt>modar:~alice</tt>).

Any selector can be negated by prepending <tt>n</tt> to its name. For example, the history selector <tt>modafter:DATE</tt> selects first modification after the given date; to select messages modified after last review, but only if modified during June 2010:

<code bash>
$ poascribe di -s modafter:2010-06 -s nmodafter:2010-07 -s modar PATHS...
</code>

Negating a history selector produces a shallow selector: while <tt>modafter</tt> is history selector, <tt>nmodafter</tt> is shallow. But the order of the two in the previous command line is not important, as the last selector is the usual <tt>modar</tt>.

Selectors can be issued in other modes too. If the PO file is big and Alice has reviewed messages up to and including entry 246 when she has to pause until another day, she can commit reviews only up to this entry by issuing the <tt>espan</tt> selector:

<code bash>
$ poascribe re -s espan::246 PATHS...
</code>

(the first parameter to <tt>espan</tt> is the first entry number, given if messages are not to be selected from the first). There is also the counterpart <tt>lspan</tt> selector, which works with referent line numbers (those of <tt>msgid</tt> keywords) instead of entry numbers.

=== Fine-Grained Reviews ===

In the introduction, several distinct types of what can go wrong in translation were described. Not all reviewers may be able to check translation against all those problems. Here is a typical scenario of this kind:

Alice is very computer-savvy and knows the translation project inside and out, which means that she can review well for context, terminology, and technical style. But, her language style leaves something to be desired, which shows through longer sentences and passages. Dan, on the other hand, is a very literary person, but not that much into the technical aspects. Dan's style reviews would thus be a perfect complement to Alice's general reviews.

<tt>poascribe</tt> can support this scenario in the following way. A ''review type'' tag for language style is defined in the ascription configuration, using the <tt>review-tags</tt> field:

<code text>
[global]
...
review-tags = lstyle
</code>

(The value to <tt>review-tags</tt> is a space-separated list of identifiers, when more than one special review type is needed.) With this addition to configuration, Alice can continue to review as she did before, without any changes to her workflow.

Dan selects messages for review quite similarly to Alice, with the exception of giving the <tt>lstyle</tt> tag as ''third'' parameter of <tt>modar</tt>:

<code bash>
$ poascribe di -s modar:::lstyle PATHS...
</code>

When committing reviews, Dan must also state this tag, in order to ascribe reviews as of language style type:

<code bash>
$ poascribe re -t lstyle PATHS...
</code>

If Dan is always going to review the language style, in order not to have to issue the selector and tag in the command line all the time, he can make them default per mode in <tt>~/.pologyrc</tt>:

<code text>
[poascribe]
user = dan
selector/diff = modar:::lstyle
tag/reviewed = lstyle
</code>

With this Dan can use plain <tt>poascribe di</tt> and <tt>poascribe re</tt>, just like Alice does.

The important point of review tags is that they make reviews by types independent. For example, Dan may come around to review the language style of the given message after several modifications and general reviews have been ascribed to it -- <tt>modar:::lstyle</tt> will simply ignore all reviews except for <tt>lstyle</tt> reviews. This is going to be reflected in the <tt>ascto:</tt> comment to marked messages:

<code po>
#...
#. ascto: charlie:m alice:r bob:m
#...
msgid "..."
msgstr "..."
</code>

Here Alice has made one review between Charlie's and Bob's modifications, and that review, being general instead of <tt>lstyle</tt>, did not cause <tt>modar</tt> to stop at it. After Dan reviews this message for language style, Alice runs selection for review and gets this:

<code po>
#...
#. ascto: bob:m dan:r(lstyle)
#...
msgid "..."
msgstr "..."
</code>

Again, since <tt>lstyle</tt> reviews do not mix with general reviews, Dan's review did not hide Bob's modification that Alice did not check so far.

(General review too has a tag assigned, the empty string, in case the reviewer needs to explicitly issue it in some context.)

== Daily Use for The Coordinator ==

After setting up the ascription system, the team coordinator should have to do very little to maintain it.

=== Ascribing Merges ===

Modifications made to summit catalogs by merging with templates must also be ascribed. Therefore, after merging the summit (<tt>posummit ... merge ...</tt>) the coordinator substitutes the VCS command:

<code bash>
$ svn commit LANG/summit/messages/ -m "Merged summit."
</code>

with the <tt>poascribe</tt> command in <tt>modified</tt> mode:

<code bash>
$ poascribe mo LANG/summit/messages/ -m "Merged summit."
</code>

Since the user is not explicitly given by <tt>-u</tt> option, this will ascribe merge modifications to the coordinator (more precisely, to the user set as default in <tt>~/.pologyrc</tt>), which is just fine. It is also possible to define a special user only for ascribing merge modifications, though there is no known advantage to that.

Since <tt>-C</tt> option is not issued, <tt>poascribe</tt> will automatically commit all modified summit and ascription catalogs when done.

=== Shuffling Ascription Catalogs ===

Sometimes summit catalogs are shuffled in the repository: moved to another module, renamed, one catalog split into two, two catalogs merged into one. Such shuffling should be exactly mirrored in the ascription tree, and this too is done on the repository side, at the same time. This relies on the ascription root being set exactly to <tt>../summit-ascript</tt> in the ascription configuration. So the team coordinator has nothing special to do here.

If instead in the central KDE repository the translation team is working in an external repository, by consequence the ascription system must be set up in that repository. But so long as <tt>process_orphans.sh</tt> script from <tt>trunk/l10n-support/scripts/</tt> is used to shuffle catalogs in the external repository as well, the ascription catalogs will be properly handled.

== Filtering for Release ==

The last component of the ascription system is how to prevent insufficiently reviewed messages from leaking into a release. In context of Pology and summit workflow, <tt>poascribe</tt> itself is used directly to this end. Instead, in the summit configuration (as opposed to ascription configuration), the team coordinator defines filters which pass messages by applying selectors.

Each top level PO tree has its own summit configuration file, named <tt>MSGTREE.extras.summit</tt>:

<code text>
...
LANG/
summit/
messages/
messages.extras.summit
docmessages/
docmessages.extras.summit
</code>

For the simple case of all reviews being general reviews, the filter is added to summit configuration like this (anywhere within <tt>*.extras.summit</tt> file):

<code python>
S.ascription_filters = [
("regular", ["nmodar"]),
]
</code>

Here the filter is named <tt>regular</tt>, and is defined as application of <tt>nmodar</tt> selector, the negation of <tt>modar</tt>. This simply means: pass all messages ''not'' modified after the last review.

When the team coordinator scatters to branches (executes <tt>posummit scatter</tt>), messages from summit POs which do not pass this filter will not be sent to branch POs. The count of stopped messages by branch PO will be reported in the output as scattering proceeds.

Why did we have to name the filter <tt>regular</tt>? (Those knowing some Python will also notice that it is defined as a list element.) Because it is possible to define more than one filter, and select which one is used on each scattering. For example, the coordinator may wish that, when the release is near and time is short to review everything, messages from a few experienced translators can be passed into release without review. If those translators are Alice and Bob, an "emergency" filter can be defined like this:

<code python>
S.ascription_filters = [
("regular", ["nmodar"]),
("emergency", ["nmodar:~alice,bob"]),
]
</code>

By default, <tt>posummit</tt> uses the first filter in the list. When the coordinator needs to do emergency scattering, he requests the emergency filter by the <tt>-a</tt> option:

<code bash>
$ cd $KDEREPO/trunk/l10n-support
$ posummit scripts/messages.summit LANG scatter -a emergency
</code>

What if several selectors are needed to pass the message? For example, the language style review (the earlier example with Alice and Dan) too may be requested for regular scattering, but omitted from emergency scattering. The filter setup for this scenario looks like this:

<code python>
S.ascription_filters = [
("regular", ["nmodar", "nmodar:::lstyle"]),
("emergency", ["nmodar:~alice,bob"]),
]
</code>

The regular filter now reads: pass the message if it has not been modified after the last (general) review ''and'' has not been modified after the last style review.

Simple combination of predefined selectors by AND-conditions may not be sufficient for more involved scenarios. When this is the case, the coordinator may write (or ask someone to write) a custom selector in Python, and plug it in as the second element in the filter tuple (instead of the list of predefined selectors).

=== Writing a Selector Function ===

((To be written.))

Localization/Workflows/PO Summit

2010-01-19T18:16:19Z

Ilic: Replaced real language code with LANG placeholder.

{{Template:I18n/Language Navigation Bar|Localization/Workflows/PO_Summit}}

{{LocalizationBrowser|
section=[[Localization#Workflows|Workflows]]|
name=Translating in Summit|
prereqs=[[Localization/Workflows/Coordinator|Language Coordinator]]|
related=[[Localization/Tools/Pology|Pology]]|
}}

== Software Branches and Translation ==

The obvious approach to releasing software is for developers to focus on one central body of code, a single "branch", fixing bugs and adding new features to it, and from time to time taking a "snapshot" of the branch and packing it as next release. One approach to make the release process more robust, is for development to proceed in two parallel branches. From the "stable" branch the actual releases are made, mostly with bugs fixed and only very important features added. The other, "unstable" branch, is used to develop new and redesign existing features, and at some point will become the next stable branch. Depending on the project, releases may be made from the unstable branch as well, in parallel with stable releases, in order for eager users to help testing the novelties.

In KDE, all three release models may be present at any given moment. Core KDE modules, which are together labeled as "the KDE" and released in unison, follow the two branch model, with stable and trunk branch, and releases from stable branch only. KDE extragear applications may do the same, but they are not required to; they may instead make releases from the trunk branch only, or also use stable and trunk branch, but make releases from both. This presents translators with the workflow as on the following figure.

[[Image:Translating_in_branches.png|center|Classic translation by branches]]

The KDE [[Localization/Workflows/Repository_Automation|repository automation]] (aka "Scripty") is preparing template POs and merging them with language POs. From the point of view of language teams, real people who perform global actions (e.g. moving around POs for all languages when an application moves from module to another) can also be grouped here. However, the translation team is still presented with two branches of POs to translate.

In general, this means that, just like programmers, at times translators have to work on both branches, and propagate fixes from one branch to another ("backport" from trunk to stable, or "forwardport" from stable to trunk). This may be prone to coordinatorial confusion, and demand extra effort and attention when updating translations. Keeping up with two branches is easier in core KDE modules, as they are released from stable branch only and with singular schedule, but could be more taxing with extragear applications.

The exact amount of effort spent due to parallel translation, porting of fixes, and coordination, depends on the translation team. For example, a well-manned and well-coordinated team, with established style and terminology guides, custom automation to support these, may be able to keep all POs in both branches fully translated at all times with ease. Or, a team may have worked out well-defined schedules, such that any given member of the team at one point switches from translating one branch to another, without looking back, thus effectively having everyone always working on one branch (even if not the same one).

If, however, you as the team coordinator have said and thought "Should do that in trunk too, bugger", "Didn't we fix that already?", "No, you should take it from stable", "It was released from WHAT branch?", more often than you would have liked, the following presents one possibility for sidestepping such issues.

== Translating in Summit ==

For a PO catalog which exists in both stable and trunk branch, a new same-named catalog can be made by ''gathering'' all the unique messages from branch PO files, i.e. the ''summit'' of branch POs. Assuming that branch POs were not all that different, since they are two versions of the same catalog, the summit PO shouldn't have much more messages than either. Translators at all times work on the summit PO, from which the messages are periodically ''scattered'' back to original branch POs. Thus, translators can always work on summit POs, not having to do any parallel translation, branch switching or porting of fixes. The summit workflow is presented by the following figure.

[[Image:Translating_in_summit.png|center|Translation in summit]]

In the summit mode, repository automation gathers summit PO templates from branch templates, and stores them separately from real branches, at {{path|trunk/l10n-support/templates/summit/}}. The language team also has a collection summit POs, at {{path|trunk/l10n-support/LANG/summit/}}, which is the sole location where translation happens. As before, repository automation handles merging of templates in branches, but merging of summit POs is done by the team coordinator; team coordinator can opt to manually merge branch POs as well (more on why-and-how of this later). From time to time, the team coordinator fills out branch POs by scattering from summit. Not to worry, each of these special actions upon the team coordinator is done with a single command.

While it is in principle obvious that two branch POs can be made into one summit PO with the union of their messages, there are some important details that should be handled the right way by the summitting system:

* What if a PO changes modules in one branch, so that it no longer belongs to the same module in both branches (e.g. application is moved from one to another module in trunk)?

* What if a PO changes its name in one branch, but not in the other (e.g. application is renamed in trunk)?

* What if a PO in one branch is split into two POs in another (e.g. extracting a library out of monolithic application in trunk)?

* Where to place messages unique to one branch in the summit PO? The original file context of a message, which messages precede it and which follow it, should be kept as much as possible.

* If source references are used to achieve good ordering of messages in summit PO, what to do if some source file paths change in one branch (e.g. application gets restructured in trunk)?

* How to handle messages with different plurality across branches (since messages are identified only by their <tt>msgctxt</tt> and <tt>msgid</tt> field, an not <tt>msgid_plural</tt>)?

And handle these details it does, the present summitting system. This also means that teams working in the summit need not take care of the first three issues above, which are affecting manual branch handling too.

Summit POs are normal, fully valid POs in their own right. A message in a summit PO is different from branch PO only by being equipped with another comment, <tt>#. +> ...</tt>, showing in which branches the message exists:

<code po>
#. +> trunk
#: kdeui/jobs/kwidgetjobtracker.cpp:469
msgctxt "The destination url of a job"
msgid "Destination:"
msgstr ""
⁠
#. +> stable
#: kdeui/jobs/kwidgetjobtracker.cpp:469
msgid "Destination:"
msgstr ""
⁠
#. +> trunk stable
#: kdeui/jobs/kwidgetjobtracker.cpp:517
msgid "&Keep this window open after transfer is complete"
msgstr ""
</code>

The first message above thus exists in trunk only, the second in stable only, and the third in both branches. The source reference always points to the source file in the first listed branch. Any extracted comments (<tt>#.</tt>) other than the branch list are also taken from the first listed branch.

Note that the two messages above are different only by context; the context was added in trunk, but not in stable, in order not to break message freeze. However, due to careful ordering of messages in summit POs, these two messages appear together, allowing translator to immediately make correction in stable branch too if the new context in trunk shows it to be necessary.

=== Setting Up and Daily Operation ===

Before initializing language summit, the team coordinator has to have all the necessary paths checked out from the KDE repository, and structured on the local machine exactly as in the repository. If the path to the root of KDE repository on the local machine is <tt>$KDEREPO</tt>, and the language code <tt>LANG</tt>, then the structure should be as follows, with leaf directories checked out in full:

<code text>
$KDEREPO/
trunk/
l10n-kde4/
scripts/
templates/
LANG/
l10n-support/
scripts/
pology/
templates/
LANG/
branches/
stable/
l10n-kde4/
scripts/
templates/
LANG/
</code>

It is important that this structure is fully under version control (rather than e.g. top directories being manually created), so if that is not the case, the following commands will fetch everything anew in proper order:

<code bash>
cd $KDEREPO
svn co --depth=empty svn+ssh://svn.kde.org/home/kde .
svn up --depth=empty branches branches/stable branches/stable/l10n-kde4
svn up --depth=empty trunk trunk/l10n-support trunk/l10n-kde4
svn up branches/stable/l10n-kde4/{scripts,templates,LANG}
svn up trunk/l10n-kde4/{scripts,templates,LANG}
svn up trunk/l10n-support/{pology,scripts,templates,LANG}
</code>

Note the trailing dot in the first <tt>svn</tt> command, and do not miss to substitute LANG for the real language code in the last three commands. <tt>--depth=empty</tt> option prevents recursive fetching, so that e.g. the first <tt>svn</tt> command does not fetch the complete repository tree. Later you can regularly update this tree with single command:

<code bash>
svn up $KDEREPO/{trunk,branches/stable}/l10n-kde4/{scripts,templates,LANG} \
$KDEREPO/trunk/l10n-support/{pology,scripts,templates,LANG}
</code>

Summit operations are performed using the <tt>posummit</tt> command, which is part of [[Localization/Tools/Pology|Pology]], residing in {{path|trunk/l10n-support/pology/}}. Therefore the first thing to do is to setup Pology, which amounts only to setting the proper path:

<code bash>
$ export PATH=$KDEREPO/trunk/l10n-support/pology/bin:$PATH
</code>

To initialize the summit, by gathering from existing translation in branches, the team coordinator executes:

<code bash>
$ cd $KDEREPO/trunk/l10n-support
$ posummit scripts/messages.summit LANG gather --create --force
</code>

Depending on the amount of translation, after some minutes the initial gathering will have been completed, and language summit located under {{path|$KDEREPO/trunk/l10n-support/LANG/summit/messages/}}. This is the only time when the coordinator performs the gather operation on language POs; it is daily done only on templates by repository automation. Then, the created language summit should be merged with current summit templates:

<code bash>
$ posummit scripts/messages.summit LANG merge
</code>

Merging the summit is something that the coordinator does periodically, with frequency of own desire. For example, it can be done daily, or with increasing frequency as the last day for translation for the next release approaches.

After the first merging, language summit is ready for active translation. The coordinator should now commit {{path|$KDEREPO/trunk/l10n-support/LANG/}}, and, importantly, notify team members to stop working on branch POs and focus exclusively on summit POs.

To scatter the summit, i.e. fill out POs in stable and trunk branch from the summit POs, the coordinator periodically executes:

<code bash>
$ posummit scripts/messages.summit LANG scatter
</code>

As with merging, there is no fixed schedule when scattering should be done. Of course, it must necessarily be done before the next release is tagged, and in between it is useful to scatter for runtime testing, or to have translation statistics by branches on l10n.kde.org up to date.

Periodic scattering and merging of the complete summit are basically all that a language team coordinator needs to do specifically to operate the summit. Also, since {{path|l10n-support/scripts/}} and {{path|l10n-support/pology/}} contain scripts and settings critical for proper functioning of summit operations, and may be tweaked at any time, they should always be updated from the repository together with PO files and templates (in fact, it is best to always update at once the whole tree as outlined above).

{{note|For documentation POs, summit setup and operation is the same, only replacing every <tt>messages</tt> with <tt>docmessages</tt> in the command lines above. The user interface and documentation summits are fully independent, so for a trial period it is reasonable to work with the interface summit only, and engage documentation summit once the trial has been deemed successfull.}}

=== Operation Targets ===

Sometimes it is advantageous to merge or scatter just a single catalog, a single module, a single branch, or any combination thereof. To this end, scatter and merge operations accept any number of ''operation targets'' after the operation keyword, specified as one of <tt>''CATALOG''</tt>, <tt>''BRANCH'':''CATALOG''</tt>, <tt>''MODULE''/</tt>, <tt>''BRANCH'':''MODULE''/</tt>, and <tt>''BRANCH'':</tt>. For example, to scatter just to Dolphin's PO in stable branch, in order to test translation at runtime, one would execute:

<code bash>
$ cd $KDEREPO/trunk/l10n-support
$ posummit scripts/messages.summit LANG scatter stable:dolphin
</code>

(note no <tt>.po</tt> ending on catalog name). Or, to scatter to every PO in <tt>kdeplasma-addons</tt> module in stable branch:

<code bash>
$ cd $KDEREPO/trunk/l10n-support
$ posummit scripts/messages.summit LANG scatter stable:kdeplasma-addons/
</code>

(the trailing slash is mandatory, or else <tt>posummit</tt> would think that <tt>kdeplasma-addons</tt> is a catalog name). Finally, to scatter to all catalogs in the stable branch (with the trailing colon for the same reason as earlier):

<code bash>
$ cd $KDEREPO/trunk/l10n-support
$ posummit scripts/messages.summit LANG scatter stable:
</code>

The command line arguments of <tt>posummit</tt> operations are intentionally arranged such that all the arguments fixed for one language are placed first. If operation targets are used frequently, then it is convenient to place the immutable part of the command line under a shell alias, with absolute path to the summit setup file:

<code bash>
$ alias posummit-kde-LANG="posummit $KDEREPO/trunk/l10n-support/scripts/messages.summit LANG"
$ cd ANYWHERE
$ posummit-kde-LANG scatter stable:dolphin
</code>

=== Externally Injected Branch Catalogs ===

Once the summit is in operation, normally the only way for a catalog to appear in branches (trunk or stable) is through scattering from the summit. However, sometimes an application which had seen considerable development outside of the infrastructure of KDE project gets moved in, drawing in any of its existing translations into branches. Such externally injected branch catalogs cause warnings to be issued on summit merging, and outright failure on scattering:

<code bash>
$ cd $KDEREPO/trunk/l10n-support
$ touch ../l10n-kde4/LANG/messages/kdetoys/foobar.po # dummy injection
$ touch ../l10n-kde4/LANG/messages/kdetoys/libfoobar.po # dummy injection
$
$ posummit scripts/messages.summit LANG merge
...
posummit: warning: No summit catalog for branch catalog
'../l10n-kde4/LANG/messages/kdetoys/foobar.po'.
posummit: warning: No summit catalog for branch catalog
'../l10n-kde4/LANG/messages/kdetoys/libfoobar.po'.
...
$
$ posummit scripts/messages.summit LANG scatter
...
posummit: error: Missing necessary summit catalogs: foobar libfoobar
$
</code>

When this happens, the injected catalogs need to be gathered into the summit. This is done just like the initial gathering, only giving injected catalogs' names as operation targets:

<code bash>
$ posummit scripts/messages.summit LANG gather --create --force foobar libfoobar
</code>

=== Summit Customization ===

File {{path|scripts/messages.summit}} (i.e. {{path|$KDEREPO/trunk/l10n-support/scripts/messages.summit}}), given as first argument to <tt>posummit</tt>, contains the general summit setup for all languages. This file is set to include customization file per language, if it exists at {{path|$KDEREPO/trunk/l10n-support/LANG/summit/messages.extras.summit}}, which contains additions and overrides to the general setup as desired by the language team. Same as the general setup file, the customization file is a Python source. It uses an external object named <tt>S</tt> to define summit settings, and can, of course, contain any helper Python code (the file is executed only once, at the beginning of a <tt>posummit</tt> run). It has the following general layout:

<code python>
# -*- coding: UTF-8 -*-
# kate: syntax Python;
# This file is included by scripts/messages.summit
# for language-specific additions/overrides.
#
# ...
# ... operations with object S and other Python code ...
# ...
</code>

Object <tt>S</tt> provides both data attributes and methods to configure different aspects of summit operation. For example, while the general setup specifies that text fields in summit catalogs should not be wrapped on column but should be wrapped on logical breaks (like some markup tags), this can be overriden using <tt>summit_wrap</tt> and <tt>summit_fine_wrap</tt> attributes:

<code python>
S.summit_wrap = True
S.summit_fine_wrap = False
</code>

Or, to get the path to a file relative to the summit customization file itself (rather than to current working directory, where <tt>posummit</tt> was executed), method <tt>resolve_path_rooted</tt> can be used:

<code python>
relpath = S.resolve_path_rooted("../whatever.txt")
</code>

The following sections will present some typical customization possibilities.

=== Fully Local Merging ===

When scattering from the summit, sometimes there will be reports of "messages missing in the summit". This happens because of time rift created by the Scripty merging branch POs, gathering summit templates, and a team coordinator merging the language summit, thus making some messages in branch POs not always present in the summit. This condition is benign, as such warnings will start to disappear with the message freeze approaching, but can be annoying. For this reason, team coordinator can stop Scripty from merging branch POs, and have the <tt>posummit ... merge</tt> command alone merge not only summit POs, but stable and trunk POs as well, such that summit and branches are always in perfect sync.

First, to stop Scripty from merging branch POs, a file named <tt>no-auto-merge</tt> (with arbitrary content) should be committed to the roots of respective trees, e.g.:

<code text>
$ touch $KDEREPO/trunk/l10n-kde4/messages/LANG/no-auto-merge
$ touch $KDEREPO/branches/stable/l10n-kde4/LANG/messages/no-auto-merge
</code>

Then, to make summit <tt>posummit ... merge</tt> merge everything, the following lines should be added into the summit customization file:

<code python>
# Set local merging for all branches.
for branch in S.branches:
branch["merge_locally"] = True
</code>

Once local merging of all branches is set, the coordinator can also use operation targets for selective merging, e.g. to merge only stable branch:

<code bash>
$ cd $KDEREPO/trunk/l10n-support
$ posummit scripts/messages.summit LANG merge stable:
</code>

=== Merging with Compendium ===

By default, <tt>msgmerge</tt> takes into account only the catalog itself to fill out near-match translations to new messages introduced by merging. i.e. to produce fuzzy messages. However, an arbitrary PO file can be given to <tt>msgmerge</tt> as another possible source of earlier translations, through the <tt>--compendium</tt> option. For maximum effect, this PO file is usually constructed as the collection of messages from all PO files project-wide, and hence called the ''compendium''.

Compendium can be used in summit merge operations simply by setting <tt>compendium_on_merge</tt> attribute in summit customization file. If the compendium is located at {{path|$KDEREPO/trunk/l10n-support/LANG/summit/messages-compendium.po}}, then:

<code python>
# Compendium to use when merging summit catalogs.
S.compendium_on_merge = S.resolve_path_rooted("messages-compendium.po")
</code>

Note that if not located as above, compendium should not be placed within {{path|.../LANG/summit/messages/}}, as then it would be considered a summit catalog itself. In case fully local merging is engaged, only summit POs will be merged with compendium; it makes no sense for branch POs, since they are not being directly translated.

See [http://www.gnu.org/software/automake/manual/gettext/Creating-Compendia.html "Creating Compendia"] section of Gettext manual for details on how to create a compendium out of current body of summit translations. Once you establish the precise commands to create the compendium (whether to collect fuzzies too, whether to include old compendium as one of sources for the new, etc.), you would periodically refresh and commit the updated compendium.

=== Vivifying Summit Catalogs ===

Translation of a summit catalog normally starts the same way as it did for branch catalogs, by copying it over from template and initializing the header. This can be done manually, but it can also be quite automatic when using project manager as sometimes provided by dedicated PO editors. However, relying on the editor's project manager can be disadvantageous at times. For example, aside from the PO editor, other, frequently command line tools may be used to process the body of translation, and these tools might need to consider non-started templates as if they were empty POs (e.g. statistics). Team members who do not have full local checkout, or no project set up in the editor, may need to jump between language and template directories when looking for files to translate.

Therefore, summit setup can be customized to automatically create ("vivify") summit catalogs for every new summit template, so that there is never the need (for a tool or human) to specifically treat templates as empty catalogs. This is done by adding the following lines into customization file:

<code python>
# Create empty summit catalog for every new summit template.
S.vivify_on_merge = True
S.vivify_w_translator = "Noone Noonian <noone.noonian@nowhere.null>"
S.vivify_w_langteam = "Neverneese <neverteam@nowhere.null>"
S.vivify_w_plurals = "nplurals=2; plural=n != 1;"
# Minimum translation state to create branch catalog on scatter.
S.scatter_min_completeness = 0.9
</code>

The <tt>vivify_w_*</tt> attributes set the necessary data to initialize headers of newly created summit catalogs. Aside from those listed above, there is also the <tt>vivify_w_charset</tt> attribute, which is by default <tt>"UTF-8"</tt>, and very probably should not be changed.

The <tt>scatter_min_completeness</tt> attribute does not refer to vivification of summit catalogs, but should invariably be set in this context. When scattering from summit, normally the branch catalog is automatically created if there is the corresponding summit catalog. When vivification is engaged, this would result in creation of empty branch catalogs too, which is not desired for two reasons. Firstly, empty branch catalogs would become part of the released language pack, for which there is no practical reason. Secondly, KDE's i18n system regards an application with installed catalog as translated, so messages coming from basic system catalogs (e.g. <tt>kdelibs4</tt>) would show through, resulting in mostly untranslated user interface with specks of translation -- many users consider this rather ungainly. Therefore, <tt>scatter_min_completeness</tt> sets how complete the translation of currently non-existing branch catalog should be after scatter (0.0 empty, 1.0 fully complete), for the branch catalog to actually be created. If the <tt>--force</tt> option is given to <tt>posummit</tt>, the branch catalog will be created regardless of its translation state, which may be handy in combination with [[#Operation Targets|operation target]] when wanting to check work in progress in running application.

If [[#Merging with Compendium|the compendium has been set]] for merging, every vivified summit catalog will also be merged against the compendium, to fill out as many of messages with approximate translations.

=== Scatter Hooks ===

((TODO. Checks, modifications...))

=== Merge Hooks ===

((TODO. Special header fields...))

== Disadvantages to Summit and Remedies ==

Although hopefully shadowed by the advantages, working in summit is not without its disadvantages. These should be weighed when deciding of whether to try out the summit workflow.

Obviously, while summit operations are made to be quite automatic, some extra aptitude is asked of the team coordinator. Reasonable shell handling, understanding of version control operations, feeling the pulse of repository automation, are all prerequisites, and some scripting ability advantageous.

After the summit is put in operation, any changes made manually in branch POs will not propagate to summit, and will be soon lost to scattering -- summit translations override everything in branches. This means that the whole team must work in the summit, it is not possible for some members to use the summit, and some not.

A summit PO file will necessarily have more messages than either of the branch files. For example, in the KDE 4.0/4.1 and 4.1/4.2 cycle, summit POs of core KDE modules had on average less than 5% more words than their stable counterparts. However, the said percent is the top, never approached limit of wasted workload due to trunk messages coming and going, given that as the next feature KDE release approaches, more and more trunk messages will find their way into it.

Another, more pressing issue with increased size of summit POs is the following scenario: a stable release is around the corner, and the team has no time to update summit POs fully, but could update only stable messages in them. E.g. there are 1000 untranslated and fuzzy messages, out of which only 100 are from the stable branch. A clever dedicated PO editor could allow jumping only through untranslated and fuzzy messages which also satisfy a general search criteria, which in this case would be that a comment matches <tt>#\.\+>.*stable</tt> regular expression. On the other hand, with some external help, it is enough if the PO editor can merely search through comments. Then, the <tt>posieve</tt> command (ready to use next to <tt>posummit</tt>) can equip untranslated and fuzzy stable messages with <tt>untranslated</tt> flag (producing <tt>#, ..., untranslated</tt> comment), and this flag can be searched for in the PO editor:

<code bash>
$ posieve tag-untranslated -sbranch:stable -swfuzzy PATHS_TO_PO_FILES_OR_DIRS
</code>

The <tt>untranslated</tt> flag does not need to be manually removed when the message is updated. It will automatically disappear on the next merge, as it is not among flags known to Gettext.

There is also the organizational issue with starting to use the summit, and, if it does not help as expected, stopping to use it. Team members have to be reminded to not send in branch POs at start, and then to be sent back to branch POs if summit is disbanded. On the plus side, disbanding summit is technically simple: just remove from the repository {{path|l10n-supprot/LANG/summit}}, possibly also <tt>no-auto-merge</tt> files if [[#Fully Local Merging|fully local merging]] was set up, and that is it.

== Another Way to Improve Branch Handling ==

If summit seems a lot to digest, or is simply an overkill for team's needs, but still some improvement to manual handling of branches would be welcomed, KDE's dedicated PO editor [[Localization/Tools/Lokalize|Lokalize]] offers a ''branch sync'' mode. It works as follows.

In Lokalize project definition, the local paths of trunk and stable PO roots are set in ''Translation directory:'' and ''Branch directory:'' fields. Then, when a trunk PO file is opened, if it has a stable counterpart with same name and location as in the trunk, this stable PO is also going to be opened. For each trunk message in the main editing pane, if such a message exists in stable PO too, the stable message will be shown in the ''Secondary Sync'' pane; changes in the translation of trunk message will reflect to the stable message, and stable PO file will also be saved when the trunk is saved.

Furthermore, any team member can personally choose to work like this, there is no need to change the workflow of the language team as whole. When sending modifications to the coordinator, team members who rely on this feature of Lokalize simply send both trunk and stable POs that got modified.

Localization/Workflows/PO Summit

2010-01-19T14:33:37Z

Ilic: Added command for upating the repository tree.

{{Template:I18n/Language Navigation Bar|Localization/Workflows/PO_Summit}}

{{LocalizationBrowser|
section=[[Localization#Workflows|Workflows]]|
name=Translating in Summit|
prereqs=[[Localization/Workflows/Coordinator|Language Coordinator]]|
related=[[Localization/Tools/Pology|Pology]]|
}}

== Software Branches and Translation ==

The obvious approach to releasing software is for developers to focus on one central body of code, a single "branch", fixing bugs and adding new features to it, and from time to time taking a "snapshot" of the branch and packing it as next release. One approach to make the release process more robust, is for development to proceed in two parallel branches. From the "stable" branch the actual releases are made, mostly with bugs fixed and only very important features added. The other, "unstable" branch, is used to develop new and redesign existing features, and at some point will become the next stable branch. Depending on the project, releases may be made from the unstable branch as well, in parallel with stable releases, in order for eager users to help testing the novelties.

In KDE, all three release models may be present at any given moment. Core KDE modules, which are together labeled as "the KDE" and released in unison, follow the two branch model, with stable and trunk branch, and releases from stable branch only. KDE extragear applications may do the same, but they are not required to; they may instead make releases from the trunk branch only, or also use stable and trunk branch, but make releases from both. This presents translators with the workflow as on the following figure.

[[Image:Translating_in_branches.png|center|Classic translation by branches]]

The KDE [[Localization/Workflows/Repository_Automation|repository automation]] (aka "Scripty") is preparing template POs and merging them with language POs. From the point of view of language teams, real people who perform global actions (e.g. moving around POs for all languages when an application moves from module to another) can also be grouped here. However, the translation team is still presented with two branches of POs to translate.

In general, this means that, just like programmers, at times translators have to work on both branches, and propagate fixes from one branch to another ("backport" from trunk to stable, or "forwardport" from stable to trunk). This may be prone to coordinatorial confusion, and demand extra effort and attention when updating translations. Keeping up with two branches is easier in core KDE modules, as they are released from stable branch only and with singular schedule, but could be more taxing with extragear applications.

The exact amount of effort spent due to parallel translation, porting of fixes, and coordination, depends on the translation team. For example, a well-manned and well-coordinated team, with established style and terminology guides, custom automation to support these, may be able to keep all POs in both branches fully translated at all times with ease. Or, a team may have worked out well-defined schedules, such that any given member of the team at one point switches from translating one branch to another, without looking back, thus effectively having everyone always working on one branch (even if not the same one).

If, however, you as the team coordinator have said and thought "Should do that in trunk too, bugger", "Didn't we fix that already?", "No, you should take it from stable", "It was released from WHAT branch?", more often than you would have liked, the following presents one possibility for sidestepping such issues.

== Translating in Summit ==

For a PO catalog which exists in both stable and trunk branch, a new same-named catalog can be made by ''gathering'' all the unique messages from branch PO files, i.e. the ''summit'' of branch POs. Assuming that branch POs were not all that different, since they are two versions of the same catalog, the summit PO shouldn't have much more messages than either. Translators at all times work on the summit PO, from which the messages are periodically ''scattered'' back to original branch POs. Thus, translators can always work on summit POs, not having to do any parallel translation, branch switching or porting of fixes. The summit workflow is presented by the following figure.

[[Image:Translating_in_summit.png|center|Translation in summit]]

In the summit mode, repository automation gathers summit PO templates from branch templates, and stores them separately from real branches, at {{path|trunk/l10n-support/templates/summit/}}. The language team also has a collection summit POs, at {{path|trunk/l10n-support/LANG/summit/}}, which is the sole location where translation happens. As before, repository automation handles merging of templates in branches, but merging of summit POs is done by the team coordinator; team coordinator can opt to manually merge branch POs as well (more on why-and-how of this later). From time to time, the team coordinator fills out branch POs by scattering from summit. Not to worry, each of these special actions upon the team coordinator is done with a single command.

While it is in principle obvious that two branch POs can be made into one summit PO with the union of their messages, there are some important details that should be handled the right way by the summitting system:

* What if a PO changes modules in one branch, so that it no longer belongs to the same module in both branches (e.g. application is moved from one to another module in trunk)?

* What if a PO changes its name in one branch, but not in the other (e.g. application is renamed in trunk)?

* What if a PO in one branch is split into two POs in another (e.g. extracting a library out of monolithic application in trunk)?

* Where to place messages unique to one branch in the summit PO? The original file context of a message, which messages precede it and which follow it, should be kept as much as possible.

* If source references are used to achieve good ordering of messages in summit PO, what to do if some source file paths change in one branch (e.g. application gets restructured in trunk)?

* How to handle messages with different plurality across branches (since messages are identified only by their <tt>msgctxt</tt> and <tt>msgid</tt> field, an not <tt>msgid_plural</tt>)?

And handle these details it does, the present summitting system. This also means that teams working in the summit need not take care of the first three issues above, which are affecting manual branch handling too.

Summit POs are normal, fully valid POs in their own right. A message in a summit PO is different from branch PO only by being equipped with another comment, <tt>#. +> ...</tt>, showing in which branches the message exists:

<code po>
#. +> trunk
#: kdeui/jobs/kwidgetjobtracker.cpp:469
msgctxt "The destination url of a job"
msgid "Destination:"
msgstr ""
⁠
#. +> stable
#: kdeui/jobs/kwidgetjobtracker.cpp:469
msgid "Destination:"
msgstr ""
⁠
#. +> trunk stable
#: kdeui/jobs/kwidgetjobtracker.cpp:517
msgid "&Keep this window open after transfer is complete"
msgstr ""
</code>

The first message above thus exists in trunk only, the second in stable only, and the third in both branches. The source reference always points to the source file in the first listed branch. Any extracted comments (<tt>#.</tt>) other than the branch list are also taken from the first listed branch.

Note that the two messages above are different only by context; the context was added in trunk, but not in stable, in order not to break message freeze. However, due to careful ordering of messages in summit POs, these two messages appear together, allowing translator to immediately make correction in stable branch too if the new context in trunk shows it to be necessary.

=== Setting Up and Daily Operation ===

Before initializing language summit, the team coordinator has to have all the necessary paths checked out from the KDE repository, and structured on the local machine exactly as in the repository. If the path to the root of KDE repository on the local machine is <tt>$KDEREPO</tt>, and the language code <tt>LANG</tt>, then the structure should be as follows, with leaf directories checked out in full:

<code text>
$KDEREPO/
trunk/
l10n-kde4/
scripts/
templates/
LANG/
l10n-support/
scripts/
pology/
templates/
LANG/
branches/
stable/
l10n-kde4/
scripts/
templates/
LANG/
</code>

It is important that this structure is fully under version control (rather than e.g. top directories being manually created), so if that is not the case, the following commands will fetch everything anew in proper order:

<code bash>
cd $KDEREPO
svn co --depth=empty svn+ssh://svn.kde.org/home/kde .
svn up --depth=empty branches branches/stable branches/stable/l10n-kde4
svn up --depth=empty trunk trunk/l10n-support trunk/l10n-kde4
svn up branches/stable/l10n-kde4/{scripts,templates,LANG}
svn up trunk/l10n-kde4/{scripts,templates,LANG}
svn up trunk/l10n-support/{pology,scripts,templates,LANG}
</code>

Note the trailing dot in the first <tt>svn</tt> command, and do not miss to substitute LANG for the real language code in the last three commands. <tt>--depth=empty</tt> option prevents recursive fetching, so that e.g. the first <tt>svn</tt> command does not fetch the complete repository tree. Later you can regularly update this tree with single command:

<code bash>
svn up $KDEREPO/{trunk,branches/stable}/l10n-kde4/{scripts,templates,cs} \
$KDEREPO/trunk/l10n-support/{pology,scripts,templates,cs}
</code>

Summit operations are performed using the <tt>posummit</tt> command, which is part of [[Localization/Tools/Pology|Pology]], residing in {{path|trunk/l10n-support/pology/}}. Therefore the first thing to do is to setup Pology, which amounts only to setting the proper path:

<code bash>
$ export PATH=$KDEREPO/trunk/l10n-support/pology/bin:$PATH
</code>

To initialize the summit, by gathering from existing translation in branches, the team coordinator executes:

<code bash>
$ cd $KDEREPO/trunk/l10n-support
$ posummit scripts/messages.summit LANG gather --create --force
</code>

Depending on the amount of translation, after some minutes the initial gathering will have been completed, and language summit located under {{path|$KDEREPO/trunk/l10n-support/LANG/summit/messages/}}. This is the only time when the coordinator performs the gather operation on language POs; it is daily done only on templates by repository automation. Then, the created language summit should be merged with current summit templates:

<code bash>
$ posummit scripts/messages.summit LANG merge
</code>

Merging the summit is something that the coordinator does periodically, with frequency of own desire. For example, it can be done daily, or with increasing frequency as the last day for translation for the next release approaches.

After the first merging, language summit is ready for active translation. The coordinator should now commit {{path|$KDEREPO/trunk/l10n-support/LANG/}}, and, importantly, notify team members to stop working on branch POs and focus exclusively on summit POs.

To scatter the summit, i.e. fill out POs in stable and trunk branch from the summit POs, the coordinator periodically executes:

<code bash>
$ posummit scripts/messages.summit LANG scatter
</code>

As with merging, there is no fixed schedule when scattering should be done. Of course, it must necessarily be done before the next release is tagged, and in between it is useful to scatter for runtime testing, or to have translation statistics by branches on l10n.kde.org up to date.

Periodic scattering and merging of the complete summit are basically all that a language team coordinator needs to do specifically to operate the summit. Also, since {{path|l10n-support/scripts/}} and {{path|l10n-support/pology/}} contain scripts and settings critical for proper functioning of summit operations, and may be tweaked at any time, they should always be updated from the repository together with PO files and templates (in fact, it is best to always update at once the whole tree as outlined above).

{{note|For documentation POs, summit setup and operation is the same, only replacing every <tt>messages</tt> with <tt>docmessages</tt> in the command lines above. The user interface and documentation summits are fully independent, so for a trial period it is reasonable to work with the interface summit only, and engage documentation summit once the trial has been deemed successfull.}}

=== Operation Targets ===

Sometimes it is advantageous to merge or scatter just a single catalog, a single module, a single branch, or any combination thereof. To this end, scatter and merge operations accept any number of ''operation targets'' after the operation keyword, specified as one of <tt>''CATALOG''</tt>, <tt>''BRANCH'':''CATALOG''</tt>, <tt>''MODULE''/</tt>, <tt>''BRANCH'':''MODULE''/</tt>, and <tt>''BRANCH'':</tt>. For example, to scatter just to Dolphin's PO in stable branch, in order to test translation at runtime, one would execute:

<code bash>
$ cd $KDEREPO/trunk/l10n-support
$ posummit scripts/messages.summit LANG scatter stable:dolphin
</code>

(note no <tt>.po</tt> ending on catalog name). Or, to scatter to every PO in <tt>kdeplasma-addons</tt> module in stable branch:

<code bash>
$ cd $KDEREPO/trunk/l10n-support
$ posummit scripts/messages.summit LANG scatter stable:kdeplasma-addons/
</code>

(the trailing slash is mandatory, or else <tt>posummit</tt> would think that <tt>kdeplasma-addons</tt> is a catalog name). Finally, to scatter to all catalogs in the stable branch (with the trailing colon for the same reason as earlier):

<code bash>
$ cd $KDEREPO/trunk/l10n-support
$ posummit scripts/messages.summit LANG scatter stable:
</code>

The command line arguments of <tt>posummit</tt> operations are intentionally arranged such that all the arguments fixed for one language are placed first. If operation targets are used frequently, then it is convenient to place the immutable part of the command line under a shell alias, with absolute path to the summit setup file:

<code bash>
$ alias posummit-kde-LANG="posummit $KDEREPO/trunk/l10n-support/scripts/messages.summit LANG"
$ cd ANYWHERE
$ posummit-kde-LANG scatter stable:dolphin
</code>

=== Externally Injected Branch Catalogs ===

Once the summit is in operation, normally the only way for a catalog to appear in branches (trunk or stable) is through scattering from the summit. However, sometimes an application which had seen considerable development outside of the infrastructure of KDE project gets moved in, drawing in any of its existing translations into branches. Such externally injected branch catalogs cause warnings to be issued on summit merging, and outright failure on scattering:

<code bash>
$ cd $KDEREPO/trunk/l10n-support
$ touch ../l10n-kde4/LANG/messages/kdetoys/foobar.po # dummy injection
$ touch ../l10n-kde4/LANG/messages/kdetoys/libfoobar.po # dummy injection
$
$ posummit scripts/messages.summit LANG merge
...
posummit: warning: No summit catalog for branch catalog
'../l10n-kde4/LANG/messages/kdetoys/foobar.po'.
posummit: warning: No summit catalog for branch catalog
'../l10n-kde4/LANG/messages/kdetoys/libfoobar.po'.
...
$
$ posummit scripts/messages.summit LANG scatter
...
posummit: error: Missing necessary summit catalogs: foobar libfoobar
$
</code>

When this happens, the injected catalogs need to be gathered into the summit. This is done just like the initial gathering, only giving injected catalogs' names as operation targets:

<code bash>
$ posummit scripts/messages.summit LANG gather --create --force foobar libfoobar
</code>

=== Summit Customization ===

File {{path|scripts/messages.summit}} (i.e. {{path|$KDEREPO/trunk/l10n-support/scripts/messages.summit}}), given as first argument to <tt>posummit</tt>, contains the general summit setup for all languages. This file is set to include customization file per language, if it exists at {{path|$KDEREPO/trunk/l10n-support/LANG/summit/messages.extras.summit}}, which contains additions and overrides to the general setup as desired by the language team. Same as the general setup file, the customization file is a Python source. It uses an external object named <tt>S</tt> to define summit settings, and can, of course, contain any helper Python code (the file is executed only once, at the beginning of a <tt>posummit</tt> run). It has the following general layout:

<code python>
# -*- coding: UTF-8 -*-
# kate: syntax Python;
# This file is included by scripts/messages.summit
# for language-specific additions/overrides.
#
# ...
# ... operations with object S and other Python code ...
# ...
</code>

Object <tt>S</tt> provides both data attributes and methods to configure different aspects of summit operation. For example, while the general setup specifies that text fields in summit catalogs should not be wrapped on column but should be wrapped on logical breaks (like some markup tags), this can be overriden using <tt>summit_wrap</tt> and <tt>summit_fine_wrap</tt> attributes:

<code python>
S.summit_wrap = True
S.summit_fine_wrap = False
</code>

Or, to get the path to a file relative to the summit customization file itself (rather than to current working directory, where <tt>posummit</tt> was executed), method <tt>resolve_path_rooted</tt> can be used:

<code python>
relpath = S.resolve_path_rooted("../whatever.txt")
</code>

The following sections will present some typical customization possibilities.

=== Fully Local Merging ===

When scattering from the summit, sometimes there will be reports of "messages missing in the summit". This happens because of time rift created by the Scripty merging branch POs, gathering summit templates, and a team coordinator merging the language summit, thus making some messages in branch POs not always present in the summit. This condition is benign, as such warnings will start to disappear with the message freeze approaching, but can be annoying. For this reason, team coordinator can stop Scripty from merging branch POs, and have the <tt>posummit ... merge</tt> command alone merge not only summit POs, but stable and trunk POs as well, such that summit and branches are always in perfect sync.

First, to stop Scripty from merging branch POs, a file named <tt>no-auto-merge</tt> (with arbitrary content) should be committed to the roots of respective trees, e.g.:

<code text>
$ touch $KDEREPO/trunk/l10n-kde4/messages/LANG/no-auto-merge
$ touch $KDEREPO/branches/stable/l10n-kde4/LANG/messages/no-auto-merge
</code>

Then, to make summit <tt>posummit ... merge</tt> merge everything, the following lines should be added into the summit customization file:

<code python>
# Set local merging for all branches.
for branch in S.branches:
branch["merge_locally"] = True
</code>

Once local merging of all branches is set, the coordinator can also use operation targets for selective merging, e.g. to merge only stable branch:

<code bash>
$ cd $KDEREPO/trunk/l10n-support
$ posummit scripts/messages.summit LANG merge stable:
</code>

=== Merging with Compendium ===

By default, <tt>msgmerge</tt> takes into account only the catalog itself to fill out near-match translations to new messages introduced by merging. i.e. to produce fuzzy messages. However, an arbitrary PO file can be given to <tt>msgmerge</tt> as another possible source of earlier translations, through the <tt>--compendium</tt> option. For maximum effect, this PO file is usually constructed as the collection of messages from all PO files project-wide, and hence called the ''compendium''.

Compendium can be used in summit merge operations simply by setting <tt>compendium_on_merge</tt> attribute in summit customization file. If the compendium is located at {{path|$KDEREPO/trunk/l10n-support/LANG/summit/messages-compendium.po}}, then:

<code python>
# Compendium to use when merging summit catalogs.
S.compendium_on_merge = S.resolve_path_rooted("messages-compendium.po")
</code>

Note that if not located as above, compendium should not be placed within {{path|.../LANG/summit/messages/}}, as then it would be considered a summit catalog itself. In case fully local merging is engaged, only summit POs will be merged with compendium; it makes no sense for branch POs, since they are not being directly translated.

See [http://www.gnu.org/software/automake/manual/gettext/Creating-Compendia.html "Creating Compendia"] section of Gettext manual for details on how to create a compendium out of current body of summit translations. Once you establish the precise commands to create the compendium (whether to collect fuzzies too, whether to include old compendium as one of sources for the new, etc.), you would periodically refresh and commit the updated compendium.

=== Vivifying Summit Catalogs ===

Translation of a summit catalog normally starts the same way as it did for branch catalogs, by copying it over from template and initializing the header. This can be done manually, but it can also be quite automatic when using project manager as sometimes provided by dedicated PO editors. However, relying on the editor's project manager can be disadvantageous at times. For example, aside from the PO editor, other, frequently command line tools may be used to process the body of translation, and these tools might need to consider non-started templates as if they were empty POs (e.g. statistics). Team members who do not have full local checkout, or no project set up in the editor, may need to jump between language and template directories when looking for files to translate.

Therefore, summit setup can be customized to automatically create ("vivify") summit catalogs for every new summit template, so that there is never the need (for a tool or human) to specifically treat templates as empty catalogs. This is done by adding the following lines into customization file:

<code python>
# Create empty summit catalog for every new summit template.
S.vivify_on_merge = True
S.vivify_w_translator = "Noone Noonian <noone.noonian@nowhere.null>"
S.vivify_w_langteam = "Neverneese <neverteam@nowhere.null>"
S.vivify_w_plurals = "nplurals=2; plural=n != 1;"
# Minimum translation state to create branch catalog on scatter.
S.scatter_min_completeness = 0.9
</code>

The <tt>vivify_w_*</tt> attributes set the necessary data to initialize headers of newly created summit catalogs. Aside from those listed above, there is also the <tt>vivify_w_charset</tt> attribute, which is by default <tt>"UTF-8"</tt>, and very probably should not be changed.

The <tt>scatter_min_completeness</tt> attribute does not refer to vivification of summit catalogs, but should invariably be set in this context. When scattering from summit, normally the branch catalog is automatically created if there is the corresponding summit catalog. When vivification is engaged, this would result in creation of empty branch catalogs too, which is not desired for two reasons. Firstly, empty branch catalogs would become part of the released language pack, for which there is no practical reason. Secondly, KDE's i18n system regards an application with installed catalog as translated, so messages coming from basic system catalogs (e.g. <tt>kdelibs4</tt>) would show through, resulting in mostly untranslated user interface with specks of translation -- many users consider this rather ungainly. Therefore, <tt>scatter_min_completeness</tt> sets how complete the translation of currently non-existing branch catalog should be after scatter (0.0 empty, 1.0 fully complete), for the branch catalog to actually be created. If the <tt>--force</tt> option is given to <tt>posummit</tt>, the branch catalog will be created regardless of its translation state, which may be handy in combination with [[#Operation Targets|operation target]] when wanting to check work in progress in running application.

If [[#Merging with Compendium|the compendium has been set]] for merging, every vivified summit catalog will also be merged against the compendium, to fill out as many of messages with approximate translations.

=== Scatter Hooks ===

((TODO. Checks, modifications...))

=== Merge Hooks ===

((TODO. Special header fields...))

== Disadvantages to Summit and Remedies ==

Although hopefully shadowed by the advantages, working in summit is not without its disadvantages. These should be weighed when deciding of whether to try out the summit workflow.

Obviously, while summit operations are made to be quite automatic, some extra aptitude is asked of the team coordinator. Reasonable shell handling, understanding of version control operations, feeling the pulse of repository automation, are all prerequisites, and some scripting ability advantageous.

After the summit is put in operation, any changes made manually in branch POs will not propagate to summit, and will be soon lost to scattering -- summit translations override everything in branches. This means that the whole team must work in the summit, it is not possible for some members to use the summit, and some not.

A summit PO file will necessarily have more messages than either of the branch files. For example, in the KDE 4.0/4.1 and 4.1/4.2 cycle, summit POs of core KDE modules had on average less than 5% more words than their stable counterparts. However, the said percent is the top, never approached limit of wasted workload due to trunk messages coming and going, given that as the next feature KDE release approaches, more and more trunk messages will find their way into it.

Another, more pressing issue with increased size of summit POs is the following scenario: a stable release is around the corner, and the team has no time to update summit POs fully, but could update only stable messages in them. E.g. there are 1000 untranslated and fuzzy messages, out of which only 100 are from the stable branch. A clever dedicated PO editor could allow jumping only through untranslated and fuzzy messages which also satisfy a general search criteria, which in this case would be that a comment matches <tt>#\.\+>.*stable</tt> regular expression. On the other hand, with some external help, it is enough if the PO editor can merely search through comments. Then, the <tt>posieve</tt> command (ready to use next to <tt>posummit</tt>) can equip untranslated and fuzzy stable messages with <tt>untranslated</tt> flag (producing <tt>#, ..., untranslated</tt> comment), and this flag can be searched for in the PO editor:

<code bash>
$ posieve tag-untranslated -sbranch:stable -swfuzzy PATHS_TO_PO_FILES_OR_DIRS
</code>

The <tt>untranslated</tt> flag does not need to be manually removed when the message is updated. It will automatically disappear on the next merge, as it is not among flags known to Gettext.

There is also the organizational issue with starting to use the summit, and, if it does not help as expected, stopping to use it. Team members have to be reminded to not send in branch POs at start, and then to be sent back to branch POs if summit is disbanded. On the plus side, disbanding summit is technically simple: just remove from the repository {{path|l10n-supprot/LANG/summit}}, possibly also <tt>no-auto-merge</tt> files if [[#Fully Local Merging|fully local merging]] was set up, and that is it.

== Another Way to Improve Branch Handling ==

If summit seems a lot to digest, or is simply an overkill for team's needs, but still some improvement to manual handling of branches would be welcomed, KDE's dedicated PO editor [[Localization/Tools/Lokalize|Lokalize]] offers a ''branch sync'' mode. It works as follows.

In Lokalize project definition, the local paths of trunk and stable PO roots are set in ''Translation directory:'' and ''Branch directory:'' fields. Then, when a trunk PO file is opened, if it has a stable counterpart with same name and location as in the trunk, this stable PO is also going to be opened. For each trunk message in the main editing pane, if such a message exists in stable PO too, the stable message will be shown in the ''Secondary Sync'' pane; changes in the translation of trunk message will reflect to the stable message, and stable PO file will also be saved when the trunk is saved.

Furthermore, any team member can personally choose to work like this, there is no need to change the workflow of the language team as whole. When sending modifications to the coordinator, team members who rely on this feature of Lokalize simply send both trunk and stable POs that got modified.

Localization/Workflows/PO Ascription

2010-01-13T23:34:05Z

Ilic: -O option renamed to -o.

{{Template:I18n/Language Navigation Bar|Localization/Workflows/PO_Summit}}

{{LocalizationBrowser|
section=[[Localization#Workflows|Workflows]]|
name=Reviewing by Ascriptions|
prereqs=[[Localization/Workflows/PO_Summit|Translating in Summit]]|
related=[[Localization/Tools/Pology|Pology]], [[Localization/Workflows/Coordinator|Language Coordinator]]|
}}

== Why Review Translations? ==

Especially to new translators, it may not be obvious to which extent the translation needs to be reviewed. If the translator has exercised due diligence, how "wrong" can the translation be? Even if the translator has good command of the source language (English in context of this article), the answer is "very wrong", when all aspects are considered. Here are some of them.

With comparatively simple grammar of English, the meaning of a short English sentence -- as typically encountered in application user interfaces -- is very dependent on the surrounding context. This context may not be obvious when the translator is going through isolated messages in the translation file, so he may commit the worst of errors from the user's viewpoint, the senseless translation. An experienced reviewer will have developed sense for troublesome contexts, and will have several means to decisively determine the context (including, for example, running the development version of the application).

Even if the context is correctly established, the translator may use "wrong" terminology, which is the next worse thing for the user. A term used in translation does not need to be wrong by itself, in fact it may be exactly the correct term -- in another translation project. The reviewer will have more experience with terminology of the present project, and be able to bring the translation in line with it.

Style in the technical sense is a consistent choice between several perfectly valid constructs in target language when applied to text in the given technical context. For example, how to translate menu titles and items, button labels, or tooltips. The choices may include noun or verb forms, particular grammar categories, tone of address, and so on. There may be a style guide to the project which details such choices, and the reviewer will know it well.

Style in the linguistic sense is especially applicable to longer texts, such as tooltips in user interfaces, and passages in documentation. A typical error of a new translator is to closely adhere to English style and grammar. This may produce translation which is semantically and grammatically valid in the target language, but very out of style -- the "translationese". Reviewer is there to naturalize such passages.

Finally, the reviewer may be an experienced translator, but that does not mean that his own translations need no review. Immersion into the source language, distraction, fatigue, will lead the reviewer into any of the above errors in translation, only with less frequency. So reviewers should also review one anothers' translations.

== Classical Reviewing by Stages ==

Classical review workflow by stages seems simple enough. Translator translates a PO file (or updates existing translation), and declares it ready to review. A reviewer reviews it, and declares it ready to "commit". Committing here should be understood generally, as inclusion into the pool from which translations are periodically shipped to end users. A committer finally commits the file. The process is iterative: the reviewer may return the file to the translator, and translator later again declare it as ready for review. There may be several stages of review (such as ''proof-reading'', ''approving''), each of which may return the translation to a previous stage, or forward it to some special stage. The process can also be more finer grained, where each message in the file goes through stages separately.

Regardless of the particularities, workflows of this kind all have the following in common. Members of the translation team are assigned ''roles'' -- such as ''translator'', ''reviewer'', ''approver'', ''committer'' -- by which they enter into the workflow (single person can have more roles). The later review stages must wait for the earlier stages to complete, and the translation cannot be updated again before the current version clears the pipeline (or the pipeline is aborted). Most importantly, once the translation is committed, it becomes part of simply "admitted" translations, with no further qualifiers.

The system of prescribed roles requires that team members assign them between themselves, stick to them, and shuffle them along the way. The prescribed review pipeline requires a tool to enforce and keep track of the stages in which translations are. This makes the review workflow complex and rigid, most probably with choke points for efficiency. Distribution of roles may become disbalanced by people coming and going, or the workflow tool may be prohibitive to some scenarios (e.g. single translator making small adjustments in dozens of files across the project, but having to upload each manually through a web interface).

Of course, "rigid", "complex", "inefficient", are comparative qualifications, so what is it that the classical review by stages can be compared to in this way?

== Reviewing by Ascriptions ==

Reviewing by ascriptions is even simpler conceptually, and yet less rigid, less complex, and much more efficient than the review by stages. It works on the message-level, rather than file-level. Anyone can simply translate some messages and directly commit modified files, without any review, but with ''ascribing'' modifications to own name. Anyone can review any committed messages at any moment, commit the modifications-on-review and ascribe reviews to own name and (possibly) to certain class -- full review, review of context, of terminology, of style, etc. Only when the translation is to be shipped to end users, the ''insufficiently'' reviewed messages are automatically omitted from the package, by evaluating the ''ascription history'' of each message.

Most importantly, based on the ascription history, the reviewer can select only some particular messages, and review only the difference between their historical and current versions. For example, Alice can select to review only messages modified since she or Bob had last reviewed them for style; she could see the difference from that last review to current version, e.g. if in the whole paragraph only a single word has changed by Charlie when he reviewed the terminology. In terms of PO workflow, the ascription history propagates through merges, so the reviewer can compare the change in original and the change in translation since the last review, to judge if one fits the other.

Since everyone just commits, translations can be efficiently kept in a version control repository, with the ascription system added on top. After having done some translating, the team member simply substitutes commit command of the version control system (VCS) with ascribe-modifications command of the ascription system (AS, which calls the underlying VCS internally). After reviewing, the team member uses ascribe-reviews command of the AS to commit reviews to ascription history (as well as modifications made during the review). To select messages for review, the team member issues diff-for-review command of the AS (with suitable parameters to narrow the set) and selected messages are marked in-place in PO files and [[Localization/Tools/Pology/PO_Embedded_Diffing|embedded with differences]], and possibly popped open in a PO editor.

When the translations are to be released, the team coordinator issues filter-for-release command of the AS, which takes the working PO files and creates final PO files with insufficiently reviewed messages removed. "Release time" is used here only figuratively: this should be a fully automatic process, so it can be performed at any interval of convenience.

What constitutes "sufficient review" can be defined in fine detail. It could be specified that messages modified by Alice need to have only review for terminology, but not necessarily for style; Charlie may belong to the group which needs to be reviewed on style, but not necessarily on context; Bob's reviews for style may be nice to have, but never blocking if missing. These decisions do not preclude released messages to be reviewed later on missing points, after higher priority reviews have been completed. The definition of sufficiency may be changed at any point, e.g. as team members get more experienced and require less review, without interfering with direct translation and review work.



In summary, with reviewing by ascriptions the lean efficiency of raw VCS operation is preserved while providing for great flexibility of review. All team members can be given commit access, no web or email detours are needed. There are no prescribed roles, but an equivalent of role assignment happens automatically at last possible moment, and can take into account both translators' and reviewers' abilities. There is no staging between completing and committing the translation, which enables translator to keep on polishing the translation undisturbed until the reviewer comes around. There is no inefficiency in handling small changes throughout many files, since single AS command commits all changes just as single VCS command would. AS in effect abstracts VCS, so general team members do not have to know the particularities of the underlying VCS. On commit operations, AS can also apply checks (e.g. decline to commit syntactically invalid PO files) and modifications (e.g. update translator's data in the PO header).

== Ascription System in Pology ==

[[Localization/Tools/Pology|Pology]] is a collection of various modular tools for supporting translation based on PO files. Among them is the script <tt>poascribe</tt>, which implements an ascription system (AS); at present, it can use Subversion or Git as the underlying VCS. <tt>poascribe</tt> is still in experimental stage, so what follows is a brief description of how to use it in context of the KDE translation project. However, very little is truly specific to KDE; the only major assumption is that there exists a VCS repository with PO files of a given language grouped together, and that the translation team can use it without special restrictions.

Very important for the AS is how branches are handled (in KDE, the rolling trunk and stable branches). AS can in principle be deployed by branch, but then there is the added complexity of porting translations between branches, which ascriptions should follow. Therefore, the AS implemented by <tt>poascribe</tt> is currently limited to assumption that there is a single branch of translations at all times. The article [[Localization/Workflows/PO_Summit|"Translating in Summit"]] explains how a KDE translation team can set up and operate such a single branch, the summit, and this is the prerequisite for the following instructions. (Note that the summit system is useful on its own, and should be conductive to any kind of review workflow.)

== Setting Up ==

The summit branch for the language <tt>LANG</tt> is positioned like this in the KDE repository:

<code text>
$KDEREPO/
trunk/
l10n-support/
LANG/
summit/
messages/
docmessages/
</code>

The team coordinator already has this part of the repository tree locally due to regular summit operations. For the same reason Pology is already set up. Setting up the ascription system is now simple. The file <tt>ascription-config</tt> is created in the root of the summit:

<code text>
...
LANG/
summit/
messages/
docmessages/
ascription-config
</code>

with the following contents:

<code text>
# ---------------------------
# Global ascription settings.

[global]

# The root of the ascription catalog tree.
ascript-root = ../summit-ascript

# The underlying version control system.
version-control = svn

# Data for updating catalog headers.
# - language code
language = LANG
# - full language name
language-team = LANGUAGE
# - email address of the team
team-email = kde-i18n-LANG@kde.org

# Default commit message.
commit-message = Translation updates.

# -----------------------
# Registered translators.

[user-alice]
name = Alice Akmalryn
original-name = Алиса Акмалрин
email = alice.akmalryn@someplacenice.org

[user-bob]
name = Bob Byomkin
original-name = Бобан Бјомкин
email = bob.byomkin@otherplacenice.org

# ...and so on.
</code>

Some notes:

* The <tt>ascript-root</tt> setting should be exactly <tt>../summit-ascript</tt>, for the reason mentioned later.

* <tt>commit-message</tt> field, if defined, allows team members to commit without providing a commit message. The value given by this field will be used by default, with translator's user name appended to the end in special syntax. For example: <tt>Translation updates. [>alice]</tt>. (Translator's user name is also appended to manually supplied commit messages.) Translators can still supply a commit message when they wish, as shown later. If this field is not set, the commit message is supplied as usual on committing.

* Team members are defined by <tt>[user-USERNAME]</tt> sections. Ascription user names can be any valid ASCII identifier: ASCII letters, digits and underscores only, digit cannot be the first character. Ascription user names have no technical relation to the underlying VCS accounts, though it is mnemonically convenient if they are the same (in case of SVN). This means that a translator who does not have a VCS account (yet) can and should be added here, with assigned user name (best one suitable as SVN account name later); why this should be done will be explained later.

* <tt>original-name</tt> field in user sections is there in case the preferred renderings of the name in English and in target language are not the same. When this is not the case, <tt>original-name</tt> can be omitted.

As soon as the <tt>ascription-config</tt> file is committed, the ascription system is ready for operation. Only regular modifications to this file are those of adding new team members. (On the other hand, team members should never be removed, because even after they no longer contribute, their ascription records remain in the system.)

=== Initial Ascription ===

The most common situation at start of ascription workflow is that there already exists a body of translations, contributed to by many different people over time. The coordinator should ascribe all existing translations as initial modifications, but to whom? It cannot be said precisely who translated what. The solution is to introduce a generic user in <tt>ascription-config</tt>, suitably known as "Unknown Hero" (or "Lost Translator", you can be inventive):

<code>
[user-uhero]
name = Unknown Hero
original-name = Незнани јунак
</code>

and ascribe all existing translations as modified and reviewed by this user. The coordinator does this with the following command:

<code bash>
$ cd $KDEREPO/trunk/l10n-support
$ poascribe reviewed -u uhero -C LANG/summit/
</code>

The argument <tt>reviewed</tt> is the ascription mode, and the <tt>-u</tt> option provides the user name to which ascriptions are made. This is an important point: ascriptions are made to a user defined in ascription configuration, and have nothing to do with VCS accounts; someone who has the account can commit in the name of someone who does not. The <tt>-C</tt> option prevent automatic committing by <tt>poascribe</tt>, which is useful for this initial step. Finally the paths which contain all summit catalogs are given.

When the <tt>poascribe</tt> command is issued, a progress bar will appear, and the following output will start to unfold:

<code text>
! LANG/summit-ascript/messages/extragear-base/rellinks.po (50)
! LANG/summit-ascript/messages/extragear-base/autorefresh.po (13)
! LANG/summit-ascript/messages/extragear-base/babelfish.po (38)
...
! LANG/summit-ascript/messages/qt/libphonon.po (13)
! LANG/summit-ascript/messages/qt/phonon-xine.po (24)
! LANG/summit-ascript/messages/qt/phonon_gstreamer.po (12)
===! Translated: 111775
===! Fuzzy: 26943
===! Obsolete translated: 2965
===! Obsolete fuzzy: 1626
! LANG/summit-ascript/messages/extragear-base/rellinks.po (50)
! LANG/summit-ascript/messages/extragear-base/autorefresh.po (13)
! LANG/summit-ascript/messages/extragear-base/babelfish.po (38)
...
! LANG/summit-ascript/messages/qt/libphonon.po (13)
! LANG/summit-ascript/messages/qt/phonon-xine.po (24)
! LANG/summit-ascript/messages/qt/phonon_gstreamer.po (12)
===! Reviewed: 143309
</code>

The number in parenthesis indicates how many messages have been ascribed in the given PO file, and at the end the totals are given. Catalogs are processed twice, first to ascribe modifications and then reviews, because a review cannot be ascribed before the modification has been ascribed. Ascribing the complete summit for the first time will take quite some time (say 15-30 minutes).



After the initial ascription has been made, the ascription tree will appear next to the summit tree. This tree will contain one ascription PO file for each summit PO file, with the same name and relative location within the tree:

<code text>
...
LANG/
summit/
messages/
kdelibs/
kcertpart.po
kdelibs4.po
...
...
docmessages/
ascription-config
summit-ascript/
messages/
kdelibs/
kcertpart.po
kdelibs4.po
...
...
docmessages/
</code>

The ascription tree can now be committed as usual (<tt>poascribe</tt> will have already added it):

<code bash>
$ svn commit LANG/summit-ascript/ -m "Initial ascription."
</code>

== Daily Use for Translators ==

Team members other than the coordinator, whether translators or reviewers, need to keep around only the <tt>trunk/l10n-support/LANG/</tt> directory. But they always need to update this directory fully (rather than just one particular module or file under <tt>.../*messages/</tt>), so that the summit tree and the ascription tree (and configuration) are kept in sync.

In order not to have to issue their own user name (<tt>-u</tt> option to <tt>poascribe</tt>) all the time, translators can set it in Pology user configuration <tt>~/.pologyrc</tt>, in <tt>[poascribe]</tt> section:

<code text>
[poascribe]
user = alice
</code>

Translators can then submit updated PO files simply by substituting <tt>svn commit</tt> (or whatever the VCS commit command is) with <tt>poascribe modified</tt> (<tt>mo</tt> for short):

<code bash>
$ poascribe mo LANG/summit/messages/kdefoo/*fooapp*.po
! LANG/summit-ascript/messages/kdefoo/fooapp.po (144)
! LANG/summit-ascript/messages/kdefoo/libfooapp.po (25)
===! Translated: 169
Sending LANG/summit/messages/kdefoo/fooapp.po
Sending LANG/summit/messages/kdefoo/libfooapp.po
Sending LANG/summit-ascript/messages/kdefoo/fooapp.po
Sending LANG/summit-ascript/messages/kdefoo/libfooapp.po
Transmitting file data ....
Committed revision 1267069.
$
</code>

<tt>poascribe</tt> will add ascription records into ascription catalogs corresponding to summit catalogs to be committed, and commit them all. Like <tt>svn commit</tt>, <tt>poascribe mo</tt> can take any number of file or directory paths, and can be issued from any working directory (it will always find ascription catalogs). If default commit message has not been set in the ascription configuration, <tt>poascribe</tt> will ask for it; or it can be given in command line through <tt>-m</tt> option.

=== Translators Without Commit Access ===

With the ascription system in place, every regular team member should have commit access. But, there may be some period of time before new translators are given accounts, revision control may be too technical for some, and even those with the account may not be able to commit temporarily for some reason.

These translators may send in their work by email, to ''any'' team member with commit access (not necessarily the coordinator or a reviewer); this team member can commit received files without any review, as review can be conducted at any later time. If Bob sends some files to Alice, she can commit them immediately by stating Bob's user name:

<code bash>
$ poascribe mo -u bob ...
</code>

For this to work, the translator who sent in the files has to be defined in the ascription configuration. There are no hidden costs or security issues to this (as opposed to opening a VCS account), so every new translator should be defined there before any work of that person is committed.

== Daily Use for Reviewers ==

The ascription system opens up all sorts of possibilities for concrete review patterns. Reviewers should keep in mind that for each message the full modification and review history is available, so that the team can think about how to make good use of it. Therefore, what follows are some examples to illustrate the review facilities that <tt>poascribe</tt> provides.

=== Basic Reviewing ===

At the very basic level (which is the only level in classical review by stages), messages can be classified into simply unreviewed and reviewed, without further qualifiers. Alice now wants to review all unreviewed messages in a group of PO files, say <tt>kdetoys</tt> module. She issues (<tt>di</tt> is short for <tt>diff</tt>):

<code bash>
$ cd $KDEREPO/trunk/l10n-support
$ poascribe di LANG/summit/messages/kdetoys/
! LANG/summit/messages/kdegames/bovo.po (2)
! LANG/summit/messages/kdegames/kdiamond.po (7)
! LANG/summit/messages/kdegames/palapeli.po (12)
===! Diffed for review: 21
</code>

Unreviewed messages have now been marked and diffed, inside the listed PO files. What is this about "diffing"? If the files had already been reviewed before, some of the messages modified since then (those marked for review) may have changed very little (e.g. a few words in a paragraph-length message, or even just punctuation). Therefore, for each message marked for review, Alice also wants to see the diff since last review to current version. Here are two messages in typical review states added by <tt>poascribe di</tt>:

<code po>
#. +> trunk stable
#. ascto: charlie:m
#: gui/mainwindow.cc:372
#, ediff
msgid "GAME OVER. {-You won-}{+Tie+}!"
msgstr "KRAJ IGRE. {-Pobeda-}{+Nerešeno+}!"

#. +> trunk stable
#. ascto: bob:m charlie:m
#: game-state.cpp:117
#, ediff-total
msgid "Click the pause button again to resume the game."
msgstr "Кликните поново на дугме паузе за наставак игре."
</code>

and the first one in Kate:

[[Image:Kate_poascribe_diff_01.png|center|Message diffed for review by poascribe in Kate.]]

In the first message, the first to note is the <tt>#. ascto:</tt> comment. This comment succinctly lists who did what with the message since the last review; here <tt>charlie:m</tt> means that Charlie is the one who modified it. Then, there is the <tt>ediff</tt> flag, which alice can use it to jump through messages marked for review. Finally, the original and translation have been diffed; here they show that, since the last review, the message was fuzzied by changing "You won" to "Tie", and what Charlie did in translation to unfuzzy it. Even on a message as short as this, the diff tells something useful to Alice: the phrase "Game over" likely has a formulaic translation, and the fact that it is not part of the diff means that the earlier reviewer had made sure it is consistent, so Alice does not have to check that.

The <tt>#. ascto:</tt> comment of the second message reveals that both Charlie and Bob had been translating it. <tt>ediff-total</tt> flag instead of plain <tt>ediff</tt> means that this message had no review at all up to now, so there are no embedded diffs in text fields.

Alice can now go through marked files and messages, review translations, and possibly make modifications. When making changes in a message with embedded diffs, she can freely edit text outside of difference segments and within <tt>{+...+}</tt> segments (as these are the ones which belong to current version of the text). While reviewing, Alice does ''not'' remove any of the added message elements while reviewing (save for an occasional difference segment, when translation should be modified), as these elements are needed for later. If a message is particularly hard and Alice wants to defer its review for later, she can add the <tt>unreviewed</tt> (or <tt>urev</tt> or <tt>nrev</tt> for short) flag to it.

Once the review is complete, Alice commits the reviewed files in <tt>reviewed</tt> mode (short <tt>re</tt>):

<code bash>
$ poascribe re LANG/summit/messages/kdetoys/{bovo,kdiamond,palapeli}.po
! LANG/summit/messages/kdegames/bovo.po (2)
! LANG/summit/messages/kdegames/kdiamond.po (7)
! LANG/summit/messages/kdegames/palapeli.po (12)
===! Cleared reviews: 21
! LANG/summit-ascript/messages/kdegames/palapeli.po (3)
===! Translated: 3
! LANG/summit-ascript/messages/kdegames/bovo.po (2)
! LANG/summit-ascript/messages/kdegames/kdiamond.po (7)
! LANG/summit-ascript/messages/kdegames/palapeli.po (12)
===! Reviewed: 21
Sending LANG/summit/messages/kdegames/palapeli.po
Sending LANG/summit-ascript/messages/kdegames/bovo.po
Sending LANG/summit-ascript/messages/kdegames/kdiamond.po
Sending LANG/summit-ascript/messages/kdegames/palapeli.po
Transmitting file data ....
Committed revision 1284220.
$
</code>

Three things have happened here. First, all review states (flags, embedded diffs, etc.) have been removed, restoring the PO file to normal. Then, any modifications that Alice have made during review are ascribed to her (here 3 out of 21 messages). Finally, all marked messages are ascribed as reviewed by Alice (any with <tt>unreviewed</tt>/<tt>urev</tt>/<tt>nrev</tt> flags would have been omitted here). When committing, the only summit catalog that got committed is the one with modifications made during review, and all the ascription catalogs were committed because of the reviews recorded in them.

When many files with few changes in each are to be reviewed, it becomes burdensome to manually open each and every diffed for review, and then to make sure that all are committed with <tt>poascribe re</tt>. To make this easier, <tt>-w torevivew.out</tt> option can be added to <tt>poascribe di</tt>, which requests that paths of all diffed PO files are written into <tt>torevivew.out</tt> file. This file can then be used to batch open POs for review in the editor, as well as fed back on <tt>poascribe re</tt> with <tt>-f torevivew.out</tt>. There is also the <tt>-o</tt> option which causes <tt>poascribe</tt> to directly open PO files in a PO editor, though this is currently applicable only to Lokalize. Putting it together, to efficiently review a whole bunch of small changes throughout many files, Alice can:

<code bash>
$ poascribe di PATHS... -w toreview.out -o lokalize
$ # ...only marked messages opened in Lokalize, review them...
$ poascribe re -f toreview.out
</code>

=== Selecting Messages for Review ===

Invocations of <tt>poascribe di</tt> without any options, as in the previous section, were actually equivalent to this:

<code bash>
$ poascribe di -s modar PATHS...
</code>

Option <tt>-s</tt> is issuing the message ''selector''. <tt>modar</tt> is the default selector for <tt>diff</tt> mode, and stands for MODified-After-Review: it selects the earliest historical modification of the message after the last (or no) review of that message, if there is any such. By selecting a historical modification of the message, the diff from it to current version can be computed and embedded into the PO file, as in previous examples.

There are various specialized selectors, and fall into two groups: ''shallow selectors'' and ''history selectors''. Shallow selectors look only into the current version of the message, and cannot select historical versions, which means that they cannot provide embedded diffs. History selectors (<tt>modar</tt> is of this type) can select messages from history and provide diffs. Several selectors can be issued on the command line, and the message is selected only if all selectors select it. Shallow selectors are then normally used as a pre-filter for history selectors. For example, to select messages modified after last reviewed, but only those found in stable branch, <tt>branch</tt> and <tt>modar</tt> selectors are chained:

<code bash>
$ poascribe di -s branch:stable -s modar PATHS...
</code>

It is important that the history selector is given last, because the last selector determines which historical message is selected. If the ordering had been reversed here, same messages would get selected, but they would not have embedded diffs, because <tt>branch</tt> is a shallow selector.

Selectors can take parameters themselves, like <tt>branch:stable</tt> in the previous example. Parameters are separated from the selector name by any non-alphanumeric character; this is colon by convention, but if a parameter contains a colon, something like slash, tilde, etc. can be used. Number of parameters can be variable, and <tt>modar</tt> in particular can take from none to three. If Alice wants to review only those messages modified ''by Charlie'' since last review, she states this by first argument to <tt>modar</tt>:

<code bash>
$ poascribe di -s modar:charlie PATHS...
</code>

If Alice does not give too much credit to other reviewers, she can request selection of messages modified after last review ''by her'' with second parameter to <tt>modar</tt>:

<code bash>
$ poascribe di -s modar::alice PATHS...
</code>

Here the first parameter ("modified by..."), which is not needed, must be explicitly skipped, before going to the second parameter ("reviewed by..."). The third optional parameter of <tt>modar</tt> will be mentioned in the next section.

When a selector parameter is a user name, normally it can also be a comma-separated list of user names (<tt>modar:bob,charlie</tt>) or prefixed with tilde to negate, i.e. select all other users (<tt>modar:~alice</tt>).

Any selector can be negated by prepending <tt>n</tt> to its name. For example, the history selector <tt>modafter:DATE</tt> selects first modification after the given date; to select messages modified after last review, but only if modified during June 2010:

<code bash>
$ poascribe di -s modafter:2010-06 -s nmodafter:2010-07 -s modar PATHS...
</code>

Negating a history selector produces a shallow selector: while <tt>modafter</tt> is history selector, <tt>nmodafter</tt> is shallow. But the order of the two in the previous command line is not important, as the last selector is the usual <tt>modar</tt>.

Selectors can be issued in other modes too. If the PO file is big and Alice has reviewed messages up to and including entry 246 when she has to pause until another day, she can commit reviews only up to this entry by issuing the <tt>espan</tt> selector:

<code bash>
$ poascribe re -s espan::246 PATHS...
</code>

(the first parameter to <tt>espan</tt> is the first entry number, given if messages are not to be selected from the first). There is also the counterpart <tt>lspan</tt> selector, which works with referent line numbers (those of <tt>msgid</tt> keywords) instead of entry numbers.

=== Fine-Grained Reviews ===

In the introduction, several distinct types of what can go wrong in translation were described. Not all reviewers may be able to check translation against all those problems. Here is a typical scenario of this kind:

Alice is very computer-savvy and knows the translation project inside and out, which means that she can review well for context, terminology, and technical style. But, her language style leaves something to be desired, which shows through longer sentences and passages. Dan, on the other hand, is a very literary person, but not that much into the technical aspects. Dan's style reviews would thus be a perfect complement to Alice's general reviews.

<tt>poascribe</tt> can support this scenario in the following way. A ''review type'' tag for language style is defined in the ascription configuration, using the <tt>review-tags</tt> field:

<code text>
[global]
...
review-tags = lstyle
</code>

(The value to <tt>review-tags</tt> is a space-separated list of identifiers, when more than one special review type is needed.) With this addition to configuration, Alice can continue to review as she did before, without any changes to her workflow.

Dan selects messages for review quite similarly to Alice, with the exception of giving the <tt>lstyle</tt> tag as ''third'' parameter of <tt>modar</tt>:

<code bash>
$ poascribe di -s modar:::lstyle PATHS...
</code>

When committing reviews, Dan must also state this tag, in order to ascribe reviews as of language style type:

<code bash>
$ poascribe re -t lstyle PATHS...
</code>

If Dan is always going to review the language style, in order not to have to issue the selector and tag in the command line all the time, he can make them default per mode in <tt>~/.pologyrc</tt>:

<code text>
[poascribe]
user = dan
selector/diff = modar:::lstyle
tag/reviewed = lstyle
</code>

With this Dan can use plain <tt>poascribe di</tt> and <tt>poascribe re</tt>, just like Alice does.

The important point of review tags is that they make reviews by types independent. For example, Dan may come around to review the language style of the given message after several modifications and general reviews have been ascribed to it -- <tt>modar:::lstyle</tt> will simply ignore all reviews except for <tt>lstyle</tt> reviews. This is going to be reflected in the <tt>ascto:</tt> comment to marked messages:

<code po>
#...
#. ascto: charlie:m alice:r bob:m
#...
msgid "..."
msgstr "..."
</code>

Here Alice has made one review between Charlie's and Bob's modifications, and that review, being general instead of <tt>lstyle</tt>, did not cause <tt>modar</tt> to stop at it. After Dan reviews this message for language style, Alice runs selection for review and gets this:

<code po>
#...
#. ascto: bob:m dan:r(lstyle)
#...
msgid "..."
msgstr "..."
</code>

Again, since <tt>lstyle</tt> reviews do not mix with general reviews, Dan's review did not hide Bob's modification that Alice did not check so far.

(General review too has a tag assigned, the empty string, in case the reviewer needs to explicitly issue it in some context.)

== Daily Use for The Coordinator ==

After setting up the ascription system, the team coordinator should have to do very little to maintain it.

=== Ascribing Merges ===

Modifications made to summit catalogs by merging with templates must also be ascribed. This ascription is made in the name of the reserved <tt>fuzzy</tt> user, which exists by default and is not defined in <tt>ascription-config</tt>. Therefore, after merging the summit (<tt>posummit ... merge ...</tt>) the coordinator substitutes the VCS command:

<code bash>
$ svn commit LANG/summit/messages/ -m "Merged summit."
</code>

with the <tt>poascribe</tt> command in <tt>modified</tt> mode:

<code bash>
$ poascribe mo -u fuzzy LANG/summit/messages/ -m "Merged summit."
</code>

Since <tt>-C</tt> option is not issued, <tt>poascribe</tt> will automatically commit all modified summit and ascription catalogs when done.

=== Shuffling Ascription Catalogs ===

Sometimes summit catalogs are shuffled in the repository: moved to another module, renamed, one catalog split into two, two catalogs merged into one. Such shuffling should be exactly mirrored in the ascription tree, and this too is done on the repository side, at the same time. This relies on the ascription root being set exactly to <tt>../summit-ascript</tt> in the ascription configuration. So the team coordinator has nothing special to do here.

If instead in the central KDE repository the translation team is working in an external repository, by consequence the ascription system must be set up in that repository. But so long as <tt>process_orphans.sh</tt> script from <tt>trunk/l10n-support/scripts/</tt> is used to shuffle catalogs in the external repository as well, the ascription catalogs will be properly handled.

== Filtering for Release ==

The last component of the ascription system is how to prevent insufficiently reviewed messages from leaking into a release. In context of Pology and summit workflow, <tt>poascribe</tt> itself is used directly to this end. Instead, in the summit configuration (as opposed to ascription configuration), the team coordinator defines filters which pass messages by applying selectors.

Each top level PO tree has its own summit configuration file, named <tt>MSGTREE.extras.summit</tt>:

<code text>
...
LANG/
summit/
messages/
messages.extras.summit
docmessages/
docmessages.extras.summit
</code>

For the simple case of all reviews being general reviews, the filter is added to summit configuration like this (anywhere within <tt>*.extras.summit</tt> file):

<code python>
S.ascription_filters = [
("regular", ["nmodar"]),
]
</code>

Here the filter is named <tt>regular</tt>, and is defined as application of <tt>nmodar</tt> selector, the negation of <tt>modar</tt>. This simply means: pass all messages ''not'' modified after the last review.

When the team coordinator scatters to branches (executes <tt>posummit scatter</tt>), messages from summit POs which do not pass this filter will not be sent to branch POs. The count of stopped messages by branch PO will be reported in the output as scattering proceeds.

Why did we have to name the filter <tt>regular</tt>? (Those knowing some Python will also notice that it is defined as a list element.) Because it is possible to define more than one filter, and select which one is used on each scattering. For example, the coordinator may wish that, when the release is near and time is short to review everything, messages from a few experienced translators can be passed into release without review. If those translators are Alice and Bob, an "emergency" filter can be defined like this:

<code python>
S.ascription_filters = [
("regular", ["nmodar"]),
("emergency", ["nmodar:~alice,bob"]),
]
</code>

By default, <tt>posummit</tt> uses the first filter in the list. When the coordinator needs to do emergency scattering, he requests the emergency filter by the <tt>-a</tt> option:

<code bash>
$ cd $KDEREPO/trunk/l10n-support
$ posummit scripts/messages.summit LANG scatter -a emergency
</code>

What if several selectors are needed to pass the message? For example, the language style review (the earlier example with Alice and Dan) too may be requested for regular scattering, but omitted from emergency scattering. The filter setup for this scenario looks like this:

<code python>
S.ascription_filters = [
("regular", ["nmodar", "nmodar:::lstyle"]),
("emergency", ["nmodar:~alice,bob"]),
]
</code>

The regular filter now reads: pass the message if it has not been modified after the last (general) review ''and'' has not been modified after the last style review.

Simple combination of predefined selectors by AND-conditions may not be sufficient for more involved scenarios. When this is the case, the coordinator may write (or ask someone to write) a custom selector in Python, and plug it in as the second element in the filter tuple (instead of the list of predefined selectors).

=== Writing a Selector Function ===

((To be written.))