Localization/Tools/Lbundle Check

Lbundle Checker

On Localization	Tools
Prerequisites	Subversion Ops, Localizing Non-Text Resources
Related Articles	n/a
External Reading	n/a

About

The lbundle_check.py script checks and records the state of localized non-text resources organized in lbundles, allowing translators to track their relation to original resources. It lives in l10n-kde4/scripts/ directory in the appropriate branch in the KDE repository.

lbundle_check.py can be used to track both out-of-source and in-source lbundles, but the setup needed for these two modes is somewhat different. First the setup is explained for each type of lbundle, followed by the details of operation.

Setup

Before going through the setup examples, refresh the examples on organization of lbundles in the repository (localizing the splash screen of the imaginary KDingus application).

As it will become apparent below, to be able to track lbundle states, the translator (one of the coordinators) needs to have all the original counterparts to localized resources checked out from the KDE repository, and in the same relative positions. E.g. for KDingus example as kdeutils application, checked out files could have the following structure: $HOME/kde-svn

   trunk/
       l10n-kde4/
           aa/
       KDE/
           kdeutils/
               kdingus/

To have it like this without checking out full parent directories, one can use -N option to svn checkout, which avoids recursion into subdirectories.

Out-of-Source Lbundles

When it was assumed that KDingus is living in kdeutils, a core KDE module, its out-of-source lbundle for the language aa was organized like this: /trunk/l10n-kde4/aa/data/kdeutils/

   CMakeLists.txt
   kdingus/
       CMakeLists.txt
       pics/
           CMakeLists.txt
           l10n-spec
           l10n/
               aa/
                   kdingus-splash.png

except that this listing includes one new file, the l10n-spec. This is the setup file for tracking the lbundle, or rather, all lbundles on its level and beneath. The files outside of l10n/ subdirs are going to be called unbundled, as they are not considered a part of lbundle proper.

The l10n-spec files are composed of key = value fields per line. For the example above the content of l10n-spec would be:

1. l10n-spec for KDingus' out-of-source bundle.

source-root = trunk/KDE/kdeutils/kdingus source-vcs = svn bundle-vcs = svn languages = aa track-unbundled = CMakeLists.txt

The fields used in the example are:

source-root

the root of the KDingus' sources in the KDE repository. This is needed to link the original counterparts to localized files. The path of the original file relative to the repository top will be composed of this, plus the relative path of the localized file in the lbundle (relative to the l10n-spec file), and stripped of the l10n/aa/ subpath.

source-vcs, bundle-vcs

the version control systems used by the source code and the lbundle, respectively. These are needed so that the checker script knows how to update the sources (if requested), and to avoid trying to track version control bookkeeping files in the lbundle (e.g. .svn subdirs for SVN). Currently the only known VCS is svn, but the script abstracts this internally, so that other may be added in the future.

languages

space-delimited list of languages expected in l10n subdirs. This is an optional field, which serves to check for naming errors with language subdirs. It may be left out, preventing such checks, but there is no reason to do so (especially for in-source bundles described below).

track-unbundled

space-delimited list of relative file paths, or pairs of file paths, of unbundled files which should be tracked nevertheless. In this example, CMakeLists.txt should be tracked because the installation instructions for the lbundle may need to change when those for the original resources change. A pair of file paths, as two space-separated paths within parenthesis (e.g. (foo.txt bar.txt)), can be given instead of a single file path when the local relative path needs to be different from that of the tracked original. A shell glob can be specified instead of a particular path (e.g. *.txt), but only for single paths, not in pairs. Backslash serves as escape character, e.g. when the file name contains a space.

Several other fields, not used in this example, are available:

source-root-rel

relative path to the sources, from l10n-spec in which it is defined. Useful when the repository organizes branches as subdirectories, since then the path relative to repository top (as provided by source-root) would change between branches.

source-root-repl

construct path to sources by replacing a part of current bundle's path. The replacement specification is in form findstr:replstr. Useful when there are many bundles which would have exactly the same l10n-spec files when the path to sources is specified like this.

ignore-unbundled

space-delimited list of relative file paths which are to be completely ignored by the lbundle checker. For an out-of-source bundle, a file which is unbundled should be either tracked too (using track-unbundled) or ignored using this field, otherwise the checker will complain about it. Shell globs also possible.

strict-state

a boolean value stating whether the tracking state is to be strictly imposed. Non-strict means that out-of-sync states between localized and original resources are recorded only in the track file, whereas strict state means that the localized files names will be changed too to reflact the out-of-sync state. More details in section on operation.

ignore-substr

space-delimited list of substrings to ignore in names of bundled files when determining their original counterparts. This is never needed in KDE, but in other environments some localized files may be handled differently than a plain substitute for the original file, based on a substring in their name (e.g. a localized image may be overlayed over the original, rather than replacing it).

track-by-subdir

by default there is only one tracking file per l10n-spec file (see operation for tracking files); when this option is enabled (a boolean), there will be instead one tracking file per subdirectory (excluding l10n/ directories) starting from the level of l10n-spec file and below. Usefull when frequent moving of subdirectories within the repository is expected, and there are a lot of localized resources to track.

The usual #-comments are allowed in l10n-spec files, as well as line continuation by trailing backslashes (e.g. when several file paths are needed in track-unbundled field).

In-Source Lbundles

For KDingus living as an extragear app, its directory structure with the in-source lbundle having aa and bb languages, and an l10n-spec file, would be this: /trunk/extragear/utils/

   kdingus/
       CMakeLists.txt
       pics/
           CMakeLists.txt
           kdingus-splash.png
           l10n-spec
           l10n/
               aa/
                   kdingus-splash.png
               bb/
                   kdingus-splash.png

and the contents of l10n-spec somewhat simpler than for out-of-source lbundles:

1. l10n-spec for KDingus' in-source bundle.

source-vcs = svn languages = aa bb

Since the lbundle is kept together with the application, sharing same root directory and version control system, there is no need for the source-root and bundle-vcs fields. In fact, presence or lack of source-root field identifies the bundle as out-of-source or in-source to the checker script.

Field track-unbundled (or ignore-unbundled) are not present either. All files which are unbundled are silently ignored. E.g. it is assumed that the KDingus' maintainer will modify and test install instructions so as to not break installation of localized resources.

languages field could be omitted here as well, but for in-source lbundles it is even more important to keep tight check of wrongly named language subdirs.

Greedy Bundling

The default assumption behind lbundles is that only a small part of all resources need to be localized. For example, if a new original resource is added, it is upon translator to notice if it needs localization, make the localized version and add it to lbundle where it will get tracked.

When instead it would be better to track all original resources, the bundle tracking can be set to greedy mode: any original resource that does not have a localized counterpart yet, will be tracked as missing. I.e. when a new original resource gets added, translator will be warned of it by the tracking process.

Fields in l10n-spec used in greedy mode are as follows:

greedy-bundling

option to engage greedy mode (a boolean value, set to true). Other fields controlling greedy bundling have no effect if this option is not set.

greedy-monolingual

states whether the bundle is monolingual (boolean). This should make sense only for out-of-source bundles, where there may be one per language. Greedy mode needs this piece of information in order to know how to report and track missing localized files, as bundled (multilingual lbundle) or unbundled (monolingual).

greedy-from-level

if the lbundle covers a tree of original subdirectories, this field can be used to limit the greedy collection and reporting of missing files only to subdirectories at this level and below. 0 is the level of the l10n-spec file itself, 1 is one level below, etc.

greedy-only-started

normally all original resources missing in localization are reported as such (except the ignored ones), no matter where in the original subdirectory tree covered by the lbundle they reside. This field (a boolean) is used to limit greedy bundling only to those subdirectories already existing in the lbundle. Useful in several scenarios, e.g. when starting the resource localization to avoid being engulfed in listings of missing localized resources. Automatically enabled when track-by-subdir is in effect.

Operation

If the repository is checked out as exemplified above, to check all lbundles for the language aa in trunk, lbundle-check.py can be run like this: $ cd $HOME/kde-svn/ $ lbundle-check.py -s $HOME/kde-svn -l aa trunk/ This will process all lbundles found in trunk/, whether out-of-source or in-source, containing the language aa. (If -l option weren't specified, it would check all languages in trunk/.) Option -s states the top local directory of the repository, which is needed to resolve original resource paths for out-of-source bundles (it is prepended to paths in source-root fields from l10n-spec files; where instead source-root-rel or source-root-repl fields are given, the top local directory is not used).

Or, to check only out-of-source bundles in the trunk/l10n-kde4/aa/data, one would use: $ cd $HOME/kde-svn/trunk/l10n-kde4 $ lbundle-check.py -s $HOME/kde-svn aa/data where -l aa is omitted since out-of-source bundles will contain only their own language.

So, what does lbundle-check.py actually do, and what does it report? For brevity, let's limit to the out-of-source lbundle for the KDingus as kdeutils app. After setting up the l10n-spec for that case (as detailed above), running lbundle-check.py for the first time would output: $ cd $HOME/kde-svn/trunk/l10n-kde4 $ lbundle-check.py -s $HOME/kde-svn aa/data ! aa/data/kdeutils/kdingus/pics/l10n-track

---

Added to tracking: 2

 aa/data/kdeutils/kdingus/pics/CMakeLists.txt
 aa/data/kdeutils/kdingus/pics/l10n/aa/kdingus-splash.png

$ creating the file aa/data/kdeutils/kdingus/pics/l10n-track in the process, with the following content:

1. Do not edit manually, except to remove complete lines.

1. -

ok ¦CMakeLists.txt¦ 865f7...e926d 755260

1. aa

ok ¦l10n/aa/kdingus-splash.png¦ 37c4a7...8fe3a 755647 (For a check, executing the same command line immediately for the second time would produce no output, nor change any files.)

For each l10n-spec file encountered, lbundle-check.py will create one of these l10n-track files. l10n-track file, in each non-comment, non-empty line, contains four fields, in order: the state of the localized resource against the original, the relative path to the bundled resource, the checksum of the original resource, and the VCS revision string of the original file which has this checksum.

At this point, when l10n-spec file has been manually written, and l10n-track created by running the checker, both should be added and committed to version control.

So long as the original resource does not change, rerunning lbundle-check.py in the same way will do nothing, as everything is in sync. Obviously, the original resources should be updated to the latest repository version prior to checking, and lbundle-check.py can do that itself if started with -u option: $ lbundle-check.py -s $HOME/kde-svn -u aa/data svn up /home/.../kdeutils/kdingus/pics/CMakeLists.txt At revision 762512. svn up /home/.../kdeutils/kdingus/pics/kdingus-splash.png At revision 762512. $

Once the original resource is modified, e.g. KDingus' splash screen is souped up, lbundle-check.py will report the following: $ lbundle-check.py -s $HOME/kde-svn aa/data ! aa/data/kdeutils/kdingus/pics/l10n-track

---

Newly fuzzied: 1

 aa/data/kdeutils/kdingus/pics/l10n/aa/kdingus-splash.png

and its entry in l10n-track will state: fuzzy ¦l10n/aa/kdingus-splash.png¦ 37c4a7...8fe3a 755647 i.e. only the status will have changed from ok to fuzzy. Now the translator has the information needed to compare what has changed in the original splash image: this entry still states the revision of the previous original splash, on which the localized one was based.

If the original resource is renamed, moved, or deleted, the report will be slightly different: $ lbundle-check.py -s $HOME/kde-svn aa/data ! aa/data/kdeutils/kdingus/pics/l10n-track

---

Newly obsoleted: 1

 aa/data/kdeutils/kdingus/pics/l10n/aa/kdingus-splash.png

and the entry in l10n-track: obsolete ¦l10n/aa/kdingus-splash.png¦ 37c4a7...8fe3a 755647 Now the translator must find out, using the old revision string, what exactly happened to the original resource, and do the same for the localized one too.

When the fuzzy or obsolete state has been resolved, localized resource updated to reflect changes in the original, it should be recorded as such. To do this, the entry for the resource is first manually deleted from l10n-track file (the whole line is removed), and then lbundle-check.py is rerun. This will recreate the entry with a fresh ok state.

If strict-state field has been set to true in l10n-spec, then not only the state of the entry in l10n-track is changed, but the tracked file itself is renamed to contain ~fuzzy or ~obsolete marker just before the extension. This mode is called strict because unless the out-of-sync state is corrected, the file is "misnamed" from the point of view of the runtime system, and will not be used instead of the original (it may even not get installed, depending on precise install instructions).

To resolve the fuzzy or obsolete state in strict mode, it is not necessary to manually remove the entry from l10n-track. Instead, it is enough to put an updated version of the file (without the state maker in its name) next to the out-of-sync version, and rerun lbundle-check.py. It will remove the old version and update the entry in l10n-track. (The renaming/removal are version control aware, i.e. proper version control commands will be used for these operations if *-vcs fields have been set in the spec file.)

When greedy bundling is engaged, each original file which doesn't have a localized counterpart will be reported in the track file (either as bundled or unbundled, depending on greedy-monolingual option), with it's state set to missing (except for the ignored files, given by the ignore-unbundled option). When the localized file is placed at the appropriate location, next run of lbundle-check.py will change the state to ok.