Development/Tutorials/Localization/i18n Challenges

Common i18n Challenges and Solutions

Tutorial Series	Localization
Previous	Writing Applications With Localization in Mind Incorporating i18n Into the Build System
What's Next	n/a
Further Reading	n/a

Abstract

The localization process can, at times, be tricky. This tutorial covers challenges that you may eventually run into such as: translating handbooks and other data that exists outside of the source code, merging and handling obsolete .po files, dealing with freezes, coding in languages other than English and creating independent releases, or moving applications between KDE modules.

Note

The references to Makefile.am's needs to be ported to CMake.

Message Freezes

As a release date starts getting close, most software projects (including the KDE project) will freeze all changes to user visible strings. This allows translators a chance to do their work without it being constantly disrupted by changes to strings thereby wasting their efforts. If you change i18n strings in your application during such freezes it creates more work (and confusion) for the teams and could prevent your application from being released with translations.

Note

If your application is part of KDE that is periodically announced and released, you must pay attention to announcements on the mailing lists for message freezes. When a message freeze is in place, you must not change any of the translatable strings in your application or its Handbook.

If you find a GUI string that you feel must be fixed or added during a message freeze, ask your translation teams first. In the KDE project, that would be done by emailing the kde-i18n-doc list. Changing a string or adding a new one will not invalidate the entire translation, but annoys the translation teams who are trying to provide a complete and proper translation for your application.

Fixing Messages During A Freeze

It's message freeze time and you forgot to code with the i18n() function or forgot to include a file in the Makefile.am messages target, or you need to make some changes to existing strings. What should you do?

If you need to change an existing string, ask the translators first for permission first before you make the fixes. In KDE, that would be the kde-i18n-doc mailing list.

If you need to add untranslated strings due to forgotten i18n() or mistakes in the build system, go ahead and make the fixes. When you commit the fixes, write "untranslated strings" in the commit message so that translators will know why you are committing string changes during a message freeze. It is a good idea to email the translation teams to let the translators know that you've added additional strings.

Note

If the translator's don't have time to translate the additional strings, they will display in English at runtime.

It is considered a violation of the message freeze to add a brand new translated string to your code where the string did not exist untranslated before. So changing this:

QMessageBox::warning(0, "Starting KTTSD Failed", error );

to this:

QMessageBox::warning(0, i18n( "Starting KTTSD Failed"), error );

is OK, but adding the call to QMessageBox where it didn't exist before will raise objections from the translators.

In the latter case, ask on the kde-i18n-doc mailing list first. In general, the criteria for modifying or adding strings is a potentially severe misunderstanding of a message, especially if it means a risk of losing data.

Tip

You may see translators refer to "fuzzy" strings. Fuzzy strings come about when a string has already been translated and then is changed a little bit later. xgettext leaves the original translation in the .po file, but marks the string as fuzzy. The translator must look at the translation to determine if a change is needed and remove the fuzzy flag, otherwise the translation will not be used at runtime. In short, if you change strings in your app, they will generate fuzzies for the translators.

Handbooks

Handbooks especially should not be touched during a message freeze. This is because changing a Handbook (even a small change) can prevent the entire translated Handbook from being released. For this reason, you should watch the release cycle plans carefully and try to complete your Handbook at least 3 to 4 weeks before the message freeze.

If you do make a late change to a Handbook, it will be up to the translation team whether to apply the fix to their translation or not. If the team has already generated the translated Handbook, the uncorrected version may end up being released.

Freezes are often announced in two phases. A feature freeze means GUI string changes should not be made without first asking permission on the kde-i18n-doc mailing list. Handbook changes are OK at this time. A hard freeze means you must not make any more changes to code or to the Handbook.

If you find a Handbook string that you feel must be fixed or added during a freeze, ask first on the kde-doc-english mailing list. Changing the Handbook, even a tiny change, can invalidate the entire Handbook. Once a hard freeze is in place, fixes should be sent to kde-doc-english as patch files (svn diff). The fixes will not be translated, but they will be incorporated into the English-only version of the Handbook.

Questions about Handbook translation should be sent to the kde-doc-english mailing list.

Independent Releases

If your application is not part of an official KDE release cycle (e.g. it is in KDE Extra Gear or in an outside repository altogether), it will be up to you to plan releases and coordinate with the translation teams. When you want to do a release, send an email to the kde-i18n-doc mailing list with the following information:

• The name of your application and its repository location.
• Your planned message freeze date, the date after which you will not make any more changes that affect i18n strings.
• Your planned code freeze date, the date after which no more changes except for severe bug fixes will be made.
• Your planned release date.

You should provide at least 3 weeks between the message freeze date and release date - more if your application is large and complex. If your application has a Handbook, you should provide even more time.

It is a good idea to avoid time periods when KDE or KOffice are in message freeze, since the translators will be very busy at these times.

If you are distributing your code to translators via a source tarball, you should first ask for volunteers. Of course, the message freeze occurs when you send them the tarball. Your tarball must have all the .pot files and prepared po folders.

KDE Extra Gear Releases

Applications in the extragear module have a special procedure for handling translations. When an extragear application becomes stable (mostly at message freeze), the maintainer should copy the application to the appropriate extragear directory in branches/stable/extragear. In addition the whole set of translation template files (.pot) and translation files (.po), including eventual documentation, of the application must be copied to the appropriate places in branches/stable/l10n. This serves as a signal to translators that the application is ready for final translation work in preparation for release.

Translating UI, RC, and KCFG files

User visible strings usually origin not only from code files, but also from three other file types, ui, rc and kcfg files. UI files are the user interface descriptions, which are edited using QtDesigner. Here the text of labels and other ui elements are in need of localized versions. Likewise for the RC files, which are KXmlGui interface descriptions for embedded components. The KCFG files are the configuration data descriptions, which are used to automatically create configuration control code. While that is not related to the user interface, the same KCFG files are also used as the base for the generic runtime editing with KConfigEditor, so the parameter descriptions also need localized versions.

The strings are handled by the translation system by having them extracted by a program called extractrc, which creates a temporary code file that is than simply included by gettext. So extend the messages target in Makefile.am this way:

messages: rc.cpp
    $(EXTRACTRC) */*.kcfg >> rc.cpp
    $(EXTRACTRC) */*.ui >> rc.cpp
    $(EXTRACTRC) */*.rc >> rc.cpp
    $(XGETTEXT) rc.cpp */*.cpp */*.h -o $(podir)/my_program.pot

If there are strings translators need context information for, they can be supplied via context attribute in RC and KCFG files:

<text context="Create new cup of tea">&New...</text>
<label context="Fancy for mixture">Concoction</label>
<whatsthis context="A concoction">Not entirely unlike tea</whatsthis>

and in QtDesigner as the "disambiguation" property to text labels (or "comment" prior to Qt 4.5), which become comment attributes in the UI file itself.

Also, if that is more suitable, a parameter can be added to the call to extractrc by which the same context is provided to all strings extracted from the file. Example:

messages: rc.cpp
    $(EXTRACTRC) --context=my_context my_config.kcfg >> rc.cpp
    $(EXTRACTRC) --context=other_context */*.ui >> rc.cpp
    $(EXTRACTRC) --context=and_some_more */*.rc >> rc.cpp
    $(XGETTEXT) rc.cpp */*.cpp */*.h -o $(podir)/my_program.pot

Translating Data

If your application uses data that you want the translators to translate for you, there are a couple of solutions. If the data is a single string, and you want to use a translation of the string that is different from the user's desktop language setting, you can create a .desktop file and extract the translated string from it. For example, the KDE text-to-speech daemon (KTTSD) has the following kcmkttsd_testmessage.desktop file:

[Desktop Entry]
Encoding=UTF-8
NoDisplay=true
Name=The text-to-speech system seems to be functioning properly.

Notice the NoDisplay=true line. The translators will translate the Name line and scripty will insert the translations back into the .desktop file. At runtime, KTTSD uses the following code to get a test message to speak in the language of the synthesizer's voice, not the user's desktop language:

QString key = "Name[" + languageCode + "]";
QString result;
QString def;
QFile file(locate("data", "kttsd/kcmkttsd_testmessage.desktop"));
if (file.open(IO_ReadOnly))
{
  QTextStream stream(&file);
  stream.setEncoding(QTextStream::UnicodeUTF8);
  while ( !stream.atEnd() ) {
    QString line = stream.readLine(); // line of text excluding '\n'
    QStringList keyAndValue = QStringList::split("=", line);
    if (keyAndValue.count() == 2)
    {
      if (keyAndValue[0] == key)
      {
        result = keyAndValue[1];
        break;
      }
      // Use English default if not found.
      if (keyAndValue[0] == "Name") def = keyAndValue[1];
    }
  }
  file.close();
}
if (result.isEmpty()) {
  result = def;
}
return result;

If your program has larger amounts of data, an XML file is a good way to go. For example, KTTSD maintains a list of Festival synthesizer voices in an XML file called voices. Here's a sample portion from that file:

<?xml version="1.0" encoding="UTF-8"?>
<voices>
<voice>
  <syntaxhighlight lang="text">kal_diphone

 <language>en_US</language>
 <codec>ISO 8859-1</codec>
 <gender>male</gender>
 <name>American Male</name>

</voice> </syntaxhighlight>

Warning

This section needs improvements: Please help us to

cleanup confusing sections and fix sections which contain a todo

The Makefile.am extracts the name strings from this file like this:

messages: rc.cpp
    $(EXTRACTRC) */*.rc */*/*.rc >> rc.cpp
    $(EXTRACTRC) */*.ui */*/*.ui >> rc.cpp
    $(EXTRACTRC) --tag=name --context=FestivalVoiceName plugins/festivalint/voices >> rc.cpp
    $(XGETTEXT) rc.cpp */*.cpp */*.h */*/*.cpp */*/*.h -o $(podir)/kttsd.pot

Notice that the translators are given a context to help them to translate the voice names. At runtime, KTTSD parses the voices file and displays the translated name of each voice (in the user's desktop language) using code like this:

QDomNode childNode = node.namedItem("name");
if (!childNode.isNull())
  QString name = i18n("FestivalVoiceName", childNode.toElement().text().utf8());

Examining .po Files

You can look at the translated .po files to see if a string is present and has been translated. For example, here's a portion of l10n/de/messages/kdebase/konqueror.po:

#: konq_mainwindow.cc:3707 konq_tabs.cc:84
msgid "&New Tab"
msgstr "Neues &Unterfenster"

The string to be translated is given on the msgid line. The translated string is on the msgstr line. Notice that the exact same string to translate came from konq_mainwindow.cc and also from konq_tabs.cc. This saves the translators from having to translate the same identical string twice.

You can also use KBabel to examine .po files. KBabel is part of the kdesdk module.

Coding in a Foreign Language

If you are not comfortable with the English language, you may have coded your application using strings in your native language. You must convert all your code to English before it can be translated by the translation teams.

One solution is to get someone who understands both English and your language to help you with this. You can also ask for a volunteer on the kde-i18n-doc mailing list. You will create your .pot file (see Distributing message catalogs [*]) and send it to them. They will translate it to English and send you back a .po file. Using cut and paste, put the English strings into your code.

If you do this, it is still a good idea to have an English speaker review your code for any mistakes. You can use poreviewtool, a tool which makes the review of the English strings easier (relays also on .po files, but makes the handling much easier if you want to review continuously.)

Merging .po Files

If you made the mistake of creating a single .pot file for all the components in my application and the translators have already created the .po files what should you do so that no work (and translations) are lost ?

Correct your code and the build system. Commit the changes to the repository. Ask one of the KDE admins to merge the existing .po files. She will need to know the exact names of all the .po files, which module they are in, and the name of the desired merged .po file. The admin will have to wait for scripty to generate your new .pot file before doing the merge.

Deleting obsolete .pot files

If you change the build for you application so that it no longer generates a .pot file, or generates a .pot file with a different name, the old .pot file will not automatically disappear from the code repository.

You must delete it yourself from the templates folder in the l10n module. You should also send an email to the kde-i18n-doc mailing list advising translators that they may delete the .po file. If you do not do this, the translators will waste their time working on an obsolete file.

Handbook entity errors

Sometimes, when building Handbooks in the l10n/xx/docs folders, you may get entity errors. First of all, make sure your kdelibs/kdoctools folder is up-to-date and installed. If that doesn't fix the problem, it may be that the translator made an error, leaving an entity in the .docbook file that is defined in English but not in the translated language. In many cases, the translators haven't finished the translation. Contact the translation team.

For example, the English entity eg is defined in the kdelibs/kdoctools/customizations/en/user.entities file but is not defined in the German kdelibs/kdoctools/customizations/de/user.entities. It must be replaced by the translator with the entity zb (Zum Beispiel).