Development/Tutorials/Localization/Unicode - Revision history

AnneW at 12:46, 13 July 2012

2012-07-13T12:46:26Z

Icwiener: Fix syntax highlighting.

2011-06-27T16:29:52Z

Fix syntax highlighting.

DrSlowDecay: Added language navigation

2007-12-15T17:05:00Z

Added language navigation

Pippin: /* File Handling */

2007-07-03T17:22:51Z

‎File Handling

Tampakrap: /* Sample Unicode Files and Fonts */

2007-05-21T23:13:19Z

‎Sample Unicode Files and Fonts

Tampakrap: improved wording

2007-05-21T23:08:25Z

improved wording

Tampakrap: improved inaccuracies, left excruriating details to external sites

2007-05-21T23:05:44Z

improved inaccuracies, left excruriating details to external sites

Tampakrap: improved wording, removed bad example

2007-05-21T22:45:20Z

improved wording, removed bad example

Tampakrap: removed duped content, corrected 16-bit nonsense, explanation for identic codepoints

2007-05-21T22:38:02Z

removed duped content, corrected 16-bit nonsense, explanation for identic codepoints

Tampakrap: Unicode is about scripts, not languages

2007-05-21T22:24:43Z

Unicode is about scripts, not languages

← Older revision		Revision as of 12:46, 13 July 2012
Line 1:		Line 1:
−	~~{{Template:I18n/Language Navigation Bar\|Development/Tutorials/Localization/Unicode}}~~

	{{TutorialBrowser\|		{{TutorialBrowser\|

@@ Line 42: / Line 42: @@
 Getting an appropriate Unicode QTextCodec is quite simple:
-<code cppqt n>QTextCodec Utf8Codec  = QTextCodec::codecForName("utf-8");
+<syntaxhighlight lang="cpp-qt">
-QTextCodec Utf16Codec = QTextCodec::codecForName("utf-16");</code>
+QTextCodec Utf8Codec  = QTextCodec::codecForName("utf-8");
 == Display ==
@@ Line 55: / Line 57: @@
 To read Unicode data from an in-memory buffer, such as a QByteArray, one can use the codecs created earlier in this section:
-<code cppqt n>
+<syntaxhighlight lang="cpp-qt">
 QByteArray buffer;
@@ Line 62: / Line 64: @@
 // UTF-8
-QString string = Utf8Codec->toUnicode(buffer);</code>
+QString string = Utf8Codec->toUnicode(buffer);
 Writing to a buffer looks very similar:
-<code cppqt n>
+<syntaxhighlight lang="cpp-qt">
 QString string; // my Unicode text
 QByteArray array; // Buffer to store UTF-16 Unicode
@@ Line 72: / Line 75: @@
 textStream.setEncoding(QTextStream::Unicode);
-textStream << string;</code>
+textStream << string;
 == File Handling ==
@@ Line 80: / Line 84: @@
 Reading Unicode data from a text file is demonstrated in the following example using the QTextCodecs we created earlier for UTF-8 data:
-<code cppqt n>QTextStream textStream;
+<syntaxhighlight lang="cpp-qt">
 QString line;
@@ Line 90: / Line 95: @@
 // UTF-8
 textStream.setCodec(Utf8Codec);
-line = textStream.readLine();</code>
+line = textStream.readLine();
 Writing is equally straight-forward, also using the QTextCodecs we created earlier:
-<code cppqt n>QTextStream textStream;
+<syntaxhighlight lang="cpp-qt">
 QString line;
@@ Line 103: / Line 110: @@
 // UTF-8
 textStream.setCodec(Utf8Codec);
-textStream << line;</code>
+textStream << line;
 == Unicode Character Input ==
@@ Line 113: / Line 121: @@
 An easy way to get Unicode sample data is to take the kde-i18n package and convert some files from there to Unicode using the <tt>recode</tt> command line tool:
-<code>recode KOI8-R..UTF-8      ru/messages/kdelibs.po
+<syntaxhighlight lang="bash">
 recode ISO-8859-1..UTF-16 de/messages/kdelibs.po
-recode ISO-8859-3..UTF-8  eo/messages/kdelibs.po</code>
+recode ISO-8859-3..UTF-8  eo/messages/kdelibs.po
 The majority of operating systems today come packaged with Unicode TrueType fonts, see [http://en.wikipedia.org/wiki/Unicode_font#List_of_Unicode_fonts].
 [[Category:Programming]]

← Older revision		Revision as of 17:05, 15 December 2007
Line 1:		Line 1:
		+	{{Template:I18n/Language Navigation Bar\|Development/Tutorials/Localization/Unicode}}
		+
	{{TutorialBrowser\|		{{TutorialBrowser\|

@@ Line 74: / Line 74: @@
 == File Handling ==
-For storage and retrieval to and from a file KDE applications should provide the option to store text data in Unicode instead of the locale character set. This way the user of your application can read and save documents with the locale charset as the default while having the option to store it in Unicode as an option. Both UTF-8 and UTF-16 should be supported. European users will generally prefer UTF-8 while Asian users may prefer UTF-16.
+For storage and retrieval to and from a file KDE applications should provide the option to store text data in Unicode instead of the locale character set. This way the user of your application can read and save documents with the locale charset as the default while having the option to store it in Unicode. Both UTF-8 and UTF-16 should be supported. European users will generally prefer UTF-8 while Asian users may prefer UTF-16.
 Reading Unicode data from a text file is demonstrated in the following example using the QTextCodecs we created earlier for UTF-8 data:

@@ Line 115: / Line 115: @@
 recode ISO-8859-3..UTF-8  eo/messages/kdelibs.po</code>
-The majority of operating systems today come packaged with TrueType fonts that cover the full spectrum of the Unicode character range. If you do not have an Unicode fonts on your system there are several that can be downloaded from the Internet and a quick Internet search should provide links to various options.
+The majority of operating systems today come packaged with Unicode TrueType fonts, see [http://en.wikipedia.org/wiki/Unicode_font#List_of_Unicode_fonts].
 [[Category:Programming]]

@@ Line 36: / Line 36: @@
 Storing Unicode strings in memory and displaying Unicode on screen is very easy with Qt: QString and QChar provide full Unicode support transparently to your application.
-QString and QChar both internally represent characters using their 16bit value. If you read in text pieces from or write it to an 8bit source such as QTextStream you must use a QTextCodec to convert the text from or to its 16bit representation. Normally you would use the UTF-8 codec to convert a Unicode string between a QByteArray and a QString.
+QString and QChar both internally represent characters using UTF-16. If you read in text pieces from or write it to an 8-bit source such as QTextStream you must use a QTextCodec to convert the text between representations. Normally you would use the UTF-8 codec to convert a Unicode string between a QByteArray and a QString.
 Getting an appropriate Unicode QTextCodec is quite simple:

@@ Line 26: / Line 26: @@
 == UTF-8 and UTF-16 ==
-UTF stands for "Unicode Transfer Format". The two variants, UTF-8 and UTF-16, define how to express unicode characters in bits.
+UTF stands for "Unicode Transformation Format". The two most commonly used variants, UTF-8 and UTF-16, define how to express a Unicode character's assigned codepoint in bits.
-UTF-16 signifies that every character is represented by the 16bit value of its Unicode number. For example, the Latin-1 characters in UTF-16 have a hex representation of 00nn where nn is the hexadecimal representation in Latin-1.
+UTF-16 signifies that every character is represented by the 16-bit value of its Unicode number. For example, in UTF-16 <tt>b</tt> (LATIN SMALL LETTER B) has a hex representation of 0x0062. There are also surrogate pairs used to encode characters with codepoints beyond 0xFFFF.
-UTF-8 signifies that the Unicode characters are represented by a stream of bytes. An 8-bit value whose value lies between 0 to 127 coresponds to an ASCII character. Other characters are represented by more than one byte. Because the characters "\0" and "/" cannot occur in a multibyte character in UTF-8, you can treat UTF-8 strings as null-terminated C-Strings.
+UTF-8 signifies that the Unicode characters are represented by a stream of bytes in a special encoding scheme described in [http://www.cl.cam.ac.uk/~mgk25/ucs/utf-8-history.txt] and [http://tools.ietf.org/html/rfc3629]. Unlike UTF-16, it is compatible to ASCII. Higher codepoints take two or three bytes to encode, rarely more than that.
-−
-Below is a simple depiction showing how the bits of Unicode character codes are put into UTF-8 bytes:
-−
-{|
-|+ UTF-8 Representations
-|-
-! Bytes !! Usable Bits !! Representation
-|-
-| 1 || 7 || <tt>0vvvvvvv</tt>
-|-
-| 2 || 11 || <tt>110vvvvv 10vvvvvv</tt>
-|-
-| 3 || 16 || <tt>1110vvvv 10vvvvvv 10vvvvvv</tt>
-|-
-| 4 || 21 || <tt>11110vvv 10vvvvvv 10vvvvvv 10vvvvvv</tt>
-|}
 == Unicode in KDE Applications ==

@@ Line 20: / Line 20: @@
 == Why use Unicode? ==
-If you use a local 8-bit character set you can only create single language documents and interchange them only with people using the same charset. Even then, many languages contain more symbols than can be uniquely address by an 8-bit number.
+If you use a legacy character set like ISO 8859-15 it is difficult to create documents containing more than only a couple of languages. You can interchange them only with people using the same character set. Even then, many writing systems contain more characters than can be uniquely addressed by an 8-bit number.
-Therefore Unicode is the only sensible way to mix languages in a document and to interchange documents between people with different locales. Unicode makes it possible to do things such as easily write a Russian-Hungarian dictionary or store documents in Asian languages which may have thousands of possible symbols.
+Therefore Unicode is the best of the sensible ways to mix different scripts and languages in a document and to interchange documents between people with different locales. Unicode makes it possible to do things such as writing a Russian-Hungarian dictionary.
 == UTF-8 and UTF-16 ==

@@ Line 16: / Line 16: @@
 == What Is Unicode? ==
-Unicode is a standard that provides a unique number for every character and symbol in the various languages used around the world. It is hardware, development platform and language independent. The current assigned codes are all within the range of a 16-bit integer with the first 256 entries being identical to [http://en.wikipedia.org/wiki/ISO_8859 ISO 8859 or Latin-1]. This also means that the first 128 characters are identical to the familiar ASCII mapping.
+Unicode is hardware, development platform and language independent. As of version 5 there are codepoints assigned up to 0x10FFFF (but not all are used), the first 256 entries being identical to [http://en.wikipedia.org/wiki/ISO_8859-1 ISO 8859-1 a.k.a. ISO Latin-1]. This also means that the first 128 characters are identical to the familiar [http://en.wikipedia.org/wiki/ISO_646 ASCII] mapping. Codepoints were assigned with ease of implementation for legacy character sets in mind, so Unicode is littered with such coincidences to keep translation tables and logic short.
 == Why use Unicode? ==

@@ Line 12: / Line 12: @@
 == Abstract ==
-Unicode is a standard that provides a unique number for every character and symbol in the various languages used around the world which makes it the basis for translatable software and international content creation. This tutorial provides a brief introduction to Unicode and how to work with Unicode data using Qt.
+Unicode is a standard that provides a unique number for every character and symbol in various writing systems used around the world which makes it the basis for translatable software and international content creation. This tutorial provides a brief introduction to Unicode and how to work with Unicode data using Qt.
 == What Is Unicode? ==