Development/KDevelop-PG-Qt Introduction - Revision history

AnneW at 13:47, 18 July 2012

2012-07-18T13:47:03Z

The User: /* Further Configuration and Output */ it is lxBEGIN_IDX

2012-05-06T13:43:36Z

‎Further Configuration and Output: it is lxBEGIN_IDX

The User: /* Chomsky Type-2 Grammars */

2012-02-06T15:27:15Z

‎Chomsky Type-2 Grammars

AnneW: Marked this version for translation

2012-01-23T14:53:20Z

Marked this version for translation

The User: /* Defining available Tokens */ -TODO

2012-01-15T04:06:25Z

‎Defining available Tokens: -TODO

The User: /* Further Configuration and Output */

2012-01-15T03:35:36Z

‎Further Configuration and Output

The User: Lexer features, removed unneccessary semicola

2012-01-15T03:20:40Z

Lexer features, removed unneccessary semicola

Show changes

AnneW: /* Weblinks */

2011-12-22T19:57:54Z

‎Weblinks

AnneW: Marked this version for translation

2011-12-06T16:13:48Z

Marked this version for translation

Show changes

AnneW at 16:13, 6 December 2011

2011-12-06T16:13:17Z

← Older revision		Revision as of 13:47, 18 July 2012
Line 1:		Line 1:
−	~~{{Template:I18n/Language Navigation Bar\|Development/KDevelop-PG-Qt Introduction}}~~
−
−	~~{{Warning\|1=''Note to Translators'' <br /> Please do not translate using the old-style language bar, as changes in the master page are not tracked by the old system}}~~

	<languages />		<languages />

@@ Line 248: / Line 248: @@
 lxLENGTH    // length of the current lexeme (as presented in the input)
 lxBEGIN_POS // position of the first character of the current lexeme
-lxBEGIN_ID  // corresponding index
+lxBEGIN_IDX  // corresponding index
 lxNAMED_TOKEN(⟨name⟩, ⟨type⟩) // create a variable representing named ⟨name⟩ with the token type ⟨type⟩
 lxTOKEN(⟨type⟩)               // create such a variable named “token”

@@ Line 274: / Line 274: @@
 == How to write Grammar-Files  == <!--T:38-->
-=== Chomsky Type-2 Grammars  === <!--T:39-->
+=== Context-Free Grammars  === <!--T:39-->
 <!--T:40-->
-'''KDevelop-PG-Qt''' uses so called [http://en.wikipedia.org/wiki/Chomsky_hierarchy Type-2-grammars] use a concept of non-terminals (nodes) and terminals(tokens). While writing the grammar for the basic structure of your language, you should try to mimic the semantics of the language. Lets take a look at an example:
+'''KDevelop-PG-Qt''' uses so called [http://en.wikipedia.org/wiki/Context-free_grammars context-free grammars] using a concept of non-terminals (nodes) and terminals(tokens). While writing the grammar for the basic structure of your language, you should try to mimic the semantics of the language. Lets take a look at an example:
 <!--T:41-->

@@ Line 107: / Line 107: @@
 '''KDevelop-PG-Qt''' can generate lexers being well integrated into its architecture (you do not have to create a token-stream-class invoking lex or something like that). See examples/foolisp in the code for a simplistic example, there is also an incomplete PHP-Lexer for demonstration purposes.
-<!--T:33-->
+==== Regular Expressions ==== <!--T:33-->
 ==== Regular Expressions ====
 Regular expressions are used to write rules using the KDevelop-PG-Qt, we use the following syntax (α and β are arbitrary regular expressions, a and b characters):
 *α|β accepts any word accepted by α or accepted by β
@@ Line 127: / Line 126: @@
 *{⟨name⟩} accepts any word accepted by the regex named ⟨name⟩
 All regular expressions are case sensitive. Sorry, there is currently no way for insensitivity.
-==== Known Escape Sequences ====
+==== Known Escape Sequences ==== <!--T:179-->
 There are several escape sequences which can be used to encode special characters:
 *\n, \t, \f, \v, \r, \0, \b, \a like in C
@@ Line 137: / Line 137: @@
 *\y, \Y followed by binary digits, same, but in binary representation
-==== Predefined named regex ====
+==== Predefined named regex ==== <!--T:180-->
 Some regexes are predefined and can be used using braces {⟨name⟩}. They get imported from the official Unicode data, some important regexes:
 *{alphabetic} any alphabetic character
@@ Line 147: / Line 147: @@
 *{math}
-==== Rules ====
+==== Rules ==== <!--T:181-->
 Rules can be written as:
 {{Input|1=
@@ Line 153: / Line 153: @@
 }}
 Then the Lexer will generate the token TOKEN for lexemes matching the given regular expression. Which token will be chosen if there are multiple options? We use the ''first longest match'' rule: It will take the longest possible match (eating as many characters as possible), if there are multiple of those matches, it will take the first one.
 Rules can perform code actions and you can also omit tokens (then no token will be generated):
 {{Input|1=
@@ Line 162: / Line 164: @@
 }}
 There is rudimentary support for ''lookahead'' and so called (our invention) ''barriers'':
 {{Input|1=
@@ Line 169: / Line 172: @@
 The first rule will only accept words if they match the first regular expression and are followed by anything matching the expression specified using %la. The second rule will accept words matched by the first regular expression but will never run into a character sequence matching the regex specified by %ba. However, currently only rules with fixed length are allowed in %la and %ba (for example foo|bar, but not qux|garply). When applying the “first longest match” rule the %la/%ba expressions count, too.
 You can create your own named regexes using an arrow:
 {{Input|1=
@@ Line 175: / Line 179: @@
 The first character of the identifier should not be upper case.
 Additionally there are two special actions:
 {{Input|1=
@@ Line 181: / Line 186: @@
 }}
 %fail will stop tokenization. %continue will make the matched characters part of the next token.
-==== Rulesets ====
+==== Rulesets ==== <!--T:188-->
 A grammar file can contain multiple ''rulesets''. A ruleset is a set of rules, as described in the previous section. It gets declared using:
 {{Input|1=
@@ Line 192: / Line 199: @@
 }}
 For your main-ruleset you omit the name (the name will be “start”).
 Usually the start-ruleset will be used. But you can change the ruleset in code actions using the macro lxSET_RULE_SET(⟨name⟩). You can specify code to be executed when entering or leaving a ruleset by using %enter [: ⟨code⟩ :]; or %leave [: ⟨code⟩ :]; respectively inside the definition of the ruleset.
-==== Further Configuration and Output ====
+==== Further Configuration and Output ==== <!--T:192-->
 The standard statements %lexer_declaration_header and %lexer_bits_header are available to include files in the generated lexer.h/lexer.cpp.
 By using %lexer_base you can specify the baseclass for the lexer-class, by default it is the TokenStream class defined by KDevelop-PG-Qt.
 After %lexerclass(bits) you can specify code to be inserted in lexer.cpp.
 You have to specify an encoding the lexer should work with internally using %input_encoding "⟨encoding⟩", possible values:
 *ASCII (7-Bit)
@@ Line 211: / Line 223: @@
 *UTF-32 (32-Bit, full Unicode)
 With %input_stream you can specify which class the lexer should use to get the characters to process, there are some predefined classes:
 *QStringIterator, reads from QString, required (internal) encoding: UTF-16 or UCS-2
@@ Line 220: / Line 233: @@
 *QUtf8ToAsciiIterator, reads from UTF-8 QByteArray, will ignore all non-ASCII characters, reqired encoding: ASCII
 Whether you choose UTF-8, UTF-16 or UTF-32 is irrelevant for functionality, but it may significantly affect compile-time and run-time performance (you may want to test your Lexer with ASCII if compilation takes too long). For example you want to work with a QByteArray containing UTF-8 data, and you want to get full Unicode support: You could either use the QByteArrayIterator and UTF-8 as internal encoding, or the QUtf8ToUtf16Iterator and UTF-16, or the QUtf8ToUcs4Iterator and UTF-32.
 You can also choose between %table_lexer; and %sequence_lexer; In the first case transitions between states of the lexer will get represented by big tables while generating the lexer (cases in the generated code). In the second case sequences will get stored in a compressed data structure and transitions will get represented by nested if-statements. For UTF-32 %table_lexer is infeasible, thus there %sequence_lexer is the onliest option.
 Inside your actions of the lexer you can use some predefined macros:
 {{Input|1=

@@ Line 379: / Line 379: @@
 <!--T:67-->
-'''TODO: explain process of writing Lexer/Tokenizer and using the parser Tokens'''
+'''TODO: explain process of using the parser Tokens'''
 === Special Syntax...  === <!--T:68-->

@@ Line 220: / Line 220: @@
 *QUtf8ToAsciiIterator, reads from UTF-8 QByteArray, will ignore all non-ASCII characters, reqired encoding: ASCII
-Whether you chose UTF-8, UTF-16 or UTF-32 is irrelevant for functionality, but it may significantly affect compile-time and run-time performance (you may want to test your Lexer with ASCII if compilation takes too long). For example you want to work with a QByteArray containing UTF-8 data, and you want to get full Unicode support: You could either use the QByteArrayIterator and UTF-8 as internal encoding, or the QUtf8ToUtf16Iterator and UTF-16, or the QUtf8ToUcs4Iterator and UTF-32.
+Whether you choose UTF-8, UTF-16 or UTF-32 is irrelevant for functionality, but it may significantly affect compile-time and run-time performance (you may want to test your Lexer with ASCII if compilation takes too long). For example you want to work with a QByteArray containing UTF-8 data, and you want to get full Unicode support: You could either use the QByteArrayIterator and UTF-8 as internal encoding, or the QUtf8ToUtf16Iterator and UTF-16, or the QUtf8ToUcs4Iterator and UTF-32.
 === Using Flex === <!--T:34-->

@@ Line 712: / Line 712: @@
 <!--T:177-->
 [[Category:Getting_Started]]
 </translate>

@@ Line 1: / Line 1: @@
 {{Warning|1=''Note to Translators'' <br />  Please do not translate using the old-style language bar, as changes in the master page are not tracked by the old system}}
-{{Template:I18n/Language Navigation Bar|Development/KDevelop-PG-Qt Introduction}}
+<languages />
 == Preface ==
@@ Line 574: / Line 577: @@
 * [http://the-user.org/category/computing/kde/kdev-pg-qt Some blog-posts by “The User” about KDevelop-PG-Qt (informative for recently added features)]
 * [http://stud4.tuwien.ac.at/~e0127236/thesis/jpetso-javaparser.pdf Jakob Petsovits' bachelor thesis about using KDevelop-PG for Java Java-Parsers]- It is a good in-depth introduction to everything you might want to know for writing your own grammar. Keep in mind that it is partly outdated. In doubt, refer to this page for updated syntax. Also some of the shortcomings of KDevelop-PG layed out in the thesis have been fixed in the meantime.