Translations:Development/KDevelop-PG-Qt Introduction/29/en
- converting keywords and chars with special meanings to tokens
- converting literals and identifier to tokens
- clean out anything that doesn't change the semantics, e.g. comments or whitespace (the latter of course not in Python)
- while doing the above, handling character encoding (we recommend using UTF8 as much as possible)