Translations:Development/KDevelop-PG-Qt Introduction/29/en

From KDE TechBase
  • converting keywords and chars with special meanings to tokens
  • converting literals and identifier to tokens
  • clean out anything that doesn't change the semantics, e.g. comments or whitespace (the latter of course not in Python)
  • while doing the above, handling character encoding (we recommend using UTF8 as much as possible)