Getting Started/Build/Windows/emerge: Difference between revisions

    From KDE TechBase
    No edit summary
    (→‎Git client: Linking to the official, up-to-date Windows command line client)
    (113 intermediate revisions by 21 users not shown)
    Line 1: Line 1:
    ''emerge'' is a tool to build the KDE sources and its third party requirements on MS Windows. It is the '''easy''' way to build KDE on MS Windows.
    ''emerge'' is a tool to build the KDE sources and its third-party requirements on MS Windows. It is the '''easy''' way to build KDE on MS Windows.


    == Introduction ==
    == Introduction ==
    Emerge is a tool that can build the different parts of KDE and its dependencies under windows. We created this tool to automate and simplify the build process under windows. We try to build all packages that we offer in the KDE installer with emerge. That has some advantages for us:
    Emerge is a tool that can build the different parts of KDE and its dependencies under Windows. We created this tool to automate and simplify the build process under Windows. We try to build all packages that we offer in the KDE installer with emerge. That has some advantages for us:
    * it is easy for people to join us:
    === Easy for people to join us ===
    Before emerge it was quite some work to set a system up for development. There were some quirks, which were documented in some mailing lists, but you had to remember them or you ran into an already solved problem again, etc.
    Before emerge it was quite some work to set a system up for development. There were some quirks, which were documented in some mailing lists, but you had to remember them or you ran into an already solved problem again, etc.
    Now to get a development machine you need a windows computer, need to install python and subversion and do the emerge checkout. Then execute emerge to build what you want to build. This is easy for developers coming from windows to KDE, and also for KDE developers coming to windows.
    Now to get a development machine you need a Windows computer, need to install Python and Git and do the emerge checkout. Then execute emerge to build what you want to build. This is easy for developers coming from Windows to KDE, and also for KDE developers coming to Windows.
    * it is easy for us to do (nightly/continuous/release/reproducable/...) builds:
    === Easy for us to do build ===
    With emerge you can build the whole software stack (lowlevel libs, qt, kdelibs, things above that) with only one command. You can start that build, and some hours later you can check if it worked, or if something broke. So we can spot problems easier and earlier. We can also start with a "naked" windows computer without any other installed software and bootstrap kde on it. That ensures, that no hidden dependencies on some pieces of software sneak in, because then the builds on a "naked" computer would break and show the problem.
    With emerge you can build the whole software stack (low-level libs, Qt, kdelibs, things above that) with only one command. You can start that build, and some hours later you can check if it worked, or if something broke. So we can spot problems easier and earlier. We can also start with a "naked" Windows computer without any other installed software and bootstrap kde on it. That ensures, that no hidden dependencies on some pieces of software sneak in, because then the builds on a "naked" computer would break and show the problem.
    * it is easier to collaborate:
    === Easy to collaborate ===
    We can test the same emerge build description for a package on different windows versions/computers before we do binary releases. People can also add build descriptions for new packages to the subversion repository.
    We can test the same emerge build description for a package on different Windows versions/computers before we do binary releases. People can also add build descriptions for new packages to the Subversion repository.


    This emerge tool was inspired by the Gentoo emerge tool.
    This emerge tool was inspired by the Gentoo emerge tool.
    Line 15: Line 15:
    == Set up the environment ==
    == Set up the environment ==
    === Root directory ===
    === Root directory ===
    Create a directory if possible in your harddrive's root e.g. C:\kderoot or D:\kderoot (You will need this PATH later). This directory will contain the whole kde installation later. We will refer to it as %KDEROOT%.
    Create a directory if possible in your harddrive's root e.g. C:\kderoot or D:\kderoot (You will need this PATH later). This directory will contain the whole kde installation later. We will refer to it as KDEROOT.


    === Python interpreter ===
    === Python interpreter ===
    <tt>emerge.bat</tt> invokes an <tt>emerge.py</tt> script written in [http://en.wikipedia.org/wiki/Python_%28programming_language%29 Python] programming language, so you first need to [http://www.python.org/download/ install a Python Interpreter]. The ''python'' installation directory will be added to the PATH later by <tt>%KDEROOT%\etc\kdesettings.bat</tt> script.
    <tt>emerge.bat</tt> invokes an <tt>emerge.py</tt> script written in [http://en.wikipedia.org/wiki/Python_%28programming_language%29 Python] programming language, so you first need to [https://www.python.org/downloads/ install the Python 3.4 Interpreter]. The ''python'' installation directory will be added to the PATH later by <tt>KDEROOT\etc\kdesettings.ini</tt>.


    === Subversion client ===
    === Git client ===
    The latest source code for windows ''emerge'' and the rest of KDE is stored in a repository created and managed using the [http://subversion.tigris.org/ Subversion] version control tool. You need a Subversion client for the first checkout. There are at least two applications:
    The latest source code for Windows ''emerge'' and a lot of the rest of KDE is stored in a repository created and managed using the [http://git-scm.com/ Git] version control tool. You need a git client for the first checkout of emerge. There are at least two applications:
    *a command line client, available at [http://subversion.tigris.org/servlets/ProjectDocumentList?folderID=91 subversion.tigris.org] '''(required by emerge to get the source code from KDE Subversion repository)''', aimed at developers or power users accustomed with the command line,
    *a command line client, available at [https://git-scm.com/download/win git-scm.com], aimed at developers or power users accustomed with the command line,
    *GUI program like [http://tortoisesvn.tigris.org/ TortoiseSVN], optional, useful for displaying differences between various versions of files in the repository in a graphical way.
    *a GUI program like [https://tortoisegit.org/ TortoiseGit], optional, useful for displaying differences between various versions of files in the repository in a graphical way.


    Note 1: If you experience problems with the checkout of Qt (subversion doesn't work correctly) please remove any other subversion binaries out of the path that you do have. '''The different versions of the Apache portable runtime (APR) are incompatible!'''
    Emerge uses its own git client for checking out all KDE source code. You can find it in KDEROOT\dev-utils\git\bin. It will be used by emerge even if you have another git executable installed.


    Note 2: Make sure to use a copy of Subversion that was built on Windows so that checked-out files do not use UNIX line endings. If you check out with UNIX line endings, the ''patch'' program will fail when attempting to apply a patch whose line endings don't match the system's.
    === Getting the ''emerge'' tool ===
    The source code of the ''emerge'' tool and the recipes for creating KDE packages are located at <tt>git://anongit.kde.org/emerge.git</tt>, which is an URL based on the git-specific ''git'' protocol.


    === Check out the ''emerge'' tool ===
    You need to clone the source code from the ''emerge'' git repository into a new directory below your root directory (the root directory is normally called %KDEROOT% here) or get it inside a self extracting archive (not tested yet).
    The source code of the ''emerge'' tool and the recipes for creating KDE packages are located at <tt>svn://anonsvn.kde.org/home/kde/trunk/kdesupport/emerge</tt>, which is an URL based on Subversion-specific ''svn'' protocol.


    You need to check out the source code from the ''emerge'' Subversion directory into a new directory, which in this example we will call %KDEROOT%.
    ==== Check out using the 'git' command ====


    ==== Check out using the 'svn' command ====
    With the ''git'' command line tool, you can accomplish this with the following commands:
    :: <pre>cd KDEROOT</pre>
    *if you will only use anonymous (read-only) access to the KDE git repository:<pre>git clone git://anongit.kde.org/emerge.git</pre>
    *if the git and ssh ports are blocked, and only http is allowed, it is better to clone over the http port for getting (read-only) access to the KDE git repository:<pre>git clone http://anongit.kde.org/emerge.git</pre>
    *or, if you plan to use write access (commit) to the emerge repository using your existing account & OpenSSH private key: <pre>git clone [email protected]:emerge.git</pre>


    *Option 1: With the ''svn'' command line tool, you can accomplish this with the following commands:<pre>cd %KDEROOT%</pre>
    === Configure the ''emerge'' tool ===
    **if you will only use anonymous (read-only) access to the KDE svn repository:<pre>svn co svn://anonsvn.kde.org/home/kde/trunk/kdesupport/emerge</pre>
    # Create the directory <tt>KDEROOT\etc</tt>.
    **or, if you plan to use write access (commit) to the KDE svn repository
    # Copy the file <tt>KDEROOT\emerge\kdesettings.ini</tt> as <tt>KDEROOT\etc\kdesettings.ini</tt> and change its contents according to your needs.
    ***via https:<pre>svn co --username yourusername https://svn.kde.org/home/kde/trunk/kdesupport/emerge</pre>
    ***via a [[Getting_Started/Build/KDE4/Windows/subversion|puTTY tunnel]] using your existing account & OpenSSH private key: <pre>svn co svn+putty://svn.kde.org/home/kde/trunk/kdesupport/emerge</pre>


    Note 0: '''Read the comments in that file very carefully'''


    This would result with:
    Note 1: '''Be sure that you neither have the msys/bin nor the cygwin/bin in your path. If so you have to definitely remove it from the path.'''


    Error validating server certificate for '<nowiki>https://svn.kde.org:443</nowiki>':
    Note 2 '''from a user: The applications gimp, inkscape and graphviz are also a problem. To make sure that there's nothing wrong I stripped my path to contain only what I needed to build.'''
      - The certificate is not issued by a trusted authority. Use the
        fingerprint to validate the certificate manually!
    Certificate information:
      - Hostname: svn.kde.org
      - Valid: from Wed, 11 May 2005 09:08:21 GMT until Sat, 09 May 2015 09:08:21 GMT
      - Issuer: SVN, KDE e.V., Nuernberg, Bavaria, DE
      - Fingerprint: xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx
    (R)eject, accept (t)emporarily or accept (p)ermanently?


    enter ''p'' here to permanently accept the certificate:
    emerge has several branches which contain specific package versions. E.g. if you want to build the 4.13 branch of KDE (or one of the 4.13 releases) you should checkout the ''kde-4.13'' branch of emerge; in the ''master'' branch, you can find the KDE Frameweorks 5.


      Authentication realm: <<nowiki>https://svn.kde.org:443</nowiki>> KDE SVN account
      To view all branches, use the following command:
    Password for 'yourusername': ***************
    <pre>git branch -a</pre>
    A    emerge\kdeenv.bat
    To change the branch of emerge, do the following:
    A    emerge\portage
    <pre>cd emerge && git checkout kde-4.13</pre>
    A    emerge\portage\kdesupport
    [....]


    The password and cache for the certificates is saved in <tt>%APPDATA%\Subversion\auth</tt> directory.
    === [Optional, Advanced] emerge-boost-config.jam ===
    Add a file emerge-boost-config.jam in the <tt>%KDEROOT%\etc</tt> directory to build boost in a specific way. The only current use case is for the following problem:
    If you build 32bit binaries with emerge on a 64bit system you need to link boost-python against a 32bit python library. In case your standard python installation is 64bit though, you can specify the python installation in the following way:
    <syntaxhighlight lang="text">
    # ---------------------
    # Python configuration.
    # ---------------------


    ==== Check out using the TortoiseSVN ====
    # Configure specific Python version.
    using python : 3.2 : C:\\Python32_x86 ;
    </syntaxhighlight>
    This way boost-python would be using the headers & libraries from C:\\Python32_x86 instead of the default one.
    Please remember that for mingw compilers you must regenerate the import library for the python dll (also for the mingw 64bit compiler):
    <syntaxhighlight lang="text">
    R:\> emerge pexports
    R:\> pexports C:\Python32_x86\python32.dll > C:\Python32_x86\libs\python32.def
    </syntaxhighlight>
    NOTE: In case you generate the import library for the 64bit compiler, add the following line to the file python32.def:
    <syntaxhighlight lang="text">
    Py_InitModule4 = Py_InitModule4_64
    </syntaxhighlight>
    For both compilers you should now run:
    <syntaxhighlight lang="text">
    dlltool -d C:\Python32_x86\libs\python32.def -l C:\Python32_x86\libs\libpython32.dll.a
    </syntaxhighlight>
    Now you should be able to do emerge -i boost-python-src without errors.


    *Option 2: If you use TortoiseSVN:
    == Using emerge ==
    *#right-click on your %KDEROOT% folder and select ''SVN Checkout...'' command from the context menu,
    *#paste <tt>svn://anonsvn.kde.org/home/kde/trunk/kdesupport/emerge</tt> URL into the ''URL of repository'' text box (replace with <tt><nowiki>https://svn.kde.org/home/kde/trunk/kdesupport/emerge</nowiki></tt> for read-write access)
    *#add <tt>\emerge</tt> to the folder name in the ''Checkout directory'' box and click OK to continue
    *#if you picked the read-write access, you will be asked for accepting the SSL certificate of the SVN server (click "Premanent") and then for username and password. For convenience select "Save authentication" checkbox too (the password and cache for the certificates will be saved in <tt>%APPDATA%\Subversion\auth</tt> directory).


    After the checkout you should have the directory <tt>%KDEROOT%\emerge</tt>. If you don't, you move your ''emerge'' directory to that location.
    To use emerge you need to start a Powershell window and point that to <tt>KDEROOT\emerge</tt>. For example:


    === Configure the ''emerge'' tool ===
    C:
    # Create the directory <tt>%KDEROOT%\etc</tt>.
    cd KDEROOT\emerge
    # Copy the file <tt>%KDEROOT%\emerge\kdesettings-example.bat</tt> as <tt>%KDEROOT%\etc\kdesettings.bat</tt> and change its contents according to your needs. The options are described in the ''rem'' lines in the file itself.


    The kdesettings.bat script will be called by the main kdeenv.bat script.
    Then in the windows powershell you have to load the script with


    Note 1: '''Be sure that you neither have the msys/bin nor the cygwin/bin in your path. If so you have to definitely remove it.'''
    ". .\kdeenv.ps1"


    Note 2 from a user: The applications gimp, inkscape and graphviz are also a problem. To make sure that there's nothing wrong I stripped my path to contain only what I needed to build.
    This tells emerge about your environment settings (e.g. paths). It will load your configuration from <tt>KDEROOT\etc\kdesettings.ini</tt>. It should not give any error messages, otherwise emerge will not work as expected. The output should look similar to this one (of course with your paths):


    == Running emerge ==
    PS C:\kderoot\emerge>.\kdeenv.ps1
    KDEROOT    : r:\
    KDECOMPILER : mingw-w64
    KDESVNDIR  : s:\
    KDEGITDIR  : q:\
    DOWNLOADDIR : t:\
    PYTHONPATH  : C:\python34
    PS C:\kderoot\emerge>


    To use emerge you need to start a console window and point that to <tt>%KDEROOT%\emerge</tt>. For example:
    ''Note: There is a short path option in kdesettings which you will need if you want to build Qt in a directory that has a pathlength of more then around 5 characters. This problem is due to limitations of the command line length and the Qt build system.''


    C:
    Next, if you have configured your kdesettings.ini to use svn+ssh for your subversion checkout, then you need to run:
    cd \%KDEROOT%\emerge


    Then you have to execute
    <syntaxhighlight lang="text">plink <your-svn-username>@svn.kde.org
    plink <your-kde-username>@git.kde.org</syntaxhighlight>


    kdeenv.bat
    This will prompt you to accept the fingerprint of the server, otherwise svn will hang forever when trying to download from the server.


    This tells emerge about your environment settings (e.g. paths). It will load your configuration from <tt>%KDEROOT%\etc\kdesettings.bat</tt>. It should not give any error messages, otherwise emerge will not work as expected. The output should look similar to this one (of course with your paths):
    Now you should be able to use emerge. To get some help on usage:


      C:\kderoot\emerge>kdeenv.bat
      emerge --help
    kdesettings.bat executed
    KDEROOT    : C:\kderoot
    KDECOMPILER : mingw
    KDESVNDIR  : C:\kderoot\svn
    PYTHONPATH  : C:\python25
    DOWNLOADDIR : C:\kderoot\download
     
    C:\kderoot\emerge>


    Now you should be able to use emerge. Type
    To get a list of available packages:
     
      emerge --print-installable
      emerge --help


    to get some help on usage.
    To get a list of currently installed packages:
    emerge --print-installed


    == Setting up a compiler ==
    == Setting up a compiler ==
    Currently emerge supports both the MinGW and MS Visual C++ (msvc) compilers. We did not add dependencies for the compilers, so you have to make sure to install a compiler by yourself. There are three ways to set up a compiler for emerge.
    Currently emerge supports both the MinGW and MS Visual C++ (msvc) compilers. We did not add dependencies for the compilers, so you have to make sure to install a compiler by yourself. There are three ways to set up a compiler for emerge.
    We assumed you have set KDECOMPILER variable properly in the <tt>%KDEROOT%\etc\kdesettings.bat</tt>.
    We assumed you have set KDECOMPILER variable properly in the <tt>KDEROOT\etc\kdesettings.ini</tt>.


    In the following sections you can find information on how to install or reuse an existing compiler.
    In the following sections you can find information on how to install or reuse an existing compiler.


    === Install the MinGW compiler with emerge ===
    === Install the MinGW compiler with emerge ===
    Let emerge install the MinGW compiler:
    Let emerge install the MinGW compiler, as soon as emerge needs MinGW it will automatically fetch the correct version for you.


    To install the MinGW ("Minimalist GNU for Windows") compiler with emerge, type
    === Install MS Visual C++ ===
    emerge mingw
    Read [[../MS_Visual_Studio#The_Compiler|here]].
    and wait until it is finished.  


    If you encounter an error like
    === Point to an existing MS Visual C++ installation ===
    Assertion failed: hunk, file ../patch-2.5.9-src/patch.c, line 354
    You need to point emerge to an existing msvc installation. This is run automatically for you from kdeenv.bat if configured properly in kdesettings.ini. Check your kdesettings.ini file to know where to set it.
    try to edit line 51 of file <tt>mingw-x.y.z.py</tt> (<tt>%KDEROOT%\emerge\portage\dev-util\mingw</tt>) by adding the <tt>--binary</tt> after <tt>-p1</tt> parameter. The line should then look like this:
    cmd = "cd %s && patch -p1 --binary < %s" % \
    This is probably because the different line break types (Linux vs Dos) in the files and the bug in patch.exe. Althought this is an ugly hack and should be fixed somewhere else, it works for the current versions of patch (2.5.9) and mingw (3.4.5).


    === Install MS Visual C++ ===
    == Installing the base system ==
    Read [[../MS_Visual_Studio#The_Compiler|here]].
    Once you have emerge and a compiler installed and working, try:
     
    You are now ready to start building KDE, it is recommended to do so progressively, relying on emerge to automatically resolve the required dependencies at each set step:
     
    * Enter <tt>emerge qt</tt>.  This will fetch and install Windows versions of numerous UNIX-like utilities and libraries, then checkout, compile and install Qt.  This will take several hours.
    * Enter <tt>emerge frameworks</tt>. This will checkout, compile and install the kde frameworks 5 modules.
    * Enter <tt>emerge kde-baseapps</tt>. This will checkout, compile and install kdebase-runtime, and kdebase-apps.


    === Point to an existing MinGW installation ===
    You will now have successfully installed a base KDE system and can now install other KDE modules as required.
    * Point emerge to an existing MinGW installation:


    '''This option is not recommended for now''', because it only adds one more point of failure, and does not gain something in comparison to the option above.<br/>NOTE from a user: be sure that path to \mingw\bin has been set correctly, by default it is pointing to: %KDEROOT%\mingw\bin which does not apply to most installations. If you see an error about <tt>cc1plus</tt> not being found, either add MinGW's <tt>\libexec\gcc\mingw32\3.4.5</tt> to your ''PATH'' (in command line <tt>set PATH=%PATH%;path\to\directory</tt>) variable or copy the contents of this directory to MinGW's <tt>bin</tt> directory.  The prior is preferred.<br/><br/>Under Vista, the mingw directory may need to be moved to c:\ in order to compile properly.
    Note that this will install the development version of KDE (trunk in svn repositories, master in git repositories), if you wish to install a particular stable branch then you should change the branch of emerge to the specific branch of that release. '''NOTE:''' You should not mix kde packages from different branches.


    === Point to an existing MS Visual C++ installation ===
    It is strongly recommended you do not choose to manually install any of the utilities and libraries yourself, as you may install the wrong version and cause installation failures.  Instead allow emerge to resolve the dependencies for you.
    You need to point emerge to an existing msvc installation. To do that, execute vcvarsall.bat before running <tt>%KDEROOT%\emerge\kdeenv.bat</tt>. In recent versions this is run automatically for you from kdeenv.bat if configured properly in kdesettings.bat. Check your kdesettings.bat file to know if you need to run this manually or not.


    Notes related to Vista:
    Every time you want to update or install a package, you should first update your emerge checkout (simply run emerge --update emerge) to ensure you are using the latest package recipies.
    *Note that for debug builds MS Visual Studio 2005 Service Pack 1 is required due to the use of manifest files and using pre-built packages for some dependecies.
    *Notes related to Vista: If you open the command prompt under Vista by right clicking and running as administrator, you don't get the UAC issues with Vista trying to unsuccesfully run patch as an installer in a seperate environment. You may want run Visual Studio with administrative rights anyway under Vista, as this is recommended by Microsoft (perhaps Visual Studio 2008 would not force you to do that...).


    == What emerge does ==
    == What emerge does ==
    ''emerge'' will fetch Windows versions of numerous UNIX-like utilities and libraries from the Internet, putting them in '''kderoot\bin''', then get the Win32 support files, then Subversion, then Perl and the Qt libraries, etc.
    Emerge can be thought of as performing many of the functions of automated tools like cmake, but in a flexible Python scripting framework.  The benefit of this is that new libraries with idiosyncratic installation procedures, conflicting library and header installation names, and complex rules for building on different setups can be generated automatically, and all directory management should be taken care of without the user's input.


    Then ''emerge'' compiles the Qt libraries, this takes hours.
    The primary logic for the program is contained in the /bin folder of the Git repository.  The script emerge.py serves as the entry point to the system; it runs appropriate code according to the command line arguments.  The basic command '''emerge ''packageName'' ''' performs the five most important actions <tt>--fetch</tt>, <tt>--unpack</tt>, <tt>--compile</tt>,  <tt>--install</tt>, and <tt>--qmerge</tt>.  The definition for each of these steps is defined using a flexible system called Portage, after the Gentoo package management system. The basic goals are:


    '''emerge ''package'' ''' performs the separate actions <tt>--fetch</tt>, <tt>--unpack</tt>, <tt>--compile</tt>, <tt>--install</tt>, <tt>--manifest</tt>, and <tt>--qmerge</tt>.
    1. Fetch action retrieves either a binary or the source code for the package
     
    2. Unpack action installs the source code in a source folder and applies KDE-specific patches.
     
    3. Compile action runs package-dependent configure make steps.
     
    4. Install action installs the headers and compiled library and executable outputs. 
     
    5. Qmerge action does something, but what?
     
     
    Emerge also offers functionality to document dependency trees, create patches to upload tweaks and fixes, and update and clean existing installs.
     
    The actual commands for fetching, unpacking etc. are defined by three increasingly specialized levels of logic.  The first level is the code in the /bin folder and determines the overall order, steps should be taken, reading environment variables to configure the build environment and compiler set by <tt>kdeenv.bat</tt>, and parsing the directory tree. 
     
    The second set of logic is found in the /bin/BuildSystem, /bin/Package, /bin/Packager, and /bin/Source folders.  This is used to determine general procedures for different classes of packages.  For example, the "Source" folder contains the logic for running the <tt>--fetch step</tt> for compressed files, git repositories, SVN, and so on.  The "Package" system contains logic for libraries that need to be configured with e.g. CMake, QMake, or internal make systems. 
     
    The final set of logic is at the per-package level.  This is what is contained in the /portage/ directory.  Emerge is able to automatically search through the Portage folders to find the name of the package you specify.  This is where dependencies, special build configurations and special commands are set up.  Individual patch files and different version configuration information is also stored here.  It is relatively straightforward to add a new package to Portage, especially if the package itself can be downloaded and installed with CMake using minimal configuration. 
     
    A good way to prepare a package for wider distribution is to create a simple <tt>CMakeLists.txt</tt> it.  You can format the addition of this file as a patch, and create a Portage script which merges the patch into the public code repository.


    == ''emerge'' command line options and settings ==
    == ''emerge'' command line options and settings ==
    Line 168: Line 196:
    |<tt>EMERGE_VERBOSE</tt>
    |<tt>EMERGE_VERBOSE</tt>
    |width="5%"|
    |width="5%"|
    |This option sets the verbosity level. Currently the highest verbosity level is 3 (<tt>-v -v -v</tt>). A verbosity level of 0 should give no output and equals to <tt>-q</tt>. You can set <tt>EMERGE_VERBOSE=3</tt> instead in the environment of the commandline or within your <tt>kdesettings.bat</tt> file.
    |This option sets the verbosity level. Currently the highest verbosity level is 3 (<tt>-v -v -v</tt>). A verbosity level of 0 should give no output and equals to <tt>-q</tt>. You can set <tt>EMERGE_VERBOSE=3</tt> instead in the environment of the commandline or within your <tt>kdesettings.ini</tt> file.
    |-valign="top"
    |<tt>--nocopy</tt>
    |<tt>EMERGE_NOCOPY</tt>
    |
    |This very useful option suppresses copying the sources from the local subversion tree to a directory within the build directory. It shouldn't be used while packaging; in the other cases it reduces the amount of harddisk used though and removes the copying time. You can set <tt>EMERGE_NOCOPY=True</tt> or <tt>=False</tt> instead.
    |-valign="top"
    |-valign="top"
    |<tt>--offline</tt>
    |<tt>--offline</tt>
    Line 185: Line 208:
    |This option enables or disables KDE4 buildtests for KDE modules. Other packages will not change. Use <tt>EMERGE_BUILDTESTS=True</tt> or <tt>=False</tt>.
    |This option enables or disables KDE4 buildtests for KDE modules. Other packages will not change. Use <tt>EMERGE_BUILDTESTS=True</tt> or <tt>=False</tt>.
    |-valign="top"
    |-valign="top"
    |<tt>--buildtype=</tt>
    |<tt>--print-targets</tt>
    |<tt>Debug</tt>
    |
    |
    |This option will display all "targets" a certain package has. Normally targets are fixed releases or different branches. They are defined in the portage file.
    |-valign="top"
    |<tt>--target=TARGET</tt>
    |
    |
    |This sets a specific target for this package. If not added, the default target is used, which can be checked by looking at the output of '''--print-targets'''.
    |-valign="top"
    |<tt>-i</tt>
    |
    |
    |This option ignores that a package is already installed. It builds it completely new, but keeps the dependencies.
    |-valign="top"
    |<tt>--update</tt>
    |
    |
    |This option enables full debugging mode for the build. '''Recommended if you plan to debug the runtime or provide more valuable feedback to developers about software defects.''' You can also change the 'set EMERGE_BUILDTYPE=RelWithDebInfo' line in the <tt>kdesettings.bat</tt> file.
    |
    |This option ignores that a package is already installed but doesn't cleanup an already existing build directory. Thus you will only rebuild files that have changed since the last build.
    |}
    |}
    == Emerge works, now what? ==
    *Once you have ''emerge --help'' working, try ''emerge --print-installable'' to get a list of valid package names.  If you're unsure which are the packages you want, try selecting one from the main "kde" group (e.g. kdegames).


    == Hints ==
    == Hints ==
    ===Updating packages===
    === Updating packages ===
    *Once you have ''packagename'' built, type <code>emerge --unmerge packagename --noclean --target=svnHEAD packagename</code> to update <tt>packagename</tt> from the subversion and compile it without removing the build dir.
    *Once you have ''packagename'' built, type <syntaxhighlight lang="text">emerge --update packagename</syntaxhighlight> to update <tt>packagename</tt> from the Subversion and compile it without removing the build dir or <syntaxhighlight lang="text">emerge --update-all</syntaxhighlight> to update all packages that can be rebuild (they are rebuild with --update).


    ===Altering locale and country settings===
    === General setup ===
    *To change locale for all users within the KDE environment, edit KDEROOT/share/config/kdeglobals file and add:
    For Fine Tuning see here:
    <code>
    [[Projects/KDE_on_Windows/Installation#Fine-tuning|Fine-tuning]]
    [Locale]
    Country=**
    Language=**
    </code>
    Replace ** with your lowercase [http://en.wikipedia.org/wiki/ISO_3166-1_alpha-2 alpha-2 country code], e.g. pl for Poland. You can edit your $HOME/.kde/share/config/kdeglobals file instead to alter your local settings, not for all users.
     
    == Notes ==
     
    ''emerge'' can mostly cooperate with the [[Projects/KDE_on_Windows/Installation#KDE_Installer_for_Windows|kdewin-installer]] but we're currently still working on some packages which are packaged in a wrong way.
    It is not recommended to use another layout then '''installer''' for '''directory_layout''' in the '''kdesettings.bat''' anymore (see that file for more detailed information).
     
    ''emerge'' creates lots of files in '''\kderoot\tmp''' during build.
    After a package is successfully installed
    (check '''\kderoot\etc\portage\installed''' or the directory '''\kderoot\manifest\'''), you can delete its temporary directory.
     
    Windows ''emerge'' is derived from the Gentoo portage system, but we are currently not enforcing compatibility. If you have questions about that please contact us at the channel #kde-windows on irc.freenode.net.


    ==Vista issues==
    ==Vista issues==
    *[[User:Jstaniek|jstaniek]] 12:02, 15 January 2008 (CET): UAC has infamous heuristics that make programs like patch.exe treat as installers and try to run them with admin rights (!). This heuristics can be tricked by renaming patch.exe to something like pch.exe ([http://nevali.net/2007/01/update-workaround-for-the-cygwin-uac-problem/ example]) but we did not want to add item to our infrastructure. Instead it is possibleto turn off the heuristics (see the screenshot [http://theinvisiblethings.blogspot.com/2007/02/running-vista-every-day.html here in the security blog calling the heuristics 'severe hole in the design of UAC']). If you happen to disable the UAC, as many annoyed users and devs do (msvc demands admin rights anyway!), patch.exe should already work for you as in older Windows. Alternatively you may want to disable UAC [http://www.howtogeek.com/howto/windows-vista/disable-user-account-controluac-for-administrators-only/ for admins only], but this makes no sense if you are the only user of your machine and use only the admin account.
    *[[User:Jstaniek|jstaniek]] 12:02, 15 January 2008 (CET): UAC has infamous heuristics that make programs like patch.exe treat as installers and try to run them with admin rights (!). This heuristics can be tricked by renaming patch.exe to something like pch.exe ([http://nevali.net/2007/01/update-workaround-for-the-cygwin-uac-problem/ example]) but we did not want to add item to our infrastructure. Instead it is possibleto turn off the heuristics (see the screenshot [http://theinvisiblethings.blogspot.com/2007/02/running-vista-every-day.html here in the security blog calling the heuristics 'severe hole in the design of UAC']). If you happen to disable the UAC, as many annoyed users and devs do (msvc demands admin rights anyway!), patch.exe should already work for you as in older Windows. Alternatively you may want to disable UAC [http://www.howtogeek.com/howto/windows-vista/disable-user-account-controluac-for-administrators-only/ for admins only], but this makes no sense if you are the only user of your machine and use only the admin account.
    * MBitter 15:50, 12 September 2008 (CET): Another workaround for the Vist-Bug is to create a file patch.exe.manifest in the bin directory with the following content:
    * [http://ben.versionzero.org/wiki/Fixing_the_way_Vista_Auto-detects_Installers This wiki page] lists instructions on how to use program manifest to disable privilege elevation for a single binary and makes patch play nice with UACThis [https://bugs.kde.org/show_bug.cgi?id=186712 should] eventually be integrated to emerge scripts.  
      <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
      <assembly xmlns="urn:schemas-microsoft-com:asm.v1" manifestVersion="1.0">
      <assemblyIdentity version="1.0.0.0"
          processorArchitecture="X86"
          name="patch.exe"
          type="win32"/>
      <!-- Identify the application security requirements. -->
      <trustInfo xmlns="urn:schemas-microsoft-com:asm.v3">
        <security>
          <requestedPrivileges>
            <requestedExecutionLevel
              level="asInvoker"
              uiAccess="false"/>
            </requestedPrivileges>
            </security>
      </trustInfo>
    </assembly>
       
       


    [[Category:MS Windows]]
    [[Category:MS Windows]]

    Revision as of 16:59, 18 August 2015

    emerge is a tool to build the KDE sources and its third-party requirements on MS Windows. It is the easy way to build KDE on MS Windows.

    Introduction

    Emerge is a tool that can build the different parts of KDE and its dependencies under Windows. We created this tool to automate and simplify the build process under Windows. We try to build all packages that we offer in the KDE installer with emerge. That has some advantages for us:

    Easy for people to join us

    Before emerge it was quite some work to set a system up for development. There were some quirks, which were documented in some mailing lists, but you had to remember them or you ran into an already solved problem again, etc. Now to get a development machine you need a Windows computer, need to install Python and Git and do the emerge checkout. Then execute emerge to build what you want to build. This is easy for developers coming from Windows to KDE, and also for KDE developers coming to Windows.

    Easy for us to do build

    With emerge you can build the whole software stack (low-level libs, Qt, kdelibs, things above that) with only one command. You can start that build, and some hours later you can check if it worked, or if something broke. So we can spot problems easier and earlier. We can also start with a "naked" Windows computer without any other installed software and bootstrap kde on it. That ensures, that no hidden dependencies on some pieces of software sneak in, because then the builds on a "naked" computer would break and show the problem.

    Easy to collaborate

    We can test the same emerge build description for a package on different Windows versions/computers before we do binary releases. People can also add build descriptions for new packages to the Subversion repository.

    This emerge tool was inspired by the Gentoo emerge tool.

    Set up the environment

    Root directory

    Create a directory if possible in your harddrive's root e.g. C:\kderoot or D:\kderoot (You will need this PATH later). This directory will contain the whole kde installation later. We will refer to it as KDEROOT.

    Python interpreter

    emerge.bat invokes an emerge.py script written in Python programming language, so you first need to install the Python 3.4 Interpreter. The python installation directory will be added to the PATH later by KDEROOT\etc\kdesettings.ini.

    Git client

    The latest source code for Windows emerge and a lot of the rest of KDE is stored in a repository created and managed using the Git version control tool. You need a git client for the first checkout of emerge. There are at least two applications:

    • a command line client, available at git-scm.com, aimed at developers or power users accustomed with the command line,
    • a GUI program like TortoiseGit, optional, useful for displaying differences between various versions of files in the repository in a graphical way.

    Emerge uses its own git client for checking out all KDE source code. You can find it in KDEROOT\dev-utils\git\bin. It will be used by emerge even if you have another git executable installed.

    Getting the emerge tool

    The source code of the emerge tool and the recipes for creating KDE packages are located at git://anongit.kde.org/emerge.git, which is an URL based on the git-specific git protocol.

    You need to clone the source code from the emerge git repository into a new directory below your root directory (the root directory is normally called %KDEROOT% here) or get it inside a self extracting archive (not tested yet).

    Check out using the 'git' command

    With the git command line tool, you can accomplish this with the following commands:

    cd KDEROOT
    • if you will only use anonymous (read-only) access to the KDE git repository:
      git clone git://anongit.kde.org/emerge.git
    • if the git and ssh ports are blocked, and only http is allowed, it is better to clone over the http port for getting (read-only) access to the KDE git repository:
      git clone http://anongit.kde.org/emerge.git
    • or, if you plan to use write access (commit) to the emerge repository using your existing account & OpenSSH private key:
      git clone [email protected]:emerge.git

    Configure the emerge tool

    1. Create the directory KDEROOT\etc.
    2. Copy the file KDEROOT\emerge\kdesettings.ini as KDEROOT\etc\kdesettings.ini and change its contents according to your needs.

    Note 0: Read the comments in that file very carefully

    Note 1: Be sure that you neither have the msys/bin nor the cygwin/bin in your path. If so you have to definitely remove it from the path.

    Note 2 from a user: The applications gimp, inkscape and graphviz are also a problem. To make sure that there's nothing wrong I stripped my path to contain only what I needed to build.

    emerge has several branches which contain specific package versions. E.g. if you want to build the 4.13 branch of KDE (or one of the 4.13 releases) you should checkout the kde-4.13 branch of emerge; in the master branch, you can find the KDE Frameweorks 5.

    To view all branches, use the following command:
    
    git branch -a

    To change the branch of emerge, do the following:

    cd emerge && git checkout kde-4.13

    [Optional, Advanced] emerge-boost-config.jam

    Add a file emerge-boost-config.jam in the %KDEROOT%\etc directory to build boost in a specific way. The only current use case is for the following problem: If you build 32bit binaries with emerge on a 64bit system you need to link boost-python against a 32bit python library. In case your standard python installation is 64bit though, you can specify the python installation in the following way:

    # ---------------------
    # Python configuration.
    # ---------------------
    
    # Configure specific Python version.
    using python : 3.2 : C:\\Python32_x86 ;
    

    This way boost-python would be using the headers & libraries from C:\\Python32_x86 instead of the default one. Please remember that for mingw compilers you must regenerate the import library for the python dll (also for the mingw 64bit compiler):

    R:\> emerge pexports
    R:\> pexports C:\Python32_x86\python32.dll > C:\Python32_x86\libs\python32.def
    

    NOTE: In case you generate the import library for the 64bit compiler, add the following line to the file python32.def:

    Py_InitModule4 = Py_InitModule4_64
    

    For both compilers you should now run:

    dlltool -d C:\Python32_x86\libs\python32.def -l C:\Python32_x86\libs\libpython32.dll.a
    

    Now you should be able to do emerge -i boost-python-src without errors.

    Using emerge

    To use emerge you need to start a Powershell window and point that to KDEROOT\emerge. For example:

    C:
    cd KDEROOT\emerge
    

    Then in the windows powershell you have to load the script with

    ". .\kdeenv.ps1"
    

    This tells emerge about your environment settings (e.g. paths). It will load your configuration from KDEROOT\etc\kdesettings.ini. It should not give any error messages, otherwise emerge will not work as expected. The output should look similar to this one (of course with your paths):

    PS C:\kderoot\emerge>.\kdeenv.ps1
    KDEROOT     : r:\
    KDECOMPILER : mingw-w64
    KDESVNDIR   : s:\
    KDEGITDIR   : q:\
    DOWNLOADDIR : t:\
    PYTHONPATH  : C:\python34
    
    PS C:\kderoot\emerge>
    

    Note: There is a short path option in kdesettings which you will need if you want to build Qt in a directory that has a pathlength of more then around 5 characters. This problem is due to limitations of the command line length and the Qt build system.

    Next, if you have configured your kdesettings.ini to use svn+ssh for your subversion checkout, then you need to run:

    plink <your-svn-username>@svn.kde.org
    plink <your-kde-username>@git.kde.org
    

    This will prompt you to accept the fingerprint of the server, otherwise svn will hang forever when trying to download from the server.

    Now you should be able to use emerge. To get some help on usage:

    emerge --help
    

    To get a list of available packages:

    emerge --print-installable
    

    To get a list of currently installed packages:

    emerge --print-installed
    

    Setting up a compiler

    Currently emerge supports both the MinGW and MS Visual C++ (msvc) compilers. We did not add dependencies for the compilers, so you have to make sure to install a compiler by yourself. There are three ways to set up a compiler for emerge. We assumed you have set KDECOMPILER variable properly in the KDEROOT\etc\kdesettings.ini.

    In the following sections you can find information on how to install or reuse an existing compiler.

    Install the MinGW compiler with emerge

    Let emerge install the MinGW compiler, as soon as emerge needs MinGW it will automatically fetch the correct version for you.

    Install MS Visual C++

    Read here.

    Point to an existing MS Visual C++ installation

    You need to point emerge to an existing msvc installation. This is run automatically for you from kdeenv.bat if configured properly in kdesettings.ini. Check your kdesettings.ini file to know where to set it.

    Installing the base system

    Once you have emerge and a compiler installed and working, try:

    You are now ready to start building KDE, it is recommended to do so progressively, relying on emerge to automatically resolve the required dependencies at each set step:

    • Enter emerge qt. This will fetch and install Windows versions of numerous UNIX-like utilities and libraries, then checkout, compile and install Qt. This will take several hours.
    • Enter emerge frameworks. This will checkout, compile and install the kde frameworks 5 modules.
    • Enter emerge kde-baseapps. This will checkout, compile and install kdebase-runtime, and kdebase-apps.

    You will now have successfully installed a base KDE system and can now install other KDE modules as required.

    Note that this will install the development version of KDE (trunk in svn repositories, master in git repositories), if you wish to install a particular stable branch then you should change the branch of emerge to the specific branch of that release. NOTE: You should not mix kde packages from different branches.

    It is strongly recommended you do not choose to manually install any of the utilities and libraries yourself, as you may install the wrong version and cause installation failures. Instead allow emerge to resolve the dependencies for you.

    Every time you want to update or install a package, you should first update your emerge checkout (simply run emerge --update emerge) to ensure you are using the latest package recipies.

    What emerge does

    Emerge can be thought of as performing many of the functions of automated tools like cmake, but in a flexible Python scripting framework. The benefit of this is that new libraries with idiosyncratic installation procedures, conflicting library and header installation names, and complex rules for building on different setups can be generated automatically, and all directory management should be taken care of without the user's input.

    The primary logic for the program is contained in the /bin folder of the Git repository. The script emerge.py serves as the entry point to the system; it runs appropriate code according to the command line arguments. The basic command emerge packageName performs the five most important actions --fetch, --unpack, --compile, --install, and --qmerge. The definition for each of these steps is defined using a flexible system called Portage, after the Gentoo package management system. The basic goals are:

    1. Fetch action retrieves either a binary or the source code for the package.

    2. Unpack action installs the source code in a source folder and applies KDE-specific patches.

    3. Compile action runs package-dependent configure make steps.

    4. Install action installs the headers and compiled library and executable outputs.

    5. Qmerge action does something, but what?


    Emerge also offers functionality to document dependency trees, create patches to upload tweaks and fixes, and update and clean existing installs.

    The actual commands for fetching, unpacking etc. are defined by three increasingly specialized levels of logic. The first level is the code in the /bin folder and determines the overall order, steps should be taken, reading environment variables to configure the build environment and compiler set by kdeenv.bat, and parsing the directory tree.

    The second set of logic is found in the /bin/BuildSystem, /bin/Package, /bin/Packager, and /bin/Source folders. This is used to determine general procedures for different classes of packages. For example, the "Source" folder contains the logic for running the --fetch step for compressed files, git repositories, SVN, and so on. The "Package" system contains logic for libraries that need to be configured with e.g. CMake, QMake, or internal make systems.

    The final set of logic is at the per-package level. This is what is contained in the /portage/ directory. Emerge is able to automatically search through the Portage folders to find the name of the package you specify. This is where dependencies, special build configurations and special commands are set up. Individual patch files and different version configuration information is also stored here. It is relatively straightforward to add a new package to Portage, especially if the package itself can be downloaded and installed with CMake using minimal configuration.

    A good way to prepare a package for wider distribution is to create a simple CMakeLists.txt it. You can format the addition of this file as a patch, and create a Portage script which merges the patch into the public code repository.

    emerge command line options and settings

    There are some options that can be used when building with emerge.

    Command line switch Command line argument Description
    -v EMERGE_VERBOSE This option sets the verbosity level. Currently the highest verbosity level is 3 (-v -v -v). A verbosity level of 0 should give no output and equals to -q. You can set EMERGE_VERBOSE=3 instead in the environment of the commandline or within your kdesettings.ini file.
    --offline This option suppresses the update step of the local tree - which needs some time. Be aware though that you have to have existing sources already if you want to use this option.
    -t EMERGE_BUILDTESTS This option enables or disables KDE4 buildtests for KDE modules. Other packages will not change. Use EMERGE_BUILDTESTS=True or =False.
    --print-targets This option will display all "targets" a certain package has. Normally targets are fixed releases or different branches. They are defined in the portage file.
    --target=TARGET This sets a specific target for this package. If not added, the default target is used, which can be checked by looking at the output of --print-targets.
    -i This option ignores that a package is already installed. It builds it completely new, but keeps the dependencies.
    --update This option ignores that a package is already installed but doesn't cleanup an already existing build directory. Thus you will only rebuild files that have changed since the last build.

    Hints

    Updating packages

    • Once you have packagename built, type
      emerge --update packagename
      
      to update packagename from the Subversion and compile it without removing the build dir or
      emerge --update-all
      
      to update all packages that can be rebuild (they are rebuild with --update).

    General setup

    For Fine Tuning see here: Fine-tuning

    Vista issues

    • jstaniek 12:02, 15 January 2008 (CET): UAC has infamous heuristics that make programs like patch.exe treat as installers and try to run them with admin rights (!). This heuristics can be tricked by renaming patch.exe to something like pch.exe (example) but we did not want to add item to our infrastructure. Instead it is possibleto turn off the heuristics (see the screenshot here in the security blog calling the heuristics 'severe hole in the design of UAC'). If you happen to disable the UAC, as many annoyed users and devs do (msvc demands admin rights anyway!), patch.exe should already work for you as in older Windows. Alternatively you may want to disable UAC for admins only, but this makes no sense if you are the only user of your machine and use only the admin account.
    • This wiki page lists instructions on how to use program manifest to disable privilege elevation for a single binary and makes patch play nice with UAC. This should eventually be integrated to emerge scripts.