User:Mark Gaiser/KDirchainRebuild

    From KDE TechBase

    Introduction

    This all started a few years ago when i was profiling Dolphin. I wasn't happy with it's general performance and was even less happy when it came to massive folders. Added to that was the - back then - brand new stuff called "QML". So i wanted to create a file browser that is fast, memory efficient and with a GUI done in QML so that i could add those nifty animations. So i set out to restructure all of the directory handling from KIO from as low UDSEntry till the user facing models. I named that "KDirchainRebuild" since it was rebuilding the chain of events when you open a directory.

    That's how it all started a few years ago. Fast forward to today and you see an entirely different landscape. Dolphin is now very fast even on large folders. I didn't make any patches for it, but my initial research in why it was performing poorly was apparently a spark for the Dolphin devs to take a good close look at it and improve it greatly. The biggest thing i did here was investigating how entries where being batched in KIO. That is improved to do real batching which in turn allows an application to show data more quickly to the user. A big thank you here for David Faure since the new batching triggered another weird socket issue that he solved.

    So is there still a reason to even consider this "KDirchainRebuild" over the existing classes? Well, yes. The new classes are aimed at simplicity, small memory footprint and being as fast as possible (which sometimes conflicts with simplicity), and - via models - being usable in QML.

    Why new classes instead of improving the existing ones?

    Initially the idea was to take the current classes, remove all function content and re-implement them in my view. Later i abandoned that idea because there seemed to be quite a few functions that i wanted to handle in a different way along with removing methods altogether. I wasn't really interested in marking a dozen functions as "deprecated". Specially not as long as this project was highly experimental and just for my own pet project. So the choice was easiest to just go for my own class names without the burden of having to re-implement the existing classes.

    The Classes

    Before i present the new classes and the API they have, i first want to take a moment to say that the classes are not done yet! The API is far from finished and code-wise it already can use a bit of cleanup. All the information below is just to indicate it's current state, what is still planned for it and where i see potential issues (and resolutions).

    KDirectoryEntry

    KDirectoryEntry compares mostly to KFileItem only with a greatly reduced memory footprint (and a lot of missing functions).

    Data stored

    Inherits from no other classes.

    • 1 d pointer (KDirectoryEntryPrivate)
    • 1 KIO::UDSEntry
    • 1 bool to know if details are loaded (details 0 vs 2). I still like to get rid of this one.

    Ideas

    You always need an UDSEntry if you want to know something about a file. It might be beneficial for the UDSEntry itself to know (with a bool) if it has all the file details or not. It adds a bool value, but that can be used later on to prevent needless hash lookups if you have no details loaded. I might be worth adding this or at least experiment with it.

    For the new classes. I just like to have details with true or false (bool) instead of a string (0 or 2). Easy to implement on my side, but will it cause any other difficulties later on? I don't know.

    Functions

    Some of the below documentation is a direct copy from KFileItem.

       /**
        * Returns the raw file name with extension as it comes from UDSEntry.
        * @return QString
        */
       const QString name() const;
    
       /**
        * Returns the owner of the file.
        * @return QString
        */
       const QString user() const;
    
       /**
        * Returns the group of the file.
        * @return QString
        */
       const QString group() const;
    
       /**
        * Returns the base name of a file. This is the name without extension.
        * @return QString
        */
       const QString basename() const;
    
       /**
        * Returns the extension of a file without leading dot.
        * @return QString
        */
       const QString extension() const;
    
       /**
        * Returns the icon name based on the output of QMimeType
        * @return QString
        */
       const QString iconName() const;
    
       /**
        * Returns the icon comment based on the output of QMimeType
        * @return QString
        */
       const QString mimeComment() const;
    
       /**
        * Returns the file size from UDSEntry if details where loaded. 0 for no details or if the current entry is a folder.
        * @return QString
        */
       const KIO::filesize_t size() const;
    
       /**
        * This is used internally when constructing this object with an UDSEntry, or used externally to update this object. For example when a rename happens. Users using this class cannot use this object to get notified of changes. The KDirectory object containing this instance will notify you of changes.
        */
       virtual void setUDSEntry(const KIO::UDSEntry& entry, const QString& details = "0");
    
       /**
        * Returns true if this item represents a link in the UNIX sense of
        * a link.
        * @return true if the file is a link
        */
       bool isLink() const;
    
       /**
        * Returns true if this item represents a directory.
        * @return true if the item is a directory
        */
       bool isDir() const;
    
       /**
        * Returns true if this item represents a file (and not a a directory)
        * @return true if the item is a file
        */
       bool isFile() const;
    
       /**
        * Checks whether the file or directory is readable. In some cases
        * (remote files), we may return true even though it can't be read.
        * @return true if the file can be read - more precisely,
        *         false if we know for sure it can't
        */
       // STUB! To be implemented
       bool isReadable() const;
    
       /**
        * Checks whether the file or directory is writable. In some cases
        * (remote files), we may return true even though it can't be written to.
        * @return true if the file or directory can be written to - more precisely,
        *         false if we know for sure it can't
        */
       // STUB! To be implemented
       bool isWritable() const;
    
       // STUB! To be implemented
       bool isExecutable() const;
    
       // STUB! To be implemented
       bool isModified() const;
    
       // STUB! To be implemented
       bool isSystem() const;
    
       /**
        * Requests the modification, access or creation time, depending on @p which.
        * @param which the timestamp
        * @return the time asked for, QDateTime() if not available
        * @see FileTimes
        */
       QDateTime time(FileTimes which) const;
    
       /**
        * Checks whether the file is hidden.
        * @return true if the file is hidden.
        */
       bool isHidden() const;
    
       /**
        * Checks whether the file details are loaded.
        * @return true if details are loaded, false otherwise.
        */
       bool detailsLoaded() const;
    

    KDirectory

    You can't compare KDirectory to anything currently living in KDE (that i know of). It is somewhat inspired by QDir but even there not fully comparable.

    KDirectory is basically a data management class. If you open a folder, this class will do a call to KIO::listDir and retrieve it's output. What you as a user can do is set properties how the output should be accessed. You can set filters and flags just like QDir. This class then applies those filters and flags to all incoming data and stores it (internally) in two lists. Once for those that pass, one for those that don't. No data is duplicated there. The number of items in both lists combined is the number of items you would see when you execute "ls -l | wc -l" in the same folder on the command line. This class only gives you access to the items that passed the filters/flags. If you want access to the items that didn't pass then you have to change your filters/flags to make them pass. Or set no filters/flags which makes all items pass. For bookkeeping, this class stores quite a bit of additional data.

    Ideas

    KDirectory should also manage file/folder changes in it's directory. However, the signals emitted by KDirWatch and QFileSystemWatcher are too generic. I need more detailed signals to let it be useful. Right now adding a file to a directlry simply means re-indexing the entire directory because we don't have enough details in our signals. Details that inotify (the backend) does provide. I plan to revive some old patches for QFileSystemWatcher on the frameworks sprint and make it work again. Once that is in working order, KDirectory can use them. Right now KDirectory just doesn't use any mechanism to monitor a folder for changes. Remember, work in progress!

    Functions

       /**
        * Returns all entries that passed the filters and flags.
        * @return QVector<KDirectoryEntry>
        */
       virtual const QVector<KDirectoryEntry>& entries();
    
       /**
        * Entry returns the KDirectoryEntry object if it's index is in the filteredEntries.
        * @param index
        * @return KDirectoryEntry at index
        */
       virtual const KDirectoryEntry& entry(int index);
    
       /**
        * String of the full path for this directory.
        * @return QString
        */
       virtual const QString& url();
    
       /**
        * Number of entries that passed the filters and flags
        * @return int
        */
       virtual int count();
    
       /**
        * Set the details for the current directory. It can be 0 (no details) or 2 (full details). If the value passed is different then the current details value (and valid) then the directory will be re-scanned according to the new details value.
        * @param QString details. Either 0 or 2.
        * @return KDirectoryEntry at index
        */
       virtual void setDetails(const QString& details);
    
       // For those, see QDir documentation.
       QDir::Filters filter();
       void setFilter(QDir::Filters filters);
       QDir::SortFlags sorting();
       void setSorting(QDir::SortFlags sort);
    
       /**
        * Loads the entry details and passes it to the KDirectoryEntry that needs the information.
        * Be aware that this function is executing a (slow) stat call!
        * @param int id the id of the file to load. This is the QVector index id.
        */
       void loadEntryDetails(int id);
    

    Signals

       /**
        * New entries in this folder have been processed.
        * @param KDirectory* directory pointer to the current directory. This pointer is given
        *        because you're likely to use multiple KDirectory objects so you wouldn't easily
        *        know which KDirectory object spawned this signal.
        */
       void entriesProcessed(KDirectory* dir);
    
       /**
        * Done loading entries.
        * @param KDirectory* directory pointer to the current directory. This pointer is given
        *        because you're likely to use multiple KDirectory objects so you wouldn't easily
        *        know which KDirectory object spawned this signal.
        */
       void completed(KDirectory* dir);
    
       /**
        * Done loading entries.
        * @param KDirectory* directory pointer to the current directory. This pointer is given
        *        because you're likely to use multiple KDirectory objects so you wouldn't easily
        *        know which KDirectory object spawned this signal.
        * @param int id is the id that you can use to get the KDirectoryEntry object (entry(id)).
        */
       void entryDetailsChanged(KDirectory* dir, int id);
    

    KDirListerV2

    Source:


    KDirListerV2 had the intention to be comparable with KDirLister, but when i decided to move away from the current KDE class structure and go for my own versions i kept the name (KDirListerV2). Right now it can't be compared to KDirLister. The name of this class is also confusion now since it's not actually listing. It's just a store from url (QString) to KDirectory mapping. KDirectory is then executing all the difficult parts. This class still has value because it maintains (and keeps) the url's that have been passed through it. It's acting as a cache for those and returns them when requested. Much like KDirLister and KDirListerCache work.

    I guess this class can be renamed to something more appropriate like KDirectoryLoader.

    Data stored

    Just one. Although the private class KDirListerV2Private is also having a pointer back to KDirListerV2. This is old cruft from earlier versions. I still need to remove it.

    • QHash<QString, KDirectory*> m_cache;

    Ideas

    Rename to KDirectoryLoader or some other name.

    Perhaps some kind of LRU caching mechanism should be used. I did try this before, but was very annoying during development so i removed it. I was using the ALT-LRU classes. Perhaps good for later re-inclusion.

    Regarding the openUrl function. I think i'd like to get rid of the flags parameter (and the OpenUrlFlags enum). The only real use for that was opening a folder or reloading one. Opening should be done through this route and reloading should be done directly on the KDirectory object representing the url (KDirectory::reload(), which doesn't exist yet). The Keep flag would be lost. Not a big loss imho. Right now reloading is done through KDirListerV2 which first removes the KDirectory object before creating a new one for a "reload". Seems a bit wasteful.

    DirectoryFetchDetails

    Just for reference since this will give some sense to the openUrl where this object is passed in.

       struct DirectoryFetchDetails {
           QString url;
           QString details = "0";
           QDir::Filters filters = QDir::NoFilter;
           QDir::SortFlags sorting = QDir::NoSort;
           KDirListerV2::OpenUrlFlags openFlags = OpenUrlFlag::NoFlags;
       };
    

    Functions

       /**
        * Open a new URL to be listed.
        *
        * @param url the directory URL.
        * @param flags. Can be NoFlags, Keep or Reload
        * @return true if successful, false otherwise.
        */
       virtual bool openUrl(const QString& url, OpenUrlFlags flags = NoFlags);
    
       /**
        * Open a new URL to be listed.
        *
        * @param dirFetchDetails
        * @return true if successful, false otherwise.
        * @see DirectoryFetchDetails
        */
       virtual bool openUrl(DirectoryFetchDetails dirFetchDetails);
    
       /**
        * Test if a given URL is being listed.
        *
        * @return true if directory is listed, false otherwise
        */
       virtual bool isListing(const QString& url);
       *** I'm thinking of removing this function. It doesn't seem to make much sense.
    
       /**
        * Return a KDirectory pointer representing the given URL.
        *
        * @param url
        * @return KDirectory pointer if url is known, nullptr otherwise.
        */
       virtual KDirectory* directory(const QString& url);
    

    Signals

       /**
        * NOTE: pay close attention here, This signal is returning the internal
        * KDirectory object which has ALL entries inside up till the emit moment!
        * It does not emit incremental updates!
        *
        * Also, there is no guarantee that this pointer will be the same for the
        * same folder. You should therefore not rely on this pointer.
        *
        * @param items
        */
       void directoryContentChanged(KDirectory* directoryContent);
    
       /**
        * @brief completed
        * @param items
        */
       void completed(KDirectory* directoryContent);
    

    DirListModel

    Source:

    This is a flat model inheriting from QAbstractListModel. It exposes nearly all functions from a KDirectoryEntry class along with some extra stuff. Like a pixmap for previews. Although the pixmap support is currently broken :)

    All roles are also exported by re-implementing roleNames. This means that QML can use this model without hassle. This model should be enough for exposing a filesystem to QML and using QML to graphically represent it.

    FlatDirGroupedSortModel

    Data stored

    Ideas

    Functions